Why does bad software get released?

Have you ever come across a piece of software that just  does not work and wonder, ‘How the heck did this get released?’

Or maybe you have actually worked on a project were you tell the powers to be that the software should not be released, but lo and behold, they go and do it anyway. I had cause to reflect on this question just recently …….. why does bad code get released?

There are of course a myriad of reasons and circumstances that lead to poor code being released, however once cause can be the company culture.

If developers are highly valued, and testers are seen of less value, then it is easy to see why the voice of the tester can go unheard and their view ignored or trodden over.

If testers are no tsufficiently technical or unable to write a coherent and convincing report, their message can get shot down in the general debate, or considered of little value due to the poor presentation.

If the release decision is taken place behind closed doors with no opportunity for the test report to be presented and discussed then the information regarding the true
state of the software might never be made available to those making the decisions.

Of course, there may be a number of reasons why, even though the test report is fully understood, the release is still sanctioned and bad code is released.  However these tend to be few and far between
in my experience, the main reason is that the test department of one reason or another is excluded from the process.

Who do I blame?   The test department. We need to do more to ensure we are respected, valued and listened to.

The chart below takes a slightly tongue in cheek look at an all too familiar release decision tree.

My favourite Testing Related Web Places

.Since I started to blog, and more recently tweet I have started to discover some really great places to visit.

This blog lists some of my favourite spots, old and new.

New on the block is TechWell (http://test.techwell.com), from the guys at Stickyminds (which is also a favourite of mine.) This stuff is high quality and covers a wealth of topics. I really love the look of the site as well.

The Software Testing Club (www.SoftwareTestingClub.com) is a site that has grown at an impressive rate over the past couple of years and Rosie (the founder) should be very proud of what she has achieved. It has a relaxed home grown feel about it. Some of the content can be a bit benign, but it is a great place to go for a quick answer from your peers if you get stuck.

Utest has a nice blog page (http://blog.utest.com) and manages to provide a very lively flow of content including interviews, articles information and posts. It has a nice clean layout that makes browsing easy on the eye.

The only blog that has caused me to write ‘fan mail’ is the fantastically honest QAHatesYou (http://qahatesyou.com/wordpress). As you can imagine from the name, he pulls very few punches. Add to that he has a certain warped mind that means that some of his post are incredible funny …….check out his songs for testers videos!

A nice site from the quality frog Ben Simo, is dedicated to software failures,, and should provide material to liven up a slide or two if you need to explain why testing is important, check it out. (http://blog.isthereaproblemhere.com)

 

I could go on, but that’s enough of that for now I think. I have set up a Software Bloggers Group on LinkedIN (http://www.linkedin.com/groups/Software-Testing-Bloggers-3945121?trk=myg_ugrp_ovr). I am hoping a bunch of bloggers might join and use it to spark off each other. I would like to issue a blogging challenge from time to time via the group. Check it out and let me know what you think.

Also why not post you own favourite testing sites via the comment box.

So you think you know the answer

Testing is more about knowing the answer than it is about knowing the question!

Let me justify that stamen by starting with a little quiz.

I will give you the question, and you mark the options below as ‘pass’ or ‘Fail’ as if this were a software test,

1: What was King George VI’s first name?

A: William
B: George
C: Albert
D: Charles

2: How long did the ‘100 year war’ last?

A: 93 Years
B: 100 years
C: 101 years
D: 116 years

3: What is the acceptable load time for a standard web page

A: 2 seconds or less
B: Under 4 seconds
C: 10 seconds if containing rich media
D: 20 seconds

Of course if you know the answers, the decision to mark any result as a pass or a fail is easy, however if you just have the questions, you may not be so sure. The question may seem obvious to you, but how do you actually know what the correct answer is to generate a ‘pass’

The correct ‘pass’ criteria for question 1 is answer ‘C’ and for 2, it is ‘D’, however for question 3, there is no universally accepted ‘correct’ answer.

So you see, knowing the answer is much more important than knowing the question when it comes to testing. Requirements can sometimes hint at the answer. Lets work though an example;

‘Website links must be obvious and accessible’

Ok so my test would be,

‘Is this link obvious?’

But what is the answer?

This link has an’ under line ….. Pass
This link does not…. Fail…. Except that the link itself is blue and all the other text is black … so Pass … or fail .. er no pass!

You see the dilemma, unless it has been agreed what ‘obvious’ for each class of link will mean, then the result is open to interpretation, just like question 3 above.

The problem is further compounded by the fact that there will exceptions from time to time, so…

‘All textual links will be underlined and coloured blue’ is great, until you decided that actually, the small footer links (privacy policy etc.) will be underlined and grey. Again, knowing the answer for each link is more important than knowing the question (Is this link obvious?).

Of course in any real project you seldom get all the answers upfront, and so asking the question is an important way of getting the answer and I am certainly not saying you should wait till everything is nailed down before you write your tests. However I am suggesting that as testers we should not get too wedded to our tests and that if as a result of a test, an answer gets changed, then we should keep in mind that the answer is the important thing that drives us.

Get the right answer and you can ask the right question.

 

 

Tony Simms is Principal Consultant at Roque Consulting (www.roque.co.uk). He can be contacted via tony.simms@roque.co.uk.

‘Toggle’ – Or what ‘Boggle’ teaches us about testing

Ever Played Boggle?

The current Mrs Simms and I have now reached the age where the highlight of a romantic night in will be two or three rounds of Boggle. You probably know the game, where you have a 3 x 3 grid of random letters and you have to try to make as many words as possible in a given time. It always amazes me that we can come to the last few seconds and neither of us can find any more words, yet when we add up the scores, she has five or six words I did not see and I have seven or eight she did not see.  So what does that teach us about testing?

Two heads are better than one, three are better than two.

I have just received back review comments on my DR Test Plan from four or five different people. Many make the same comment about this or that, but they all have additional comments none of the others have. They not only see the same things differently, they also have different agendas, responsibilities and experiences that influence that review process.

The same is true of test execution, having just had a report back from a session of crowd testing its clear that things have been spotted and logged as bugs that none of the previous reviews and test cycles picked out. Again it has to do with how our brains are all wired differently and how that influences what we see.

Results from the testing of common pieces of code is the combination of scientiffic experience of years suffixed by years of personal growth and development.

If you ask people to count the number of times the letter ‘f’ appears in the sentence above, it is interesting to see how many different answers you get. (by the way with the double ‘f’ in scientific I am pretty sure there are 9 or is it 10?) 

 Different is Good, but… Different can be Helped

Having agreed that different people see things differently and that brings value to the review and test process, that is not to say that you can’t teach some uniformity in to what to look for.

Below is a set of guidelines for reviewing documents, taken from the excellent book

‘Software testing In the real World’  by Edward Kit.

The check list below is adapted from the Boeing Computer Services Requirements Checklist

1. Complete.          All items needed to specify the solutions to the problem have been included.
2. Correct.              Each item is free from error.

1. Precise,               unambiguous , and clear. Each item is exact and not vague; there is a single interpretation; the meaning of each item is understood; the specification is easy to read.

4. Consistent.       No item conflicts with another item in the specification.
5. Relevant.           Each item is pertinent to the problem and its solution.
6. Testable.            During program development and acceptance testing, it will be possible to determine whether the item has been satisfied.
7. Traceable.        Each item can be traced to its origin in the problem environment.
8. Feasible.            Each item can be implemented with the available techniques, tools, resources, and personnel, and within the specified cost and schedule constraints.
9. Free of unwarranted design detail.          The requirement specifications are a statement of the requirements that must be satisfied by the problem solution, and they are not obscured by proposed solutions to the problem.
10. Manageable.  The requirements specification can be controlled in such a way that each item can be changed without excessive impact on other items. Changes to the completed requirements specification can be controlled; each proposed change can be traced to an existing requirement; and the impact of the proposed change can be accessed.

Some methods of transforming specifications to reveal ambiguities, errors and/or misunderstandings:

1. Read the sentence several times, varying the stress pattern to reveal possible ambiguities.
2. When a term is defined explicitly somewhere, try substituting that definition in place of the term to see if the definition fits in this context
3. When a structure is described in words, try to sketch a picture of the structure being described.  This is especially useful where hierarchical structures are being described.
4. When a picture describes a structure, see if redrawing the picture reveals different possible meanings
5. When there is an equation, try expressing the meaning of the equation in words.
6. When a calculation is specified or implied in words, try expressing it in an equation.
7. When a calculation is specified, work at least two examples by hand and give them as examples in the specifications.
8. Look for statements that in any way imply certainty and then look for proof.  Words such as ‘always’, ‘every’, ‘all’, ‘none’ and ‘never’ are useful clues to indicate unproved certainty. 
1. When you are searching behind certainty statements, PUSH THE SEARCH BACK as many levels as are needed to achieve the kind of certainty a computer will need. 
2. Be on the lookout for words that are supposed to be persuasive, such as CERTAINLY, THEREFORE, CLEARLY, OBVIOUSLY, or ‘AS ANY FOOL CAN PLAINLY SEE’.  If its not obvious to you then there is a good chance that others will interpret it differently from the author’s intended meaning.
3. Watch for vague words, such as SOME, SOMETIMES, OFTEN, USUALLY, ORDINARILY, CUSTOMARILY, MOST, or MOSTLY.  These must be backed by  conditions that you can test (and the developers can code to)
4. When lists are given, but not completed, make sure that there is a complete understanding of the nature of the subsequent items.  Watch out for ETC., AND SO FORTH, AND SO ON, or SUCH AS.
5. In attempting to clarify lists, as in (12), we sometimes state a rule.  Be sure that the rule doesn’t contain unstated assumptions.
6. Look for lists without examples or examples that are too few or too similar to each other to explicate the rule.
7. Beware of vague verbs such as HANDLED, PROCESSED, REJECTED, SKIPPED, or ELIMINATED.
8. PASSIVE VOICE constructions are also traps.  Since the passive voice does not name an actor, it’s easy to overlook who is doing the work.  An example is ‘The log file is deleted later’.  Who or what deletes the log file?
9. Be especially on the lookout for comparatives without referents.
10. Pronouns are often clear to the writer and not to the reader.

UK 2011 Census – Not written by testers

Its Census time in the UK, an event that comes round once every 10 years and that requires every householder to ensure they collect a full set of personal data, or face a £1000 fine.

My census questionnaire dropped through the letter box last week and immediately my tester senses start tingling!

The message from Jil Matheson (our National Statistician) , in bold, on the front page is,

 ‘Act Now’ 

 but read on three sentences and it says  

‘complete your census questionnaire on the 27  March (in bold).

So I need to  act now, on the 27^th ! This confusing message certainly looks like a case of confused requirements to me. So back to the business for an answer…

 “Jil, which one is it, should I act now, or wait till the 27^th?”

Not everyone considers the census the harmless fun I do, and so last time there were a number of census parties where as many people as possible got together in one location to skew the results. This time I think there may be a clue on how to do that in Jil’s name. I note that Jil Matheson is an anagram of “Jam in hotels”!

CARS

Next as I glace through the pages of questions I spot Question H14.

 It asks “how many cars are available for use by members of the household?”

For four or more cars you have to give a number, but the input box is only two digits long.

Well given that the question does not exclude loan cars, hire cars, purchasing new cars, stealing cars etc., I am going to have to answer “thousands” which would be the correct answer to the question as phrased, but that won’t fit in the two boxes.

 Healthy?

Question 13 just does not give enough information to be able to answer the question. It want to know how good my health is, the options are Very Good, Good, Fair, Bad, Very Bad.

How can I answer that? Where is the benchmark by which I can make a comparison? I have poor eyesight, am overweight and my blood pressure is above average, though as far as I know I don’t have TB, radiation sickness or scurvy. Does that mean I am good, fair or bad?

I am not asked how I consider my health, but how my health actually is in general and I have no way of knowing what the baseline is that I am supposed to judge by. I guess I will have to leave that one blank and claim that I am not medically qualified to make the statement.

 Enough is Enough

I could go on (indeed I do go on and on) but you get the idea. You would think that given 10 years to prepare the Office for National Statistics could make a better job of framing the questions. Indeed it is not only the questions that need addressing. I called the helpline  (an outsourced private concern) and asked this simple question.

“If I fill the form in by hand, and want to post it in rather than hand it to a collector, what if  the date by which I am legally obliged to post the form or to get it to you.?”

After one and a half hours on the phone, I still do not have an answer. It seems no one can actually find the date by which you have to return the form!

You may ask  “what is the point of this blog?” Well really I just wanted someone to know that I will be available in 10 years’ time to review the next questionnaire if required!

‘No email’ Thursday

I once worked for a client who had a ‘No Email on Thursday’ policy. The rule was, you did not send or respond to emails on a Thursday, you got on the phone or off your backside and you talked to people. It was surprisingly effective and as a test manager it made me stop and think why I was using email.

I used to think I used email as it provided a fast and effective way to communicate, but actually the developers on that particular project, spread over a number of offices and sites used IM heavily and that proved a much better and more instant form of communication than my emails.

In fact email has often slowed down my communication, where a phone call has sped it up. Today for example I emailed three versions of an integration test diagram and reviewed the response before I finally called the architect and talked through the correct design on the phone.

So why do I use email?

Some of the reasons that surprised me when I came to think about it are;

I use email to drop others in it.

Of course at the time I am sure I don’t think of it like that, I am sure I justify it along the lines of, ‘for your information’. What I find myself doing is copying in someone on the email trail so that they are aware of an issue or problem I am experiencing. In fact I was once told that the most important names on an email are often those that appear in the CC field.

I am not saying this is good or bad practice, (or rather I am saying that it is both good and bad practice) just that you need to be aware why you are copying this or that person in. Ask yourself, do I really need to copy in his or my boss, I do I really need to pick up the phone and sort this out.

I use email as an audit trail.

Again I am not saying this is bad, in fact I find this one of the most important uses of email both in terms of tracking my own actions and those of others. I can’t remember the times I have gone back to an old email and re-sent it with a comment along the lines of, “I told you ‘this’ and I told you ‘then’. (Of course, emails trails are becoming more and more used in court cases and so it is always worth considering what you are putting in an email, remember just because you deleted it, that does not mean it is deleted on your server and the recipients server or inbox.)

The problem with this is that I sometimes hide behind the fact that I sent an email four weeks ago, and so, this or that should have been done, instead of taking responsibility to check that things are progressing. This week I had to call my project manager and explain that I had taken my eyes of the ball around work that should have been going on to enable my testing. Yes I had emailed requesting this work, but no, it had not been done and yes I should have checked.

I use email as an excuse not to do real work.

I am an email junkie, so if I am not responding to email from work, I am often checking my personal email account. Why I check my personal account is always a mystery to me, as even with spam filters and rules I spend half my time deleting offers from Nigerian widows who want me to launder cash, Russian singles who want to find a loving man and the current Mrs Simms who keeps forwarding me emails about weekend theatre breaks and exotic holiday destinations. In fact I can easily spend half an hour reading emails rather than reviewing a specification or writing a test plan!

Long live email

Now before anyone things I have a big downer on email, let me say I love email, I think it is a powerful and essential business tool, it provides functionality that could not easily be achieved in any other way and is the life blood of most project teams. However…..

I wonder if you have ever really thought about how and why you send emails, and if you would ever consider a ‘No Email Thursday’. 

Tony Simms is the Principal Consultant at Roque Consulting
(www.roque.co.uk) he can be contacted via email, tony.simms@roque.co.uk