Why does bad software get released?

Have you ever come across a piece of software that just  does not work and wonder, ‘How the heck did this get released?’

Or maybe you have actually worked on a project were you tell the powers to be that the software should not be released, but lo and behold, they go and do it anyway. I had cause to reflect on this question just recently …….. why does bad code get released?

There are of course a myriad of reasons and circumstances that lead to poor code being released, however once cause can be the company culture.

If developers are highly valued, and testers are seen of less value, then it is easy to see why the voice of the tester can go unheard and their view ignored or trodden over.

If testers are no tsufficiently technical or unable to write a coherent and convincing report, their message can get shot down in the general debate, or considered of little value due to the poor presentation.

If the release decision is taken place behind closed doors with no opportunity for the test report to be presented and discussed then the information regarding the true
state of the software might never be made available to those making the decisions.

Of course, there may be a number of reasons why, even though the test report is fully understood, the release is still sanctioned and bad code is released.  However these tend to be few and far between
in my experience, the main reason is that the test department of one reason or another is excluded from the process.

Who do I blame?   The test department. We need to do more to ensure we are respected, valued and listened to.

The chart below takes a slightly tongue in cheek look at an all too familiar release decision tree.

So you think you know the answer

Testing is more about knowing the answer than it is about knowing the question!

Let me justify that stamen by starting with a little quiz.

I will give you the question, and you mark the options below as ‘pass’ or ‘Fail’ as if this were a software test,

1: What was King George VI’s first name?

A: William
B: George
C: Albert
D: Charles

2: How long did the ‘100 year war’ last?

A: 93 Years
B: 100 years
C: 101 years
D: 116 years

3: What is the acceptable load time for a standard web page

A: 2 seconds or less
B: Under 4 seconds
C: 10 seconds if containing rich media
D: 20 seconds

Of course if you know the answers, the decision to mark any result as a pass or a fail is easy, however if you just have the questions, you may not be so sure. The question may seem obvious to you, but how do you actually know what the correct answer is to generate a ‘pass’

The correct ‘pass’ criteria for question 1 is answer ‘C’ and for 2, it is ‘D’, however for question 3, there is no universally accepted ‘correct’ answer.

So you see, knowing the answer is much more important than knowing the question when it comes to testing. Requirements can sometimes hint at the answer. Lets work though an example;

‘Website links must be obvious and accessible’

Ok so my test would be,

‘Is this link obvious?’

But what is the answer?

This link has an’ under line ….. Pass
This link does not…. Fail…. Except that the link itself is blue and all the other text is black … so Pass … or fail .. er no pass!

You see the dilemma, unless it has been agreed what ‘obvious’ for each class of link will mean, then the result is open to interpretation, just like question 3 above.

The problem is further compounded by the fact that there will exceptions from time to time, so…

‘All textual links will be underlined and coloured blue’ is great, until you decided that actually, the small footer links (privacy policy etc.) will be underlined and grey. Again, knowing the answer for each link is more important than knowing the question (Is this link obvious?).

Of course in any real project you seldom get all the answers upfront, and so asking the question is an important way of getting the answer and I am certainly not saying you should wait till everything is nailed down before you write your tests. However I am suggesting that as testers we should not get too wedded to our tests and that if as a result of a test, an answer gets changed, then we should keep in mind that the answer is the important thing that drives us.

Get the right answer and you can ask the right question.

 

 

Tony Simms is Principal Consultant at Roque Consulting (www.roque.co.uk). He can be contacted via tony.simms@roque.co.uk.

‘Toggle’ – Or what ‘Boggle’ teaches us about testing

Ever Played Boggle?

The current Mrs Simms and I have now reached the age where the highlight of a romantic night in will be two or three rounds of Boggle. You probably know the game, where you have a 3 x 3 grid of random letters and you have to try to make as many words as possible in a given time. It always amazes me that we can come to the last few seconds and neither of us can find any more words, yet when we add up the scores, she has five or six words I did not see and I have seven or eight she did not see.  So what does that teach us about testing?

Two heads are better than one, three are better than two.

I have just received back review comments on my DR Test Plan from four or five different people. Many make the same comment about this or that, but they all have additional comments none of the others have. They not only see the same things differently, they also have different agendas, responsibilities and experiences that influence that review process.

The same is true of test execution, having just had a report back from a session of crowd testing its clear that things have been spotted and logged as bugs that none of the previous reviews and test cycles picked out. Again it has to do with how our brains are all wired differently and how that influences what we see.

Results from the testing of common pieces of code is the combination of scientiffic experience of years suffixed by years of personal growth and development.

If you ask people to count the number of times the letter ‘f’ appears in the sentence above, it is interesting to see how many different answers you get. (by the way with the double ‘f’ in scientific I am pretty sure there are 9 or is it 10?) 

 Different is Good, but… Different can be Helped

Having agreed that different people see things differently and that brings value to the review and test process, that is not to say that you can’t teach some uniformity in to what to look for.

Below is a set of guidelines for reviewing documents, taken from the excellent book

‘Software testing In the real World’  by Edward Kit.

The check list below is adapted from the Boeing Computer Services Requirements Checklist

1. Complete.          All items needed to specify the solutions to the problem have been included.
2. Correct.              Each item is free from error.

1. Precise,               unambiguous , and clear. Each item is exact and not vague; there is a single interpretation; the meaning of each item is understood; the specification is easy to read.

4. Consistent.       No item conflicts with another item in the specification.
5. Relevant.           Each item is pertinent to the problem and its solution.
6. Testable.            During program development and acceptance testing, it will be possible to determine whether the item has been satisfied.
7. Traceable.        Each item can be traced to its origin in the problem environment.
8. Feasible.            Each item can be implemented with the available techniques, tools, resources, and personnel, and within the specified cost and schedule constraints.
9. Free of unwarranted design detail.          The requirement specifications are a statement of the requirements that must be satisfied by the problem solution, and they are not obscured by proposed solutions to the problem.
10. Manageable.  The requirements specification can be controlled in such a way that each item can be changed without excessive impact on other items. Changes to the completed requirements specification can be controlled; each proposed change can be traced to an existing requirement; and the impact of the proposed change can be accessed.

Some methods of transforming specifications to reveal ambiguities, errors and/or misunderstandings:

1. Read the sentence several times, varying the stress pattern to reveal possible ambiguities.
2. When a term is defined explicitly somewhere, try substituting that definition in place of the term to see if the definition fits in this context
3. When a structure is described in words, try to sketch a picture of the structure being described.  This is especially useful where hierarchical structures are being described.
4. When a picture describes a structure, see if redrawing the picture reveals different possible meanings
5. When there is an equation, try expressing the meaning of the equation in words.
6. When a calculation is specified or implied in words, try expressing it in an equation.
7. When a calculation is specified, work at least two examples by hand and give them as examples in the specifications.
8. Look for statements that in any way imply certainty and then look for proof.  Words such as ‘always’, ‘every’, ‘all’, ‘none’ and ‘never’ are useful clues to indicate unproved certainty. 
1. When you are searching behind certainty statements, PUSH THE SEARCH BACK as many levels as are needed to achieve the kind of certainty a computer will need. 
2. Be on the lookout for words that are supposed to be persuasive, such as CERTAINLY, THEREFORE, CLEARLY, OBVIOUSLY, or ‘AS ANY FOOL CAN PLAINLY SEE’.  If its not obvious to you then there is a good chance that others will interpret it differently from the author’s intended meaning.
3. Watch for vague words, such as SOME, SOMETIMES, OFTEN, USUALLY, ORDINARILY, CUSTOMARILY, MOST, or MOSTLY.  These must be backed by  conditions that you can test (and the developers can code to)
4. When lists are given, but not completed, make sure that there is a complete understanding of the nature of the subsequent items.  Watch out for ETC., AND SO FORTH, AND SO ON, or SUCH AS.
5. In attempting to clarify lists, as in (12), we sometimes state a rule.  Be sure that the rule doesn’t contain unstated assumptions.
6. Look for lists without examples or examples that are too few or too similar to each other to explicate the rule.
7. Beware of vague verbs such as HANDLED, PROCESSED, REJECTED, SKIPPED, or ELIMINATED.
8. PASSIVE VOICE constructions are also traps.  Since the passive voice does not name an actor, it’s easy to overlook who is doing the work.  An example is ‘The log file is deleted later’.  Who or what deletes the log file?
9. Be especially on the lookout for comparatives without referents.
10. Pronouns are often clear to the writer and not to the reader.