Practical Process Improvement (Part 5)

The first four posts (Part 1, Part 2, Part 3, Part 4) dealt with working through a process for handling bugs and improving that process. But what about preventing bugs to begin with? In this part I’ll talk about some preventative measures to help eliminate bugs at the requirements and development phase.

Let’s start with functional bugs. These are the bugs that really shouldn’t be in applications. Now, I am a realist and I understand that it is impractical to try to eliminate all bugs (you can do it but at what cost?) in most applications. This does not hold true for life-critical applications in which there is no margin for error. However, how do you eliminate all of the functional bugs? Testers are not the way to eliminate bugs. If a bug gets to the testing stage you’ve already introduced waste. You want to ensure that before the software gets to the testers, that there are no bugs.

Note: In Lean theory, there would be no testers (or “inspectors”) because all they would be doing is finding a problem that shouldn’t have been introduced in the first place. But in software development I believe this is impractical in most cases.

Let me pose this question to you (this is more applicable if you are a developer but if you aren’t, put yourself in the developers’ shoes): How does a developer know when they are done writing code?

Seems like a simple question, right? I commonly get five answers when I ask this question:

1. You’re never done coding a requirement, it’s an ongoing, evolving process with no end in sight.
2. When my code does what the requirement says it should do.
3. You’re done when the analyst says you’re done.
4. You’re done when all of the unit tests pass.
5. You’re done on the day the code is due.

The fourth answer is usually from groups doing Test Driven Development or heavy unit testing in a true agile process such as Extreme Programming (XP). I’ll address questions about methodology later in this book.

Looking at answers 1, 2, 3 and 5 you can almost feel the depression. Certainly answer number 1 is depressing – essentially you never finish and you can never know when you’re finished. Answer 5 means that the developer feels that maybe they don’t get the time to finish the way they would like to – again, depressing. Answers 2 and 3 are essentially the same (you hope anyway) but there is a major problem. As I’ve noted already, requirements are subjective which means that while you think you’re done you probably aren’t done.

Test Cases before Code

A better answer is: “I’m done when all of the test cases for this requirement pass.” Now this presents some problems. First, most organizations do not create test cases at the same time as they create requirements. Second, those organizations that do create test cases at this stage do not create repeatable test cases (a.k.a. flimsy test cases). Finally, I have not worked with any organizations that require the users to sign off on test cases the same way they have to sign off on requirements. It kind of doesn’t make sense does it? To have customers sign off on documents which are mostly vague but not have the customers sign off on documents that provide definitive proof that a requirement has been met.

So then, the solution to this problem is simple, assign testers to work with analysts and write test cases for requirements before the requirements are handed to developers. Sounds simple, right? This is where a cost justification comes in. You need additional resources earlier in the process which will save costs later in the process. It’s hard to prove a negative to management and get the appropriate resources and funding.

Another issue with this simple solution is that depending on the type of methodology you are using, it may be difficult to break people of old habits. The manners in which requirements are documented make a huge difference in the ease of creating related test cases. For instance, the best format (let me qualify this – in my opinion) for requirements in relationship to test cases are use cases. A sample use case is shown in the table below.

While you can probably find lots of other situations, alternate paths and exception paths, this serves as a good example for entering items you may find at any retail store.

The reason why use cases are the best starting point for test cases is because they already describe the interaction between the user and the system so you know what the user is supposed to be providing and you know what information the system is supposed to be responding with.

If you are using a methodology such as Scrum which advocates user stories there may be some difficulties. Not because there is anything inherently wrong with user stories, but you can’t write code from a user story. User stories are designed to note a high level requirement and describe the users’ use of the functionality to provide perspective and context for the developers. What many forget to do is to provide more detailed requirements (which can be in the form of use cases) which can form the basis of test cases (if you are going to write use cases based off of user stories, then the description would be the user story). In other circumstances there is no formal method for writing requirements documents which means the structure of each document is entirely up to the analyst who writes the requirements.

Let’s look at a sample test case which tests the use case in the table above.

HOLD EVERYTHING! What’s wrong with this test case (besides that it isn’t complete)? This is the typical test case that I see when I work with customers all of the time. If you are a professional tester reading this you know the answer to this: This test case is not repeatable. Believe it or not, I have had to disabuse people of the notion that this type of test case is a good test case. But let’s see what the problem is here. In step 1 I log on as the user. Well, what user? It assumes that the logon was successful so that means I need to log on with a valid user. What user though? This is a test system and will hopefully not have production values in it. In step 3 I’m supposed to scan a barcode for the test. What barcode? Can I use the paperback book I’m reading at lunch for this purpose or maybe the barcode from the candy bar that I bought for lunch? Again, the system is supposed to display the appropriate information which assumes I’ve picked an item actually in the inventory. This makes for a poor testing and a poor coding experience. Let’s try this test case again.

Now this is almost a repeatable test case. What’s missing? There really is only one thing missing here (there are a few minor things but I’m sticking with the major issues) – the setup. How does the system know the tax amount is 8.25%? It can be different depending on where the machine is located. Test cases need to included necessary infrastructure setup and in this case the test would have to be run with different sales tax amounts (or no sales tax amount if we want to assume that maybe this is being used for internet sales in some cases).

In order to have a complete battery of tests this test case would have to be repeated with numerous different values. How many different scenarios do you test? What are the likelihood of the different scenarios occurring and what is the cost if they do occur and the software fails. In the above example, one may assume that you could simply ignore the situation in which the item isn’t found in inventory. After all, if the item is on the store shelf it had to have been received and if it was received it was scanned into inventory. Maybe the chance of this occurring is very, very low but what if it does occur? Without being able to manually enter the item you would not be able to sell the item to the customer. So the cost of this mistake is the amount of the item plus some of the stores reputation. This may be too high a price. It is critical to understand the importance of each of the scenarios when making the decision of what to test.

Well, why don’t we just test everything? The short answer is that it isn’t feasible (again, this does not apply to life-critical applications but it does apply to virtually everything else). The cost in time, resources and equipment to test everything has to be balanced against the ROI. I believe it is possible to make error free applications but at what cost? If it costs 10,000 to test every aspect of the alternate paths but those only become a reality once every 5,000 sales for low dollar amounts, does it provide a benefit? The answer is probably not. As with almost everything else, use the 80/20 rule. That is, 80% of the usage of your application is going to involve 20% of the code (we write an enormous amount of code to handle exception paths and alternate situations). Test that 20% of the code 100%. The other 80% of the code can be tested as problems are discovered or as time permits.

Now that you have a series of good test cases which concretely detail what the system must do, a developer can now answer the question, “When do you know you are done coding?” The answer is, “When the functional test cases I was provided pass.” This is the only right answer (but not usually the one the developers can use).

Comments

By Jeff Levinson on March 10th, 2009 at 7:49 am

Great article.

Trackbacks

• Team System News : VSTS Links - 12/31/2008

Featured Video