Practical Process Improvement (Part 2)
In my last post I talked about why metrics are important and what the goals are in using metrics. In this post I’ll talk about the bug process. How does your team handle bugs, how can you gather metrics on it and which metrics are important? This post covers the process itself. So let’s take a look at an “ideal” bug handling process:
Let’s look at each of the states so we understand what the state represents. And if this looks like the start of a new methodology, well, it is and it isn’t. It is because I’m going to detail out of this series of posts how to handle numerous items and there is a lot of methodology and process displayed here. But it isn’t because much of this is just common sense. Also, note that you do not need to implement every step. This process that I’m describing here can be scaled based on your needs.
The states and their descriptions are these:
Proposed
This state indicates that a reported bug has been received. It does not indicate that any action has been taken with regard to working on the reported bug.
The key item to note is the reason which in this case tells us what group filed the bug. This is important because as has already been noted, it costs more to fix a bug the later it is found in the process. This allows you to note where the bugs are found and determine where certain weaknesses lie.
Under Investigation
This state provides status to the end user to indicate that the bug report is currently being triaged.
The state transition provides us one very useful piece of information – the time it takes from the bug being received until someone starts working on it. This is the first of the areas where we find waste.
This state requires some work though – no free rides here. Several things need to happen:
1. A duplicate bug must be checked for. If a duplicate is found, then this bug needs to be closed and linked to the original bug report.
2. Find the test case that relates to the functionality at issue.
3. The analyst needs to determine what the correct functionality is supposed to be. If it works correctly (as designed) then what the user is actually reporting is a change request and a change request needs to be created. This bug can be closed and linked to the change request.
4. The bug needs to be reproduced. If it can’t be then it is closed.
5. If the bug can be reproduced, then the test case needs to be modified (or created).
What does this information provide us?
If the bug is a duplicate it means at least one thing and maybe two. First it means that this bug occurs in a commonly used piece of code and is therefore a cause for concern. Secondly, depending on the lag time between the original bug report being filed and the duplicate bug being filed may indicate that the bugs aren’t being fixed in a reasonable period of time.
If you cannot find the associated test case(s) then there is another issue. First, why was no test case created? Was it because this is in a less frequently used part of the system? If so, that may be acceptable (maybe it isn’t part of the normal path and so minimal testing was done on it), but it means that this part of the system is being used and additional test cases may need to be created to proactively identify any additional, related bugs. If you do find the test case, what about the test case missed the bug? Is there a gap in the test case? Are all paths not tested? Can you go back and review the code coverage report to see if there is a path through the code that was missed? These questions and their answers will help you proactively identify other areas in this particular area of code so you can take mitigating steps.
What if the functionality is as designed? The user is then requesting a change to the functionality – maybe. They may not understand how the functionality is supposed to work. But what does this tell you? Were the users trained in the correct use of the application? Is there online help for the application and is it effective? If the users haven’t been trained and you receive a high number of these items then maybe it’s time for a training class. If they have been trained and there is online help, then maybe the help isn’t effective or effectively integrated into the application. This may indicate that changes in those areas are needed – but only if you know what to look for.
If the bug can’t be reproduced, it must be closed. However, it must not drop off of the radar. When duplicate bugs are found you need to remember to check the closed bugs as well. If you find a number of non-reproducible bugs being duplicated, it may be time to activate the bug and try to have the developers spend some time investigating the situation.
Finally, if the bug is validated, that isn’t good enough. There needs to be a test case which shows what the correct behavior of the system is supposed to be so that the developers know when they are done fixing the bug and they have a way to verify that fix. If there is an existing test case and it is incorrect, then you need to look back at the requirements to determine if the test case was incorrect or whether the requirements were incorrect and make the appropriate changes (each of these points to a different area that needs to be looked at for improvement).
Investigation Complete
This state indicates that the bug report has been verified and is now an actual bug. And again, it provides a solid indication to the end user of the status of their bug report.
The transition from Under Investigation to Investigation Complete provides the first piece of information on work performed in order to fix the bug (after all, you can’t fix the bug until you have verified it).
At this point the assignment of work can commence. However, there may be a high number of work items to assign so it may be up to a Change Control Board (CCB) to accept and validate work for a release. If that is the case then the list of Investigation Complete items will make up part of the agenda. This however can present problems. See the section entitled “Waste” later in this chapter for the problems related to this and some ideas on how to fix them.
Assigned
This state is an indicator that the bug has been assigned to a developer for work.
This is called a buffer state. The transition between Investigation Complete and Assigned is basically unimportant. What is important is that this transition says is that you are planning to fix the bug. Now, what it doesn’t tell you is what release you are planning on fixing it in. Using TFS you can simply re-assign the bug to an appropriate future iteration. The assigned state also tells users that no one has started working on the bug yet. Additionally this state provides feedback to the CCB about what work is scheduled to be done but hasn’t started yet so the CCB doesn’t accept more bugs than can be reasonably fixed.
Active
This state indicates that the bug is actually being fixed. No bug must ever move to this state without having an accompanying test case.
This state isn’t as important as the transition to this state. The time between the bug being assigned and the bug being active is lag time and is therefore waste.
One of the activities that should be performed during this phase is the creation of a unit test to verify the existence of the bug in code. This will help create a large library of unit tests which target previous bugs which makes regression tests easier to run.
In this part I’ve started to describe what happens in each state and what information you can gather from each state. In the next post I will round out the steps in this particular process and then I’ll move on to discussing the benefits of the metrics. Note that this process covers virtually every type of metric you may need (there are a few it doesn’t cover but I’ll talk about that in the next post). Note also that you can pick and choose which states you want. For example, if you’re reasonably confident that you don’t have bugs sitting around for a long period of time, then maybe the early buffer states aren’t needed. Or if you aren’t sure where the problem is occurring, use broad categories and introduce additional steps to pinpoint a problem as needed.
Stay tuned for the next post!
Comments
Trackbacks