Click the image for a link to Tech Ed! And be sure to drop by and visit us at the Team System booth!

CAUTION – THIS INVOLVES GOING INTO THE TFS DATABASE – DANGER WILL ROBINSON, DANGER

Okay, having given you that warning, here’s what you want to do:

Look in the database server for the TfsIntegration database. Run the following selects:

select * from tbl_subscription

select * from tbl_security_identity_cache

The first table gives you the list of subscriptions. However, the users aren’t displayed in the table, only the SID’s. Look in the second table for the SIDS. You can delete rows out of the tbl_subscription table without harm which will remove the alert.

Do NOT delete the first four alerts (which are not assigned to a user and don’t have a filter).

TAKE A BACKUP FIRST.

It never hurts

Let’s start with functional bugs. These are the bugs that really shouldn’t be in applications. Now, I am a realist and I understand that it is impractical to try to eliminate all bugs (you can do it but at what cost?) in most applications. This does not hold true for life-critical applications in which there is no margin for error. However, how do you eliminate all of the functional bugs? Testers are not the way to eliminate bugs. If a bug gets to the testing stage you’ve already introduced waste. You want to ensure that before the software gets to the testers, that there are no bugs.

Note: In Lean theory, there would be no testers (or “inspectors”) because all they would be doing is finding a problem that shouldn’t have been introduced in the first place. But in software development I believe this is impractical in most cases.

Let me pose this question to you (this is more applicable if you are a developer but if you aren’t, put yourself in the developers’ shoes): How does a developer know when they are done writing code?

Seems like a simple question, right? I commonly get five answers when I ask this question:

1. You’re never done coding a requirement, it’s an ongoing, evolving process with no end in sight.
2. When my code does what the requirement says it should do.
3. You’re done when the analyst says you’re done.
4. You’re done when all of the unit tests pass.
5. You’re done on the day the code is due.

The fourth answer is usually from groups doing Test Driven Development or heavy unit testing in a true agile process such as Extreme Programming (XP). I’ll address questions about methodology later in this book.

Looking at answers 1, 2, 3 and 5 you can almost feel the depression. Certainly answer number 1 is depressing – essentially you never finish and you can never know when you’re finished. Answer 5 means that the developer feels that maybe they don’t get the time to finish the way they would like to – again, depressing. Answers 2 and 3 are essentially the same (you hope anyway) but there is a major problem. As I’ve noted already, requirements are subjective which means that while you think you’re done you probably aren’t done.

Test Cases before Code

A better answer is: “I’m done when all of the test cases for this requirement pass.” Now this presents some problems. First, most organizations do not create test cases at the same time as they create requirements. Second, those organizations that do create test cases at this stage do not create repeatable test cases (a.k.a. flimsy test cases). Finally, I have not worked with any organizations that require the users to sign off on test cases the same way they have to sign off on requirements. It kind of doesn’t make sense does it? To have customers sign off on documents which are mostly vague but not have the customers sign off on documents that provide definitive proof that a requirement has been met.

So then, the solution to this problem is simple, assign testers to work with analysts and write test cases for requirements before the requirements are handed to developers. Sounds simple, right? This is where a cost justification comes in. You need additional resources earlier in the process which will save costs later in the process. It’s hard to prove a negative to management and get the appropriate resources and funding.

Another issue with this simple solution is that depending on the type of methodology you are using, it may be difficult to break people of old habits. The manners in which requirements are documented make a huge difference in the ease of creating related test cases. For instance, the best format (let me qualify this – in my opinion) for requirements in relationship to test cases are use cases. A sample use case is shown in the table below.

While you can probably find lots of other situations, alternate paths and exception paths, this serves as a good example for entering items you may find at any retail store.

The reason why use cases are the best starting point for test cases is because they already describe the interaction between the user and the system so you know what the user is supposed to be providing and you know what information the system is supposed to be responding with.

If you are using a methodology such as Scrum which advocates user stories there may be some difficulties. Not because there is anything inherently wrong with user stories, but you can’t write code from a user story. User stories are designed to note a high level requirement and describe the users’ use of the functionality to provide perspective and context for the developers. What many forget to do is to provide more detailed requirements (which can be in the form of use cases) which can form the basis of test cases (if you are going to write use cases based off of user stories, then the description would be the user story). In other circumstances there is no formal method for writing requirements documents which means the structure of each document is entirely up to the analyst who writes the requirements.

Let’s look at a sample test case which tests the use case in the table above.

HOLD EVERYTHING! What’s wrong with this test case (besides that it isn’t complete)? This is the typical test case that I see when I work with customers all of the time. If you are a professional tester reading this you know the answer to this: This test case is not repeatable. Believe it or not, I have had to disabuse people of the notion that this type of test case is a good test case. But let’s see what the problem is here. In step 1 I log on as the user. Well, what user? It assumes that the logon was successful so that means I need to log on with a valid user. What user though? This is a test system and will hopefully not have production values in it. In step 3 I’m supposed to scan a barcode for the test. What barcode? Can I use the paperback book I’m reading at lunch for this purpose or maybe the barcode from the candy bar that I bought for lunch? Again, the system is supposed to display the appropriate information which assumes I’ve picked an item actually in the inventory. This makes for a poor testing and a poor coding experience. Let’s try this test case again.

Now this is almost a repeatable test case. What’s missing? There really is only one thing missing here (there are a few minor things but I’m sticking with the major issues) – the setup. How does the system know the tax amount is 8.25%? It can be different depending on where the machine is located. Test cases need to included necessary infrastructure setup and in this case the test would have to be run with different sales tax amounts (or no sales tax amount if we want to assume that maybe this is being used for internet sales in some cases).

In order to have a complete battery of tests this test case would have to be repeated with numerous different values. How many different scenarios do you test? What are the likelihood of the different scenarios occurring and what is the cost if they do occur and the software fails. In the above example, one may assume that you could simply ignore the situation in which the item isn’t found in inventory. After all, if the item is on the store shelf it had to have been received and if it was received it was scanned into inventory. Maybe the chance of this occurring is very, very low but what if it does occur? Without being able to manually enter the item you would not be able to sell the item to the customer. So the cost of this mistake is the amount of the item plus some of the stores reputation. This may be too high a price. It is critical to understand the importance of each of the scenarios when making the decision of what to test.

Well, why don’t we just test everything? The short answer is that it isn’t feasible (again, this does not apply to life-critical applications but it does apply to virtually everything else). The cost in time, resources and equipment to test everything has to be balanced against the ROI. I believe it is possible to make error free applications but at what cost? If it costs 10,000 to test every aspect of the alternate paths but those only become a reality once every 5,000 sales for low dollar amounts, does it provide a benefit? The answer is probably not. As with almost everything else, use the 80/20 rule. That is, 80% of the usage of your application is going to involve 20% of the code (we write an enormous amount of code to handle exception paths and alternate situations). Test that 20% of the code 100%. The other 80% of the code can be tested as problems are discovered or as time permits.

Now that you have a series of good test cases which concretely detail what the system must do, a developer can now answer the question, “When do you know you are done coding?” The answer is, “When the functional test cases I was provided pass.” This is the only right answer (but not usually the one the developers can use).

Roles & Responsibilities

The figure below shows who is responsible for transitioning items between states in a perfect world.

This figure very succinctly states the following:

1. A user, analyst, developer or tester can file a bug report.
2. A project manager must assign the bug report to someone to investigate. The person they will assign it to is an analyst.
3. An analyst can determine if the investigation is complete. They can also close a bug for the reasons noted in the previous section.
4. A project manager assigns the bug to a developer (or not if the bug is being fixed in a later release).
5. A developer will set the bug to Active when they start working on it.
6. A developer determines when the bug is fixed.
7. Depending on the release process, either a Release Manager or QA Manager will note when the bug is ready for testing (and assign it to a tester). This is because there may be a configuration management (i.e. branching structure) which the code fix needs to be promoted through so that the testers can actually get it in a good build.
8. A tester will note that the bug is being tested.
9. A tester will note that the bug fix has been verified. If the bug has not been fixed, a tester can also re-assign the bug to a developer.
10. A release manager will determine which release the bug fix will be deployed in and prepare the appropriate documentation.
11. Upon successful release, the project manager will close the bug.

Reports

So, you have this great bug tracking process now and you know what information you can get from each step. How do you get it easily? What reports are most important? Where do you cut down on waste?

Waste

Let’s start with looking at a report showing waste and determine how it might be eliminated. The next figure shows a chart of the average flow time for bugs in a system using the above process.

This chart is based on the following data:

Note that I have included only working hours – no weekends or after hours. While this would normally be an average of these values, I wanted to demonstrate how time is determined. Still, if this were an average, you would note that the average time to fix bugs and deploy them is 67.5 hours or almost three days on average. Where can you reduce the amount of time wasted?

Note

In Lean, batching is considered a bad process. The reason for this is that when you batch it means that someone is waiting to receive all of the times that make up that batch and therefore time is wasted. However, in software development, and specifically when dealing with change and release configuration management there really is no other way because fixes almost always must go in together. Promoting them one at a time would take more time than it would save and is also a major configuration headache.

Looking at the chart and table, the actual time spent working on the bug is 15.25 hours (the hours spent investigating, fixing and testing). The time wasted is 50.25 hours (the hours spent waiting while the bug was waiting to be investigated, between the investigation being completed and the bug being assigned, between the bug being assigned and the bug being worked, between the bug being fixed and it being tested and finally between the time the bug was tested until the time the bug was deployed). Again, going back to the principle of waste in Lean theory, the 50.25 hours is complete waste. Now having said that, let’s take a reality check since not all of Lean can apply to software development. Where can you reasonably shave hours?

The first thing to do is to understand why an item may be in a given state for an extended period of time. Let’s look at a hypothetical process and see what may happen in some of these states where waste is occurring.

The next post in this series will cover the hypothetical process. We’ll also start covering different scenarios and organization size when dealing with issues because this is NOT a one-size fits all process.

Fixed, unverified

This state indicates the bug has been fixed according to the developer.

This means that the developer has fixed the code, checked it in and associated it with the appropriate bug work item. They have run the test case provided by the analyst in the Under Investigation state. In addition, the time between the bug being in the Active state and the Fixed, unverified state is the time it took the developer to fix the bug. In other words, we now have an approximation of the developer effort to fix the bug. It doesn’t provide an exact time but it provides an overall throughput time.

Ready for Testing

This state indicates the bug has been assigned to a tester but the testing has not yet started.

There are a number of reasons for this state. The first is that your branching model is such that a tester does not have access immediately after the code has been finished (i.e. you are waiting to promote code to the QA branch). The second is that there is simply other work going on and that the tester just can’t get to it right now. The transition between Fixed, unverified and Ready for Testing should be made by a Test Manager to acknowledge that the bug has been assigned to a tester (or to the testing group in general) and that it is on the right code branch for testing. The transition to this state does not provide any other meaningful information.

In Testing

This state indicates that the bug is actively being tested.

The other important information provided by this state is the amount of time between the bug being ready for testing and when the testing started. This again shows waste.

Fixed, verified

This state indicates that the bug has been fixed according to independent testers.

The time between this state and the In Testing state provides the amount of time spent testing this bug and is considered work time.

Note: For those familiar with Lean theory, this work would be classified as waste. The reason is because work should never pass a station with defects – it means the process for building the code needs to be fixed. However, as I have mentioned before, I believe this is mostly impractical in software development.

Scheduled for Deployment

This state simply indicates that the fix is scheduled to be released.

An important piece of information to capture at this state is the release that the fix is being deployed in. This helps create a Release Notes document to provide the user explaining the changes in the upcoming release.

Closed

The bug is closed after the release that it is included in has been deployed.

The time between the bug report being Proposed and the bug being Closed provides the entire throughput for the bug being fixed. This information is a good metric of whether or not your process is improving (if reducing the amount of time to accept and fix bugs is one of your goals). While removing individual pieces of waste is important, you have to keep your eye on the overall goal.

A note about UAT

You may be wondering about the lack of inclusion of User Acceptance testing steps to track users’ acceptance of the bug. There are two reasons why this isn’t included. The first is, does it provide you any additional information? Sure, it lets you know the users accepted it, but you can do that as part of the QA process. Second, by doing UAT separately it means that you have to have separate UAT environments from QA which many organizations can’t afford. Is the cost worth it? This may be something you need to take into account as you think through your process.

This finished the process and description of each step in the process. The next post in this series will cover Roles & Responsibilities and Reports and how to track improvements in the process.

Let’s look at each of the states so we understand what the state represents. And if this looks like the start of a new methodology, well, it is and it isn’t. It is because I’m going to detail out of this series of posts how to handle numerous items and there is a lot of methodology and process displayed here. But it isn’t because much of this is just common sense. Also, note that you do not need to implement every step. This process that I’m describing here can be scaled based on your needs.

The states and their descriptions are these:

Proposed

This state indicates that a reported bug has been received. It does not indicate that any action has been taken with regard to working on the reported bug.

The key item to note is the reason which in this case tells us what group filed the bug. This is important because as has already been noted, it costs more to fix a bug the later it is found in the process. This allows you to note where the bugs are found and determine where certain weaknesses lie.

Under Investigation

This state provides status to the end user to indicate that the bug report is currently being triaged.

The state transition provides us one very useful piece of information – the time it takes from the bug being received until someone starts working on it. This is the first of the areas where we find waste.

This state requires some work though – no free rides here. Several things need to happen:

1. A duplicate bug must be checked for. If a duplicate is found, then this bug needs to be closed and linked to the original bug report.

2. Find the test case that relates to the functionality at issue.

3. The analyst needs to determine what the correct functionality is supposed to be. If it works correctly (as designed) then what the user is actually reporting is a change request and a change request needs to be created. This bug can be closed and linked to the change request.

4. The bug needs to be reproduced. If it can’t be then it is closed.

5. If the bug can be reproduced, then the test case needs to be modified (or created).

What does this information provide us?

If the bug is a duplicate it means at least one thing and maybe two. First it means that this bug occurs in a commonly used piece of code and is therefore a cause for concern. Secondly, depending on the lag time between the original bug report being filed and the duplicate bug being filed may indicate that the bugs aren’t being fixed in a reasonable period of time.

If you cannot find the associated test case(s) then there is another issue. First, why was no test case created? Was it because this is in a less frequently used part of the system? If so, that may be acceptable (maybe it isn’t part of the normal path and so minimal testing was done on it), but it means that this part of the system is being used and additional test cases may need to be created to proactively identify any additional, related bugs. If you do find the test case, what about the test case missed the bug? Is there a gap in the test case? Are all paths not tested? Can you go back and review the code coverage report to see if there is a path through the code that was missed? These questions and their answers will help you proactively identify other areas in this particular area of code so you can take mitigating steps.

What if the functionality is as designed? The user is then requesting a change to the functionality – maybe. They may not understand how the functionality is supposed to work. But what does this tell you? Were the users trained in the correct use of the application? Is there online help for the application and is it effective? If the users haven’t been trained and you receive a high number of these items then maybe it’s time for a training class. If they have been trained and there is online help, then maybe the help isn’t effective or effectively integrated into the application. This may indicate that changes in those areas are needed – but only if you know what to look for.

If the bug can’t be reproduced, it must be closed. However, it must not drop off of the radar. When duplicate bugs are found you need to remember to check the closed bugs as well. If you find a number of non-reproducible bugs being duplicated, it may be time to activate the bug and try to have the developers spend some time investigating the situation.

Finally, if the bug is validated, that isn’t good enough. There needs to be a test case which shows what the correct behavior of the system is supposed to be so that the developers know when they are done fixing the bug and they have a way to verify that fix. If there is an existing test case and it is incorrect, then you need to look back at the requirements to determine if the test case was incorrect or whether the requirements were incorrect and make the appropriate changes (each of these points to a different area that needs to be looked at for improvement).

Investigation Complete

This state indicates that the bug report has been verified and is now an actual bug. And again, it provides a solid indication to the end user of the status of their bug report.

The transition from Under Investigation to Investigation Complete provides the first piece of information on work performed in order to fix the bug (after all, you can’t fix the bug until you have verified it).

At this point the assignment of work can commence. However, there may be a high number of work items to assign so it may be up to a Change Control Board (CCB) to accept and validate work for a release. If that is the case then the list of Investigation Complete items will make up part of the agenda. This however can present problems. See the section entitled “Waste” later in this chapter for the problems related to this and some ideas on how to fix them.

Assigned

This state is an indicator that the bug has been assigned to a developer for work.

This is called a buffer state. The transition between Investigation Complete and Assigned is basically unimportant. What is important is that this transition says is that you are planning to fix the bug. Now, what it doesn’t tell you is what release you are planning on fixing it in. Using TFS you can simply re-assign the bug to an appropriate future iteration. The assigned state also tells users that no one has started working on the bug yet. Additionally this state provides feedback to the CCB about what work is scheduled to be done but hasn’t started yet so the CCB doesn’t accept more bugs than can be reasonably fixed.

Active

This state indicates that the bug is actually being fixed. No bug must ever move to this state without having an accompanying test case.

This state isn’t as important as the transition to this state. The time between the bug being assigned and the bug being active is lag time and is therefore waste.

One of the activities that should be performed during this phase is the creation of a unit test to verify the existence of the bug in code. This will help create a large library of unit tests which target previous bugs which makes regression tests easier to run.

In this part I’ve started to describe what happens in each state and what information you can gather from each state. In the next post I will round out the steps in this particular process and then I’ll move on to discussing the benefits of the metrics. Note that this process covers virtually every type of metric you may need (there are a few it doesn’t cover but I’ll talk about that in the next post). Note also that you can pick and choose which states you want. For example, if you’re reasonably confident that you don’t have bugs sitting around for a long period of time, then maybe the early buffer states aren’t needed. Or if you aren’t sure where the problem is occurring, use broad categories and introduce additional steps to pinpoint a problem as needed.

Stay tuned for the next post!

I’ll be giving three talks on the new Team System 2010, Architecture and Test editions.

Take advantage of this offer – especially if you own the Development Edition as the Database Edition provides numerous advantages.

Enjoy!

Any and all comments are appreciated – especially if you think I got something wrong.

This part lays the groundwork for practical process improvement.

At this point you’ve made one of two decisions – you are going to implement process for the first time (your current process is referred to as chaos) or you are going to improve your existing process. Process for process improvements sake is a poor reason to change or implement a process. So, what are you trying to improve (a.k.a. where are you now)? Before you know where you want to go you have to have some type of baseline to measure your progress against. Implementing a process is all about small, measured steps – try to do everything at once and you are guaranteed to fail.

You want to work in small iterations in order to improve your process – even if you are working on a formal (or waterfall) based project. Why small iterations? Did I just tell you to change to an agile development process? Well, no, not exactly. Short iterations allow for rapid feedback and the ability to measure and introduce change quickly and at well defined points. This does not mean you have to start using XP or Crystal or Scrum or any of a dozen other agile methodologies. And these methodologies don’t work particularly well in every situation – especially if you are working on a much more structured project such as a life-critical system.

The goal here is not so much to measure velocity (although that in itself is highly useful), unless that’s an area you want to improve on which I’ll talk about in a much later part of the series, as to be able to introduce a change and gauge its effectiveness.

The other thing that is critical is metrics. If you’re making changes because you think you need to make a change (aka “the gut feeling reason”) then you aren’t going to be able to prove whether the change helped you or not except annecdotally. While we as developers don’t mind this, management really does. They don’t want to plow money into process changes without knowing the benefit they are getting for their dime. So let’s give them what they want and also help ourselves in the process. Don’t make any changes without having a baseline – ever. In addition, when you are doing a retrospective on whether the change worked or not, do it by comparing metrics (as well as gut feeling) to determine the effectiveness.

Next, pick and choose what you want to change (aka get the low hanging fruit first). Choose items that are more simple to change and are easier to measure. In light of this, I’m going to start off this series with simple areas (or at least, in what our experiences with customers have taught us are simple areas) and move to more difficult areas later on.

In preparation for the next part of this series, the first area I’m going to walk you through improving is the bug handling process. This doesn’t cover how to reduce the number of bugs you get initially (well, it does in part but that isn’t the focus) but how you handle the bugs in your process. As I mentioned before, you need baseline metrics to determine the effectiveness of the changes. To begin with, spend the next week or two and measure how long it takes from when you receive the bug (the date and time – to the hour) to when the bug fix is released into production.

In the next part I’ll talk about how to reduce that time frame (obviously this fix will only be important if you feel that it takes a large period of time to implement the fix and that your customers are unhappy) and what you should be doing at each stage.