Bryan's Coding Corner

Pop quiz: What are the different levels where you can catch defects in your code? Try to think of at least 5, I’ll wait …

I came up with nine:

1. Compiler
2. Static Code Analysis
3. Code Contracts/Assert Statements
4. Developer Testing
5. Unit/Regression Testing
6. Code Reviews
7. Dynamic Code Analysis
8. QA Testing
9. Users

This is an important list to have in the back of your mind as you program because as defects travel through what I call the Circles of Defensive Programming they get more difficult and costly to fix. If a bug gets all the way to your users it can become very costly for you and your company to fix. Now, not only will it take a lot of time and money to fix the bug, the solution will need to be deployed at an additional cost and your users probably won’t be too happy either.

So how can we make sure that any bugs that manage to get past our amazing coding skillz get caught as quickly as possible? I don’t know. If I knew I would be a multi-bazillionaire and I’d take over the world by systematically buying it (My easiest and most likely plan for world domination by the way). It takes hard work and dedication from the entire team at every stage of development just to make software that isn’t bug-filled trash. There is no silver bullet that will allow you to build quality software except hard work . But, there are some things you can do to catch defects as quickly as possible and make sure few if any get to your users.

• Make sure you have a good system at each of these levels. Each is good at catching different kinds of bugs and missing even one leaves a large gap in your bug stopping defenses. Get a system in place at each level and constantly improve it by making the weakest level stronger. How do you know which circle is the weakest? Keep track! Every time you find a bug figure out which circle should have caught it. That circle is weaker than it needs to be and if you get a few bugs that all point to one area you know where to focus your improvements.
• Automate as much as possible so it doesn’t take extra time from anyone. Bug checking that isn’t automatic and painfully simple (i.e. part of the automatic build process) is one of the first things to get tossed when things get a little behind schedule because, after all, we all write perfect code and every minute spent bug checking is a minute that could be spent trying to catch up to the schedule.
• Make all of the circles as tight as possible without making them burdensome. Strict checks will catch more defects but if they get too strict they might start finding a lot of false positives or make the code more difficult to work with, taking up more time and money than they save. That’s not even considering the fact that programmers do what they do because they like to solve problems. If your bug catching system becomes a problem they will solve it, and almost always in a way that is less than optimal for either of you. Experiment with how strict you can make things, don’t try to go too fast, and don’t forget to collect and listen to feedback from the developers.
• If a bug gets past the level of Code Contracts/Assert Statements don’t just fix the bug and move on! Fix the bug and try to write the code in such a way that the next time it happens it gets caught at least one level sooner. If you can move the check all the way to a compiler error the program won’t even be able to be built until the bug is fixed. At that point the cost of fixing the defect might as well be free.

Okay, okay, I know what you’re thinking. “That’s all well and good Bryan. I’ve heard of unit testing and code reviews but I don’t even know what static or dynamic code analysis are and since I’ve never needed them before they can’t be that helpful. Can they?”

You have a valid point. You probably have never heard of static or dynamic code analysis before and I’m sure no bugs have ever been deployed to your customers right? I thought so. Well first I’d like to tell you that if you use a modern IDE you have used at least rudimentary static and dynamic code analysis, apparently without even knowing it, and I bet it helped you catch a lot of bugs before they got out the door too. It’s probably best for me to just explain each of the levels of maintainability and give an example of each (from the .Net stack).

1. Compiler – Your first line of defense against things like typos, misspelled variable names, access violations, and all manner of other problems that mean what your trying to do is illegal in your chosen language. It catches so many bugs that some programmers will say “It builds, ship it!”, we call these programmers interns who will never be hired. The compiler is your first and most powerful line of defense against bugs, everything else is just the reserves that get called up when things manage to slip past. So Use It! Listen to compiler warnings (most result from static code analysis done by the compiler by the way), turn all of them on and get rid of the ones that are there. True, sometimes they are superfluous but you can suppress them on a case by case basis. Always treat a compiler warning just like an error and you’ll spend less time tracking down bugs and more time actually getting stuff done.
2. Static Code Analysis – Most modern IDEs have some static code analysis built in. Many of the warnings your compiler generates are the result of static code analysis. A more powerful analyzer like FxCop, StyleCop, lint, or to some degree ReSharper will let you know if you do things that probably weren’t what you intended. Things like “you never used this variable”, “you’re assigning in this IF statement rather than testing equality”, or “you’re raising the PropertyChanged event with a property name that doesn’t match the property it’s in”. Okay, I don’t know if any of them will give you that last one but it would be really helpful if they did. Interestingly enough, you can make it easier for a static code analyzer to find possible bugs by doing the same things you would to make a program more readable for a human. Most of the problems people tend to have with using a static code analyzer is that they don’t actually use one daily. Make it an automatic part of the build process if it can complete without doubling the compile time of your application. Otherwise make it a one-click process for a developer to run and put it on the build server as well. If the analyzer on the build server flags something then consider the build broken.
3. Code Contracts/Assert Statements – First, I’m just going to say assert statements when I mean both because code contracts are usually just a more powerful framework for assert statements. I always thought asserts were great for the hardcore anal-retentive types but they were superfluous in the real world because “I’m never going to be stupid enough to call this function with a negative number.” I was completely wrong. Assertions document the assumptions your code makes so that other developers know that null is never a valid argument for this code. If you have asserts throughout your code you’ll catch most instances of bad data as soon as it’s generated. One of the best things about asserts is that they are usually made so they don’t show up in release builds so you can happily assert away and know your team will be seeing error after error if the code is wrong but your normal error handling mechanisms will take care of the job when the code is actually released. This is a key point: Assert statements are not a replacement for proper error handling! You still have to handle any errors as if the assert statement wasn’t there. An assert statement is for documentation and catching things that, while they won’t throw an error, will cause your program to go all wonky. So how should you go about using Assert statements? Start in your Select-Case statements, if the case statements should run the whole range of inputs then you should assert that the default case never hit. Next go through the public functions of your classes and put some asserts in there to catch unexpected input arguments, but you need to make sure you still handle any errors that result from bad input. You’ll find many other places for a few nice assert statements but those should get you started. I know you’re thinking “Why should I use an assert rather than throwing an exception when I have to do both anyway?” That’s simple. An error can be caught and handled without anyone ever knowing it, an assert can’t. You can also add an assert while handling the invalid input nicely so the developers know it’s an error but your program continues to act normally. Even if you aren’t convinced, take the time to check out Code Contracts, I promise you won’t be disappointed.
4. Developer Testing – A lot of companies put way to much emphasis on developer testing. If the developer had thought of the error or missed requirements already they would have put in code to handle it. Developer testing is a good way to make sure the code does what the developer intended it to do in the cases they thought through. Unless some unexpected input is received the developer is only going to find and fix the most obvious bugs and then send it on. Even if you have a policy that the developer tests everything extremely thoroughly before it’s checked in they still won’t catch most of the bugs and it certainly won’t be the most cost effective way to get rid of defects in your program. Don’t get me wrong, developers should still test all of their code before checking it in but you shouldn’t expect them to find a very large percentage of the bugs that have survived until this point.
5. Unit/Regression Testing – Unit testing and Test Driven Development (TDD) are all the rage nowadays, and for good reason. Unit tests exercise the code with a large range of inputs that you aren’t likely to encounter in standard testing and regression tests make sure you don’t repeat bugs you’ve had before. There’s a lot of literature out therestating the case for unit testing much better than I ever could, take a look. Reading about unit testing might make it sound like it is the be-all and end-all of software quality and bug fixing. It isn’t, it is merely one more piece of a much larger puzzle. It’s also worth the time to check out Pex. It hasn’t yet reached its full potential but it’s developing quickly and will be incredibly powerful when combined with normally written unit tests.
6. Code Reviews – If you don’t have some sort of peer code review system in place, do it now! I’m not kidding. My company started using Code Collaborator around six months ago and it has proved invaluable. I’m planning on writing a review of it later but I’ll say it’s a great choice and well worth the price. Code reviews catch a large percentage of the defects that get to this point, IF the reviews are kept simple and only do one thing at a time. Fix one bug, implement one feature, etc. I’ve found that many developers don’t completely grok source control so multiple change-sets and switching among difference views can get confusing quickly. Keep it simple and you’ll find a lot of bugs, let the review get more complex and you’ll only find spelling errors. What code reviews do is get more sets of eyes on a section of code meaning more error conditions thought of and guarded against, less confusing code (reviewers should always speak up if they don’t get what the code is trying to do), and dissemination of knowledge of how the section of code works. Take Code Collaborator for a spin (they have a trial) or at least take a look at the articles, blog posts, and book they’ve put out on the best practices for performing code reviews and start reviewing your code.
7. Dynamic Code Analysis – Dynamic code analysis is usually used to find memory leaks, security errors, and sometimes Heisenbugs. Most modern IDEs also contain some dynamic code analysis, most commonly referred to as the debugger. Basically they watch your program as it runs to make sure it’s not doing anything it’s no supposed to. You can also use them to track memory usage or execution time if you have constraints on either of those. Honestly, outside of the debugger and profiling there aren’t a lot of compelling tools out there that I know of at this time but they are coming along quickly with the recent rise of dynamic languages. If you are running in a memory or execution time constrained environment they are definitely something to look into now, otherwise there will be some excellent tools coming along in the next couple of years so you should keep your ears open.
8. QA Testing – Testers are your last line of defense against bugs getting out to your users. It’s their job to test the program in every way possible and figure out the steps to reliably reproduce every bug they find. They will also probably give a lot of usability input. Not just anyone can be a tester, it takes a very rare person and there will be as much variability in the skills of the testers as there is in the skills of developers. Don’t skimp on your testers! Make sure the good ones are well paid, not bored, and have a clear career advancement path. Also, good testing takes a long time, don’t expect someone to be able to go through an entire application in many different situations before lunch. Delegate time to test a feature just as you did to implement it.
9. Users – You can always hope that no bugs actually make it to your users, but we all know that some will sneak out. Make sure you have a good support staff with an actual human on the phone and quick response times. They should be capable of troubleshooting the problem and collecting all of the information that is relevant. Once they get it to developers and get their feedback they can give the customer a good idea of when to expect the issue to be fixed.

Recommended References:
Code Complete 2nd Edition by Steve McConnell, specifically Chapters 20-22
Best Kept Secrets of Peer Code Review by Jason Cohen
Microsoft Devlabs: Code Contracts

Tweet