Monthly Archives: January 2011

Where do software bugs come from?

Last night, I came across this question on StackOverflow, essentially asking how you can be a zero-bug programmer. Many of the responses can be boiled down to “Don’t write any code.”, which is pretty much how I’d respond to that question.

If you continue to throw ever-increasing amounts of time and money at a software project, you can certainly get closer and closer to feeling confident you have no problems in your code, but almost any non-trivial piece of software is likely to have some (albiet, probably rare and minor) bugs in it.

I don’t, however, think an appropriate response to this fact is to throw up our hands and say “oh well, what can you do?”. Bugs are a serious problem that pretty much every software developer faces – from those building small sites in PHP to engineers writing software to guide spacecraft. Bugs are embarrassing when they are your fault as the developer. They are costly for project teams to fix. They cause your users frustration, inconvenience them, cause them to lose money, and may even kill them.

So, if it is impossible (or at least highly impractical) to create software that is guaranteed to have 0 bugs, how can we at least minimize the number of bugs (especially common and/or serious ones)?

“We cannot solve our problems with the same thinking we used when we created them.” - Albert Einstein

Whenever talking about software quality (or lack thereof), the knee-jerk response is always “write more tests!” It always strikes me as a little amusing to think that developers who are capable of writing a piece of software with a bug in it are also capable of writing 100% comprehensive tests that are completely free of bugs themselves.

Tests are, of course, incredibly valuable and I don’t want to demean their value – indeed, they are useful beyond just testing for bugs, and I’ve found them helpful as supplemental documentation and as ways to improve my APIs. I do, however, think that blindly writing more tests as a solution to buggy software is misguided. To understand why that is, we must first understand how we get bugs in our software in the first place.

Where do bugs come from?

A lot of the time, we naively assume that a bug happened because a programmer did something wrong. Technically, this may be true, but it is neither useful nor interesting to think of it this simplistically. What are some more specific reasons?

  1. Programmer error. Perhaps, the programmer did indeed screw up. Maybe it was a typo, or perhaps they didn’t understand a particular API and misused it. Even within this category, there are a bunch of useful things to consider – is it because of lack of training? Not enough documentation of APIs a developer is using? Hiring inexperienced programmers?
  2. Insufficient requirements. Often, specific pieces of functionality are incompletely and ambiguously defined. The code may be completely bug free for every situation the developer was told to think of, but there are holes in the requirements that don’t manifest themselves until later.
  3. Seams in functionality. The Agile methodology has, on the whole, been hugely positive for the software development industry. One negative aspect to how some teams implement it, however, is that while each user story (piece of functionality) may be completely bug free, the seams between those pieces of functionality are often messy, poorly tested, and often buggy – partially owing to the fact that you may have had one developer and one qa resource working on one part, and a completely different pair working on the other part. In my own personal experience, a disproportionate number of bugs appear in the seams between functionality.
  4. Third party libraries. Sometimes the code we have little or no control over contains bugs.
  5. External resources. Sometimes resources outside of the control of our program contain bugs.

I could go on, but hopefully I’ve given you the idea that a lot of factors cause or contribute to bugs in software products and that it is useful to try to think about these reasons.

How do we avoid making bugs?

So, knowing how we end up with bugs, how can we minimize them?

  1. Develop better. Whether this means more training, more code reviews, or just hiring better developers.
  2. More, and better, tests.
  3. Static analysis tools.
  4. Better requirements.
  5. More manual QA.
  6. Many, many more ways….

Hopefully as you read through my causes of bugs above (and thought of your own), you realized why I think tests aren’t necessarily the best method for dealing with buggy software in all cases. If your problem is that specifications are poor and incomplete, it is impossible for more tests to help you. If your programmers are making a lot of mistakes, they don’t need to write more tests – they need to write fewer bugs. If most of your bugs manifest themselves in the seams of functionality, one functional test is probably going to be worth more than a thousand unit tests.

You can’t improve what you can’t measure

Back around 2000, I started a company to track the results of online advertising. Companies were spending ghastly amounts of advertising with little idea of which advertisements were making them a lot of sales and which ones were complete wastes. That company didn’t do incredibly well because I was 18, kind of an idiot, and had absolutely no money. I still, however, believe in the power of hard data and analytics to help us make better decisions and gauge our success. I also believe that these concepts can be applied to making better software. However, nobody – including myself – is bothering to track how bugs actually happen and how we can most effectively prevent them.

Indeed, I’ve used a number of high-quality open-source and commercial issue tracking tools. They are all really good at recording bugs, prioritizing them, tracking them as they get resolved, and helping us remember when we fixed them. But I’ve yet to see an issue tracking tool that actually tries to help you make better decisions about where to concentrate your future software quality efforts. This is unfortunate, because I think issue tracking tools are ideally situated to be the place where causes and possible solutions are recorded and measured.

Half the money I spend on advertising is wasted; the trouble is I don’t know which half.” – John Wanamaker

Software projects spend a lot of money in an effort to ensure a quality product. Much like the problem John Wanamaker had with his advertising money, I think projects waste a lot of money trying to improve quality.

Consider a product that currently has 50% test code coverage and too many bugs. A manager may say “We need to raise our test coverage to 75%!”, and the developers dutifully go off and spend a few weeks writing tests to get test coverage up to an acceptable level. At the end of the exercise, they have 75% test coverage – a 50% increase!

Has the quality of their product gone up by 50%? My experience tells me “No”. The reason is usually  a combination of a few factors – lack of testing probably wasn’t the whole problem, and test coverage is raised by testing the easiest (and probably least-buggy) stuff first. All of that time spent increasing code coverage was probably a waste, or at least not as effective as other efforts could have been – perhaps the requirements could have been documented more precisely, or rather than trying to hit a test coverage target, problematic portions of the application could have been identified and tested more thoroughly.

Conclusion

I have no product to sell, nor do I have a comprehensive solution to software quality to propose. I do hope that this post maybe sparks some thought and discussion on how we can better analyze our past coding mistakes and figure out better, more efficient ways of preventing them in the future.

Visual Studio 2010 – A bunch of code completion/navigation features I miss from Eclipse

For those of you who don’t know me, I’m a long-time Java developer (although I’ve spent plenty of time playing around with other languages and platforms). I’m also a big believer in developer tools – I create developer tools as part of my day job, and take advantage of tools as part of all the programming I do. Finally, I’m incredibly lazy and I’d spend all day playing video games if I could, so saving time is important to me  – every millisecond I save typing code is a millisecond more I can play World of Warcraft or Gran Turismo 5.

This weekend, I spent a bunch of time playing around with C#, ASP.NET MVC 3, and Visual Studio 2010. There are a lot of compelling things for me about this platform – C# is incredibly close to Java (even looking at IL code, I was surprised at how similar it was to Java bytecode) but has some features we Java developers can only dream about, there is a lot of progress within the whole .NET ecosystem, and the deployment mode of ASP.NET websites has potential to solve some problems I have deploying Java applications.

Overall, my experience making a few small ASP.NET websites this weekend was pretty good. For the purpose of this post, however, I’m going to concentrate on my experiences with Visual Studio’s code editor – specifically, some things I really miss from the JDT in Eclipse. It is important to keep in mind that this isn’t meant to be a comparison of the two tools and a declaration of  which one is better – the scope of my wishes are pretty narrow, and frankly, I don’t care which one is better since they rarely directly compete. It also isn’t meant to be a rant on how much Visual Studio sucks, as I think it is generally a good tool that, like any other tool, has  some things that I wish worked a little differently.

Also, I freely admit that some of my complaints/wishes may be invalid – maybe there is a way to do what I want, maybe what I want to do is stupid, or maybe there is a good reason I can’t do what I want. Feel free to correct me in the comments :)

With that rambling preamble out of the way, here are the things that I miss from Eclipse:

1. Open Type

It is really handy to be able to open a class by its name. Open Type.. is a considerably more efficient way to navigate to an arbitrary class than finding it in the project tree and it has saved me countless hours. Here is what it looks like in Eclipse:

A couple of things to note – 1) It can open any type in your workspace, whether it is a class you’ve written, one from a referenced library, or from the JDK. 2) It has pretty smart matching (ie, you can use * liberally or do something like NPE to have it find NullPointerException. 3) It is really fast.

I can’t tell you how much I use this every day. Visual Studio has similar capabilities, so let’s see what it looks like:

Not bad, but there are a few differences:

  • It doesn’t include referenced libraries. Actually, the scope of the navigation was a little tricky to figure out. It would occasionally return an external class, but only if I already had it open in another tab.
  • It also includes member variables and other items. I can see how this would sometimes be useful, but it can get noisy – it would be nice if there were an easy way to filter results.

2. Class Hierarchy

Another handy thing is to be able to quickly view the class hierarchy for any given type. It can be really useful to instantly know stuff like “What are this class’ super types?” or “What known implementations of this interface are there?”. Eclipse has a Type Hierarchy view that you can access by selecting any class name (it doesn’t have to be at the class declaration) and either right-clicking and selecting ‘Open Type Hierarchy’ or simply hitting F4. This opens a nice view of the type hierarchy that looks like this:

Visual Studio does have a type hierarchy viewer, but I couldn’t figure out how to get it to show any arbitrary type – you have to use the tree view to find the particular type you are looking for.

3. Code Complete

Code completion is a lazy developer’s best friend, and I’d heard great things about IntelliSense. My experience was that while it worked in more contexts than in Eclipse, the code complete in C# code felt really sub-optimal. First, let’s compare aut0-completing a method on the string class in both languages. Note: I’ve butchered these a little bit to get them to fit correctly in my blog. In both cases, they look a little better in the actual IDE.

Eclipse:

A couple of things that I want to point out about what you see in the code-complete popup:

  • It shows the full method signature, as well as other info. You see the return type (immediately after the method signature) and which class that method actually comes from (to the right of the return type). If there are overloaded methods, you see all of them.
  • If there are any JavaDocs associated with that method, you’ll see them on the right.
  • The symbol on the far left tells you what kind of method this is – public, protected, or private.

Now, let’s take a look at what a similar situation looks like in Visual Studio:

We get a roughly similar view, but the information is harder to process. Parameter information and the return type is instead located on the right-popout that you only see if you hover your mouse over the auto-complete candidate for a second. Worse, it is poorly formatted and the information doesn’t jump out at you like it does in the Eclipse scenario. Finally, if there are overloaded methods, you can’t see their signatures in this view.

4. Code Complete, Part 2

So, what happens once we hit ‘enter’ on one of those items?

Eclipse:

You’ll note that it not only inserts the rest of the method name, but stubs out the entire call for me with placeholders. As I type in real values for each of the placeholders, I can hit ‘tab’ to go to the next one. It also helpfully shows the parameter types in the tooltip above it. Really useful.

Visual Studio:

This is what it looks like in Visual Studio immediately after hitting ‘enter’. This is not very helpful. Hitting control + space at this point does nothing.

I have to think there is a way to get VS to do something more useful here (let me know if there is!), but really, it shouldn’t take another key combination or other action to get it to do something close to what Eclipse does.

5. Code Complete, Part 3

The scope of IntelliSense seemed really limited, too. For example, I was trying to use the DbContext class from System.Data.Entity. So, I typed DbC and then hit ctl+space, and wasn’t presented with anything useful. In Eclipse, if you were to do something similar, you’d see something like this:

Selecting one of those would cause Eclipse to automatically import the proper package.

Visual Studio clearly knows enough to do this – when I right-click on DbContext, it offers me the option of importing the correct namespace. Why couldn’t it do this as part of the IntelliSense functionality?

6. Sundry

There a ton of little things that annoyed me or felt sub-optimal, but didn’t warrant screenshots or their own section:

  • Eclipse has a feature where you can tell it to automatically put semi-colons and braces at the end of the line when you type them. This saves you having to cursor over or hit ‘end’ to do that (I told you I was lazy!).
  • Code Snippets are pretty cool – Eclipse has a similar feature. I have 2 problems with Visual Studio’s implementation: 1) They aren’t editable or createable through a simple interface in the IDE (that I can tell). 2) They show in the same IntelliSense popup as everything else, yet you hit ‘tab’ instead of ‘enter’ to activate them. Why?
  • It is kind of annoying to have to manually build the project to get certain things to happen. Why can’t it be continuously building like Eclipse Java projects? I realize this occasionally causes problems in Eclipse, but you can turn it off if you want.

Conclusion

Admittedly, when you use a tool for 8 or so hours per day for a number of years, you develop a ton of muscle memory, and using anything else feels really weird. I think, though, that some of the things I’ve pointed out above aren’t just ‘different’ in Eclipse, but are actually better, and it would be nice to see them in Visual Studio. I’m pretty open-minded, though, so if I’m somehow wrong, feel free to let me know via the comments!