This past month, over the course of a few weeks, I got several tickets stating:

DELETEs are broken – they show as a POST on my server.

I was a little concerned when the first one came in – something as simple as sending the correct verb shouldn’t be broken, but if it were, that was pretty bad. A quick test showed that DELETEs were, in fact, working. After requesting a saved .httpreq file, I quickly pinpointed the problem – DELETEs normally worked, but when a request body was present, they were magically turned into POSTs.

Further investigation revealed the problem to be in the library I use to actually assemble and send the HTTP requests (this behavior is actually present in a number of HTTP libraries, it turns out).W hile fixing the problem (version 1.0.6 contains the fix and should be available on the app store soon), I wondered – Is it valid to include a body with a DELETE request?

I spent a bunch of time reading through the HTTP specification, and saw nothing that would indicate this situation was explicitly forbidden. In practice, however, a number of HTTP clients and servers seem to not approve – as I mentioned, a number of clients silently turn them into POSTs, and per this Stack Overflow thread, a number of servers silently discard the body of a DELETE request.

Regardless of whether it is allowed, or how it is implemented by popular clients and serveres, I think it is a bad idea in general. If you are consuming a service, you don’t really have a lot of choice in the matter, but if you are creating one, I think you shouldn’t create DELETE services that require bodies. This is due to how inconsistently it is implemented, and the fact that it doesn’t make a lot of sense to include a body with a delete in the first place – a body is supposed to represent the entity, but you are deleting it, so why do you need to send it?

There are a few reasons I think folks may try to develop services that rely on a body being sent with the DELETE request, and I think there are better ways to accomplish them:

1. To identify WHAT is to be deleted.

This should be specified by the URI itself.

2. To specify metadata about the delete request itself – for example, who deleted it or a comment related to the action.

In most cases, a header is much more appropriate for these metadata fields.

This isn’t my day job, and I know I’ll never get rich doing it. My main goals are to get some more experience developing for iOS, earn back my expenses, and help pay for NFL Sunday Ticket.

Last year, my app was sold for $.99, and I made a small, but reasonable amount of money (something between $400-$500). This year, I decided to see what the results would be like if I offered it for free, but monetized it using advertising. At the last minute, I decided to make another version that cost $.99, but was identical except for the fact that it lacked ads – ads are sometimes kind of annoying, and as a user I’ll often opt to pay $.99 rather than deal with them.

Now that the fantasy football draft season is over, I thought it would be kind of interesting to take a look at the results. The limited shelf-life of this app (drafting season is at most 2 months long, and the average user won’t use it more than a few hours per year) makes a less than ideal case study for ad-supported vs paid, but I think there is some interesting data anyway. So, here is some stuff I learned from this:

1) Free apps get downloaded way, way more than even $.99 apps

This is obvious, of course, but the magnitude of the difference is worth pointing out. The free version of my app was downloaded 11,450 times, whereas the $.99 version was downloaded only 239 times.

My takeaway from this is, if you’re going to try to sell an app that is going to have free competitors, you’re really going to have to work hard to differentiate yourself from the free versions. Free apps are simply going to eat up the lion’s share of the market.

2) But they don’t make nearly as much, per-user

Those 11,450 downloads translated into several hundred thousand ad views, resulting in total revenue of $111.25 for me. This is very slightly less than $.01 per download. The paid version brought in $167.34 from its 239 downloads – despite having roughly 1/50th of the downloads, it actually out-earned the ad-supported version by ~$55.

I almost certainly made a lot less money this year by offering a free version supported by ads – if even 2% of the users who downloaded my free version would have shelled out $.99 for the paid version had it been the only option, I would have come out ahead.

Had this app been something people would use year-round, it might have worked out a little better.

Lessons for next time

Despite this not being the ideal outcome, I learned a few lessons from this. Your app has to be immensely popular to make any sort of worthwhile revenue from advertising. I don’t think niche apps or apps with limited lifespans are good candidates for ad-supported versions. Also, I think there are probably better ways to utilize a free app than just throwing ads on it.

Probably the best use of an ad-supported app is to use it to promote additional, paid functionality. You could do this through In-App Purchase or by selling a different version of the app altogehter. The important thing is to differentiate the two beyond just advertising – the paid version should also offer more features. I would have done something like this, but I given the short life-span of these apps and my late decision to offer both versions, I didn’t have time.

Going this route gives you a chance to use your likely more popular free download as an opportunity to promote the paid functionality. Just be careful to offer enough functionality in the free version to make it worth downloading – free apps that are utterly useless platforms to promote paid functionality get (rightly) harsh reviews in the app store.

After a few support requests and 1 one-star review related to this issue, I figured it was at the very least worth a deeper explanation. If you’ve googled this and are just looking for the answer here it is:

You need to set the Content-Type header to ‘application/x-www-form-urlencoded’.

That said, I think it is worth spending a few minutes to learn about HTTP messages and why this is required. Removing levels of abstraction and learning more about what is going on underneath makes you a better developer and is usually a lot of fun, too. This is a big part of the reason I’m giving up my evenings and weekends to go back to school for a CS degree.

Anyway, here is a quick primer on HTTP messages and why you have to add the header above…

The HTTP message

When you make an HTTP request, you’re connecting to an HTTP server at a given IP on a certain port and sending it a message. It then processes that message and gives you some sort of response.

What does that message look like? It consists of 3 parts:

Request Line. This tells the server what resource you’re looking for and what HTTP method (ie, one of GET, POST, PUT, DELETE, etc..) you are using. It also specifies the version of the HTTP protocol to use.
Headers. These give metadata about the message, such as the Content-Type, the Host, what type of content you’ll accept, and a whole bunch of other things.
[CRLF]
Optional message body.

So, for example, if you came to the homepage of my blog, your browser sent a message to my server that looked something like this:

GET / HTTP/1.1
Host: www.spenceruresk.com
[CRLF]

Pretty simple, right? Let’s look at a slightly more complex example. Let’s say I was hosting a Twitter-like service that allowed you to post status updates with simple HTTP POSTs. A request you’d send might look something like this:

POST /twitterClone/updateStatus HTTP/1.1
Host: www.spenceruresk.com
Content-Type: text/plain
[CRLF]
Check out this crappy cellphone pic I took at a concert!

In this case, we’re POSTing some data and telling the server what kind of content it is – in this case, it is just plain text. Pretty simple, right?

Content-Type and message bodies

One header in particular – Content-Type – is relevant to the issue and is worth talking about. When you send a message body, it can represent any number of types of data. For example, it could be plain text, a GIF, form data, or a JSON document. In order for the server to properly decode the body, you have to tell it what it is you’re sending it. The Content-Type header is how you do this. For a list of common MIME types, this Wikipedia article is useful.

Submitting forms

So, what HTTP message is generated and sent to the server when a user submits a form in their browser? The answer is that it depends. If the form is submitted via GET request, the message would look something like this:

GET /signupform?name=Spencer&age=29 HTTP/1.1
Host: www.spenceruresk.com
[CRLF]

Your web framework will then take the key/value pairs in the query string and make them available as request parameters.

What if it is a POST?

POST /signupform HTTP/1.1
Host: www.spenceruresk.com
Content-Type: application/x-www-form-urlencoded
[CRLF]
name=Spencer&age=29

The Content-Type line is critical – it tells the server “I’m sending you URL-encoded form data in the message body.” Most/All web frameworks see that, then decode the body and make each key/value pair available as request parameters.

If you don’t tell it that you are sending URL-encoded form data, it doesn’t decode the body and make the parameters available.

Conclusion

Hope that helps make it at least a little bit clearer. In the next version of the app, I’m thinking about adding some sort of Content-Type autodetection (that you can turn off) to help avoid people getting confused and frustrated in situations like this.

On a side note – I spent some time playing around with NoSQL technologies (Riak and MongoDB, specifically) over the weekend. One interesting thing about the current crop of NoSQL solutions is that many of them offer an HTTP interface. I found this tool to be quite handy when interacting with Riak and Mongo, and I’m now brainstorming ideas for making it even more useful for working with NoSQL solutions – I am kind of excited by some of the possibilities! If this interests you at all, please let me know what ideas you have.

New Features in 1.0.3

1. Improved header entry

Probably the worst thing about Graphical HTTP Client in its current state is the way you enter and update request headers. It works, but it is kind of clunky and some unexpected things can happen if you aren’t careful. It turns out, a table view isn’t the easiest thing to work with for this kind of data. In 1.0.3, I’ve modified it so that there is a new sheet for adding and updating headers. Clicking on the + button will bring up this sheet, which lets you pick from a list of header names. You can also enter your own if it is a non-standard header. To edit a header, just double-click on its entry in the table. It won’t let you save over another header, which was a problem with the old table (actually, it would sometimes lock up if you even tried).

I think this is a much better interaction, and in future releases I’m going to make it even better. Here are some plans for this functionality for the next release:

- Documentation for standard headers, including example values.
- Some form of auto-complete for values (for example, if you select ‘Accept’ or ‘Content-Type’, we can give you a list of standard content types to choose from).

Also, double-clicking on a response header row will pop up a read-only sheet that shows the full header name and header value – this can be helpful if you are dealing with longer header values.

2. New Options

Above the ‘Request Headers’ box, there is a new ‘Options’ button. This will grow over time, but for now it has 4 options:

- Validate Certificates. True by default. This allows you to tell it to NOT validate SSL certificates, which can be handy if you are using self-signed or internal certificates for your sites/services.
- Follow Redirects. True by default. Unchecking this will cause the tool to stop when it encounters a redirect.
- Compress Request Body. False by default. If checked, this will compress the request body.
- Allow Compressed Responses. False by default. If checked, this will allow the server to send GZIPped responses.

All of these options are scoped to the request and will be saved along with the request, so if you load it back up later, it will remember your options.

3. Redirect Count

This is a fairly trivial feature that shows you the number of times your request was redirected. In the future, I plan to record information (url and headers) about each redirect and provide a way for you to view that data, which could be helpful for diagnosing redirect issues.

4. If the response is an image, it will show up instead of the text box

If we can see that the response is an image (by looking at the content-type header), we’ll try to show it instead of the blank text view. Not a huge feature, but if you are ever loading images via the tool, it will save you from having to save the image and go to another program to view it.

Bug Fixes

1. Fixed an issue where the ‘Copy to Clipboard’ functionality wouldn’t work properly in certain cases.
2. Fixed an issue where sometimes old data hung around when you loaded a request from a file.

What’s coming in 1.0.4

1. The ability to upload binary data. This is something I badly wanted to get into this release, but it will for sure make it into the next one.
2. More improvements to the header functionality, as mentioned above.
3. Also as mentioned above, more information about redirects.
4. The ability to set preferences for certain things.
5. It will save your last X number of URLs, so you can re-use them easily.

1.1 and beyond

1. The #1 feature folks have requested is the ability to use OAuth integration. This is useful for working with services like Twitter.
2. Better formatting for non-JSON responses.
3. Color-coding for response bodies (ie, JSON and XML).
4. The ability to paste in a CURL command and have it be parsed into the tool. This was suggested by someone on the suggestion forum, and once I went through the Riak tutorial, I realized how handy it could be.

Thanks to everyone who uses this tool, has provided feedback, and left ratings/reviews on the app store. If you have feedback on how this tool could improve, don’t hesitate to let me know.

C# feels like a slightly-improved version of Java, the tooling is pretty good (even though I’m still not thrilled with the code editor in VS2010), the Razor syntax used in views is quite clean, and the whole thing is pretty easy to work with. It isn’t perfect, and I haven’t tried to go to production with my website yet, but all-in-all, it is a pretty good developer experience.

Working on my website this weekend, I ran into a problem that I – like probably every other web developer – have run into many times before: I needed to take some data that had line breaks in it and display it on a page, and somehow figure out how to convert those line endings into <br> tags. A lot of languages and frameworks have something built-in to handle this – PHP’s nl2br(), for example – but I couldn’t find anything in the documentation far ASP.NET MVC. This is a fairly trivial problem, but it is also a pretty common one and doing it wrong can lead to security problems, so I figured it was worth spending a few minutes figuring out the best way to do it and documenting it.

Doing some research, it seems like folks were recommending 2 main approaches to solving this:

1. Convert the line-endings to <br>’s before saving the content to the database. I didn’t care for this because saving HTML into the database complicates Cross-Site Scripting (XSS) defense. It also makes it problematic if you later want to display that same content outside of HTML (for example, in a mobile phone app).

2. Call Replace(“\r\n”, “<br>”) on the string when displaying it. I don’t care for this either – it seems repetitive (I’m lazy – I hate having to do something once, much less 500 times), and even worse, since you have to use @Html.Raw() to display it, it can open you up to XSS attacks.

Thankfully, MVC’s Html Helpers and C#’s extension methods provide a way to come up with a relatively simple and robust solution. My requirements were:

1. It had to be easy to use.
2. It had to protect against XSS attacks.
3. It should probably try to take into account the fact that there are (naturally) a number of different ways to end a line: \r\n on Windows, \r on really old Macs, and \n just about everywhere else.

This is what it ended up looking like:

static Regex LineEnding = new Regex(@"(\r\n|\r|\n)+");
 
public static MvcHtmlString Nl2br(this HtmlHelper html, string text, bool isXhtml = true)
{
  var encodedText = HttpUtility.HtmlEncode(text);
  var replacement = isXhtml ? "<br />" : "<br>";
  return MvcHtmlString.Create(LineEnding.Replace(encodedText, replacement));
}

And here is how you’d use it:

@Html.Nl2br(@Model.Description)

And here are a few tests for good measure:

[TestMethod]
public void TestCRLF()
{
    var input = "Hello\r\nWorld!";
    string result = MyHelpers.Nl2br(null, input).ToHtmlString();
    var expected = "Hello<br />World!";
    Assert.AreEqual(expected, result);
}
 
[TestMethod]
public void TestLF()
{
    var input = "Hello\nWorld!";
    string result = MyHelpers.Nl2br(null, input).ToHtmlString();
    var expected = "Hello<br />World!";
    Assert.AreEqual(expected, result);
}
 
[TestMethod]
public void TestNonXhtmlOnOldMac()
{
    var input = "Hello\rWorld!";
    string result = MyHelpers.Nl2br(null, input, false).ToHtmlString();
    var expected = "Hello<br>World!";
    Assert.AreEqual(expected, result);
}
 
[TestMethod]
public void TestXSS()
{
    var input = "Hello\nWorld!\r\nSpecial Message: <script>alert('pwned!');</script>";
    string result = MyHelpers.Nl2br(null, input).ToHtmlString();
    var expected = "Hello<br />World!<br />Special Message: &lt;script&gt;alert(&#39;pwned!&#39;);&lt;/script&gt;";
    Assert.AreEqual(expected, result);
}

A few notes:

- You’ll note that we HtmlEncode the raw text first, then replace the line breaks with <br> tags. This is so that any potentially malicious user-supplied content gets escaped, but our HTML doesn’t.
- If the page you are working with is XHTML, the <br> tags should be self-closing. I’ve made this the default in this case, but you could pass in false to denote that they shouldn’t be self-closing.
- It is worth noting that you can’t pass dynamic parameters (such as ViewBag.Message) into an extension method, so you’ll have to explicitly cast them to a string.
- I’m quite new to C# and the .NET platform, so any comments or criticisms about this approach are certainly welcome.

Anyway, extension methods make it pretty easy to create your own HTML helpers – I’ve already ended up making handful of them to save time.

Actually, though, I did read Fred Wilson’s blog post on how the current patent system, abused more and more by lowly patent trolls, is a threat to innovation.

This isn’t a new problem – I’ve had discussions about the problems with software (and business process) patents over a decade ago, and even then it probably wasn’t new. What is changing, however, is the willingness of patent trolls to go after smaller and smaller businesses, and the fact that technology is so much more a part of every day life for everyone today.

Before, patent suits were largely behind-the-scenes affairs between big companies that resulted in boring settlements or cross-licensing agreements. Or, as I’ve seen firsthand, the threat of patent lawsuits prevented small startups from receiving financing. Unfair and a huge impediment to innovation? Yes. Exciting and interesting to the average American citizen? Not really.

But as actions like the Lodsys suit against mobile apps that utilize in app purchasing become more common, the effects of a horribly-implemented patent system, abused by unscrupulous patent trolls like Lodsys, has the potential to affect the every-day lives of your average person – and perhaps shed some more light on the problem.

The scope of the Problem

The scope of the problem with software patents isn’t obvious at first, but it is an important part of realizing why they are such a threat to innovation and economic growth. Let’s go through a short little exercise -

Grab your smartphone or tablet if you have one. Scroll through the pages and pages of apps you’ve downloaded. Pop open your browser and look through the bookmarks and history items that contain all of your favorite websites. Look through the list of applications listed on your computer.

What do all of these have in common? The vast majority (if not all) of them violate one or more patents. Indeed, any marginally non-trivial app, service, or website probably violates dozens of patents.

Who are these people who have been providing you with applications or services chock-full of patent violations? Are they hardened criminals, numb to the harm they cause others? Or, are they shameless copycats who copy whatever they can find? Or, perhaps, they are hackers who’ve used their computer skills to steal secret designs from competitors?

None of the above. Most patents (especially the ones used by patent trolls) don’t exist as an implemented product, a documented library one can buy, or anything of that nature. They exist solely as one of millions of software patents filed with the PTO, written in legalese so heavy you or I would have a hard time knowing what they are describing.

Figuring out which patents your technology violates is so expensive and time consuming that it is effectively impossible for any small or medium business.

Think about that for a minute. Every small business that creates a technology product or service is sitting on dozens of bombs, any one of which can kill it. How is that fair? How is that good policy for promoting innovation and economic growth?

The worst kind of tax

When it comes to talking about economic growth (or lack thereof), it is hard to get very far without talking about the economic impact of taxes. Patents are a form of a tax, but they are the very worst kind of tax – a tax that is unpublished, seemingly random, and can wipe out your business.

The worst part is that for many of the small victims of these patent trolls, there is very little that can be done to fight this government-enforced monopoly – the very act of defending themselves will at worst case leave them bankrupt, and at best case cost countless hours and tens of thousands (at the very least) of dollars merely to continue to be able to exist.

It isn’t a bad business model – document some basic and broad idea, wait for others to come up with and implement the same idea, then sue them for a percentage of the profits.

Arguments for the patent system

There are lots of arguments for some form of a patent system. The main one is that if I think someone is going to steal my idea (which presumably took a lot of work to come up with), then I won’t bother coming up with new stuff because it will be hard for me to make money off of my work.

As a society that very highly values competition, we are willing to effectively kill competition by granting a monopoly on an idea in exchange for the value of someone coming up with it. This requires 2 things to be true – 1) Your ‘idea’ must be genuinely novel and take a lot of investment to come up with, and 2) The only way you were willing to invest in coming up with the idea was if you were granted a temporary monopoly on it.

A government-enforced monopoly is a pretty drastic – and potentially harmful – thing, so if we are going to buy into that concept we’d better be getting a lot out of it. As I’ll try to explain in the next few sections, I don’t think we are getting even close to enough out of the deal (at least with regards to software patents) to make it worth the cost.

Necessity, the mother of invention

Software development, as a profession, is largely about coming up with solutions to problems. For a given problem, there are generally only a handful of reasonable ways to solve it, and developers independently working on a similar problem will likely come up with generally similar solutions.

If you were to get 100 developers who were ignorant of the Lodsys patents (which, until Lodsys began firing off lawsuits, could have been pretty much any developer) and tell them you need to be able to sell expansions of your game to your users, and give them each an afternoon to come up with a rough solution, you’d likely get back ideas that could be grouped into half a dozen or so similar distinct ideas. Of those half-dozen distinct approaches, at least a few of them would probably violate the Lodsys patents.

The fact is that most of us solve similarly-difficult problems on a consistent basis – it is part of our job, and it is part of the reasons software developers are compensated somewhat more than average. The business side of the organization comes to us and says “We need to be able to do X.” So, we sit in front of a whiteboard for an hour or whatever, and then start implementing it.

Most of us don’t think “Hey, that’s a good idea! I should write my name on it and send it to the PTO so nobody else can do that without my blessing.” If we’re nice enough (and/or our egos are big enough), we’ll even write about it so others can use our idea.

Some people, however, feel like every idea they come up with is special and that nobody else should be able to do something like it unless they first pay for a license. Those people are assholes.

The cost of an idea

When I look at patents involved in some of these lawsuits, one thing almost always jumps out at me – once you get past all the lawyer-speak and understand what the patent is describing, it becomes clear that there wasn’t a lot of work involved in coming up with the idea. In fact, I wouldn’t be shocked to find that the effort to file the patent significantly outweighed the effort in coming up with the original idea.

If your patent is for a medication that your company spent years of effort and billions of dollars to research and produce, I can get behind patenting it. If your software patent describes something I wrote in High School using Perl, I’m going to have a hard time accepting your idea as a patentable one.

Counterpoint: Open Source Software

Open sourcing your software is pretty much the opposite of patenting it – rather than claim a monopoly on my idea, I’m going to go to great lengths to give it away to others so they can use it. It is a powerful counterpoint to the argument that patents foster innovation.

Go back to the original exercise I had you do earlier. Now consider this – all of those apps, services, and websites most likely rely heavily on open-source software (yes, even iOS uses plenty of open-source components). This simple blog relies heavily on open-source: The software used is WordPress – an open-source blogging platform, which is written in PHP – an open-source programming language. The content is stored in mySQL – an open-source database. All of it is running on Linux – an open-source operating system. To write this post, I’m using Chrome – an open-source web browser, which is running on Mac OS X – an operating system that relies heavily on open-source components.

Even you – one of my 3 or 4 readers – are almost certainly utilizing a wide array of open-source software to view this post.

The fact that open-source software plays such a pivotal role in everything we do is clear, convincing evidence to me that we can have invention and innovation on a breathtakingly large scale without the need to patent every idea we come up with.

The ‘small inventor’ myth

The final defense patent supporters like to throw out is a hypothetical one – patents are needed to help the small inventor who has his ideas ripped off by a big, evil corporation. I have to ask – how many times has a small inventor ‘invented’ some software and successfully used a patent to keep a big corporation from stealing it? We have a long history of patent trolls abusing software patents, but have we ever seen someone legitimately use one in this way? I haven’t.

Conclusion

So, to summarize – most software patents encapsulate something that an average developer could reasonably come up with on their own, are overly broad, and don’t typically represent a large amount of investment. The people at the PTO should hate themselves for approving them. Even worse – patents don’t even seem to be required to get people to innovate and invent new software. So why are we continuing to allow them?

A lot of people talk about half-measures that could be used to improve the patent system, while still allowing software patents – and a lot of the ideas are decent, but come with their own set of drawbacks.

The real thing that scares me, though, is the fact that much of the damage has already been done. Millions of incredibly simple, overly broad software patents already litter the landscape – much like how cluster bombs and land mines make an area unsafe even long after the conflict has been resolved. Even if we were to stop issuing software patents immediately, the ones already issued are broad enough to hinder innovation for the next decade.

How will we get out of this mess we’ve allowed to happen? I don’t know. But like Fred Wilson said, enough is enough. At some point, this madness has to end.

Nostalgia aside – I’ve released a new Mac (desktop) app to the Mac App Store called ‘Graphical Http Client’. This is a developer tool aimed at helping developers test and interact with REST-based services. Typically, when experimenting with or testing these services, you’ll use a command-line tool (like curl), access it with your browser (if it is a really simple GET request), or write tests to work with these services. My software aims to make this interaction somewhat easier. Some features include:

- Perform any HTTP method (GET, POST, PUT, DELETE, etc..)
- Ability to set request headers
- Ability to set authentication (Basic or Digest)
- When you perform the request, you’ll see the HTTP status code, how long the request took, the response headers, and the response body
- Nice formatting for requests that return JSON
- Ability to view HTML responses in a web view
- You can save the response body as a file (useful for when it returns binary data like an image)
- You can also save your requests and open them up later to save time
- Plus a bunch of other stuff

You can view it on my website here, or directly on the Mac App Store.

If you do download it, please leave feedback in the App Store and/or at my User Voice site.

1. A Short History of Nearly Everything by Bill Bryson

A Short History of Nearly Everything is a fun examination physics, chemistry, biology, and general science. The book is quite enjoyable to read and Bill Bryson is quickly becoming one of my favorite authors (Life at Home: A Short History of Private Life is another book of his that I recently read and really enjoyed).

One major theme that stands out is just how tenuous our existence on Earth is. Consider this quote:

“To appreciate just how narrow, you have only to look at Venus. Venus is only twenty-five million miles closer to the Sun than we are. The Sun’s warmth reaches it just two minutes before it touches us. In size and composition, Venus is very like Earth, but the small difference in orbital distance made all the difference to how it turned out. It appears that during the early years of the solar system Venus was only slightly warmer than Earth and probably had oceans. But those few degrees of extra warmth meant that Venus could not hold on to its surface water, with disastrous consequences for its climate.”

The book is full of examples of how tiny changes to one of a number of variables result in us not being here – a truly humbling notion. If you are looking for a fun, easy to read introduction to a great deal of interesting science, this book is worth picking up – I got the illustrated version for my Dad for his birthday, and he’s been enjoying it as much as I did.

2. 1776 (Illustrated Edition) by David McCullough

David McCullough has written a number of well-researched, interesting historical books. My interest in his work was kicked of by 1776, which examines the military side of the beginning of the founding of our country.

Calling this new version of 1776 “Illustrated” is selling it a bit short – there are indeed a lot of interesting illustrations, but it also includes a number of sealed pouches that contain maps, letters, and historical articles that help bring the most important year of our country to life. McCullough’s writing also has a way of bringing the main cast of characters in the revolution to life.

Much like Bryson shows us how fragile our very existence is, McCullough shows us how victory in our fight for independence was never assured and was often close to failing, only sustained by unlikely and lucky events – Washington’s crossing of the East River, aided by a fortuitous fog comes to mind.

1776 is a great read, and the documents that accompany the illustrated edition make it come alive even more.

Other books from David McCullough I’ve really enjoyed: The Great Bridge – a history of the Brooklyn Bridge and the men who built it. Truman – A look at an unlikely president who oversaw a lot of important events in American history.

3. American Prometheus: The Triumph and Tragedy of J. Robert Oppenheimer by Kai Bird and Martin J. Sherwin

I have a huge interest in World War II history, and I’m also interested in science and technology, so this book about Oppenheimer seemed like it would be a good read. I found the book to be an interesting look at Oppenheimer’s life, the development of the atomic bomb, and the terrible way in which he was treated after the war.

I find it somewhat disgraceful the way the western allies treated some of their scientists after the war – these were men who applied their genius to hasten the end of the war and undoubtedly save millions of lives. Oppenheimer and many of his fellow Manhattan project scientists became casualties of McCarthyism. Alan Turing, the British scientist instrumental in breaking German ciphers and an important figure in the history of computer science, was relentlessly persecuted for being a homosexual and eventually committed suicide in 1954 (Britain issued an official public apology for this in 2009).

Interestingly, Truman is portrayed somewhat less charitably in this book than in McCullough’s “Truman” – as Oppenheimer is dealing with feelings of guilt about developing the atomic bomb, Truman (who ultimately approved its use) grows impatient with him, perhaps dealing with his own feelings about it.

4. Team of Rivals by Doris Kearns Goodwin

This year marks the 150th anniversary of the start of the Civil War, and perhaps no single individual is more central to this period of history than Abraham Lincoln. Team of Rivals concentrates on Lincoln’s relationships with important members of his cabinet – Seward, Bates, Chase, and Stanton – who were once his bitter political enemies. Unlike many political and business leaders who surround themselves with yes-men, Lincoln recognized the intelligence and usefulness of his political rivals (who, for the most part, had little respect for him at the time) and included them in his cabinet, knowing full-well they would challenge him.

Team of Rivals shows the intelligence and shrewdness of Lincoln, and how he eventually earned respect and friendship from his former enemies.

5. Conspiracy of Fools by Kurt Eichenwald

Greed, corruption, and incompetence have been major components of corporate america and wall street for the last few decades, and I really enjoy books that go into the details of some of our biggest scandals. Conspiracy of Fools takes a look at one of the more infamous scandals – Enron.

Eichenwald tells a fascinating story about what went on at Enron, and how the company was ultimately ruined by greedy executives who did anything they could to make more money. The large amount of research that went into Conspiracy of Fools is evident, as is some of the bias in that research – Ken Lay is portrayed as almost completely oblivious to any wrongdoing going on, Jeff Skilling’s portrayal is likewise somewhat sympathetic, and others like Andy Fastow, Dick Causey, and David Duncan are portrayed quite negatively.

Ultimately, it is a fun and interesting inside look at one of the bigger business scandals in recent history.

If you continue to throw ever-increasing amounts of time and money at a software project, you can certainly get closer and closer to feeling confident you have no problems in your code, but almost any non-trivial piece of software is likely to have some (albiet, probably rare and minor) bugs in it.

I don’t, however, think an appropriate response to this fact is to throw up our hands and say “oh well, what can you do?”. Bugs are a serious problem that pretty much every software developer faces – from those building small sites in PHP to engineers writing software to guide spacecraft. Bugs are embarrassing when they are your fault as the developer. They are costly for project teams to fix. They cause your users frustration, inconvenience them, cause them to lose money, and may even kill them.

So, if it is impossible (or at least highly impractical) to create software that is guaranteed to have 0 bugs, how can we at least minimize the number of bugs (especially common and/or serious ones)?

“We cannot solve our problems with the same thinking we used when we created them.” - Albert Einstein

Whenever talking about software quality (or lack thereof), the knee-jerk response is always “write more tests!” It always strikes me as a little amusing to think that developers who are capable of writing a piece of software with a bug in it are also capable of writing 100% comprehensive tests that are completely free of bugs themselves.

Tests are, of course, incredibly valuable and I don’t want to demean their value – indeed, they are useful beyond just testing for bugs, and I’ve found them helpful as supplemental documentation and as ways to improve my APIs. I do, however, think that blindly writing more tests as a solution to buggy software is misguided. To understand why that is, we must first understand how we get bugs in our software in the first place.

Where do bugs come from?

A lot of the time, we naively assume that a bug happened because a programmer did something wrong. Technically, this may be true, but it is neither useful nor interesting to think of it this simplistically. What are some more specific reasons?

1. Programmer error. Perhaps, the programmer did indeed screw up. Maybe it was a typo, or perhaps they didn’t understand a particular API and misused it. Even within this category, there are a bunch of useful things to consider – is it because of lack of training? Not enough documentation of APIs a developer is using? Hiring inexperienced programmers?
2. Insufficient requirements. Often, specific pieces of functionality are incompletely and ambiguously defined. The code may be completely bug free for every situation the developer was told to think of, but there are holes in the requirements that don’t manifest themselves until later.
3. Seams in functionality. The Agile methodology has, on the whole, been hugely positive for the software development industry. One negative aspect to how some teams implement it, however, is that while each user story (piece of functionality) may be completely bug free, the seams between those pieces of functionality are often messy, poorly tested, and often buggy – partially owing to the fact that you may have had one developer and one qa resource working on one part, and a completely different pair working on the other part. In my own personal experience, a disproportionate number of bugs appear in the seams between functionality.
4. Third party libraries. Sometimes the code we have little or no control over contains bugs.
5. External resources. Sometimes resources outside of the control of our program contain bugs.

I could go on, but hopefully I’ve given you the idea that a lot of factors cause or contribute to bugs in software products and that it is useful to try to think about these reasons.

How do we avoid making bugs?

So, knowing how we end up with bugs, how can we minimize them?

1. Develop better. Whether this means more training, more code reviews, or just hiring better developers.
2. More, and better, tests.
3. Static analysis tools.
4. Better requirements.
5. More manual QA.
6. Many, many more ways….

Hopefully as you read through my causes of bugs above (and thought of your own), you realized why I think tests aren’t necessarily the best method for dealing with buggy software in all cases. If your problem is that specifications are poor and incomplete, it is impossible for more tests to help you. If your programmers are making a lot of mistakes, they don’t need to write more tests – they need to write fewer bugs. If most of your bugs manifest themselves in the seams of functionality, one functional test is probably going to be worth more than a thousand unit tests.

You can’t improve what you can’t measure

Back around 2000, I started a company to track the results of online advertising. Companies were spending ghastly amounts of advertising with little idea of which advertisements were making them a lot of sales and which ones were complete wastes. That company didn’t do incredibly well because I was 18, kind of an idiot, and had absolutely no money. I still, however, believe in the power of hard data and analytics to help us make better decisions and gauge our success. I also believe that these concepts can be applied to making better software. However, nobody – including myself – is bothering to track how bugs actually happen and how we can most effectively prevent them.

Indeed, I’ve used a number of high-quality open-source and commercial issue tracking tools. They are all really good at recording bugs, prioritizing them, tracking them as they get resolved, and helping us remember when we fixed them. But I’ve yet to see an issue tracking tool that actually tries to help you make better decisions about where to concentrate your future software quality efforts. This is unfortunate, because I think issue tracking tools are ideally situated to be the place where causes and possible solutions are recorded and measured.

“Half the money I spend on advertising is wasted; the trouble is I don’t know which half.” – John Wanamaker

Software projects spend a lot of money in an effort to ensure a quality product. Much like the problem John Wanamaker had with his advertising money, I think projects waste a lot of money trying to improve quality.

Consider a product that currently has 50% test code coverage and too many bugs. A manager may say “We need to raise our test coverage to 75%!”, and the developers dutifully go off and spend a few weeks writing tests to get test coverage up to an acceptable level. At the end of the exercise, they have 75% test coverage – a 50% increase!

Has the quality of their product gone up by 50%? My experience tells me “No”. The reason is usually  a combination of a few factors – lack of testing probably wasn’t the whole problem, and test coverage is raised by testing the easiest (and probably least-buggy) stuff first. All of that time spent increasing code coverage was probably a waste, or at least not as effective as other efforts could have been – perhaps the requirements could have been documented more precisely, or rather than trying to hit a test coverage target, problematic portions of the application could have been identified and tested more thoroughly.

Conclusion

I have no product to sell, nor do I have a comprehensive solution to software quality to propose. I do hope that this post maybe sparks some thought and discussion on how we can better analyze our past coding mistakes and figure out better, more efficient ways of preventing them in the future.

This weekend, I spent a bunch of time playing around with C#, ASP.NET MVC 3, and Visual Studio 2010. There are a lot of compelling things for me about this platform – C# is incredibly close to Java (even looking at IL code, I was surprised at how similar it was to Java bytecode) but has some features we Java developers can only dream about, there is a lot of progress within the whole .NET ecosystem, and the deployment mode of ASP.NET websites has potential to solve some problems I have deploying Java applications.

Overall, my experience making a few small ASP.NET websites this weekend was pretty good. For the purpose of this post, however, I’m going to concentrate on my experiences with Visual Studio’s code editor – specifically, some things I really miss from the JDT in Eclipse. It is important to keep in mind that this isn’t meant to be a comparison of the two tools and a declaration of  which one is better – the scope of my wishes are pretty narrow, and frankly, I don’t care which one is better since they rarely directly compete. It also isn’t meant to be a rant on how much Visual Studio sucks, as I think it is generally a good tool that, like any other tool, has  some things that I wish worked a little differently.

Also, I freely admit that some of my complaints/wishes may be invalid – maybe there is a way to do what I want, maybe what I want to do is stupid, or maybe there is a good reason I can’t do what I want. Feel free to correct me in the comments

With that rambling preamble out of the way, here are the things that I miss from Eclipse:

1. Open Type

It is really handy to be able to open a class by its name. Open Type.. is a considerably more efficient way to navigate to an arbitrary class than finding it in the project tree and it has saved me countless hours. Here is what it looks like in Eclipse:

A couple of things to note – 1) It can open any type in your workspace, whether it is a class you’ve written, one from a referenced library, or from the JDK. 2) It has pretty smart matching (ie, you can use * liberally or do something like NPE to have it find NullPointerException. 3) It is really fast.

I can’t tell you how much I use this every day. Visual Studio has similar capabilities, so let’s see what it looks like:

Not bad, but there are a few differences:

• It doesn’t include referenced libraries. Actually, the scope of the navigation was a little tricky to figure out. It would occasionally return an external class, but only if I already had it open in another tab.
• It also includes member variables and other items. I can see how this would sometimes be useful, but it can get noisy – it would be nice if there were an easy way to filter results.

2. Class Hierarchy

Another handy thing is to be able to quickly view the class hierarchy for any given type. It can be really useful to instantly know stuff like “What are this class’ super types?” or “What known implementations of this interface are there?”. Eclipse has a Type Hierarchy view that you can access by selecting any class name (it doesn’t have to be at the class declaration) and either right-clicking and selecting ‘Open Type Hierarchy’ or simply hitting F4. This opens a nice view of the type hierarchy that looks like this:

Visual Studio does have a type hierarchy viewer, but I couldn’t figure out how to get it to show any arbitrary type – you have to use the tree view to find the particular type you are looking for.

3. Code Complete

Code completion is a lazy developer’s best friend, and I’d heard great things about IntelliSense. My experience was that while it worked in more contexts than in Eclipse, the code complete in C# code felt really sub-optimal. First, let’s compare aut0-completing a method on the string class in both languages. Note: I’ve butchered these a little bit to get them to fit correctly in my blog. In both cases, they look a little better in the actual IDE.

Eclipse:

A couple of things that I want to point out about what you see in the code-complete popup:

• It shows the full method signature, as well as other info. You see the return type (immediately after the method signature) and which class that method actually comes from (to the right of the return type). If there are overloaded methods, you see all of them.
• If there are any JavaDocs associated with that method, you’ll see them on the right.
• The symbol on the far left tells you what kind of method this is – public, protected, or private.

Now, let’s take a look at what a similar situation looks like in Visual Studio:

We get a roughly similar view, but the information is harder to process. Parameter information and the return type is instead located on the right-popout that you only see if you hover your mouse over the auto-complete candidate for a second. Worse, it is poorly formatted and the information doesn’t jump out at you like it does in the Eclipse scenario. Finally, if there are overloaded methods, you can’t see their signatures in this view.

4. Code Complete, Part 2

So, what happens once we hit ‘enter’ on one of those items?

Eclipse:

You’ll note that it not only inserts the rest of the method name, but stubs out the entire call for me with placeholders. As I type in real values for each of the placeholders, I can hit ‘tab’ to go to the next one. It also helpfully shows the parameter types in the tooltip above it. Really useful.

Visual Studio:

This is what it looks like in Visual Studio immediately after hitting ‘enter’. This is not very helpful. Hitting control + space at this point does nothing.

I have to think there is a way to get VS to do something more useful here (let me know if there is!), but really, it shouldn’t take another key combination or other action to get it to do something close to what Eclipse does.

5. Code Complete, Part 3

The scope of IntelliSense seemed really limited, too. For example, I was trying to use the DbContext class from System.Data.Entity. So, I typed DbC and then hit ctl+space, and wasn’t presented with anything useful. In Eclipse, if you were to do something similar, you’d see something like this:

Selecting one of those would cause Eclipse to automatically import the proper package.

Visual Studio clearly knows enough to do this – when I right-click on DbContext, it offers me the option of importing the correct namespace. Why couldn’t it do this as part of the IntelliSense functionality?

6. Sundry

There a ton of little things that annoyed me or felt sub-optimal, but didn’t warrant screenshots or their own section:

• Eclipse has a feature where you can tell it to automatically put semi-colons and braces at the end of the line when you type them. This saves you having to cursor over or hit ‘end’ to do that (I told you I was lazy!).
• Code Snippets are pretty cool – Eclipse has a similar feature. I have 2 problems with Visual Studio’s implementation: 1) They aren’t editable or createable through a simple interface in the IDE (that I can tell). 2) They show in the same IntelliSense popup as everything else, yet you hit ‘tab’ instead of ‘enter’ to activate them. Why?
• It is kind of annoying to have to manually build the project to get certain things to happen. Why can’t it be continuously building like Eclipse Java projects? I realize this occasionally causes problems in Eclipse, but you can turn it off if you want.

Conclusion

Admittedly, when you use a tool for 8 or so hours per day for a number of years, you develop a ton of muscle memory, and using anything else feels really weird. I think, though, that some of the things I’ve pointed out above aren’t just ‘different’ in Eclipse, but are actually better, and it would be nice to see them in Visual Studio. I’m pretty open-minded, though, so if I’m somehow wrong, feel free to let me know via the comments!