HTTP DELETE requests that include a body

Posted by Spencer Uresk on November 7, 2011 2 comments

I’ve had an enjoyable time getting down into a lot of gritty details about HTTP over the past year or so as I’ve worked on some webservices and a HTTP client tool for OS X. Every month, it seems like, I learn a little more about it based on the way my users use my app.

This past month, over the course of a few weeks, I got several tickets stating:

DELETEs are broken – they show as a POST on my server.

I was a little concerned when the first one came in – something as simple as sending the correct verb shouldn’t be broken, but if it were, that was pretty bad. A quick test showed that DELETEs were, in fact, working. After requesting a saved .httpreq file, I quickly pinpointed the problem – DELETEs normally worked, but when a request body was present, they were magically turned into POSTs.

Further investigation revealed the problem to be in the library I use to actually assemble and send the HTTP requests (this behavior is actually present in a number of HTTP libraries, it turns out).W hile fixing the problem (version 1.0.6 contains the fix and should be available on the app store soon), I wondered – Is it valid to include a body with a DELETE request?

I spent a bunch of time reading through the HTTP specification, and saw nothing that would indicate this situation was explicitly forbidden. In practice, however, a number of HTTP clients and servers seem to not approve – as I mentioned, a number of clients silently turn them into POSTs, and per this Stack Overflow thread, a number of servers silently discard the body of a DELETE request.

Regardless of whether it is allowed, or how it is implemented by popular clients and serveres, I think it is a bad idea in general. If you are consuming a service, you don’t really have a lot of choice in the matter, but if you are creating one, I think you shouldn’t create DELETE services that require bodies. This is due to how inconsistently it is implemented, and the fact that it doesn’t make a lot of sense to include a body with a delete in the first place – a body is supposed to represent the entity, but you are deleting it, so why do you need to send it?

There are a few reasons I think folks may try to develop services that rely on a body being sent with the DELETE request, and I think there are better ways to accomplish them:

1. To identify WHAT is to be deleted.

This should be specified by the URI itself.

2. To specify metadata about the delete request itself – for example, who deleted it or a comment related to the action.

In most cases, a header is much more appropriate for these metadata fields.

Ad supported vs $.99 – My experience

Posted by Spencer Uresk on September 11, 2011 0 comments

I love football, and I can’t tell you how excited I am that it is back on this week. I love playing fantasy football even more, and have even made an iPhone app for fantasy football drafts the past few years.

This isn’t my day job, and I know I’ll never get rich doing it. My main goals are to get some more experience developing for iOS, earn back my expenses, and help pay for NFL Sunday Ticket.

Last year, my app was sold for $.99, and I made a small, but reasonable amount of money (something between $400-$500). This year, I decided to see what the results would be like if I offered it for free, but monetized it using advertising. At the last minute, I decided to make another version that cost $.99, but was identical except for the fact that it lacked ads – ads are sometimes kind of annoying, and as a user I’ll often opt to pay $.99 rather than deal with them.

Now that the fantasy football draft season is over, I thought it would be kind of interesting to take a look at the results. The limited shelf-life of this app (drafting season is at most 2 months long, and the average user won’t use it more than a few hours per year) makes a less than ideal case study for ad-supported vs paid, but I think there is some interesting data anyway. So, here is some stuff I learned from this:

1) Free apps get downloaded way, way more than even $.99 apps

This is obvious, of course, but the magnitude of the difference is worth pointing out. The free version of my app was downloaded 11,450 times, whereas the $.99 version was downloaded only 239 times.

My takeaway from this is, if you’re going to try to sell an app that is going to have free competitors, you’re really going to have to work hard to differentiate yourself from the free versions. Free apps are simply going to eat up the lion’s share of the market.

2) But they don’t make nearly as much, per-user

Those 11,450 downloads translated into several hundred thousand ad views, resulting in total revenue of $111.25 for me. This is very slightly less than $.01 per download. The paid version brought in $167.34 from its 239 downloads – despite having roughly 1/50th of the downloads, it actually out-earned the ad-supported version by ~$55.

I almost certainly made a lot less money this year by offering a free version supported by ads – if even 2% of the users who downloaded my free version would have shelled out $.99 for the paid version had it been the only option, I would have come out ahead.

Had this app been something people would use year-round, it might have worked out a little better.

Lessons for next time

Despite this not being the ideal outcome, I learned a few lessons from this. Your app has to be immensely popular to make any sort of worthwhile revenue from advertising. I don’t think niche apps or apps with limited lifespans are good candidates for ad-supported versions. Also, I think there are probably better ways to utilize a free app than just throwing ads on it.

Probably the best use of an ad-supported app is to use it to promote additional, paid functionality. You could do this through In-App Purchase or by selling a different version of the app altogehter. The important thing is to differentiate the two beyond just advertising – the paid version should also offer more features. I would have done something like this, but I given the short life-span of these apps and my late decision to offer both versions, I didn’t have time.

Going this route gives you a chance to use your likely more popular free download as an opportunity to promote the paid functionality. Just be careful to offer enough functionality in the free version to make it worth downloading – free apps that are utterly useless platforms to promote paid functionality get (rightly) harsh reviews in the app store.

HTTP Message basics, and POSTing form data in Graphical HTTP Client

Posted by Spencer Uresk on September 4, 2011 0 comments

“The POST body isn’t working correctly! I add in something like name=spencer&age=29, but the server doesn’t get the parameters. Please fix your program!”

After a few support requests and 1 one-star review related to this issue, I figured it was at the very least worth a deeper explanation. If you’ve googled this and are just looking for the answer here it is:

You need to set the Content-Type header to ‘application/x-www-form-urlencoded’.

That said, I think it is worth spending a few minutes to learn about HTTP messages and why this is required. Removing levels of abstraction and learning more about what is going on underneath makes you a better developer and is usually a lot of fun, too. This is a big part of the reason I’m giving up my evenings and weekends to go back to school for a CS degree.

Anyway, here is a quick primer on HTTP messages and why you have to add the header above…

The HTTP message

When you make an HTTP request, you’re connecting to an HTTP server at a given IP on a certain port and sending it a message. It then processes that message and gives you some sort of response.

What does that message look like? It consists of 3 parts:

Request Line. This tells the server what resource you’re looking for and what HTTP method (ie, one of GET, POST, PUT, DELETE, etc..) you are using. It also specifies the version of the HTTP protocol to use.
Headers. These give metadata about the message, such as the Content-Type, the Host, what type of content you’ll accept, and a whole bunch of other things.
[CRLF]
Optional message body.

So, for example, if you came to the homepage of my blog, your browser sent a message to my server that looked something like this:

GET / HTTP/1.1
Host: www.spenceruresk.com
[CRLF]

Pretty simple, right? Let’s look at a slightly more complex example. Let’s say I was hosting a Twitter-like service that allowed you to post status updates with simple HTTP POSTs. A request you’d send might look something like this:

POST /twitterClone/updateStatus HTTP/1.1
Host: www.spenceruresk.com
Content-Type: text/plain
[CRLF]
Check out this crappy cellphone pic I took at a concert!

In this case, we’re POSTing some data and telling the server what kind of content it is – in this case, it is just plain text. Pretty simple, right?

Content-Type and message bodies

One header in particular – Content-Type – is relevant to the issue and is worth talking about. When you send a message body, it can represent any number of types of data. For example, it could be plain text, a GIF, form data, or a JSON document. In order for the server to properly decode the body, you have to tell it what it is you’re sending it. The Content-Type header is how you do this. For a list of common MIME types, this Wikipedia article is useful.

Submitting forms

So, what HTTP message is generated and sent to the server when a user submits a form in their browser? The answer is that it depends. If the form is submitted via GET request, the message would look something like this:

GET /signupform?name=Spencer&age=29 HTTP/1.1
Host: www.spenceruresk.com
[CRLF]

Your web framework will then take the key/value pairs in the query string and make them available as request parameters.

What if it is a POST?

POST /signupform HTTP/1.1
Host: www.spenceruresk.com
Content-Type: application/x-www-form-urlencoded
[CRLF]
name=Spencer&age=29

The Content-Type line is critical – it tells the server “I’m sending you URL-encoded form data in the message body.” Most/All web frameworks see that, then decode the body and make each key/value pair available as request parameters.

If you don’t tell it that you are sending URL-encoded form data, it doesn’t decode the body and make the parameters available.

Conclusion

Hope that helps make it at least a little bit clearer. In the next version of the app, I’m thinking about adding some sort of Content-Type autodetection (that you can turn off) to help avoid people getting confused and frustrated in situations like this.

Graphical HTTP Client 1.0.3 submitted to the App Store

Posted by Spencer Uresk on July 6, 2011 0 comments

Graphical HTTP Client has been in the app store for roughly 2 months now and has had a few updates, but I haven’t quite added as much as I’d original hoped to. I’ve been able to devote more time to it recently and have just submitted 1.0.3 to the App Store. If history is any indication, it should be available in roughly a week. I plan to submit 1.0.4 in the next few weeks and generally update the app more frequently over the next few months.

On a side note – I spent some time playing around with NoSQL technologies (Riak and MongoDB, specifically) over the weekend. One interesting thing about the current crop of NoSQL solutions is that many of them offer an HTTP interface. I found this tool to be quite handy when interacting with Riak and Mongo, and I’m now brainstorming ideas for making it even more useful for working with NoSQL solutions – I am kind of excited by some of the possibilities! If this interests you at all, please let me know what ideas you have.

New Features in 1.0.3

1. Improved header entry

Probably the worst thing about Graphical HTTP Client in its current state is the way you enter and update request headers. It works, but it is kind of clunky and some unexpected things can happen if you aren’t careful. It turns out, a table view isn’t the easiest thing to work with for this kind of data. In 1.0.3, I’ve modified it so that there is a new sheet for adding and updating headers. Clicking on the + button will bring up this sheet, which lets you pick from a list of header names. You can also enter your own if it is a non-standard header. To edit a header, just double-click on its entry in the table. It won’t let you save over another header, which was a problem with the old table (actually, it would sometimes lock up if you even tried).

I think this is a much better interaction, and in future releases I’m going to make it even better. Here are some plans for this functionality for the next release:

- Documentation for standard headers, including example values.
- Some form of auto-complete for values (for example, if you select ‘Accept’ or ‘Content-Type’, we can give you a list of standard content types to choose from).

Also, double-clicking on a response header row will pop up a read-only sheet that shows the full header name and header value – this can be helpful if you are dealing with longer header values.

2. New Options

Above the ‘Request Headers’ box, there is a new ‘Options’ button. This will grow over time, but for now it has 4 options:

- Validate Certificates. True by default. This allows you to tell it to NOT validate SSL certificates, which can be handy if you are using self-signed or internal certificates for your sites/services.
- Follow Redirects. True by default. Unchecking this will cause the tool to stop when it encounters a redirect.
- Compress Request Body. False by default. If checked, this will compress the request body.
- Allow Compressed Responses. False by default. If checked, this will allow the server to send GZIPped responses.

All of these options are scoped to the request and will be saved along with the request, so if you load it back up later, it will remember your options.

3. Redirect Count

This is a fairly trivial feature that shows you the number of times your request was redirected. In the future, I plan to record information (url and headers) about each redirect and provide a way for you to view that data, which could be helpful for diagnosing redirect issues.

4. If the response is an image, it will show up instead of the text box

If we can see that the response is an image (by looking at the content-type header), we’ll try to show it instead of the blank text view. Not a huge feature, but if you are ever loading images via the tool, it will save you from having to save the image and go to another program to view it.

Bug Fixes

1. Fixed an issue where the ‘Copy to Clipboard’ functionality wouldn’t work properly in certain cases.
2. Fixed an issue where sometimes old data hung around when you loaded a request from a file.

What’s coming in 1.0.4

1. The ability to upload binary data. This is something I badly wanted to get into this release, but it will for sure make it into the next one.
2. More improvements to the header functionality, as mentioned above.
3. Also as mentioned above, more information about redirects.
4. The ability to set preferences for certain things.
5. It will save your last X number of URLs, so you can re-use them easily.

1.1 and beyond

1. The #1 feature folks have requested is the ability to use OAuth integration. This is useful for working with services like Twitter.
2. Better formatting for non-JSON responses.
3. Color-coding for response bodies (ie, JSON and XML).
4. The ability to paste in a CURL command and have it be parsed into the tool. This was suggested by someone on the suggestion forum, and once I went through the Riak tutorial, I realized how handy it could be.

Thanks to everyone who uses this tool, has provided feedback, and left ratings/reviews on the app store. If you have feedback on how this tool could improve, don’t hesitate to let me know.

Nl2br example in ASP.NET MVC 3

Posted by Spencer Uresk on June 5, 2011 0 comments

I’ve been playing around with ASP.NET 3.5 this weekend, and am building my first real website with it. Much to my horror, as a long-time Java developer, I’m finding it to be incredibly delightful to work with.

C# feels like a slightly-improved version of Java, the tooling is pretty good (even though I’m still not thrilled with the code editor in VS2010), the Razor syntax used in views is quite clean, and the whole thing is pretty easy to work with. It isn’t perfect, and I haven’t tried to go to production with my website yet, but all-in-all, it is a pretty good developer experience.

Working on my website this weekend, I ran into a problem that I – like probably every other web developer – have run into many times before: I needed to take some data that had line breaks in it and display it on a page, and somehow figure out how to convert those line endings into <br> tags. A lot of languages and frameworks have something built-in to handle this – PHP’s nl2br(), for example – but I couldn’t find anything in the documentation far ASP.NET MVC. This is a fairly trivial problem, but it is also a pretty common one and doing it wrong can lead to security problems, so I figured it was worth spending a few minutes figuring out the best way to do it and documenting it.

Doing some research, it seems like folks were recommending 2 main approaches to solving this:

1. Convert the line-endings to <br>’s before saving the content to the database. I didn’t care for this because saving HTML into the database complicates Cross-Site Scripting (XSS) defense. It also makes it problematic if you later want to display that same content outside of HTML (for example, in a mobile phone app).

2. Call Replace(“\r\n”, “<br>”) on the string when displaying it. I don’t care for this either – it seems repetitive (I’m lazy – I hate having to do something once, much less 500 times), and even worse, since you have to use @Html.Raw() to display it, it can open you up to XSS attacks.

Thankfully, MVC’s Html Helpers and C#’s extension methods provide a way to come up with a relatively simple and robust solution. My requirements were:

1. It had to be easy to use.
2. It had to protect against XSS attacks.
3. It should probably try to take into account the fact that there are (naturally) a number of different ways to end a line: \r\n on Windows, \r on really old Macs, and \n just about everywhere else.

This is what it ended up looking like:

static Regex LineEnding = new Regex(@"(\r\n|\r|\n)+");
 
public static MvcHtmlString Nl2br(this HtmlHelper html, string text, bool isXhtml = true)
{
  var encodedText = HttpUtility.HtmlEncode(text);
  var replacement = isXhtml ? "<br />" : "<br>";
  return MvcHtmlString.Create(LineEnding.Replace(encodedText, replacement));
}

And here is how you’d use it:

@Html.Nl2br(@Model.Description)

And here are a few tests for good measure:

[TestMethod]
public void TestCRLF()
{
    var input = "Hello\r\nWorld!";
    string result = MyHelpers.Nl2br(null, input).ToHtmlString();
    var expected = "Hello<br />World!";
    Assert.AreEqual(expected, result);
}
 
[TestMethod]
public void TestLF()
{
    var input = "Hello\nWorld!";
    string result = MyHelpers.Nl2br(null, input).ToHtmlString();
    var expected = "Hello<br />World!";
    Assert.AreEqual(expected, result);
}
 
[TestMethod]
public void TestNonXhtmlOnOldMac()
{
    var input = "Hello\rWorld!";
    string result = MyHelpers.Nl2br(null, input, false).ToHtmlString();
    var expected = "Hello<br>World!";
    Assert.AreEqual(expected, result);
}
 
[TestMethod]
public void TestXSS()
{
    var input = "Hello\nWorld!\r\nSpecial Message: <script>alert('pwned!');</script>";
    string result = MyHelpers.Nl2br(null, input).ToHtmlString();
    var expected = "Hello<br />World!<br />Special Message: &lt;script&gt;alert(&#39;pwned!&#39;);&lt;/script&gt;";
    Assert.AreEqual(expected, result);
}

A few notes:

- You’ll note that we HtmlEncode the raw text first, then replace the line breaks with <br> tags. This is so that any potentially malicious user-supplied content gets escaped, but our HTML doesn’t.
- If the page you are working with is XHTML, the <br> tags should be self-closing. I’ve made this the default in this case, but you could pass in false to denote that they shouldn’t be self-closing.
- It is worth noting that you can’t pass dynamic parameters (such as ViewBag.Message) into an extension method, so you’ll have to explicitly cast them to a string.
- I’m quite new to C# and the .NET platform, so any comments or criticisms about this approach are certainly welcome.

Anyway, extension methods make it pretty easy to create your own HTML helpers – I’ve already ended up making handful of them to save time.