Category Archives: Software

The response time paradox

Where I work, we have a number (let’s say around 100) of backend services that are used by front-end code to render our website. These services are usually centered around one part of our businesses – we have, for example, a service for managing users and getting information about them, another one for getting order information, etc. We have a lot of monitoring on these services, mostly through NewRelic, that makes us feel like we have pretty good visibility into them most of the time.

It isn’t uncommon for one of these services to have spikes in calls that range from a few minutes to a few hours – usually because of some batch process or maybe a spike in web traffic or something. Sometimes there is an interesting phenomenon that occurs – despite a large spike in traffic, response times appear to go DOWN. What is even more perplexing, every endpoint in the service seems to be responding slower due to the load, but overall response time appears to decrease.

It’s interesting to think about why this is the case. If you get a bunch of smart developers and operations people into a room, a bunch of theories will start to come out. Maybe the extra requests are keeping a specific thread hot and so there is less context switching? Maybe something is being cached and re-used repeatedly? Maybe an increase in L2 cache hits? And so on…

Some of these may have some element of truth to them, but it’s hard to imagine any of them improving response times by any measurable amount. The answer, it turns out (at least in every case I’ve looked into so far), is a little surprising to some people – the services aren’t actually responding faster, they are responding slower, and in some cases quite a bit slower. The way we report response times misleads us and makes us think otherwise, however.

To illustrate, let’s imagine a service that manages users, and that service has two endpoints: /read for getting information about a user, and /write for creating users. Our read endpoint is really fast and called quite frequently, but our write endpoint does more work and is much slower, but is called relatively infrequently. Assume that under normal circumstances, our read endpoint is called 100 times per minute and takes 10 ms to respond, whereas our write endpoint is called 5 times per minute and takes a whopping 500 ms to respond.

What is our mean response time? Well, ((100*10) + (5*500)) / 105 = 33 ms.

Now, what happens if our read traffic goes up by 10x for a minute, while the write traffic stays the same, and due to increased load, everything is 20% slower?

((1000*12) + (5*600)) / 1005 = 15 ms.

It looks like our response time has dropped in half, even though it has gotten 20% worse in reality. This is a fairly typical result.

There are two fairly easy criticisms you can make:

1) You shouldn’t be aggregating across endpoints

This is sort of fair, but at some point, you’ll have so many endpoints that doing so at that level makes it really hard to get useful information. It is also useful to aggregate at the service or process level because that is one of the data points you are using to gauge how they are doing and know when to scale them. Finally, we can have the same situation happen on a single endpoint – what if most of our users were really fast, but a small subset were power users who had a lot more data attached to them and thus loaded a lot more slowly? This is a scenario I’ve run into before.

2) The whole problem is that you are using the mean

This is also a good point – anyone with even a fairly basic background in statistics can tell you why using the mean may not be the best way to measure aggregate response times.

Using the median might make situations like this less likely (but you can still contrive examples of them), but hides data in its own way – it is easy, for example, for it to hide the fact that you have a number of really slow endpoints if you have slightly more fast ones.

Using percentiles or a histogram of response times avoid most of these problems, but require a lot more data to be kept around versus calculating the mean on the fly (i suspect this is why NewRelic shows the mean by default and only retains limited history of other metrics). Indeed, the 99% metric is the only one that has actually shown load times increasing when we’ve had these events in the past, although some of the spikes have been so large that even the 99% isn’t sensitive enough.

tl;dr Sometimes the way metrics are reported can be really misleading in extreme cases, and if something feels counterintuitive, sometimes it really isn’t exactly as it seems. I generally prefer to look at the 95/99% graph when I’m looking at a service – it tells you if you really have stuff you need to look into and often gives you an idea of what the future looks like for your service.

CloudStudio for AWS now available

I’m excited to announce the initial release of a project I’ve been working on – CloudStudio.

Amazon’s AWS provides a ton of great services, but not all of them have easy UIs for testing and interacting with them. As a frequent user of services like Kinesis and Firehose, I’ve often wished I could have an easy way to push a message onto a stream, or take a peek at messages coming in, so I started building a suite of tools to just do that.

This initial release focuses on Kinesis, Firehose, and SQS. You can send messages to all 3, and you can poll messages from Kinesis, all with a simple and easy-to-use GUI. It is available now for Mac for $9.99, but I’m wrapping up work on Windows and Linux versions in the next week or so as well.

It may be starting out simple, but I have some really big plans to make it even more useful in the near future, by doing things like allowing you to stream to Kinesis from a file, tail an S3 file, and more. I’ve made my roadmap open on Trello, and I’ve added a forum to solicit feedback on what features you want.

I want this to be a really powerful tool for AWS users, so I hope you’ll check it out here and give me some feedback on how I can make it more useful for you.

Running a single Mocha test file in VS Code

I’ve been doing a lot of Node development lately, after doing mostly Java the past 10 years. There are lots of comparisons between the two, and I’ve come away with a better understanding of where one is better than the other and what I wish I could take from each one.

One thing I miss about doing Java development is being able to right-click in a single test, run it, and be able to easily debug it. You can get close to this in Node-land, but it is nowhere near as simple or seamless. You can, however, get kind of close with VS Code (which I’m loving more and more every day) by creating a custom launch configuration that lets you debug a single mocha file.

1. Create a launch configuration that only runs the current file

This creates a launch configuration that passes the current file to the mocha command.

2. If you have a mocha.opts, you may need to override it

A lot of projects have a mocha.opts file that has something like this ‘–recursive test/.’

Command-line args should override options in mocha.opts, but it looks like the file specification part does not get overridden. So, what I did was create a dummy mocha-debug.opts that is empty, then point to it in the config:

3. You can now run and debug a single Mocha file.

Combining data from multiple sources with Spark and Zeppelin

I’ve been doing a lot with Spark lately, and I love how easy it is to pull in data from various locations, in various formats, and have be able to query/manipulate it with a unified interface.

Last week, I had a situation at work where I needed to get a list of users (from our mySQL database), filtered with ancillary data from JSON files in S3 and a CSV file that had been sent to me and was sitting on my local machine. Using Spark and Zeppelin, I was able to do this in just a few minutes – analyzing a few GBs of data from multiple sources in multiple formats from my local machine took only a few minutes to execute, too (this approach would work with much larger data also, you just would want to run it on a cluster..).

I wanted to share a simplified, adapted version of that with others to show the power of Spark and Zeppelin. This is a really trivial example with a tiny amount of data, but hopefully it gives an idea of what is possible – and while it may seem kind of gimmicky, I’ve already used something very similar once for an actual business use case, and I can see lots of other situations where it could come in handy as well.

The Task

Marketing has sent us a list of users – some of whom they sent an email, some they didn’t. They want to know how many pages the users who got sent our email viewed versus users who did not. Unfortunately, our analytics data only has the user id, and marketing only has their email address – we’ll have to use the data in our database to bridge them.


AWS S3 (You can obviously change this to a local file, if you want)


I’ve uploaded samples for all of the assets you need, as well as the Zeppelin notebook itself:

SQL file for creating the users table and the users in it
JSON file with analytics data to be loaded from S3
Email list with all of the users that marketing emailed in CSV format
Zeppelin file that has all of the code

I’ve posted screenshots in this post, if you’d like the code, grab the Zeppelin file.

Step 1: Imports

We need to import a mySQL driver, the AWS SDK (so we can load data from S3), and the databricks library that can create RDDs from CSV files.


Step 2: Load users from mySQL

We want to create a DataFrame from our users table in our database. Note that it will parse the schema and turn it into a DataFrame with similar column names as are in the table. We now have all of our users and their ids/names/emails available to us.


Step 3: Load JSON file from S3

Spark is really awesome at loading JSON files and making them queryable. In this case, we’re doing a little extra work to load it from S3 – just give it your access key, secret key, and then point it at the right bucket and it will download it and turn it into a DataFrame based on the JSON structure.

Now we have a record of all events from all of our users.


Step 4: Load the email list CSV from our local filesystem

We have a simple CSV file in the format of <email address>,<promo email sent: true|false>. Let’s read it in now, too, so we know who got an email and who didn’t. In this case, our CSV has a header file that Spark will use to create the DataFrame columns, and it will attempt to infer the schema based on the data in the file – we can also pass in an explicit schema if we need to.


Step 5: Query

Now, let’s write a query that joins them all together and tells us who got an email and how many events they produced. As you can see, this looks like any other SQL query, only it is pulling data in from a mySQL database, a JSON file we pulled from S3, and a local CSV file.


Step 6: Graph

We can even use the %sql interpreter in Zeppelin to write a query without any Scala code and graph the results:



As you can see, the combination of Spark and Zeppelin is incredibly powerful. Data exploration and complex reporting – even with terabytes of data spread across multiple data sources – is fairly easy to do.

Aliased index rebuilds in Elasticsearch

Aliases are really useful tools in Elasticsearch. They allow you to create something that looks and acts like an index, but is really a pointer to another index (or multiple indices).

One really handy use of aliases is in doing full-index rebuilds. There are a handful of reasons you may need to do a rebuld:

  1. You only rebuild an index periodically – either on a schedule or in response to some sort of event (such as an import of some sort).
  2. You do online updates to your index, but want to periodically rebuild it to keep it in sync with the system of record in cases where you might miss new/updated data for various reasons.
  3. You do online updates to your index, but want to do a full rebuild in background in response to certain events, because handling them online would be inefficient or cause delays to real-time events.
  4. You want to update your index in a way that is backwards compatible, and do so in a way that there is no downtime.

The general idea behind an aliased index rebuild is that you never query an index directly – instead, you query an alias that points to the real index. When you need to rebuild the index, you create a new index, populate it, and then point your alias at it.

There are several advantages to doing this:

  1. Your index can be rebuilt as often as you’d like, with no downtime – even if it takes a long time to rebuild.
  2. You can keep a history (space allowing) of past index instances – if there is a problem with new data, you can easily roll back to a previous index.

There are two main strategies for how you structure your backing indexes:

  1. Rolling index. You choose to have 2 or more backing indices. When you start, it will be on index 0. Subsequent rebuilds increment the index counter, until it reaches the amount of indices you want to keep, then roll over and start at the beginning. So, for example, if you chose to only have 2 indices (one hot, one cold), the first time you built your index, it would point to 0. Then when you rebuilt the index, index 1 would be created and when it was done being populated, the alias would point to it. When you rebuilt it a second time, index 0 would be deleted and rebuilt with new data, then the alias would switch from 1 to 0.
  2. Time-based. You start with an index that has some element of time in it (for example, 2016-04-09) and then create new ones also based on time. The alias always points at the latest-built one. Pruning old indices when they are already needed is done by a separate process.

We’re going to look at how to do the first one – using a rolling index – in this example. Sample code that you can run can be found here. The steps are roughly as follows:

  1. Get a lock to make sure no other process can also attempt to rebuild the index. This would leave the alias/index in a bad state and would possibly result in downtime.
  2. Figure out which index (i), if any, the alias currently belongs to. If the alias doesn’t exist or doesn’t point to any index yet, we’ll start at 0. Otherwise, we start at i + 1. The underlying index name will be <alias name>-<index number>. So, if our alias was called ‘cars’ and we were building it for the first time, the index would be named ‘cars-0’.
  3. If an index with this name already exists, delete it.
  4. Create the index, with the given properties (index name, shards, replica count).
  5. Put the type mapping, if needed.
  6. Rebuild the index, using a user-supplied function.
  7. Create the alias (if it doesn’t exist), or switch an existing alias to point to our newly-created and populated index.
  8. Release the lock (if any errors occur, this probably should happen as well).

The sample project will go through all of these steps using the Java API. It isn’t production-ready code, but it will run out of the box against Elasticsearch 2.3 (run net.uresk.samples.elastic.aliasing.sample.AliasRebuildApp) and should get you pointed in the right direction. All of the interesting code is in net.uresk.samples.elastic.aliasing.AliasingRebuildService, which I’ll walk through now.

Step 1 – Get a lock

We want to create a distributed locking mechanism to ensure that only one thing is attempting to rebuild any given index at a time. If two process were to rebuild an index at the same time, they would overwrite each other in various ways, and would almost certainly result in temporary availability issues with the index.

We can do this by creating an empty document in a special locks index in Elasticsearch. If we try to index a document and set create = true, it will fail if the document already exists. This guarantees that if we successfully create a document, we now hold the lock and can proceed.

This code will attempt to create a document. It will return true if it does, and false if it cannot. If the result is false, we assume something else is rebuilding the index and terminate.

One thing we don’t address here is a timeout on the lock – Elasticsearch does support TTLs on documents, and we could use this so that a lock would expire after a certain amount of time. This is useful to handle cases where a rebuild job crashes for whatever reason and you don’t want future jobs to get blocked. However, figuring out a good TTL value takes a bit of work, and I’ve left it out of this example.

Step 2 – Decide which index to use

Here we want to figure out which index we should be writing to, based on where the alias is currently pointing. If the alias doesn’t exist, we’ll start with <alias name>-0. Otherwise, we’ll use <alias name>-(i+1), where i is the current index. If we’ve reached the maximum number of indices to create, we’ll start back at 0.

These 3 methods are useful for doing this. If we find an index name that matches the alias name we are trying to use, or if the alias is pointing at more than one index, we throw an exception and stop processing because we are in an unknown state and don’t want to break the current setup.

Step 3 – Delete the index if it exists

This is pretty straightforward. If we are writing to a previously used index, we want to delete it first.

Step 4 – Create the index

Again, pretty straightforward – we create the index with our name (derived from the alias name) and the shard/replica counts specified.

Step 5 – Put the type mapping, if needed

In the sample, I’m not doing this, but it is often necessary to specify type mappings (ie, to define how certain should be stored or analyzed, or to tell it to not analyze certain fields). This will load JSON from src/main/mappings/<aliasName>.json and use it to create a type mapping.

Step 6 – Rebuild the index

Here we just call a user-defined function for rebuilding the index. We can pass it the actual index name so it knows where to index documents. It can also return data about its results (a count, or a list of ids) so we can report it back.

Step 7 – Create/Move the alias

Now it is time to either create the alias, or move an existing alias to a new index. It is important that moving/renaming the alias be done atomically, otherwise, you could end up in an inconsistent state (an alias pointing at 2 indices) or without an alias at all.

Step 8 – Release the lock

Now that we’re done, we can release the lock by deleting the document we created in step 1.

That’s it! Now our index is rebuilt and search traffic is hitting it, without any downtime.

Note: If you are doing online updates in addition to rebuilds, you’ll probably want to pre-create all of your indices ahead of time and do your online updates to all of them – this will minimize the amount of stale data that sits in your index due to data that changed or was inserted while the rebuild job was running.

If you have any problems, questions, or ideas for making this better, please let me know!

Also – If you are working with Elasticsearch, I’ve found Elasticsearch: The Definitive Guide to be a really useful resource.