Reading a Latency Histogram: Why p99 Is the Number That Pages You

When you're monitoring distributed systems, the average is nearly worthless because it masks the behavior that matters most: the latency outliers that lead to actual system stalls. Many outliers are missed for every sample you measure, so you must catch the tail percentages.

But here's another thing that's easy to miss: the way you measure the tail can be wrong in a way that makes the tail look good. That's the problem of coordinated omission, and it's so common and predictable that the HdrHistogram library has special features in it to detect and correct for it.

Why p99 Lies

In some circles, there's an idea, there's an idea that p99 is the most important tier. But this falls into a gaffing trap. The tail is important, but only after it has been corrected for the way it was measured.

Coordinated Omission

That correction is necessary because latency measurement is not a high-impedance probe, but rather a low-impedance load generator, and the low-impedance load generator coordinates itself with the timing of the workload it is measuring. Specifically, in a closed-loop testing system, the load generator shuts off for a moment when it sees a stall. This looks to the load generator like a little perfect calm, so the tail is snipped off at the top of the latency histogram.

This phenomenon has been so thoroughly described that programmers have nicknamed it, in one forum discussion, coordinated omission. Gil Tene discussed the ideas in the Mechanical Sympathy forum in 2013, and later summed up the problem at a technical talk where he said:

Coordinated omission exists when you measure a system's response-times at one frequency, but want to confidently describe the system's p99 response-time. The p99 will occur far less frequently than once every 100 measurements, so is likely to be unmeasured. If the p99 of the real system is similar to or only slightly larger than the typical observed response-time interval, you will likely know about it. But if the real system's p99 is much larger than the measured measurement interval, the measured p99 will be too low and you are coordinated-omitting the higher values from your measurement.

When coordinated omission exists, you're measuring p99 of the moments between outages, not of the system in real life.

It has been widely discussed in technical circles. Coordinated omission acts like a perpetual motion machine for p99, but the p99 values it measures play out like a mirage—once the machine starts, the illusion collapses.

It turns out this problem has been common in distributed tracing, but improvements are being made. Improved instrumentation is making it harder to mishandle p99.

How HdrHistogram Fixes It

Latency histogram libraries now do a type of coordinated omission correction that involves more than just waiting. HdrHistogram, for instance, generates missing values when it finds them. This is not a random sampling, but a step back from the actual value to an earlier, shorter interval. It remembers those and when it sees the next actual value, it rechecks the interval. In other words, it stitches a new tail by stitching in synthetic values higher up in the tail.

These corrections are described in the HdrHistogram documentation and have been in the library for years. But even this is not enough: you have to instrument for coordinated omission from the start.

What “Good” Measurement Requires

To get coordinated omission right, you need to understand what your own system is hiding. When you instrument, that means making a choice. Either you gather at fine enough granularity and quality at the source so that the tail you measure is real, or you correct for the tail in the processing.

For the first choice, you need to decide how the instrument in the system is connected to the workload or to the timing. The timing is the easiest: most systems will already be timing distributed tracing, or they are easy to instrument if a timer is present. But connecting the workload to the timing is the hard part: it means your system has to have a way to trace requests through the services that manage them.

For the second choice, you need to understand how often your system is going to be gathering latency information, and you can't just average out the values you get from those times. That's why p99 needs to be measured from the histogram you get from combined systems, not from p99 measured from each system combined.

This kind of attention to detail can expose the true shape of a service's behavior and put an end to the rise of bad latency figures: tail latency amplification when systems coordinate and hide p99 values from view. Coordinated omission made us blind, but it's happening less and less now that we're teaching people what p99 means.

Why Fleet p99 Must Be Recomputed

One trap for p99 is making the same mistake we discussed above: averaging the p99s of multiple samples. Instead, to get the fleet’s real p99, you need to combine the histograms that measure individual services, and get a fleet-wide p99 from the histogram of their combination.

The Alerting Question

Finally, p99 is a useful indicator of service health. But how do you decide when it's normal, or when it’s a problem that you need to take action on? Making a decision about p99 like that isn't something you can do scientifically, which draws the concern that p99 is more of a psychological construct than a metric of reality.

What the Metric Really Means

The lesson of coordinated omission is clear: whenever there is a p99 line chart, there should be a measured interval. And if the correction process is not consistent across the data, it's going to be an illusion at the end.