What Live Means in Live Video: The Latency Budget From Camera to Screen

Low-latency live video has transformed the streaming landscape, but how can streaming professionals achieve these ultra-fast viewing experiences? The answer lies in understanding the latency budget, which is the total amount of time it takes for a video signal to go from the camera lens to the viewer's screen. In streaming, this budget is a complex accounting that's spread across video capture, encoding, packaging, transport, and finally the buffer in the video player.

Most measurements of end-to-end latency—from video capture to TV playback—converge on between 3 to 10 seconds, according to DASH-IF, an industry consortium. Low-latency streaming, as the name implies, aims to compress this time budget.

MPEG-DASH, or Dynamic Adaptive Streaming over HTTP, is the ISO standard that enables this. It's designed to work with existing HTTP infrastructure, including servers, Content Delivery Networks (CDNs), proxies, and caches. So, you don't need a special low-latency streaming setup in order to deliver live video at this lower latency.

The latency superiority of DASH comes from its media packaging and playback behavior, not from fast encoding. There are two key approaches:

1. DASH's low-latency segment mode, which is a mode for video segments. It minimizes the delay before the first packet of a segment is sent, even before the entire segment has been encoded and uploaded. The manifest file keeps updating as the encoded video is transmitted, rather than being created after the segment is finalized.

2. Chunked delivery, where the chunk of data is available for playback as it arrives and is not blocked behind the rest of the segment. This is DASH's approach to delivery, where a small buffer is maintained and the video begins playing before the segment is complete.

Under the hood, this involves SegmentTimeline files formatted with $Number$ or $Time$ to specify the arrival and availability of segments. A ServiceDescription element declares the target max and min latencies, along with playback speeds. It's designed to reduce latency while maintaining video quality.

In contrast, HLS—Apple's adaptive streaming protocol—takes a different approach, favoring reliability and adaptive bitrate performance over latency. Apple published a protocol extension in 2019 that supports low-latency HLS, but there's less detail on how the extension improves the time budget.

Latency isn't a static measurement, and it's not a single quote-unquote number. So talking of a stream being "optimized" for low latency without linking it to metric improvements, especially when more detail is available in the references, transgresses the line for this writing.

In fact, the DASH-IF report concludes that overall latency isn't comparable to broadcast specs, which conflate capture, packaging and bufffer with transport. Instead, the industry is changing its expectations around what's possible by measuring the time from the point of acquisition to the viewer's screen. But this is also not universal and varies from setup to setup.

What remains consistent across streaming and playback systems is the reliance on buffers. Streaming video can endure a few seconds' latency in video action as it arrives piece-by-piece, chunk-by-chunk, at the TV. But extend that, and the user feels the lack of synchrony. So even at its fastest, low-latency streaming still carries a so-called latency budget, an allocation that's made across media capture, video encoding, media packaging, transport over the internet, and buffering. Taking any one of these slices of the latency budget—whether by cutting out the measurement step, or through chunking or segmenting segments in novel ways—as the key to low-latency performance misunderstood the real tradeoffs.

What streaming developers need to understand, to think about when they are designing a video streaming system, is that there is no true real-time low-latency. There won't be a live stream with no delay, a live stream with no re-buffering interruptions in the middle of the game. With adaptive streams, the latency will scale with the bitrate - picking up new action faster is done at the expense of video quality, or loss of bandwidth.

Low-latency streaming, as with any engineering challenge in broadcasting, trading off speed for quality, is ultimately about calculating the real-time delay impacts of saving on buffering and segment packaging. Apple engineers can discuss their latency metrics, but latency is always budgeted, segment by media segment.