Why Your Dashboard and Your Database Disagree

The numbers can change after you see them. For analysts using online dashboards, it's a fact of real-time analytics: the same metric displayed at two different times will frequently show two different values. This is a core aspect of stream processing systems, and few know the vocabulary for it. To interpret real-time trends and estimates, business analysts must understand how systems manage the arrival of data over time. Event time and processing time are different, and watermarks are only estimates.

Event time vs Processing time

A system defines events and processes data differently, based on the way it handles time. When it comes to event time, Apache Beam — an open-source streaming analytics framework — defines it as the timestamps on the elements, which corresponds to the actual occurrence time of each event. Processing time, on the other hand, is the current system time at which the data is received and processed. It is the time the event reaches the system and is observed.

These two different systems use different timelines, focusing on event time or processing time to manage the flow of data. Azure Stream Analytics, a real-time analytics service for the cloud, reinforces this understanding, defining "event time" as the actual occurrence time of the event, and "processing time" as the current system time.

Why the number changes later

When you see a number on the dashboard, it's a snapshot of the data at that specific time, based on the system's time management setting.

Watermarks: Estimating completion time

The systems use watermarks to estimate when all data in a window is expected to have arrived, and results are emitted when the watermark passes the end of the window. Apache Beam describes a watermark as a "guess" about when all data in a window is expected to have arrived. It's an estimate, and it can change, causing additional firings of the results.

Late-arriving data: Using different clocks

However, processing time triggers and event-time timers fire after a certain amount of processing time or when the watermark passes the timer timestamp. Therefore, the number could change as late-arriving data is processed.

This is because data is not always guaranteed to arrive in time order or at predictable intervals. When this happens, the system may fire metrics or events based on the clock, not the actual occurrence time, to ensure processing doesn't wait for late data.

But even in this real-time processing, when there is no incoming event, the watermark is the current estimated arrival time minus the late-arrival tolerance window, according to Microsoft. And the late-arrival tolerance window is a variable, not a definitive date, that allows events to arrive late and still be processed.

What a watermark is

When using estimate-based watermarks, Apache Beam assumes that if no event has arrived with a certain timestamp, then no event will ever arrive with an earlier timestamp. In other words, they estimate that enough data has arrived and begin to process and output the data, even if some late arrivals still exist.

The analytics vocabulary readers need

This is where the complex vocabularies for timing come in. To understand what they're interpreting, analysts should:

• Know the difference between event time and processing time — event time is the actual occurrence time of an event, while processing time is the time the event reaches the system and is observed.
• Understand the role of a watermark — a watermark is an estimate of when all data has arrived for a specific time frame. It's a guess, not a hard cutoff.
• Recognize the concept of late-arrival tolerance — when using event time, systems may account for some data arriving late, within a specified window.
• Be aware of the difference between processing-time triggers and event-time timers — processing-time triggers fire after a certain amount of processing time has passed, while event-time timers fire when the watermark passes the timer timestamp.

Why dashboards diverge from databases

When configuring real-time analytics, teams must choose the timing model for their specific use case. The timing model defines the key time for each job: when an event is considered to occur, when a timer could fire, when the stream is defined as a watermark, and when the system's estimate comes from.

Dashboard discrepancies often come from late-arriving data, estimated watermarks, and different clocks in different systems. A stream processing dashboard and a downstream database might use different timing models, leading to differing counts. This could occur even when both systems are working as intended and receiving the same data.

For heuristic watermarks, the watermark is an estimate that no earlier event-time data will be observed in the pipeline. When late data arrives, different systems may have different definitions of "late" based on their watermarking and late-arrival tolerance settings.

The upshot is that two "correct" systems can show different numbers at the same time, even if they are relying on the same data sources. This can be jarring and confusing for analysts, but it's a natural outcome of the way these systems manage the timing and arrival of data.

What to watch for in product docs

When configuring real-time analytics, teams should closely review the documentation to understand how their specific platform defines and manages event time and processing time, as well as how it uses watermarks and late-arrival tolerance.

Product documentation should clearly define the timing model used and explain how metrics may change over time as late data arrives and watermarks are updated. Ideally, docs should also explain the reasoning behind these approaches and help analysts understand the impacts on their specific use case.

For business analysts who regularly use real-time metrics, understanding this fundamental aspect of stream processing systems can help them avoid confusion and better interpret the data on the dashboard.