Why real-time analytics is often overkill, and what you need instead
Streaming data pipelines are expensive to build and run, and most business decisions do not need them. A plain framework for deciding how fresh each metric has to be, what real-time really involves, and what to do instead for the few things that cannot wait.
"Real-time" is on almost every analytics product page. For most small and mid-size businesses it is the wrong goal. Data that is a few hours or a day old supports nearly every decision people make from a dashboard, and the step from daily to truly real-time is one of the most expensive in data engineering.
This post defines the terms, explains what each level of freshness involves, and gives a framework for deciding how fresh each metric needs to be.
Three levels of freshness
Real-time (sub-second to a few seconds). Each event is processed as it happens and is queryable almost immediately. This needs streaming infrastructure: an event broker such as Kafka, a stream processor such as Flink, and a store that can take continuous writes and answer queries at the same time. It runs around the clock and has to recover from failures without losing or double-counting events.
Near-real-time (about a minute to 15 minutes). Data arrives in small batches on a short schedule, or through change data capture from a database. Simpler than full streaming, but still more moving parts than a daily job.
Batch (hourly to daily). Data is extracted on a schedule, transformed and loaded. This is the classic pattern, and the easiest to debug: when a job fails, you rerun it.
What each level asks of you
The difference is not a slider; it is a step change in what you build and who has to look after it.
| Daily or hourly batch | Near-real-time | Real-time streaming | |
|---|---|---|---|
| Typical parts | Scheduled extract, a store, dashboards | Adds an event broker, micro-batch processing, a fast serving store | Adds stateful stream processing, checkpointing, exactly-once delivery |
| Failure handling | Rerun the job | Monitor lag, replay from the broker | Specialist skills to diagnose |
| Who runs it | Anyone who can schedule a job | An engineer with some streaming experience | A dedicated data platform engineer or team |
| On-call load | Low | Moderate | High |
Costs follow the same shape. Always-on compute, a managed broker and specialist engineering time are the big items, and people usually cost more than infrastructure. Get quotes for your own volumes rather than trusting any rule of thumb, including ours.
Start from how fast someone will act
Freshness only matters if someone, or something, acts on the data faster than the next refresh. Map each metric to the decision it serves.
| Decision type | Examples | Freshness that is enough |
|---|---|---|
| Strategic | Pricing, hiring plans, new markets | Weekly or monthly |
| Tactical | Budget shifts between campaigns, stock reorders, staffing next week | Daily, sometimes hourly |
| Operational | Incident response, queue management | Minutes, from a monitoring tool |
| Automated | Fraud blocking, dynamic pricing, auto-scaling | Real-time, because a machine acts |
A useful rule: real-time pays off when machines act on the data or when safety is at stake. When people act, daily or hourly is almost always enough. People read dashboards in the morning, in a weekly review, or when something prompts them. Data that changed every second in between went unread.
Put a number on the delay
When someone insists a metric must be real-time, estimate what the delay actually costs:
Value of fresher data = hours of delay removed x cost per hour of delay x how often it happens
Example: a checkout problem costs about $1,000 an hour in lost orders and happens twice a year. With daily reporting you notice it about 12 hours in on average; with hourly data, within about an hour. Moving to hourly saves roughly 11 x $1,000 x 2 = $22,000 a year. Going from hourly to real-time saves under an hour each time, so about $2,000 a year, which will not pay for a streaming stack.
The example also shows the better answer: a checkout outage should be caught by uptime and error monitoring that pages an engineer, not by a business dashboard.
Check what your sources can deliver
A pipeline is only as fresh as its slowest source:
- SaaS APIs (CRM, billing, marketing tools) are polled and rate-limited.
- Ad platforms and analytics tools revise recent data. Google Analytics 4 can take 24 to 48 hours to finalize a day, so a "real-time" revenue-per-session chart would be built on numbers that are still changing.
- Files and exports are batch by nature.
- Financial data is often only final after month-end close.
If your most important numbers come from these, real-time processing downstream adds cost without adding freshness.
What to do instead
Tier your metrics. Most businesses land somewhere like this:
- A handful of operational signals (payments failing, site down) handled by monitoring tools with alerts.
- Business performance metrics (orders, sign-ups, spend) refreshed hourly or daily.
- Planning metrics (CAC, retention, cohorts) refreshed daily or weekly.
Alert on exceptions instead of watching screens. A check that runs every hour and flags a value far outside its normal range beats a live dashboard nobody watches after the first week. The anomaly detection guide covers how to set that up without drowning in false alarms.
Offer a refresh button. For the occasional moment when someone needs current numbers, an on-demand sync costs almost nothing.
Show when data was last updated on every dashboard and report, so nobody has to guess.
Mistakes worth avoiding
- Building for a need nobody has yet. Idle streaming infrastructure still costs money and still complicates every change.
- Choosing technology for its own sake. A reliable daily pipeline is better engineering than a fragile streaming one.
- The lobby screen. A large display of "live" numbers that nobody makes decisions from is an expensive screensaver.
How clariBI handles freshness
clariBI is built for the batch end of this scale, and says so.
- Sources sync on a schedule, not continuously. A source is synced again when its data is older than its sync interval (daily for most sources) and a dashboard refresh, report or question needs it, or when you click Sync Data. Uploaded files never re-sync. The sync help article explains the details.
- Dashboards refresh on the schedule you choose: manual, hourly, daily, weekly or monthly, checked every 30 minutes. The dashboards page describes what happens on each refresh.
- Some things are live. Notifications, sync status and the progress of long-running jobs are pushed to the app as they happen. What updates live lists them.
- No streaming ingestion and no threshold alerts on metrics. For unusual values, forecasting (Starter and above) flags anomalies and trend changes when it runs, and goals show when a target has fallen behind its pace.
If you need second-by-second operational monitoring, use a monitoring tool for that and keep your business reporting on a daily cadence. For more on how the stack underneath fits together, see data warehouses: what they are and when you need one.
A next step: list every dashboard your team has, write next to each one who looks at it and how often, and set its refresh to match. Most will end up daily, and nobody will notice the difference except in the bill.