Semantic layers are usually sold as a place to put metric definitions. Write revenue once, expose it everywhere, stop arguing in meetings.

Then two teams report different revenue anyway, both technically correct, and the argument moves from the dashboard into the definition file.

The definitions were never the hard part.

Metrics are the visible layer, not the load-bearing one

A metric is an aggregation over a set of things. Change what counts as one of those things and the number changes, without the formula changing at all.

sum(amount) is unambiguous. Which amounts is where the disagreement lives:

• Is a cancelled order still an order?
• Is a customer who signed two contracts one customer or two?
• Does an order belong to the date it was placed, paid, or shipped?
• Is a refund a negative order, or a different thing entirely?

None of these are metric questions. They are questions about what an order and a customer are, and whether two records refer to the same real thing. A semantic layer that answers only the metric question inherits every one of these disagreements and gives them nowhere to live.

That missing part is the ontology: not the formulas, but the entities the formulas quantify over.

What an ontology actually commits to

Three things, and each one is a decision somebody has to make explicitly.

Identity. What makes two records the same thing. This is the one most often skipped, and the one that causes double counting for years.

Grain. What one row means. “One row per order” and “one row per order line” are different worlds, and metrics silently break when a model changes from one to the other.

Relationships. How entities connect, and — critically — the cardinality. A fan-out join that nobody declared is the single most common cause of an inflated total.

erDiagram
    CUSTOMER ||--o{ ORDER : places
    ORDER    ||--|{ ORDER_LINE : contains
    ORDER    ||--o{ REFUND : "may be refunded by"
    PRODUCT  ||--o{ ORDER_LINE : "appears in"
    CUSTOMER {
        string customer_id PK
        string account_id "identity: one per legal entity"
    }
    ORDER {
        string order_id PK
        string customer_id FK
        date   placed_at
        date   recognised_at "grain: revenue date"
        string status
    }
    ORDER_LINE {
        string order_line_id PK
        string order_id FK
        numeric amount
    }

Read the diagram as a set of commitments rather than a picture. It says an order has many lines, so any metric summing amount at the order grain is wrong. It says a customer maps to an account, so “number of customers” needs to state which of the two it means. It says revenue has its own date, distinct from when the order was placed.

Each of those is a sentence somebody could have argued about in a meeting. Written down, they stop being re-litigated every quarter.

Making it executable

An ontology that lives in a diagram decays. The version that survives is the one the query engine reads, because then it cannot drift from what the numbers actually do.

entities:
  - name: order
    primary_key: order_id
    grain: "one row per order"
    # Identity is a decision, not a technicality: two rows with the same
    # source id from different systems are the same order only after this
    # rule says so.
    identity:
      resolve_by: [source_system, source_order_id]
    relationships:
      - to: customer
        type: many_to_one
        via: customer_id
      - to: order_line
        type: one_to_many
        via: order_id

  - name: order_line
    primary_key: order_line_id
    grain: "one row per line within an order"

metrics:
  - name: revenue
    description: >-
      Net of refunds, recognised on recognised_at rather than placed_at,
      excluding cancelled orders.
    entity: order_line
    agg: sum
    expr: amount
    filters:
      - "order.status != 'cancelled'"
    time_dimension: order.recognised_at

The entity: order_line line is doing quiet, important work. It declares the grain the sum runs at, which is what stops the engine from fanning out an order-level join and counting the same money twice.

How a question resolves

The value of writing this down is that a question stops being answered by whoever composes the SQL, and starts being answered by the model.

flowchart LR
    Q["revenue by customer,<br/>last quarter"] --> M[metric: revenue]
    M --> G[grain: order_line]
    M --> F["filter: status != cancelled"]
    M --> T[time: recognised_at]
    G --> J{join path}
    J --> O[order]
    O --> C[customer]
    C --> ID["identity: account_id"]
    ID --> R[one row per account]

Two analysts asking the same question now traverse the same path. Not because they agreed, but because the disagreement was settled once, upstream, and encoded.

Where the ontology should live

The hard part is not modelling. It is ownership: an ontology maintained by a central team drifts from the domains it describes, because the people who know what a contract is do not work there.

This is the argument for pushing definitions to the domains that own the data, with the semantic layer as the contract between them rather than the place where meaning is invented.

You do not have to adopt the whole organisational model to take the useful part: the definition belongs where the knowledge is, and the layer’s job is to make it legible everywhere else.

Failure modes worth naming

The pattern across all five: the formula was fine, and the thing underneath it was never written down.

Where to start

Not with a metric catalogue. Start with the five or six entities the business argues about most, and for each one write the three sentences: what makes two of them the same, what one row means, and how it connects to the others.

It is unglamorous, and it takes an afternoon per entity. But those sentences are what every metric quietly depends on, and leaving them unwritten is how a semantic layer ends up being one more place where the same disagreement is stored.

The question arrives as a technology question. Should this integration be batch or streaming? And it gets answered like one, by comparing engines, latencies and throughput numbers.

That framing is the problem. Latency is not the requirement. It is a property you buy, and the thing worth arguing about is how much of it you actually need.

Start from the decision, not the pipeline

Data exists so somebody, or something, can act on it. So the useful question is not “how fresh can we make this?” but “how quickly does the action it triggers actually happen?”

Freshness beyond the cadence of the decision it feeds is waste. It costs money and complexity and buys nothing, because the number sits there unread until the weekly review, the nightly credit run, the moment a human opens the dashboard.

That gives a rule worth applying before any engine is chosen:

flowchart TD
    A[New integration] --> B{Does something act<br/>on this automatically?}
    B -- no --> C{How often does a<br/>person look?}
    C -- daily or less --> D[Batch]
    C -- continuously --> E[Micro-batch]
    B -- yes --> F{Is the action reversible<br/>if the data is late?}
    F -- yes --> E
    F -- no --> G[Streaming]

Most integrations land on the left half of that diagram. The ones that genuinely need streaming tend to share a shape: something acts without a human, and acting late is materially worse than not acting.

Fraud declines. Inventory that can oversell. Routing decisions. Anything where the window to respond closes on its own.

The cost you can see

The obvious cost difference is shape, not size. Batch is bursty: nothing, then a spike, then nothing. Streaming is a floor you pay whether or not anything is happening.

That last row is the one that gets left out of the estimate, and it is often the one that decides whether the system survives its second year.

The cost you find later

The expensive part of real time is not infrastructure. It is that correctness stops being obvious.

In batch, “yesterday’s orders” is a closed set. You know when it is complete because the day ended. In a stream there is no such boundary — you decide one, and every event that arrives after it forces a choice you now have to make explicit.

sequenceDiagram
    participant S as Source
    participant P as Pipeline
    participant C as Consumer
    S->>P: event t=10:00
    S->>P: event t=10:02
    P->>C: window 10:00-10:05 closed
    Note over P,C: consumer acts on the total
    S-->>P: event t=10:01 (arrives 10:07)
    Note over P: late — restate, discard,<br/>or emit a correction?

There is no free answer. Discarding it makes the number quietly wrong. Restating means every downstream consumer has to handle a total that changes after the fact. Emitting a correction means they have to understand corrections.

In batch, the same event simply lands in the next run.

What this looks like in practice

The batch version of a rolling total is a query with a boundary you can point at:

-- Incremental model: the boundary is explicit and inspectable.
select
    customer_id,
    date_trunc('day', ordered_at) as day,
    sum(amount)                   as daily_amount,
    count(*)                      as order_count
from {{ ref('orders') }}
where ordered_at >= (select coalesce(max(day), '1900-01-01') from {{ this }})
group by 1, 2

If it is wrong, you re-run it. The failure mode is a stale number, and stale is a condition you can detect and alert on.

The streaming version has to say what it believes about time:

# The windowing is the easy part. The watermark is the real decision:
# it declares how long the pipeline waits for stragglers before it
# calls a window closed — and therefore which events become "late".
orders = (
    env.from_source(kafka_source, watermark_strategy, "orders")
       .key_by(lambda o: o.customer_id)
       .window(TumblingEventTimeWindows.of(Time.days(1)))
       .allowed_lateness(Time.hours(2))
       .aggregate(SumAmount())
)

allowed_lateness(2 hours) is not a tuning parameter. It is a business statement: after two hours we would rather be wrong than wait. Somebody should agree to that explicitly, and almost nobody is ever asked.

The hybrid trap

The instinct when both are needed is to run both — a fast path for freshness, a batch path for correctness, reconciled downstream. It is a reasonable-sounding design that tends to age badly, because the same business logic now exists twice and drifts.

Jay Kreps made the counter-argument well: if the log is replayable, you can reprocess history through the same code that handles the live stream, and the second implementation stops being necessary.

The argument is worth taking seriously, and worth reading as what it is: a case for avoiding duplicated logic, not a case for making everything streaming.

What I would ask before choosing

• What acts on this, and how fast does it actually act?
• If the number were two hours stale, what would go wrong — concretely?
• Who is on call for this at 3am, and do they know this engine?
• When a value is restated after the fact, what does the consumer do?
• If we choose batch now and need streaming in a year, what does that migration cost?

The last one matters more than it looks. Batch to streaming is usually a rewrite. Streaming to batch is usually a simplification. When the answer is genuinely unclear, the asymmetry argues for starting simple.

None of these are big decisions. Each is a sentence in a design document, made in an afternoon, and each one quietly sets what the system can be asked for two years later.