Skip to content

Order-book latency: ingest → publish

How long a Binance order-book change takes to reach a subscriber, what it is made of, and what the worker's depthlevels / publishintervalms change about it. Numbers are from the shared cluster (medusa), 2026-10-21.

What is measured

The scope is ingest → publish: from the WebSocketManager receiving the venue frame (the event's ingesttime extension) to a subscriber receiving the BinanceOrderBook worker's changed event. Venue → WebSocketManager is not included -- that is a network/colocation question, not a pipeline one.

Method, per run:

  • 50 pairs, BinanceOrderBook workers, WebSocketManager connections.
  • A notebook subscriber to all 50 changed topics records receive walltime − ingesttime for every event and reports p50/p90/p99.
  • Snapshot-derived events are excluded (~1%): a snapshot response has no ingesttime.
  • Before every run the order-book streams in Redis are cleared and no workers, connections or triggers are left over. Both matter: Dapr's Redis pubsub replays a new consumer group from the stream's start, and stale connections from an earlier run double the offered rate (that alone once made a run look four times slower than it was).
  • Two sources of load. Live Binance (binance-order-book-live.verso) is the reality check, but its activity moves p50 by ±11 ms between identical runs -- more than any pipeline change buys. A/B comparisons use the fixed-rate harness (examples/harness/fake-venue, driven by binance-order-book-harness.verso or alpaca-order-book-harness.verso), which repeats within ±4 ms at p50. The venue serves Binance's protocol on /ws and Alpaca's on /alpaca, from the same books.

Results

Config p50 p90 p99 Events
full depth, every update 86.4 ms 464 ms 1,109 ms 144/s
depthlevels: 20, every update 22.2 ms 52 ms 204 ms 151/s
depthlevels: 20 + publishintervalms: 250 20.3 ms 32 ms 199 ms 64/s
  • Depth is the lever. A full BTC book is ~127 KB and one is published per applied diff; at 50 pairs that saturates the publish path. 20 levels per side is where it stops being byte-bound, hence the default.
  • Conflation does not reduce per-event latency. It thins announcements (2.3× fewer events here) and helps p90 a little, but the events it drops are not measured -- the book a consumer holds can be up to the interval older. Keep publishintervalms at 0 unless a consumer only wants periodic snapshots.
  • Expect ±5–11 ms p50 between identical runs: venue activity moves the offered rate (123–195 events/s across these runs) and latency tracks the rate. Comparisons below ~5 ms are not meaningful on live traffic.

The tail (p99)

The tail is not a slow component -- it is queueing. Each delta needs its own actor call and publish, serialized per pair, so a burst of deltas for one pair arrives at the subscriber in order but spread by the per-delta cost. The harness, at the same mean rate (150/s):

Load shape p50 p90 p99
smooth 21.2 ms 29.4 ms 44.2 ms
5-frame bursts per pair, independent phases 42.7 ms 88.8 ms 144.0 ms
5-frame bursts, 10 workers instead of 5 42.6 ms 98.9 ms 148.7 ms
5-frame bursts, every pair firing simultaneously 313.9 ms 429.3 ms 510.6 ms
  • Burstiness explains the live tail. 44 ms smooth → 144 ms bursty is the live pipeline's shape (100–200 ms p99), at an unchanged mean rate.
  • Sharding does not help. Doubling the worker count (5 → 10, i.e. 10 → 5 pairs per worker) leaves the tail where it was: this is per-delta serial application, not per-worker inbox depth.
  • Synchronization is poisonous. When every pair fires at once, p50 goes to 314 ms -- 15× the smooth value. Keep anything that touches all pairs at once (snapshot rounds, trigger schedules) staggered; the example notebooks pace worker creation by 2 s per pair for exactly this reason.

Conflation (publishintervalms) would make the p99 look better, but only because the dropped books are never delivered -- the consumer's book can be up to the interval older. It is a freshness trade, not a latency win.

Dapr's Redis queueDepth (100 → 10) made no measurable difference on the harness (21.2 → 22.0 ms p50, 29.4 → 31.1 ms p90), so the cluster keeps the default.

Alpaca vs Binance

AlpacaOrderBook shares the encode path, the depth trim, the publish gate and the background save with BinanceOrderBook (Virtufin.MarketData.DevKit), but the feeds differ: Alpaca has no snapshot/gap protocol, sends a full book per subscribe, batches messages as arrays and carries several pairs per frame. The harness puts both on the same load -- 50 pairs, 3 frames/s per pair (~150/s), depthlevels 20, 5-frame bursts (the venue's default) -- one run each:

Worker p50 p90 p99
BinanceOrderBook 0.6.0 42.7 ms 88.8 ms 144.0 ms
AlpacaOrderBook 0.4.0 40.7 ms 74.2 ms 132.1 ms

The two are indistinguishable at p50 and Alpaca is a little better at p90/p99 (smaller frames: a 5-level full book and one level per change, against Binance's per-pair raw frames). The Binance measurements transfer: the cost is the shared path -- the hops and the per-delta serial application -- not the venue's protocol. Alpaca's changed events also carry ingesttime (every market-data worker copies it through MarketDataEnvelope.Stamp), so the same measurement works without a special case.

Where the time goes

Per-leg timing in the runtime (a diagnostic build; 349 five-second intervals, 60k messages, depth 20):

Leg p50 p90 What it is
ingest → runtime enqueue 7.3 ms 15.3 ms WebSocketManager → Dapr → Redis → API fan-out → runtime
inbox wait 0.03 ms 0.14 ms the worker's ordered inbox
actor call 2.9 ms 4.9 ms the in-process worker, including the WorkManager ↔ DotNetDll bridge
outbox wait 0.05 ms 0.4 ms response queued behind the batch
publish 4.3 ms 7.6 ms the gateway's PublishEvents round trip
final delivery ~3 ms ~10 ms Redis → API subscriber → the subscribing client

Nothing is queued (inbox and outbox are effectively zero): the pipeline is not throughput-bound at depth 20, it is five network hops at roughly 3 ms each. That puts a floor of about 15–20 ms p50 on the current topology, whatever the payload or worker code does. The API is the mediator of every hop by design, so single-digit milliseconds would need fewer hops, not faster code.

Reproducing

Two notebooks, same measurement:

  • examples/dotnet/binance-order-book-live.verso -- live Binance, the reality check.
  • examples/dotnet/binance-order-book-harness.verso -- the fixed-rate fake venue (examples/harness/fake-venue), for A/B runs. Deploy the venue first; its README covers the ingress WebSocketManager's SSRF guard requires.
  • examples/dotnet/alpaca-order-book-harness.verso -- the same harness against AlpacaOrderBook, using the venue's /alpaca endpoint and an in-notebook handshake responder (login + subscribe).

Pass the knobs in the worker's config:

{
  "currencypairs": "BTC_USDT,ETH_USDT",
  "depthlevels": "20",          // default; 0 = every level
  "publishintervalms": "0",     // default; 0 = every applied update
  "pairconfig": "{\"BTC_USDT\":{\"depthlevels\":50}}"
}