A scale baseline: making insights more visible in RL experiment tracking

A single continuous run. No stitching, no sampling, no downsampled retention: every point below remains individually queryable after the fact.

One run, one configuration, a first pass. We have not yet swept the tuning parameters that govern batching, compression or partitioning, so these figures are a floor rather than a ceiling.

Parallel environments

50,000

Metrics per environment

20Scalar series, logged per step

RL steps

6,000

Env steps per RL step

64

Run duration

21 h 20 m

Disk written

2.2 / 7.5 TiBUsed of available, full-fidelity

ClickHouse

3 × i7ie.3xlargeSingle shard, 3 replicas · 12 vCPU / 96 GiB / NVMe each

Ingestion

4 workers1 vCPU / 2 GiB each, + 1 service pod

Kafka

3 nodes1 vCPU / 4 GiB each

Logger

1 process, 1 connectionVectorized · same region, separate AZ

Throughput first held at 4.4M points per second, and ClickHouse was not the reason: CPU and disk both had headroom. The limit was the network. Too much of the workload landed on one Kubernetes node and saturated its link, and AWS throttled it. Enabling compression on the ingestion workers cleared the constraint, and throughput rose past 6M per second and held there for hours, at which point ClickHouse CPU became the next bound. The run settled at 5M only because of a self-limiter left in the logger configuration, not because of a ceiling.

This matters for how the headline number should be read: every limit we reached was an infrastructure parameter, not the engine. More shards, more Kafka partitions and more ClickHouse CPU each open the next band. We quote the run conservatively, since it is synthetic, and would stand behind around 4M per second as a comfortable sustained figure on this hardware — with 5M measured, and 6M and above reachable when the headroom is spent on ingestion rather than reserved for queries and merges.

Experiments dominate the compute budget, and in RL they are also the least observable part of the stack. A policy degrades, an environment stops resetting, a reward gets gamed — and a tracker that records only top-line curves reports that something went wrong long after it did, without the environment-level detail that would say what. Retaining per-environment and per-episode data lets a change in the aggregate be traced to the specific environments, episodes and transitions that produced it. A pathology identified at step 200 rather than step 200,000 is compute not spent, which is the reason the ingestion rate has to hold under sustained load rather than in a burst.

The constraint is cardinality within a single run, not raw rate. Conventional trackers take the run as the unit — a dashboard of runs, each with a modest number of series. Environment-level RL logging inverts that: one run holds a million concurrent series, one per environment per metric, all live and individually queryable. That is a different problem from tracking many runs, and holding a million live series in one run without sampling any away is what this baseline was built to test.

A result is only interpretable alongside its limits, so we report them in the same place, at the same level of detail.

• Scope

  This is a synthetic load test

  The load was generated data from a single logging process, not a live RL training loop. It exercises the ingestion and storage path, which is the part that has to survive scale. It does not measure the training loop itself, or the overhead placed on it.

  One consequence worth stating plainly: generating random points is faster than real training, where the GPU takes time to produce each step. In a real run, throughput is bounded by how fast the training loop emits data, not by the logger. This test shows what the logger and ingestion path can absorb — the ceiling, not the typical case. A run against a live workload is the next measurement.

• Held up

  Queries stayed responsive under load

  Server-side P99 of 250–500ms during the heaviest phase, while ClickHouse was also running background merges; add roughly 100ms for the client round-trip. Rare outliers approached a second under peak merge pressure. Measured on NVMe-backed self-hosted ClickHouse.

• Known limit

  Storage is the real constraint over long runs

  The run filled 2.2 of 7.5 TiB in a day at full fidelity — comfortable here, but sustained around the clock, storage becomes the binding limit before throughput does. Continuous operation at this rate needs larger disks and a tiering policy for older runs. A genuine limit, not a tuning gap.

• Headroom

  5M per second was a single connection

  The whole run used one logger over one TCP connection, and that connection — not the backend — was the limit. Throughput scales with how the network load is spread, so distributed logging across several processes should go further. The figure was set by a single connection, not the backend.

• Reproducibility

  Throughput depends on the network between logger and cluster

  5M per second was reached with the logger in the same AWS region as the cluster, one availability zone away — close, low-latency, near-ideal. Two limits sit between a logger and the backend: our AWS L7 layer caps TCP window sizes (an infrastructure detail we can bypass, and one that matters less the closer you log to the cluster), and, from outside the cloud, the client's own upload bandwidth — which nobody can control. From a home connection in Poland the same single logger reached around 1M per second.

  Three things raise it: logging close to the cluster; distributed logging across more TCP sessions (with the caveat that sessions sharing one machine can still hit a shared upload cap); and a fast enough producer — if the training loop only emits 0.5M/s, the logger sends 0.5M/s. The high-throughput logging approach is documented.

• Unswept

  The configuration is unoptimised

  A first pass. Batching, partitioning and compression are largely untuned; enabling compression alone took the run from 4.4M to past 6M. We expect to improve on these figures and will republish when we do, including if anything moves the wrong way.

A throughput number is worth only as much as its completeness. Under sustained backpressure a pipeline can shed load silently, and the gaps surface only later, as missing spans in a plot. So the run was reconciled in full, in layers.

The statistics and headers — roughly 4 billion episode headers and 6 billion per-step stat rows — were reconciled first, since they are the fragile layer: they demand exactly-once accounting, where a single retry that double-counts would corrupt the aggregate. Every count matched. The individual data points were then verified against the per-series counters recorded in those headers, down to the level of each series: every metric held exactly its expected 384,000 points, and each series segment its expected 64 or 128.

No data point was lost, in the plots or the statistics, and every point, header and stat was queried and accounted for. The 384 billion is a reconciled count, not an ingestion count.