ScaleScope: I Built a System That Makes Autoscaling Watchable — Here's What Broke

The autoscaler is doing something. You can't see it.

You set a CPU threshold. Load increases. A few minutes later, you refresh a dashboard, and the container count has changed. Somewhere in between, a decision happened:

demand increased → threshold crossed → capacity changed → container started → latency recovered

But you weren't there to watch the chain happen. Instead, you reconstruct it afterwards from logs with different timestamps, dashboards with different sampling rates, and metrics that don't share the same clock.

That gap is why autoscaling still feels like a black box — even when you're the person who configured it.

So I wanted to build something different. Not another mocked autoscaling dashboard. Not a simulated container counter. I wanted real load hitting a real service, Zerops making the real scaling decisions, and the entire causal chain rendered live while it happens.

That's ScaleScope.

🚀 Try It Yourself

• Repository: ScaleScope on GitHub
• Live Demo: scalescope

What Is ScaleScope?

ScaleScope is an experimental autoscaling observability system built around one idea:

Make the entire scaling decision observable on one timeline.

You pick a load profile — Steady, Ramp, Spike, or Latency-target autopilot — choose a duration, and start the experiment. From there:

• A fleet of worker containers sends real HTTP requests to a real target service.
• The target burns real CPU per request.
• Zerops' own autoscaler decides when to scale — nothing is simulated.
• Container count is measured independently, not pulled from a privileged platform API.
• A digital twin predicts the container count 15 seconds into the future.
• Every experiment is recorded as an append-only event stream.
• Runs can be replayed later through the exact same rendering pipeline used live.
• Chaos experiments can kill, degrade, or partition the target mid-run.
• A scheduler can run unattended experiment suites and compare results across runs.

There are twelve services in the system, and yes — I checked that they all actually do something.

System Architecture

The key architectural decision: Postgres, ClickHouse, and Valkey are projections of the event log — they are not independent sources of truth. Everything traces back to one append-only stream in JetStream.

That's why replay is cheap: replay.js re-emits the same events — preserving their original inter-arrival gaps — into the exact SSE pipeline used by live traffic. The frontend never branches on isReplay. It just renders events. Live and replay speak the same language.

A Live Experiment, Step by Step

Here's what actually happens when you press Start Run, based on the real implementation in apps/gateway/src/orchestrator.js and apps/worker/src/fleet.js.

1. Admission

Before anything starts, the gateway checks two things stored in Valkey: the hourly credit budget and a single-active-run lock. The button is public, and the backend is credit-billed — the budget cannot be an afterthought.

2. The Two-Phase Barrier

This was one of the easiest places to accidentally introduce a race condition. The gateway first opens a subscription for CTRL.READY acknowledgements, then broadcasts CTRL.PREPARE. That ordering is deliberate — a fast worker acknowledging before anyone is listening is exactly the kind of race that might never appear during a demo, then suddenly appear under real load.

Each worker receives PREPARE and performs one throwaway warm-up request, so DNS resolution and connection-pool initialization don't pollute the first real latency bucket.

3. One Shared T0

Once enough workers respond (or the timeout fires), the gateway computes an absolute future timestamp:

const t0 = Date.now() + T0_LEAD_MS;

pub(nc, CTRL.GO, { runId, t0, config, workers });

Every worker derives its per-second buckets from that same timestamp. Five different load-fleet containers can now agree on what "second 7" means without continuously coordinating — instead of worker 1: "7.01s ago", worker 2: "6.94s ago", worker 3: "7.13s ago", they all just tick from the same origin.

4. Load Starts

Each worker maintains its own concurrency pool against the target, using a shared keep-alive HTTP agent (not fetch) for deterministic connection reuse. Concurrency is derived from the selected load profile:

The autopilot controller is intentionally asymmetric — it backs off quickly, but only increases load as fast as maxRise allows. Ramping quickly is a surprisingly effective way to DoS your own target.

5. The Target Scales

Every target response carries:

X-Instance-Id: 7f83c1a2
X-Instance-Age: 4217

const INSTANCE_ID = crypto.randomUUID().slice(0, 8);
const BOOT_MS = Date.now();

res.set('X-Instance-Id', INSTANCE_ID);
res.set('X-Instance-Age', String(Date.now() - BOOT_MS));

That's the entire container-counting mechanism — no platform API, no privileged credentials, no hidden autoscaler endpoint. The collector tracks distinct instance IDs observed during a rolling 10-second window, and that becomes the live container count.

6. Samples Become Tick Frames

Workers publish per-instance samples to NATS every second. The collector merges them into one TickFrame. For percentiles, the system takes the worst observed p95/p99 across workers rather than pretending percentiles can be averaged; for p50 it uses a request-weighted mean. The resulting frame goes to the gateway, the oracle, and ClickHouse — one second of telemetry becomes one shared representation of system state.

7. The Gateway Broadcasts It

The gateway forwards the same tick frame to connected SSE clients, filtered by runId — a viewer watching a permalink to one experiment never receives events from another active run.

8. The Event Log Becomes the Record

Events — created, armed, started, tick, scaled, chaos, prediction, slo, completed — are appended to JetStream. At finalization, the gateway reads the full event stream and folds it into a summary using one reducer: foldRun(). That same reducer powers the REST API, replay, finalization, the scheduler, and suite success evaluation.

One event stream. One reducer. Multiple projections. That decision ended up being one of the most important architectural choices in the project.

The Engineering Decisions I'm Most Proud Of

Counting containers without a platform API

There's no simple application-layer Zerops endpoint that says "you currently have 4 containers" — and even if there were, depending on a privileged API on the hot path would introduce another dependency. Instead every container identifies itself, and distinct IDs in the rolling window become the count. Simple, observable, and independently measured.

Forcing horizontal scaling

This one broke the demo before it worked. Zerops scales vertically before scaling horizontally — a sensible default for normal workloads. But ScaleScope's entire premise is watch containers appear. If the platform keeps adding CPU to the existing container, the container count never changes.

The fix was one line in the import YAML:

cpuMode: DEDICATED

Dedicated CPU enables the horizontal CPU-trigger behavior the experiment depends on. An earlier version missed this; the system vertically scaled forever, the container count stayed at one, and the demo silently failed at its most important job. A very useful bug to find early.

The digital twin doesn't use machine learning

It learns three parameters: capacity per container, scale-up lag, and scale-down lag. That's it. AutoscalerTwin updates them with exponentially weighted estimates, and capacity updates are intentionally asymmetric:

const rate =
  observed > this.params.capacityPerContainer
    ? alpha
    : alpha * 0.3;

this.params.capacityPerContainer +=
  rate * (observed - this.params.capacityPerContainer);

One overloaded tick shouldn't permanently convince the model that every container is terrible. The model only learns from ticks that are demonstrably not saturated (p95 <= setpoint * 1.1), and it refuses to persist learning from runs shorter than 20 ticks. I watched this guard fire during local testing for runs with 18, 14, and 2 ticks — no special test case, no prompt, it just refused to learn from insufficient data. Exactly what I wanted.

The chaos secret check

The chaos endpoint accepts a secret, compared in a timing-safe way:

const a = crypto.createHash('sha256').update(presented).digest();
const b = crypto.createHash('sha256').update(CHAOS_SECRET).digest();

return crypto.timingSafeEqual(a, b);

timingSafeEqual() requires equal-length buffers — comparing raw strings of different lengths can throw, and careless padding can leak secret-length information. Hashing both inputs first produces fixed 32-byte buffers before comparison. Small detail, worth getting right.

The scheduler doesn't bypass the gateway

The scheduler could publish CTRL.* messages directly to NATS. It intentionally doesn't:

If the scheduler became a second writer to the NATS control subjects, the two paths would eventually drift, bypassing credit limits, active-run locking, admission checks, and barrier logic. So the scheduler behaves like any other client — it calls the public API and polls for progress.

What Actually Broke

Three real bugs survived static checks and careful code reading, and were only discovered when the whole system actually ran.

1. A Postgres type mismatch. One update statement used the same parameter as a bigint in one clause and implicitly as numeric inside to_timestamp() in another. It read fine, passed syntax checks, and the real driver rejected it on the very first POST /api/runs. Nearly invisible in review — humans are good at inferring intent, databases are good at enforcing types.

2. Every log line said [svc]. The telemetry package cached SCALESCOPE_SERVICE at module-load time. ES module imports evaluate before the entry point's own initialization code runs, so by the time each service set its environment variable, the logger had already cached the default. Result: eight services, every log line labelled [svc]. Nothing technically broken, but debugging a distributed system where every service has the same name is painful. Fix: read the environment value at log time instead of caching it at import time.

3. The status panel "fixed itself" into being wrong. The architecture panel did a direct health check, then ~10 seconds later, some services would mysteriously flip to unknown. A heartbeat reconciliation loop was overwriting the direct health result with data from a heartbeat table that nobody was populating for those services:

It was technically reconciling state — just reconciling it with nothing. Fix: exclude those services from that reconciliation path.

None of these three bugs was caught by node --check. All twelve services passed syntax checks early. Once you have event streams, distributed workers, barriers, asynchronous services, databases, caches, and queues, you need to run the actual system — not just the files.

What I Learned Building ScaleScope

1. Static checks tell you surprisingly little about distributed systems. A service can be perfectly valid JavaScript and still be completely wrong when it interacts with seven other services. The bugs weren't syntax bugs — they were system interaction bugs.

2. One event log beats three sources of truth. Making Postgres, ClickHouse, and Valkey projections of one JetStream event stream dramatically simplified replay. If every database independently decided what happened, replay would need a completely different code path. Instead, everything is a projection, and the event stream is the history.

Final Thoughts

ScaleScope started with a simple question: What if you could actually watch an autoscaler think?

The result is a distributed experiment platform for making scaling behaviour observable, replayable, predictable, and deliberately breakable. There's still work left to do, but the core loop works:

Start a run → generate real load → watch Zerops scale → observe the causal chain → replay it afterward.

And honestly, watching that second container appear for the first time was worth every bug that came before it.