Validating a Kubernetes ownership instrument (before you trust it on harder faults)

Create a Deployment. Kubernetes quietly wires a chain under it. Deployment owns ReplicaSet. ReplicaSet owns Pods. Garbage collection and rollouts lean on those ownerReferences. Most days you never look.

Here is the awkward part. Suppose a controlling owner flickered. A Pod’s controller=true owner briefly became a different ReplicaSet, then reverted, without an expected delete/adopt pattern. Would that show up in a recorded trace? Or only as a hope that everything settled?

KOSV (Kubernetes Ownership Safety Verifier) is a small lab instrument for that measurement question. It polls ownership into JSONL, evaluates two plain oracles, injects a narrow set of lab faults, and archives the run so someone else can replay it.

This post is instrument validation. It is not a Kubernetes defect report.

Repo: k8s-ownership-safety-verifier

Pin: v0.1.5

git clone https://github.com/kazuru-chidumbwe/k8s-ownership-safety-verifier.git
cd k8s-ownership-safety-verifier
git checkout v0.1.5
make smoke-fixtures
make smoke-kind   # Docker + kind + kubectl
# archived matrix: matrix/runs/20260726T210257Z/

Figure 1. Workload and faults into Kind. Collector polls. O1/O2 verifier. Report out. On single-node Kind, eth0 netem delays host kubectl↔API, not the controller-manager informer path.

What we are actually testing under delay

E1/E2 put tc netem on the Kind node’s eth0. That does not delay the controller’s view of the API.

On single-node Kind, kube-controller-manager reaches the API at the node’s own address. Linux delivers that on lo. So eth0 netem hits the host kubectl collector path, which is the path KOSV uses to build traces.

We measured that under the same 500 ms qdisc (fault-reach archive). Host kubectl mean ≈ 1605 ms. In-node HTTPS to the same API address ≈ 23 ms.

So the delay experiments ask whether the polling instrument still produces a usable ownership trace when its own observations are late. If the instrument goes blind under moderate collector latency, we cannot trust it later when we actually stress the controller.

That is the experiment. Not “Kubernetes ownership under API latency.”

Two oracles

O1 (at most one controlling owner, snapshot).

At one observation, for one object (resource, uid):

count(ownerReferences where controller=true) ≤ 1

Two controlling owners at once is an O1 failure. Example fragment:

{"resource":"pod","uid":"…","owners":[{"uid":"rs-a","controller":true},{"uid":"rs-b","controller":true}]}
# → O1 FAIL

O2 (unintended transfer).

If the controlling owner flips from A to B without an expected orphan or delete sequence, that is an O2 failure. Orphan-then-adopt is intended and does not fail.

{"resource":"pod","uid":"…","owners":[{"uid":"rs-a","controller":true}]}
{"resource":"pod","uid":"…","owners":[{"uid":"rs-b","controller":true}]}
# → O2 FAIL (no orphan / DELETE between)

A flicker, in this instrument, is exactly that kind of O2-shaped pattern in the JSONL. The controlling owner changes for a few observations without an expected handoff, then may revert. We are not reading the controller’s informer cache. We only see what landed in the polled API snapshot.

Belief-state oracles (O3/O4) are not in this pin.

Lab bounds

v0 runs on Kind v1.31.6. The ownership path under test is Deployment → ReplicaSet → Pod. The collector is kubectl polling about once a second (plus kubectl runtime). Faults are collector-path eth0 netem, plus a clean kube-controller-manager restart during steady state.

Kind is a laboratory. Not in this pin are StatefulSets, Jobs, custom operators, multi-control-plane, or true API↔controller delay.

Threat model and scope notes live in THREAT-MODEL.md and SCOPE-ISOLATION.md.

Fault hygiene (confirmation)

The collector-path claim is not a late confession. It is the design. Evidence is archived under docs/evidence/fault-reach-2026-07-27/.

Separately, host-side delay_proxy self-tests stay within roughly ±10% of configured 500/2000 ms targets. That is delay_proxy calibration, checking a tool against a known target. It is not Kind netem as controller RTT.

Figure 2. Host delay_proxy means for E0/E1/E2. Tool calibration only.

True API-to-controller delay (lo netem / interception) is not in this pin.

E3 is a clean restart during steady state. The controller-manager restarts while the Deployment is already ready, with no in-flight rolling update. The ownership graph is essentially static across the bounce. Later campaigns can escalate to restarts during rolling updates or scale-down races.

Prove the detector first

Before arguing about clusters, make the verifier catch known lies.

• SMOKE-O1 expects O1 fail → does
• SMOKE-O2 expects O2 fail → does
• SMOKE-O2-INTENDED expects pass → does
• Cross-resource shared UID string → pass (no false O2 leak)
• SMOKE-KIND-CLEAN expects pass → does (16 events in the pin run)

Twenty-run instrument validation

Archive matrix/runs/20260726T210257Z/. Same script every time. Create Deployment (2 replicas) → wait → scale to 3 → collect → evaluate O1/O2.

• E0 baseline · 5 · PASS
• E1 500 ms collector-path netem · 5 · PASS
• E2 2000 ms collector-path netem · 5 · PASS
• E3 clean CM restart in steady state · 5 · PASS

20/20 PASS. Zero O1, zero O2, zero inconclusive.

These runs confirm that the verifier stays operational and produces clean traces under these lab conditions. That is instrument validation, a necessary prerequisite before hunting real anomalies. It is not a safety result about Kubernetes ownership.

Event counts drop under E1/E2. Thinner coverage when the measurement path is throttled. Not proof that informers went stale.

Figure 3. Mean ownership events per run (min/max). Matrix 20260726T210257Z.

One-pager write-up is in MATRIX-ANALYSIS.md.

What this buys. Planted violations are caught. Intended transfers are not false alarms. The pipeline stays consistent under these lab faults.

What this does not buy. Universal ownership safety. Production equivalence. A complete fault model. O3/O4.

With ~1 second polling, we can miss violations shorter than our sampling window. PASS means no O1/O2 violation showed up in the recorded trace.

Reports also carry a machine-checkable caveats[] field. That is part of the artifact, not only prose.

Why publish the instrument?

Later campaigns can ask harder questions about ownership under broader faults. Those questions are worthless if the ruler is bent.

KOSV is the ruler. Argue from tagged artifacts.

git checkout v0.1.5
make smoke-fixtures
make smoke-kind

Star or watch the repo if you want later tags.