The Room Can Tell Us Apart Now

We got it working, and we want to be precise about what "it" is

Two of us in the lab. For a while the machine has handled one person fine and come apart with two — losing people, swapping them, re-introducing the same human every few seconds. As of tonight it holds both of us as distinct, persistent people: through occlusion, through the tracker dropping us, through us sitting down behind desks.

One of us left dancing. It's been a long-running fight.

Now the honest part: we invented none of this. Every core technique we used has a name, a paper, and in one case a 1970 publication date. What follows is what we built, who we took each piece from, and the part that turned out to be the real work.

---

Recognition is cheaper than reconstruction

The most useful thing we understood all night: our depth camera's skeleton tracker is bad at partial bodies, and a plain 2D object detector on the colour camera is excellent at them. It will happily find a person from a torso and part of an arm.

That looks like one model simply being better. It isn't — they are answering different questions.

The detector asks "is there a person-shaped thing in this patch of pixels?" A head and one shoulder is enough evidence to answer yes. It trained on millions of crops where most of the body was already out of frame, so partial bodies are its normal case, not its failure case.

The skeleton tracker asks "where are all thirty-two of this person's joints in 3D?" That is a structured fit. Legs under a desk means no evidence for half the skeleton, so the fit either collapses or its own confidence gate rejects it. It fails on partial bodies because the thing it outputs is the whole body.

So occupancy comes from the detector, and the skeleton is an enrichment attached to a person when it happens to be available — never the reason that person exists in the system. Over 561 frames with both of us visible, the depth tracker under-reported in 78 of them. The detector caught every one.

---

The shoulders we stood on

• Coasting a track through a dropout is radar vocabulary, decades older than computer vision.
• Re-anchoring on the observation when someone reappears, instead of trusting a prediction that has been drifting blind through the occlusion, is OC-SORT's Observation-Centric Re-Update (Cao et al., CVPR 2023). It is their entire named contribution, and we reimplemented it.
• Pelvis height above the floor as the sit/stand discriminator is one of the most standard features in the depth-camera fall-detection literature (Kepski & Kwolek and that lineage).
• Refusing to classify when confidence is low, and letting the previously held belief stand, is Chow's reject option — C. K. Chow, 1970.
• A light instead of a spoken sentence to acknowledge a human is Weiser & Brown's calm technology, 1996. There is a certification programme for it now.

We list these because it is what makes the next section mean anything.

---

The papers describe a street. We live in a room.

Here is the gap we actually spent the night inside.

Multi-object-tracking benchmarks are largely built by splitting the same videos into train and test sets, so published trackers are evaluated in-domain. When researchers trained on one benchmark and tested on a different one, several state-of-the-art trackers dropped sharply; the authors named it a "generalization bottleneck."

Benchmark footage is mostly upright pedestrians translating across an outdoor scene. A room is high camera angles, people seated, furniture occluding half of everyone, extreme scale variance, and screens throwing phantom detections. The leaderboard order is not the deployment order.

Nothing is wrong with the papers. They solved a cleaner problem than the one in our building.

Concretely: the standard discriminator we reached for first — knee below hip — is ambiguous in a room full of chairs, because someone perched on a stool or leaning forward drops a seated knee as low as a standing one. It misfired in 79 of 251 frames on a seated person. We only found the right feature by dumping the actual distributions: seated pelvis clustered at 533–551mm above the floor, standing sits near 950mm. Four hundred millimetres of empty space between the two states, and one person's entire seated spread was thirteen millimetres.

Measure the distribution before you pick the threshold, not after. Two guessed constants died teaching us that.

---

Three times, the instrument was the broken part

This is the part worth writing down, because none of these were bugs in the system being built. All three were in the thing measuring it, and two of them produced confident, plausible, wrong reports that we were ready to act on.

The write that never landed — and we measured its effect. We had a theory that a frame loop was serialised and costing half our frame rate. We built the fix, copied it over the old binary, restarted, measured: no change. Reported the theory dead. The copy had failed silently — the target was root-owned, the error was swallowed, and the rest of the script ran anyway. We had measured the old binary. The conclusion happened to be right; the evidence was worthless. Those are different things, and a checksum inside the running container was the only reason we found out.

The query that searched the wrong day. Checking whether a feature had ever fired, we ran a log query with a bare --since "22:17:20". That means today at 22:17. It was past midnight. We searched an empty future window, got zero results, and concluded the feature had never worked — including when both of us stood up and walked out of the room. It had fired fourteen times. Zero is a suspicious number, and a surprisingly clean result deserves suspicion of the query before the system.

The rate limiter that laundered a broken signal. The stand-up light fired once and then politely suppressed the rapid repeats. It looked like a working feature with good hygiene. The log said otherwise: the underlying belief was flipping between seated and standing ten times in forty seconds, and the cooldown was faithfully converting a garbage signal into a plausible-looking output. Every debounce, cooldown and retry can do this. The suppression count is the health metric, not the success count. If you do not log what you threw away, you cannot see the thing that is wrong.

---

The technique we would most recommend

To validate the finished detector, we did not ask anyone to stand up repeatedly in front of the camera. We replayed thirty-two minutes of recorded sensor log through the new logic offline. It predicted fourteen stand events. The live system had logged exactly fourteen.

Offline replay of recorded sensor data is the cheapest honest test of a stateful detector, and nobody has to perform for it.

---

What we would take from this

Not an algorithm. We do not have a new one, and the next round of work here is mostly reading what other people already published and doing that instead of what we improvised.

Two things instead. Ask what would make your instrument lie before you believe what it tells you. And prove the negative case — "it fires when I stand up" is half a test; "it fires zero times in three minutes while we sit still" is the half that finds the bugs.

The room can tell us apart now. Most of the work was not the seeing.