My GPU Broker Kept Killing Inference Jobs for Games That Weren't Running

My GPU broker kept canceling live inference jobs over games that weren’t running. Most of the time, nothing had launched at all.

The service is a Go broker I run at home that arbitrates my desktop’s single GPU between gaming, Plex transcoding, and local inference through Ollama; the same broker later grew a parking layer for embedding requests caught mid-yield. When it detects gaming or Plex activity, it force-cancels whatever inference is running and unloads the model from VRAM, no exceptions.

In my house, whoever’s playing a game or watching something wins that argument. That priority order is correct. The detector deciding when to enforce it was not.

I found the bug while chasing a different crash: the LightRAG embedding crash that took nine fixes to actually stop. A bulk ingestion job that leans on the broker for embeddings kept dying partway through with a read error on the Ollama calls, which cascaded into a full pipeline halt. Nothing in the job’s own code looked wrong.

Checking the broker’s logs during the failure windows turned up the real problem: it kept flipping into a “yielding” state with nothing running.

ps aux during one of those windows showed exactly one candidate: Steam’s idle background client, doing nothing more incriminating than existing in the process table.

A single matching process was enough to cancel a running job

#

One matching line in /proc was enough to kill a running job. The detector polls /proc every three seconds for command-line substrings:

• Plex Transcoder
• Steam’s launch marker
• Heroic’s and Lutris’s runner patterns
• a bare wine .exe

The moment any one poll matched, the controller flipped to yielding and canceled whatever inference was in flight.

There was no debounce (the industry term for waiting out a signal before trusting it) and no second signal to corroborate the first. One sample counted as ground truth. That design wasn’t an oversight so much as an unexamined assumption: I’d built the hard-cancel policy deliberately, then never asked whether the thing triggering it deserved that much trust.

Plex’s own maintenance jobs look identical to real playback

#

Plex’s own support documentation confirms that Skip Intro and Credits detection, along with chapter-thumbnail generation, run as scheduled server maintenance through the same Plex Transcoder binary that handles real playback, on a cadence that has nothing to do with anyone pressing play. My detector grepped for that process name, so a 3am maintenance pass looked exactly like me starting a movie.

No amount of debounce timing fixes this: the false match isn’t a brief blip, it can run for several minutes at a stretch. Tautulli, a widely used third-party Plex monitoring tool, sidesteps the problem by reading Plex’s /status/sessions API instead of the process table, since that endpoint only reports sessions that are actually “now playing.” The real fix for the Plex side: stop grepping for the binary and ask Plex what’s actually playing.

No game launcher exposes a real “foreground game” signal

#

The gaming side is a different problem: I can’t fix it by finding a better API, because none exists. Steam’s overlay APIs report whether the overlay is active, not whether a game is running in the foreground. Heroic and Lutris expose no equivalent signal at all.

Process-name matching is the only practical option left for gaming detection. The logs showed those false matches clustering in three-to-six-second blips, much shorter than Plex’s multi-minute stretches: different noise shape, different fix.

Confirmation only gates the cancel

#

The fix makes the broker demand confirmation before it cancels a job, but not before it recovers from one. Here’s the actual change, before and after:

flowchart LR
    subgraph Before["Before: single-poll trigger"]
        A1[Poll /proc every 3s] --> A2{Any match?}
        A2 -->|1 match| A3[Cancel inference immediately]
    end
    subgraph After["After: debounced trigger"]
        B1[Poll /proc every 3s] --> B2{Match?}
        B2 -->|1st match| B3[Wait for confirmation]
        B3 --> B4{2-3 consecutive matches?}
        B4 -->|Yes| B5[Cancel inference]
        B4 -->|No, false blip| B6[Ignore, keep running]
    end

For the gaming side, the fix is the debounce pattern I should have had from the start: require several consecutive positive polls before flipping to yielding, instead of trusting a single one. I set the default at two or three consecutive matches.

Recovery, the transition back out of yielding, stays instant and undebounced. Delaying it only costs a few extra seconds of inference downtime, and never risks letting inference run over an actual game. That asymmetry is deliberate: the two directions carry different failure costs. A genuine game launch now takes a few seconds longer for the GPU to free up, a small price against jobs dying for no reason.

I’ve only shipped half of this fix. The poll-confirmation gate is small, self-contained, and went in first. The Plex session-API swap hasn’t happened yet. It needs a token Plex issues locally, and I haven’t wired that up. Until I do, a multi-minute Plex maintenance run will still trip the broker no matter how high I set the confirm-poll count: debounce only filters single-sample noise, and does nothing against a signal that stays true for five straight minutes.

I’m also not confident two or three polls is the right number for every workload this machine runs. I picked it from a general flapping-detection convention rather than from measurement on my own logs. I won’t know if it’s wrong until the false positives either stop or don’t.

Hard-canceling instead of throttling is a defensible but costly choice

#

Hard-canceling instead of throttling is the right call for my house, and it’s also why this bug turned into a real pipeline outage instead of a minor annoyance. My broker treats every real contention event as a hard stop:

• cancel the inference request
• unload the model
• hand the GPU over completely

Process Lasso does something closer to priority scheduling instead, deprioritizing background compute rather than killing it outright when a game starts. That approach would have made this whole bug far less painful: a false positive would have meant a slower inference request instead of a canceled one.

I built it as a hard cutover on purpose. I wanted a guarantee that the GPU comes back completely clean the moment someone in this house wants to play, and priority-based throttling can’t promise that as cleanly. I still think that tradeoff was right for a shared family machine.

The debounce fix is live; the Plex fix isn’t. I’ll find out whether either was tuned right the next time this job runs unattended overnight, and either survives or it doesn’t.