[{"content":"This site had no image pipeline until this week. Every image in every post loaded at its original file size and format, usually a multi-megabyte PNG screenshot, with no responsive sizing and no loading placeholder.\nDiagrams had exactly one path in: a Blowfish theme shortcode you had to remember to wrap your diagram in by hand, with no way to drop a diagram into a plain fenced code block the way you would in a GitHub README or almost anywhere else that renders Markdown.\nBoth problems share a mechanism, so I fixed them in the same pass: Hugo render hooks, which let a site override how the built-in Markdown renderer turns one specific element (an image, a code block) into HTML.\nRender hooks, not a CDN # The obvious alternative to fixing this in Hugo would have been an image CDN, a hosted service like Cloudinary or imgix that resizes and reformats images on request. I didn\u0026rsquo;t want a third-party dependency for something Hugo already does natively at build time.\nEvery image on this blog is a file checked into the repo. Hugo\u0026rsquo;s resources.Get and the image processing methods it exposes (.Resize, format conversion, quality settings) run once, during hugo build, and the output is a static file next to everything else this site already serves — no runtime cost, no external service, no new failure mode when that service has an outage.\nA render hook is Hugo\u0026rsquo;s supported way to intercept one piece of that build. Drop a template at layouts/_default/_markup/render-image.html and every Markdown image reference in every post routes through it instead of Hugo\u0026rsquo;s default renderer. Same idea for code blocks: layouts/_default/_markup/render-codeblock-mermaid.html intercepts only the fenced blocks tagged with the language name mermaid, leaving every other code block (Python, Bash, YAML, whatever) untouched.\nWebP conversion, responsive srcset, and a blur-up placeholder # The image hook does three things to every local raster image (a PNG or JPEG that isn\u0026rsquo;t an SVG and isn\u0026rsquo;t loaded from a remote URL):\nConverts it to WebP at two widths, 800px and 1280px, quality 75. WebP is a modern image format that produces meaningfully smaller files than PNG or JPEG at the same visual quality. That\u0026rsquo;s the actual win here, since the original screenshots on this blog were often 1–3MB PNGs. Builds a srcset so the browser picks whichever of the two sizes fits the reader\u0026rsquo;s screen, instead of always downloading the largest version. Generates a low-quality placeholder. LQIP stands for low-quality image placeholder: a tiny, heavily compressed preview (24px wide, WebP quality 40) encoded directly into the HTML as a base64 data URI. It shows as a blurred background while the real image loads, then swaps out once the image finishes (onload, checking the image actually has real pixels rather than firing on a broken image). Neither the resizing nor the srcset widths upscale past the source: both are capped at the image\u0026rsquo;s own width, so a small source image never exceeds its native resolution.\nTwo cases skip all of this on purpose:\nRemote images — anything with an http://, https://, or data: URL passes straight through, since Hugo can\u0026rsquo;t resize a file it doesn\u0026rsquo;t have locally. SVGs pass through unmodified: SVG is already a compact vector format, and converting one to a raster WebP would only make it bigger and blurrier. There\u0026rsquo;s also a site-wide escape hatch, a disableImageOptimizationMD parameter that reverts every image on the site to the original, unconverted file, for the rare case where exact pixel fidelity matters more than page weight.\nHere\u0026rsquo;s the decision flow the hook actually runs, from a Markdown image reference to the final rendered figure:\nflowchart TD A[Markdown image reference] --\u003e B[render-image.html hook fires] B --\u003e C{Remote URL, or local resource not found?} C --\u003e|Yes| D[Plain img tag, no conversion] C --\u003e|No, local resource found| E{SVG, or optimization disabled by site param?} E --\u003e|Yes| D E --\u003e|No| F[Responsive path] F --\u003e G[Resize to 800w and 1280w WebP, quality 75, capped at source width] F --\u003e H[Resize to 24px WebP, quality 40, base64-encode] G --\u003e I[img src + srcset + sizes] H --\u003e J[Inline background-image data URI, cleared once the real image loads] I --\u003e K[Rendered figure: responsive WebP with blur-up placeholder] J --\u003e KHere\u0026rsquo;s a real image going through that exact path, reused from the Docker Compose VPN guide on this blog rather than a synthetic test image, so the pipeline does real work here instead of showing off on a stock photo of a laptop on a beach:\nThe Docker Compose + VPN topology from an earlier post on this blog, now served as WebP with a blur-up placeholder The bug that would have shipped: images under 800px skipped WebP # Blowfish, the theme this site runs on, already had an image render hook, and the one I built started as a fork of it rather than something written from scratch. Its responsive-image logic resized to WebP only inside a conditional gated on the source image\u0026rsquo;s width, and that conditional was written so images narrower than 800px fell through without ever hitting the .Resize call.\nA screenshot that happened to be, say, 600px wide would render as a plain, unconverted PNG. No WebP. No srcset. No LQIP. No error telling anyone anything had gone wrong. I caught this during spec review, before it shipped, by deliberately testing against a narrow image instead of only the wide screenshot used elsewhere in this post. The fix: make the WebP conversion unconditional. Every local raster image gets resized to WebP now, with each target width capped at math.Min(originalWidth, 800) (or 1280 for the larger variant), so a small source image gets downsized cleanly and never upscaled.\nA conditional that silently skips work instead of erroring is invisible until someone tests the exact input it was written to exclude.\nDiagrams from a plain code fence, not just a custom shortcode # Before this, the only way to add a diagram to a post was Blowfish\u0026rsquo;s mermaid shortcode, Hugo\u0026rsquo;s mechanism for calling a custom template from inside Markdown by name, wrapped around the content it applies to. It works. But it\u0026rsquo;s specific to this theme: paste the same Markdown into GitHub, or into any other Hugo site without that exact shortcode installed, and instead of a diagram you get a wall of raw arrows and brackets sitting on the page as plain text.\nMermaid, the diagramming library, not the theme feature, has a real, portable convention for this: a fenced code block tagged with the word mermaid as its language name, the same triple-backtick-plus-language convention you\u0026rsquo;d use for any other code block, just with mermaid in place of python or bash. GitHub, GitLab, and most Markdown renderers already recognize that convention natively.\nHugo\u0026rsquo;s code-block render hook lets this site recognize it too: render-codeblock-mermaid.html intercepts any fenced block tagged that way and wraps its raw content in a \u0026lt;pre class=\u0026quot;mermaid\u0026quot;\u0026gt; element, the exact markup the existing shortcode already produced. Same CSS, same Mermaid JavaScript runtime, picked up identically no matter which syntax wrote it. The diagram earlier in this post, the one showing the image hook\u0026rsquo;s decision flow, comes from that exact fenced block instead of a mockup.\nLoading the Mermaid bundle exactly once, from either entry point # Mermaid\u0026rsquo;s JavaScript runtime is a real cost, tens of kilobytes a reader\u0026rsquo;s browser has to fetch and execute, so it should only load on pages that actually use it, and it should never load twice on the same page. Blowfish\u0026rsquo;s theme already handled the first half of that for the shortcode: a partial checks .Page.HasShortcode \u0026quot;mermaid\u0026quot; and only then fetches, minifies, concatenates, and fingerprints the Mermaid library and its config into one bundle.\nForking that theme file too would mean re-syncing it by hand on every future Blowfish update. So instead I added a second, narrower check in a site-level partial, extend-head-uncached.html. It loads the same bundle only when the page\u0026rsquo;s raw source contains a fenced block tagged mermaid and the shortcode is absent — that \u0026ldquo;and shortcode is absent\u0026rdquo; clause is the double-load guard:\nShortcode only: the theme\u0026rsquo;s own check already loads the bundle, so the new check backs off. Fenced block only: the theme\u0026rsquo;s check is false (no shortcode), so the new check fires instead. Both, like this page: the theme\u0026rsquo;s check fires and loads it, and the new check backs off, for the same reason as the shortcode-only case. One script tag, regardless of which syntax, or both, a given post uses.\nThe old shortcode syntax still renders on the same page # The diagram below is a regression check more than content in its own right: it\u0026rsquo;s written with the original mermaid shortcode syntax, sitting on the same page as the fenced diagram above, to confirm both entry points coexist without loading the Mermaid bundle twice or conflicting with each other.\nflowchart LR A[Shortcode entry point] --\u003e B[Blowfish's original loader: HasShortcode check] B --\u003e C[Same Mermaid runtime bundle] C --\u003e D[Renders next to the fenced-block diagram above] Where this stands # hugo build runs clean across every existing post plus this one. The build output confirms both diagrams render and the image above comes out as WebP with a srcset and a blur-up placeholder rather than a flat PNG.\nI also wrote real browser tests, not just a clean build, to catch a regression here automatically. They confirm:\nThe Mermaid bundle loads exactly once on this page (both syntaxes present) and stays absent on pages with neither. The diagram actually renders as an SVG, rather than sitting as unrendered text. Its colors really change between light and dark mode after clicking the appearance switcher. The LQIP placeholder clears once the real image loads, rather than just being present in the markup. One gap I\u0026rsquo;m not pretending isn\u0026rsquo;t there: there\u0026rsquo;s still no isolated fixture anywhere on this site for \u0026ldquo;fenced block only, no shortcode\u0026rdquo; or \u0026ldquo;shortcode only, no fenced block\u0026rdquo; in separate pages. This post exercises both at once, which proves the double-load guard but not each syntax fully alone. That guard\u0026rsquo;s logic is simple enough to have checked by reading the template directly, so I\u0026rsquo;m treating it as covered. ","date":"10 August 2026","externalUrl":null,"permalink":"/blog/native-hugo-image-pipeline-webp-lqip-and-mermaid/","section":"Blog Posts","summary":"This site had no image pipeline until this week. Every image in every post loaded at its original file size and format, usually a multi-megabyte PNG screenshot, with no responsive sizing and no loading placeholder.\n","title":"A Native Hugo Image Pipeline: WebP, LQIP Blur-Up, and Mermaid Diagrams","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/authors/","section":"Author","summary":"","title":"Author","type":"authors"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/blog/","section":"Blog Posts","summary":"","title":"Blog Posts","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"I\u0026rsquo;m Preston Bernstein, a full-stack developer based in Atlanta with over ten years of experience building and optimizing web applications, and the infrastructure they run on.\nMost of what I write about comes out of my own home lab: a Synology NAS, a desktop with one much-fought-over GPU, a retired laptop running Proxmox VE, and a UniFi network with Pi-hole DNS filtering underneath all of it. On top of that hardware I run self-hosted media automation, photo backup, knowledge-graph pipelines built on LightRAG, local LLM inference through Ollama, and a growing amount of AI agent tooling — Claude Code pipelines that spec, build, review, and deploy software with me supervising rather than typing.\nThe blog is where the debugging sessions and design decisions get written down honestly, including the parts that failed or stayed unresolved. Posts cover home-lab infrastructure and networking, self-hosted service placement and observability, and practical AI/LLM engineering: agent workflows, local inference, retrieval pipelines, and cost control.\nA few places to start:\nWhat a $364 Claude Code session taught me about running agents unattended Three failure modes wearing one name: running concurrent Claude Code agents Tuning LightRAG ingestion concurrency against a rate-limited Gemini API Not every Docker container belongs on the NAS You can read more about me, browse the blog, see my projects on GitHub, or connect on LinkedIn.\n","date":"10 August 2026","externalUrl":null,"permalink":"/","section":"Home","summary":"I’m Preston Bernstein, a full-stack developer based in Atlanta with over ten years of experience building and optimizing web applications, and the infrastructure they run on.\n","title":"Home","type":"page"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/home-lab/","section":"Categories","summary":"","title":"Home Lab","type":"categories"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/hugo/","section":"Tags","summary":"","title":"Hugo","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/image-optimization/","section":"Tags","summary":"","title":"Image Optimization","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/mermaid/","section":"Tags","summary":"","title":"Mermaid","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/performance/","section":"Tags","summary":"","title":"Performance","type":"tags"},{"content":"Preston Bernstein is a versatile full-stack developer with experience in both front-end and back-end technologies. He is based in Atlanta, GA.\n","date":"10 August 2026","externalUrl":null,"permalink":"/authors/preston-bernstein/","section":"Author","summary":"Preston Bernstein is a versatile full-stack developer with experience in both front-end and back-end technologies. He is based in Atlanta, GA.\n","title":"Preston Bernstein","type":"authors"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/software-architecture/","section":"Categories","summary":"","title":"Software Architecture","type":"categories"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/static-site/","section":"Tags","summary":"","title":"Static Site","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/web-development/","section":"Categories","summary":"","title":"Web Development","type":"categories"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/webp/","section":"Tags","summary":"","title":"WebP","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/ai-agents/","section":"Tags","summary":"","title":"AI Agents","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/ai-infrastructure/","section":"Categories","summary":"","title":"AI Infrastructure","type":"categories"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/claude-code/","section":"Tags","summary":"","title":"Claude Code","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/developer-workflow/","section":"Tags","summary":"","title":"Developer Workflow","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/github/","section":"Tags","summary":"","title":"GitHub","type":"tags"},{"content":"GitHub\u0026rsquo;s per-repo Agents tab is a mission-control surface, live since January 26, 2026, where GitHub-hosted coding agents pick up issues and turn them into PRs. No terminal required. Copilot\u0026rsquo;s own agent lives there by default, and Claude and Codex have been selectable alongside it in public preview since February 4, 2026. The tab is part of Agent HQ, the umbrella GitHub announced on October 28, 2025, meant to give every agent vendor one shared surface across Issues, PRs, and Actions.\nMy read after digging into how it actually works: this is a real product, not a rebrand of anything Anthropic ships, and it covers a narrower slice of my workflow than local Claude Code already handles. Whether I keep reaching for it once the novelty wears off is the part I genuinely don\u0026rsquo;t know yet.\nRunning Claude or Codex in the Agents tab draws down Copilot credits # Running Claude or Codex inside GitHub\u0026rsquo;s Agents tab requires a paid Copilot plan, per GitHub\u0026rsquo;s plan matrix:\nPro: $10/month, $15 in AI credits Pro+: $39/month, $70 in AI credits Max: $100/month, $200 in AI credits Every session the tab runs draws down those credits. GitHub moved the whole system to usage-based credit billing on June 1, 2026, so cost tracks the work actually done instead of a flat seat price.\nAnthropic\u0026rsquo;s own bridge into GitHub runs on a completely separate path: the claude-code-action GitHub App, which you install yourself by running /install-github-app from the Claude Code CLI, and which bills straight against an ANTHROPIC_API_KEY stored as a repo secret.\nI\u0026rsquo;ve already learned the hard way that usage-based agent billing punishes unattended workloads, so which bill a session lands on is not a detail I\u0026rsquo;ll shrug off. Same Claude model either way, but two different accounts get charged, and two different places end up holding the session history. It\u0026rsquo;s one Claude wearing two different name tags, depending which bill it\u0026rsquo;s on. Worth deciding that on purpose instead of defaulting into both.\nIt already reads the instructions file I wrote for a different reason # GitHub\u0026rsquo;s Copilot cloud agent reads whatever CLAUDE.md sits at a repo\u0026rsquo;s root, along with AGENTS.md and path-scoped .github/instructions/**/*.instructions.md files. No extra setup on my end. Any repo I maintain that already keeps a CLAUDE.md as its canonical instructions file is handing that same document to GitHub\u0026rsquo;s agent the instant the Agents tab gets turned on for it.\nAn excludeAgent property exists for scoping a file to specific agents, useful once Copilot needs house rules that shouldn\u0026rsquo;t also apply to Claude or Codex running in the same repo. I haven\u0026rsquo;t hit that case yet.\nGitHub caps a single instructions file around 1,000 lines before response quality reportedly drops. That\u0026rsquo;s a ceiling worth knowing before any CLAUDE.md grows past what an agent, local or cloud, can actually use. The permission model is generic where mine is already specific # The cloud agent only touches the repo it\u0026rsquo;s assigned to, and any Actions workflow its PR triggers needs write-access approval before it runs. GitHub built that sandbox to hold for any repo any customer points it at, which makes it necessarily generic.\nMy local Claude Code sessions already run a tighter, more specific version of the same idea: I decide per repo what a session is allowed to touch, and nothing runs unsupervised against something live without a change-control gate I wrote for that exact system. The two guardrails aren\u0026rsquo;t competing. They sit at different points in the pipeline, and for anything touching a running service, I still trust the gate I built over one designed to be safe for every customer\u0026rsquo;s repo at once.\nTask suitability draws the same line I already draw myself # GitHub is explicit about both sides of that line: bug fixes, doc updates, dependency bumps, test coverage, and accessibility fixes belong in the Agents tab, while complex cross-repo refactors, anything security-sensitive, and anything with requirements that aren\u0026rsquo;t already nailed down don\u0026rsquo;t. That boundary lands almost exactly where I already split unsupervised background work from the interactive sessions I sit and drive myself.\nGitHub\u0026rsquo;s own framing puts it plainly: local agents for interactive work that needs immediate feedback, cloud agents for tasks that can run all the way to a finished PR with nobody watching, and a /delegate command meant to hand a task from one mode to the other without losing context. I was already running that model before this tab existed. What\u0026rsquo;s new is a GitHub-native trigger for the cloud half, reachable from the repo UI or a phone instead of only from my own machine.\nBenchmark rankings show what the tab is actually routing to # Third-party benchmarks rank the models the tab routes to, and Copilot\u0026rsquo;s own agent isn\u0026rsquo;t near the top:\nClaude Opus: 88.6% on SWE-bench Verified Codex: 77.3% on Terminal-Bench 2.0 Cursor: ~74% on SWE-bench Copilot\u0026rsquo;s own agent: ~54% Picking Claude or Codex from inside the Agents tab, instead of defaulting to Copilot\u0026rsquo;s built-in agent, means picking the same models I already reach for locally. GitHub sits underneath that choice as a router and a billing layer, not a rival source of intelligence.\nIf Copilot\u0026rsquo;s own agent were the only option in that tab, I\u0026rsquo;d have skipped this whole investigation. But Claude sits there as a first-class pick, and the real question the tab poses is whether I want GitHub\u0026rsquo;s UI and GitHub\u0026rsquo;s bill wrapped around Claude, or my own.\nWhether it earns a permanent spot comes down to one habit I haven\u0026rsquo;t built yet # The place I can actually see this earning a spot is triage: assigning a low-risk issue to the Agents tab from my phone the second I file it, instead of sitting on it until I\u0026rsquo;m back at a keyboard to spin up a local session. That\u0026rsquo;s a real gap in how I work today: small fixes wait for keyboard time regardless of how trivial they are.\nMaybe I build that habit once the first week of novelty wears off. But I might just keep defaulting to my own Claude Code session, because I already trust its logs, its worktree lifecycle, and my own change-control gate more than a run I can only inspect through GitHub\u0026rsquo;s diff view. I\u0026rsquo;m giving it a real trial on one low-stakes repo before I decide either way. I\u0026rsquo;d rather report back after a month of actual use than guess now.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/github-agents-tab-vs-claude-code/","section":"Blog Posts","summary":"GitHub’s per-repo Agents tab is a mission-control surface, live since January 26, 2026, where GitHub-hosted coding agents pick up issues and turn them into PRs. No terminal required. Copilot’s own agent lives there by default, and Claude and Codex have been selectable alongside it in public preview since February 4, 2026. The tab is part of Agent HQ, the umbrella GitHub announced on October 28, 2025, meant to give every agent vendor one shared surface across Issues, PRs, and Actions.\n","title":"GitHub's Agents Tab Puts Claude and Codex in the Repo UI. It's a Separate Bill From Claude Code.","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/home-lab/","section":"Tags","summary":"","title":"Home Lab","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/networking/","section":"Categories","summary":"","title":"Networking","type":"categories"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/networking/","section":"Tags","summary":"","title":"Networking","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/pi-hole/","section":"Tags","summary":"","title":"Pi-Hole","type":"tags"},{"content":"I rebuilt my home network from the ISP modem outward instead of dropping in a new router and hoping the rest of the stack sorted itself out. The order was fixed:\nModem into passthrough (a mode where the ISP box stops doing routing and just hands its public IP straight through). UniFi gateway and switch as the core. Pi-hole DNS filtering on a Raspberry Pi controller. Every downstream device reconnected one at a time. Bottom-up, slowest layer first. Nothing skipped ahead of what it depended on.\nStarting at the modem forces every later phase to be honest # Most rebuild guides start at the router, because the router is the interesting box. I started at the AT\u0026amp;T modem instead, because it\u0026rsquo;s the one thing everything else would eventually depend on. Get it wrong there and you\u0026rsquo;re redoing every phase that comes after it.\nA gateway sitting behind a modem that\u0026rsquo;s still doing its own routing and NAT gets a private IP instead of the real one, and half its features either misbehave or silently don\u0026rsquo;t work. Fix that after the fact and you\u0026rsquo;re re-wiring a spine you already built. Fix it first, and every phase after stands on a foundation that\u0026rsquo;s actually solid.\nThe Pi controller has to prove itself before touching hardware # Before I unplugged a single cable, I checked that the Raspberry Pi meant to run both the UniFi controller software and Pi-hole was actually in working order. That\u0026rsquo;s a controller and a DNS filter sharing one small board, so if the board is flaky, both systems inherit the problem.\nI SSH into the Pi directly, skipping any intermediate device, and check three things:\nThe UniFi controller process is running. Pi-hole\u0026rsquo;s FTL service is active. Pi-hole\u0026rsquo;s local API responds. If any of those fail, I fix them before phase one starts. A rebuild with an unreliable controller doesn\u0026rsquo;t announce itself — it just produces mystery failures later that look like network problems and aren\u0026rsquo;t.\nPhysical inspection beats trusting old notes # The next step was confirming what the UniFi switch actually was: model, MAC address, firmware version. I had this written down from an earlier setup, but hardware gets swapped and notes go stale. I checked the label on the unit itself instead of trusting a document from months ago — skipping that step is how you end up troubleshooting a switch that isn\u0026rsquo;t the switch you think it is.\nReset the Gateway Before the Controller Ever Adopts It # I factory reset the UniFi gateway before letting the controller adopt it, instead of adopting whatever configuration state it happened to be in. Holding the reset button through a full LED flash cycle wipes prior config and puts the device back to a known default. That matters, because adopting a gateway with leftover settings from a previous topology is how you get rules that contradict what you\u0026rsquo;re about to build.\nOnce it settles, the gateway is reachable at its default local address over a direct wired connection, and that\u0026rsquo;s the state I want walking into adoption.\nAdoption is where the controller and the gateway agree to work together # Adoption is UniFi\u0026rsquo;s term for a device formally joining a controller: the controller pushes its configuration down, the device reboots into it, and from then on the controller manages it. The steps:\nConnect a laptop directly to the gateway\u0026rsquo;s LAN port. Open the controller\u0026rsquo;s web dashboard from the Pi. Adopt the gateway once it shows up as pending. Most of the time this works from the UI in a few minutes. But when it doesn\u0026rsquo;t, there\u0026rsquo;s a command-line fallback that points the device at the controller\u0026rsquo;s inform address directly, run over a direct SSH session into the gateway itself, and then the UI adoption is retried. I didn\u0026rsquo;t need the fallback this time, but I wrote it into the plan anyway, because the one time you skip documenting the fallback is the one time you need it at 11pm.\nWiring the spine follows a strict power-on order # Physical wiring came only after every device was individually verified:\nThe modem\u0026rsquo;s LAN port feeds the gateway\u0026rsquo;s WAN port. The gateway\u0026rsquo;s LAN port feeds the UniFi switch, which acts as the spine, the central point everything downstream connects through. The switch feeds the Pi controller on one port and the rest of the existing switch gear on another. Power-on order matters too. Skipping it doesn\u0026rsquo;t necessarily break anything, but it\u0026rsquo;s one more variable I didn\u0026rsquo;t need while troubleshooting a fresh spine.\nHere\u0026rsquo;s the spine those wiring steps actually build, in the order signal flows through it:\nflowchart LR A[\"ISP modem(passthrough mode)\"] --\u003e B[UniFi gateway] B --\u003e C[UniFi switch — the spine] C --\u003e D[\"Pi controller(UniFi + Pi-hole)\"] C --\u003e E[Rest of existing switch gear]Power-on order runs switch first, then gateway, then the Pi last, so the gateway always has something to talk to the moment it boots.\nPassthrough Is a Modem-Side Setting # Passthrough gets configured on the ISP modem, not on the UniFi side, which is a detail that trips people up. It lives in the modem\u0026rsquo;s own admin firewall settings, tied to the gateway\u0026rsquo;s MAC address so the modem knows which downstream device gets the real public IP.\nAfter enabling it and letting the modem reboot, I check two things:\nThe gateway\u0026rsquo;s WAN interface picked up a real public IP, instead of a private one handed out by the modem\u0026rsquo;s own NAT. The controller\u0026rsquo;s dashboard shows that same address. If those two don\u0026rsquo;t match, passthrough isn\u0026rsquo;t actually active yet, no matter what the modem\u0026rsquo;s settings page claims. Pi-hole runs on the same board as the controller, which is a real tradeoff # Pi-hole filters DNS requests before they leave the network, blocking ads and unwanted domains at the resolver instead of per-device. Running it on the same Raspberry Pi as the UniFi controller keeps the hardware footprint small, and for a home network that\u0026rsquo;s a fine tradeoff.\nIt also means a single board failure takes out both the DNS filter and the controller UI at once — a shortcut worth being honest about instead of glossing over. Where downstream devices land was a decision I hadn\u0026rsquo;t made yet # Here\u0026rsquo;s the part of the plan I can\u0026rsquo;t write up as finished, because it wasn\u0026rsquo;t. Before the rebuild, the NAS, desktop, laptop, and a couple of media devices connected straight into modem ports, flat, no managed switch in the path. Once the modem is just a passthrough bridge and the UniFi gateway is the real router, those devices need a new home: stay on the old flat ports and lose DHCP consistency with everything else, or get rewired into the managed spine and gain it.\nI listed four options in my planning notes and didn\u0026rsquo;t pick one. It touches a NAS with a bonded network connection I didn\u0026rsquo;t want to reroute on a guess (the same NAS whose workload placement got its own post), and a couple of devices whose physical cable runs I hadn\u0026rsquo;t confirmed. That\u0026rsquo;s an honest gap — I\u0026rsquo;d rather admit the plan stalled on a real unknown than pretend I closed it out.\nThe plan mattered more than the finish line # What I actually got out of this wasn\u0026rsquo;t a finished network. It was a sequence I trust: verify the controller, confirm hardware, reset before adopting, wire in a fixed order, flip passthrough, filter DNS, and only then touch the devices that depend on all of it. Each phase has a clear pass or fail condition, which means when something breaks later, I know roughly which layer to check first instead of guessing across the whole stack. The device-landing question is still sitting there unresolved, and I\u0026rsquo;d rather leave it open in writing than pretend the rebuild wrapped up neatly. It didn\u0026rsquo;t, not yet.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/rebuilding-home-network-from-the-modem-up/","section":"Blog Posts","summary":"I rebuilt my home network from the ISP modem outward instead of dropping in a new router and hoping the rest of the stack sorted itself out. The order was fixed:\n","title":"Rebuilding a Home Network from the Modem Up, One Phase at a Time","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/unifi/","section":"Tags","summary":"","title":"UniFi","type":"tags"},{"content":"A clean ClamAV scan means nothing matched a known signature. It does not mean the file is safe. I run a scan gate in front of my media-download pipeline: everything that lands from the download clients gets checked by a ClamAV daemon before it\u0026rsquo;s allowed into the library. (The pipeline sits on the placement split from Not every Docker container belongs on the NAS.)\nFor a long time I treated a clean verdict as the end of the question. It isn\u0026rsquo;t. ClamAV is a signature engine: it only catches what someone has already seen, fingerprinted, and shipped a rule for. Zero-days and packed or obfuscated executables walk right past it.\nWorse: ClamAV is open source, so anyone can download the exact detection logic and test their malware against it before release. Free QA for the bad guys.\nThat\u0026rsquo;s not a hypothetical: researchers have measured samples built specifically to dodge open-source detectors evading ClamAV 70 to 85 percent of the time, without even needing inside knowledge of the engine. Here\u0026rsquo;s the full layered gate, in the order a file actually passes through it:\nflowchart TD A[File lands from download client] --\u003e B[clamd signature scan + extension blocklist] B --\u003e C[PUA detection: DetectPUA flag] C --\u003e D[\"Third-party signature feeds:Sanesecurity, SecuriteInfo, URLhaus, MalwarePatrol\"] D --\u003e E[YARA-Forge Core rules, native in clamd] E --\u003e F{Borderline verdict?} F --\u003e|Yes| G[\"SHA-256 hash lookup:VirusTotal / MetaDefender, hash only\"] F --\u003e|No| H[Entropy / packer check: Detect It Easy] G --\u003e H H --\u003e I[Verdict: clean / flagged / infected / blocked] Signature scanning only catches what\u0026rsquo;s already been seen # Every ClamAV signature exists because someone already found and analyzed that malware sample. A brand-new keygen or crack, repacked or lightly modified, has no signature yet, and it sails through clean.\nPacked and obfuscated binaries make this worse: the payload is scrambled until runtime, so a static signature scanner has nothing to match against, even for a known threat.\nMy original scan gate had one static layer: clamd plus a blocklist on file extensions like .exe, .scr, .bat, and a handful of others. That layer stops the laziest attacks and nothing else.\nThe real threat model for a media pipeline isn\u0026rsquo;t a nation-state implant. It\u0026rsquo;s commodity crack and keygen malware bundled into an executable a downloader was told to run. That\u0026rsquo;s exactly the category built to slip past this kind of scanner.\nPUA detection targets the actual threat, with a real tradeoff # ClamAV has a flag, DetectPUA, that flags potentially unwanted applications: adware, riskware, and, most relevant here, keygens and cracks. Turning it on is a one-line config change to clamd.conf. No code touched.\nBut it\u0026rsquo;s not a free lunch. PUA signatures are less rigorously curated than core malware signatures, so expect more false positives on legitimate but aggressively-bundled installers.\nClamAV\u0026rsquo;s own category-exclusion filtering for PUA is currently broken in the shipped version I\u0026rsquo;m running, so I can\u0026rsquo;t cleanly say \u0026ldquo;flag keygens but ignore adware\u0026rdquo; and trust the exclusion list to hold. I\u0026rsquo;m turning it on anyway, tuning against real false positives as they show up. The alternative is leaving the single most on-target detection knob switched off.\nThird-party signature feeds close known gaps for free # ClamAV\u0026rsquo;s own database misses a lot that other groups have already catalogued. clamav-unofficial-sigs is a maintained aggregator that pulls in four feeds and drops them straight into the same database directory clamd already reads:\nSanesecurity SecuriteInfo URLhaus MalwarePatrol No changes to my scan gate\u0026rsquo;s code, no new dependency in the pipeline logic — just a cron job and a shared volume.\nOf everything I added, this is the best ratio of detection gained to effort spent: pure config that widens the signature set clamd already checks against on every scan.\nYARA rules run inside clamd, but only the trimmed kind # Clamd loads .yar files natively from the same database directory (no separate engine required) and applies YARA rules against files it has already unpacked from archives and installers. That\u0026rsquo;s a real advantage over running YARA standalone, since clamd\u0026rsquo;s decomposition sees inside the containers a raw file scan would miss.\nBut clamd\u0026rsquo;s YARA support is only a subset of full YARA:\nNo imports No external variables A 64-string cap per rule Minimum two-byte string segments Community rule packs written for full YARA often won\u0026rsquo;t load as-is. I\u0026rsquo;m using YARA-Forge\u0026rsquo;s curated \u0026ldquo;Core\u0026rdquo; tier instead of pulling raw rules from wherever, because unvetted community rules have a documented history of tanking scan performance.\nOne bad community rule reportedly took a three-hour scan job to seven. Curation here isn\u0026rsquo;t optional polish: it\u0026rsquo;s the difference between a scan gate that finishes and one that doesn\u0026rsquo;t.\nA hash lookup adds a second opinion without uploading anything # Signature and YARA scans both run locally against files I already have. A hash lookup asks a different question: has anyone else already seen this exact file and scored it? I compute a SHA-256 of anything the local scan flags as borderline and check it against VirusTotal\u0026rsquo;s or MetaDefender\u0026rsquo;s free tier — hash only, never the file itself.\nThat distinction matters for a pipeline that occasionally handles cracked software. Uploading the actual file to a public multi-scanner makes it permanently visible and searchable by anyone. That\u0026rsquo;s exactly the exposure I don\u0026rsquo;t want for downloads that were never meant to be public.\nThis isn\u0026rsquo;t shipped in my scan gate\u0026rsquo;s code yet: it needs a new verdict state that plugs into the same aggregation logic the gate already uses, so a \u0026ldquo;flagged, pending second opinion\u0026rdquo; result sits in the same priority chain as infected, blocked, and clean.\nEntropy and packer detection catch what hashes can\u0026rsquo;t # A hash lookup only works if someone else has already seen the file. A packer or entropy check doesn\u0026rsquo;t need that. Detect It Easy, and its CLI diec, identifies packers and protectors on executables and reports Shannon entropy. A section reading above roughly 7 bits of entropy is the standard first signal that it\u0026rsquo;s packed or encrypted rather than plain code.\nThat reading is a heuristic on its own. I plan to route it to quarantine-and-alert rather than a silent auto-block, because plenty of legitimate installers are also highly compressed, and I don\u0026rsquo;t want to nuke a real release over a false positive I can\u0026rsquo;t explain later.\nWhat I\u0026rsquo;m deliberately not building # A self-hosted dynamic-analysis sandbox (actually detonating suspicious files in an isolated VM to watch what they do) is technically doable in a home lab. CAPEv2 runs fine on a single box with nested virtualization. But I\u0026rsquo;m not building it.\nIt\u0026rsquo;s a heavyweight answer to a threat model that\u0026rsquo;s mostly commodity keygen and crack malware, not a targeted attacker who needs behavioral analysis to unmask. If one of the layers above misses something in an actual incident, that\u0026rsquo;s the trigger to revisit sandboxing.\nThe honest residual gap # None of this closes the gap completely, and I don\u0026rsquo;t think any config change could. A sufficiently novel packer that mimics legitimate compression entropy, paired with a payload built against ClamAV\u0026rsquo;s public signature set and PUA rules specifically, can still get through every layer I\u0026rsquo;ve described.\nThe hash lookup only helps once a file is already known to someone. A first-seen sample gets a pass there by definition.\nWhat changed isn\u0026rsquo;t that my scan gate is now airtight. It\u0026rsquo;s that I stopped treating a clean verdict as proof of safety, and started treating it as one data point among several, none of which is trustworthy alone. That\u0026rsquo;s a more honest place to operate from, even if it\u0026rsquo;s a less comfortable one.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/clamav-clean-scan-doesnt-mean-safe/","section":"Blog Posts","summary":"A clean ClamAV scan means nothing matched a known signature. It does not mean the file is safe. I run a scan gate in front of my media-download pipeline: everything that lands from the download clients gets checked by a ClamAV daemon before it’s allowed into the library. (The pipeline sits on the placement split from Not every Docker container belongs on the NAS.)\n","title":"A Clean ClamAV Scan Doesn't Mean the File Is Safe","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/clamav/","section":"Tags","summary":"","title":"ClamAV","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/devops/","section":"Categories","summary":"","title":"DevOps","type":"categories"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/security/","section":"Categories","summary":"","title":"Security","type":"categories"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/self-hosting/","section":"Tags","summary":"","title":"Self-Hosting","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/linux/","section":"Tags","summary":"","title":"Linux","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/proxmox/","section":"Tags","summary":"","title":"Proxmox","type":"tags"},{"content":"Proxmox VE, one LXC per workload. I think I made the right choice, and I feel good about the reuse: I wouldn\u0026rsquo;t buy a laptop to do this, but I\u0026rsquo;m happy I found an economical use for one that was already here.\nThe XPS 17 had been sitting there for a year or two. I tried giving it to a friend and he didn\u0026rsquo;t want it. I tried selling it online and it was a whole pain, I kept getting lowballed. Meanwhile my desktop was overloaded and having ethernet in/out issues with all the data transfers going through it, and it was getting annoying. There\u0026rsquo;s a server rack under my desk with some space in it. So I figured, why not.\nWhat moved: the arr stack (Sonarr, Radarr, Prowlarr, qBittorrent, NZBGet), a financial data pipeline, a research-automation pipeline, a LightRAG knowledge graph, and Prometheus/Grafana. That\u0026rsquo;s some of the compute off my main desktop and onto a laptop nobody wanted, which is the whole point.\nWhy Proxmox and not Ubuntu plus Docker # Every other machine I run is Ubuntu Server plus Docker Compose. For one app I\u0026rsquo;d do that here too. Five unrelated stacks on one kernel is a different situation: one docker compose down -v in the wrong directory takes out a volume another project mounted, and one bad apt upgrade hits all five. Proxmox gives each stack its own LXC with its own Debian userland and its own Docker daemon, and each LXC gets a filesystem snapshot I can roll back in under a minute without the other four noticing. As of Proxmox VE 9.2 it\u0026rsquo;s Debian 13 underneath, so it\u0026rsquo;s the base I already trust with a newer kernel.\nFedora Server was the other candidate. Podman by default, when I have five stacks of working docker-compose.yml, and each release is supported for about 13 months on a machine I want to rack and forget. So it was out.\nThe cost is one more SSH hop, into the Proxmox host and then into the LXC, every time I touch a container. I knew that going in.\nflowchart TD subgraph Flat[\"Flat Docker host (rejected)\"] H1[One Ubuntu host] --\u003e D1[\"5 Docker Compose stacks,shared kernel, shared blast radius\"] end subgraph Proxmox[\"Proxmox VE (chosen)\"] H2[Proxmox bare metal] --\u003e L1[LXC: media automation] H2 --\u003e L2[LXC: financial data pipeline] H2 --\u003e L3[LXC: research automation] H2 --\u003e L4[LXC: LightRAG knowledge graph] H2 --\u003e L5[LXC: Prometheus/Grafana] end The laptop parts # Nothing really bothered me. The XPS 17 has no Ethernet port, and Proxmox can\u0026rsquo;t bridge containers over Wi-Fi: the wireless card only associates for itself, so a bridged container\u0026rsquo;s frames get dropped at the access point. I had a spare USB-C-to-Ethernet dongle. Plugged it in, Proxmox reassigned nic0 to it, vmbr0 was already bridging nic0, done.\nThe lid: set the three HandleLidSwitch* directives to ignore in /etc/systemd/logind.conf and put consoleblank=300 on the kernel line. Closed the lid with an SSH session open and it stayed up.\nTwo small ones. Bare Proxmox doesn\u0026rsquo;t ship sudo, so apt install sudo as root before you set up a service user. And Proxmox 9 uses deb822 .sources files, so the enterprise-repo 401s go away by disabling pve-enterprise.sources and ceph.sources and adding pve-no-subscription.sources.\nThe one thing that is finicky is the power plug. I have to jiggle it just right and have it sit just right for it to be recognized.\nNothing is ever perfect # The arr stack and the resale-clothing monitor are running on the box now. The NBA data pipeline is running on both the desktop and this box, and I haven\u0026rsquo;t picked one yet.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/proxmox-for-the-xps-17-offload-box/","section":"Blog Posts","summary":"Proxmox VE, one LXC per workload. I think I made the right choice, and I feel good about the reuse: I wouldn’t buy a laptop to do this, but I’m happy I found an economical use for one that was already here.\n","title":"Why the XPS 17 Offload Box Runs Proxmox, Not Plain Ubuntu","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/grafana/","section":"Tags","summary":"","title":"Grafana","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/observability/","section":"Tags","summary":"","title":"Observability","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/prometheus/","section":"Tags","summary":"","title":"Prometheus","type":"tags"},{"content":"Run one shared Grafana and Prometheus stack for your whole home lab, not one per repo. I\u0026rsquo;ve got around 30 GitHub repos and 15-20 always-on self-hosted services, running mostly on one desktop plus a NAS (split by the placement framework from an earlier post).\nRecently I found two separate Grafana containers sitting on that same desktop, each spun up by a different project\u0026rsquo;s docker-compose file, each with its own dashboards, and each blissfully unaware the other one existed. That\u0026rsquo;s the anti-pattern this post argues against. It happened because \u0026ldquo;just add a Grafana container to the compose file\u0026rdquo; felt like the path of least resistance at the time.\nThe isolation argument doesn\u0026rsquo;t apply to a personal setup # Per-repo or per-tenant observability stacks solve exactly one problem: hard isolation between parties who must never see each other\u0026rsquo;s data. Grafana Labs\u0026rsquo; own guidance treats a single shared stack as the default, reserving multi-tenant splits for cases like separate customers or separate teams inside a company, where combining dashboards would be a compliance or trust violation.\nNone of that applies here. Every service on the network is mine. There\u0026rsquo;s no tenant boundary to protect, and no isolation benefit worth buying with extra containers.\nDoes splitting the stack actually save memory? # A full Prometheus, Grafana, and Loki stack runs comfortably in 500MB to 2GB of RAM on a single host, even in a single-binary \u0026ldquo;everything in one process\u0026rdquo; configuration, and that number barely moves whether it\u0026rsquo;s watching 3 services or 20. Fragmenting into two or three separate stacks doesn\u0026rsquo;t save meaningful memory. Most of that footprint is fixed cost: the databases, the web UI, the query engine. None of it scales down with fewer targets. Multiply that fixed cost across five projects instead of paying it once, and it\u0026rsquo;s pure waste.\nOn my desktop, the two duplicate Grafana instances were doing exactly that: quietly holding memory a single shared one would never have needed twice.\nHub-and-spoke is the actual pattern people run at this scale # Homelab operators running desktop-plus-NAS setups converge on the same shape: one central Prometheus/Grafana/Loki stack, plus a lightweight collection agent on every monitored host. The current standard agent is Grafana Alloy, an OpenTelemetry-based collector that replaced the older Grafana Agent (now deprecated and past end-of-life).\nAlloy ships metrics, logs, and traces from each host back to the one shared backend, using a single config file per host: one small agent per machine instead of one full stack per project. That\u0026rsquo;s the part I got backwards, letting each project\u0026rsquo;s compose file drag its own Grafana along for the ride.\nKeeping the central stack on a separate machine from the workloads it watches matters too.\nIf your monitoring stack lives on the same box as the service it\u0026rsquo;s alerting on, a crash on that box takes out your visibility into the crash at the exact moment you need it most. Splitting stack and workload physically, not just logically, is what turns \u0026ldquo;monitoring\u0026rdquo; into something you can actually trust mid-incident.\nHere\u0026rsquo;s the shape of the migration, anti-pattern on the left, target on the right:\nflowchart TD subgraph Before[\"Before: one stack per repo\"] A1[Repo A] --\u003e G1[Grafana + Prometheus A] A2[Repo B] --\u003e G2[Grafana + Prometheus B] A3[Repo C] --\u003e G3[Grafana + Prometheus C] end subgraph After[\"After: hub-and-spoke\"] H[One central Prometheus/Grafana/Loki stack] S1[Alloy agent, host 1] --\u003e H S2[Alloy agent, host 2] --\u003e H S3[Alloy agent, host 3] --\u003e H end This is the same shared-infrastructure pattern I already use # I already draw a line between shared infrastructure and project-specific code. Networking and VPN routing live in one dedicated infra repo, and shared libraries get imported by whichever project needs them instead of copy-pasted into each one.\nObservability belongs in the same category: plumbing every project needs, something no single project owns. Treating it as project-specific, letting each repo bootstrap its own copy, is the same mistake as vendoring a shared library into five places and letting the copies drift.\nThe real downside: cross-project noise and a bigger blast radius # The honest cost of consolidating: one shared stack means one shared failure domain and one shared signal-to-noise problem.\nA misbehaving data-ingestion service can spam the same Grafana instance that\u0026rsquo;s supposed to be giving a calm read on a media pipeline\u0026rsquo;s health. Without rigorous tagging and labeling, alerts from unrelated projects blur together. A stack outage now takes down visibility into everything at once, instead of just one project. Dashboard sprawl is the risk I\u0026rsquo;ll actually admit to: once ten or fifteen projects report into the same Grafana instance, the dashboard list turns into its own mess without folders and consistent naming. The fix is discipline, not pretending the problem doesn\u0026rsquo;t exist because you gave up and split the stacks anyway:\nConsistent labels Per-project dashboard folders Alert routing that filters by service What I\u0026rsquo;m actually doing about it # I\u0026rsquo;m standing up a single Prometheus, Grafana, and Loki stack in my shared infrastructure repo, with Alloy as the collector on every host instead of the deprecated Agent.\nEach service exposes a metrics endpoint where it has one. Node- and container-level metrics get scraped centrally instead of per-project. The two duplicate Grafana instances get their dashboards migrated over, then decommissioned one at a time, carefully, since one of those projects touches live financial data and I\u0026rsquo;d rather not break its alerting mid-migration. Services with zero monitoring today get wired into the shared stack as I go, instead of getting their own bespoke setup. None of this needed new hardware or a new product. Just admitting that \u0026ldquo;quick, add Grafana to this compose file\u0026rdquo; was a decision I kept making locally, one compose file at a time, that never added up to a coherent system. Observability isn\u0026rsquo;t part of each project. It\u0026rsquo;s part of the network.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/one-observability-stack-not-one-per-repo/","section":"Blog Posts","summary":"Run one shared Grafana and Prometheus stack for your whole home lab, not one per repo. I’ve got around 30 GitHub repos and 15-20 always-on self-hosted services, running mostly on one desktop plus a NAS (split by the placement framework from an earlier post).\n","title":"Run One Observability Stack, Not One Per Repo","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/ai-infrastructure/","section":"Tags","summary":"","title":"AI Infrastructure","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/gpu/","section":"Tags","summary":"","title":"GPU","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/hardware/","section":"Categories","summary":"","title":"Hardware","type":"categories"},{"content":"Form factor is the call that matters most on this build, and the popular answer gets it wrong. Every \u0026ldquo;quiet home-lab PC\u0026rdquo; guide points at mini-ITX: small, tucked in a corner, low power draw.\nI already own an RTX 3060 and want a box around it that stays quiet, stays cool, and stays upgradable, meaning I can swap the CPU, RAM, storage, and eventually the GPU without replacing the motherboard underneath them. Mini-ITX fails on all three at once. Seeing why took real bench data and practitioner threads, not case marketing copy.\nMini-ITX trades away the two things this build needs # Mini-ITX cases force two acoustic penalties that stay hidden until you look at the actual hardware inside them:\nFan size. A small case only fits small, high-RPM fans, and small fans have to spin faster than large fans to move the same volume of air. Faster fans are louder fans. PSU size. ITX all but requires an SFX power supply instead of a full ATX unit, and SFX units run louder at idle because their tiny fans work harder inside a smaller housing. Practitioner testing backs this up: builders chasing a genuinely silent PC report mATX and ATX cases as consistently quieter than ITX equivalents at equivalent airflow.\nMini-ITX also caps upgrade room in ways a spec sheet doesn\u0026rsquo;t show until you\u0026rsquo;re staring at four empty screw holes wondering where the second M.2 slot went. Most ITX boards ship two RAM slots, one M.2. That\u0026rsquo;s fine on day one. It\u0026rsquo;s a wall on day four hundred, when I want:\nA second GPU for a small inference cluster More NVMe for a growing model cache More RAM, without pulling both sticks to replace them A build I\u0026rsquo;m calling upgradable at every part can\u0026rsquo;t start on a board that\u0026rsquo;s already out of holes.\nmATX gets the noise win ITX promises but can\u0026rsquo;t deliver # mATX solves the acoustic problem ITX claims to own, without the expansion penalty. A mATX case is roomy enough for a full ATX power supply and full-size 120mm or 140mm fans. Larger fans move the same air at lower RPM. That\u0026rsquo;s the actual mechanism behind a quiet build; the case size badge on the box has nothing to do with it.\nmATX boards typically carry four RAM slots and two or three M.2 slots, plus a full-length PCIe slot for the GPU and often room for a second card down the road. I stop fighting the case for room to grow.\nThe tradeoff I\u0026rsquo;m accepting here is real. A mATX build sits noticeably larger on a desk or shelf than a genuinely compact ITX box. The Fractal Design Ridge measures around 32dB idle by itself, real engineering in a real quiet ITX case, and mATX doesn\u0026rsquo;t beat that on size.\nBut it wins on the constraint I actually have: upgrade room and noise together. If quiet in the smallest possible box is the only requirement, ITX with a case like the Ridge is still the right call. That isn\u0026rsquo;t my constraint set.\nSocket choice decides how long the board lasts # AM5 is the safer bet for a board I don\u0026rsquo;t want to replace in two years. AMD extended AM5 platform support through 2029, up from an earlier 2027 commitment, with Zen 6 and likely Zen 7 landing on the same socket.\nIntel\u0026rsquo;s next socket, LGA1954, has only a VP\u0026rsquo;s public statement pointing toward similar multi-generation support, not a locked commitment the way AMD\u0026rsquo;s is. A CPU swap two or three years out should mean unscrewing four cooler mounts. It shouldn\u0026rsquo;t mean a new motherboard, new RAM, and an OS reinstall.\nChipset tier drives idle power more than the CPU spec sheet # Chipset tier changes idle power draw on AM5 boards more than most builders expect. Measured bench data on a single-chip B650E board showed roughly 71W idle, tying the dual-chip X670E flagship board tested alongside it. The second chip on X670 and X670E boards buys nothing here. It just adds another die pulling power around the clock.\nI\u0026rsquo;m buying a single-chip B650 or B650E board and skipping X670E outright. This machine runs continuously as an inference host, and idle draw compounds over a year in a way a gaming rig\u0026rsquo;s idle time never does. That\u0026rsquo;s the same idle-power math that decided my last box purchase.\nThe CPU\u0026rsquo;s job is sitting at 20W, not winning benchmarks # The GPU carries the AI workload here, so the CPU\u0026rsquo;s real job is staying quiet at idle. A Ryzen 5 7600, non-X and without 3D V-Cache, measured around 20W idle in independent testing, a figure that held across two separate sources.\nPicking the 3D-cache or X variant would buy gaming frame rates this box has no use for. The actual work happens on the GPU sitting next to it.\nThe power supply only needs to cover the real load # An oversized power supply runs less efficiently on this build than a right-sized one. The RTX 3060 carries a 170W power spec set by Nvidia, and a Ryzen 5 7600 idles around 20W and stays well under 100W under load.\nMeasured efficiency curves tell the story:\n600-650W ATX units peak near 91% efficiency at 50% load, and dip at both the 10% and 100% ends. A Corsair RM650e held 90.9% efficiency at 50% load, with average noise measured at only 12.6 dBA. An 850W-plus unit bought for \u0026ldquo;headroom\u0026rdquo; would run this system under 20% load most of the time, off its efficiency peak, for no real benefit. 550 to 650W, full-size ATX, is the right target.\nThe board and case pick still isn\u0026rsquo;t verified # One piece of this build isn\u0026rsquo;t locked yet. I haven\u0026rsquo;t picked a specific mATX board or case, and I don\u0026rsquo;t want to dress up a guess as a confirmed pick the way the rest of this list is confirmed. Candidates worth pricing out, none backed by the same measured bench data as the CPU, chipset tier, and PSU sizing:\nBoard: ASRock B650M Pro RS or MSI B650M Mortar Case: Fractal Design Pop Air or Meshify 2 Compact Reddit\u0026rsquo;s homelab and SFF communities would probably settle this faster than another round of vendor listicles, but that search hit a wall this round.\nThe build that comes out of all this:\nPlatform: AM5, single-chip B650 or B650E board Case: mATX CPU: non-X Ryzen 5 7600 GPU: the RTX 3060 I already own PSU: 550-650W full-size ATX, sized to the real load instead of an imagined one None of the individual parts are exotic or expensive. The only decision that took real digging was form factor. The small-box answer everyone defaults to worked against what I actually needed: room to add parts later, without losing the quiet, and without running out of holes to screw them into.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/mini-itx-is-the-wrong-form-factor-for-a-quiet-ai-homelab-pc/","section":"Blog Posts","summary":"Form factor is the call that matters most on this build, and the popular answer gets it wrong. Every “quiet home-lab PC” guide points at mini-ITX: small, tucked in a corner, low power draw.\n","title":"Mini-ITX Is the Wrong Form Factor for a Quiet AI Home-Lab PC","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/pc-build/","section":"Tags","summary":"","title":"PC Build","type":"tags"},{"content":"Idle power draw, not the price tag stamped on the mini PC, decides whether a dedicated low-power compute box actually saves you money over a gaming desktop. A gaming desktop idles around 80-200W, depending on the board, the PSU, and how many drives happen to be spinning. A purpose-built low-power box, the N100-class mini PCs and similar, idles at 10-15W.\nWhether that gap means anything on your electricity bill comes down to one question: does buying the mini PC actually let the desktop power off or sleep when you\u0026rsquo;re not gaming? If the answer is no, the math falls apart. I found that out the hard way, pricing hardware for my own setup.\nThe wattage gap turns into real money over a year # Run the actual numbers and the wattage gap turns into real money fast:\nGaming desktop, 80-200W idle, left on continuously: $200-460/year in electricity (depends on local rates) Mini PC, 10-15W idle, same always-on duty: $20-43/year Savings: $150-400/year Payback: 18-24 months for a $300-500 mid-tier mini PC, or well under a year for a $90-110 used enterprise small-form-factor desktop On paper, this is a fast, boring, obviously-correct upgrade.\nBut that number only works if the desktop actually reaches that low idle draw during \u0026ldquo;off\u0026rdquo; hours instead of pulling more power doing something else. A desktop that\u0026rsquo;s rendering, transcoding, or serving requests around the clock isn\u0026rsquo;t idling at 80-200W. It\u0026rsquo;s running at whatever load those tasks add on top of that baseline.\nThe savings calculation compares two idle states. If one of your machines never reaches an idle state, you\u0026rsquo;re not comparing what you think you\u0026rsquo;re comparing.\nBuying a mini PC doesn\u0026rsquo;t save power if the desktop stays on anyway # My own desktop killed the clean version of this argument, because it never stops running long enough to go idle. It wasn\u0026rsquo;t just gaming hardware sitting idle between sessions — it was already running 24/7 to serve a stack of self-hosted services:\nA media-automation pipeline A personal trading-research pipeline A local broker that arbitrates GPU access for LLM inference None of that stops when I\u0026rsquo;m not gaming. The desktop was never going to drop to a true idle state, let alone power off, regardless of what other hardware I bought.\nThat fact kills the power-savings case outright. Adding a 10-15W mini PC next to a desktop that keeps running at its existing load doesn\u0026rsquo;t subtract 80-200W from the bill — it adds 10-15W on top of what I was already paying. Total household power draw goes up, not down.\nAnyone pricing this decision purely on wattage needs to check their own uptime pattern first, because the entire payback calculation assumes the expensive box gets to power down once the cheap box exists.\nMine didn\u0026rsquo;t, so I never got that $150-400 check to cash. The whole decision comes down to one branch:\nflowchart TD A[Considering a low-power mini PC] --\u003e B{\"Does the desktop actuallyidle down or sleep today?\"} B --\u003e|Yes, it goes idle| C[\"Mini PC saves ~150-400 dollars/yearreal payback in 12-24 months\"] B --\u003e|No, runs 24/7 for other services| D[\"Mini PC adds 10-15W on toptotal household draw goes UP\"] D --\u003e E[\"Buy it anyway? Only for isolation/reliability,not for watts\"] The case for a dedicated box shifts to reliability once power savings are off the table # Once electricity cost stopped being the argument, reliability is what actually justified building a second box, and that case turned out to be stronger than I expected. Every driver update, every Windows patch, every game that wants a reboot to apply a change takes every hosted service down with it.\nA media pipeline and a trading-research pipeline don\u0026rsquo;t care about my GPU driver version. But they go offline anyway, every time I reboot for one. Decoupling the services from the gaming machine means a driver crash or a game install no longer doubles as a service outage.\nSplitting the workloads also removes a category of risk that has nothing to do with watts: a misbehaving game, a bad driver, or a resource-hungry mod shouldn\u0026rsquo;t be able to starve a database import or a scheduled job of the CPU and memory it needs.\nContention on a shared machine is invisible until it isn\u0026rsquo;t. I\u0026rsquo;d rather not find out about it during something genuinely time-sensitive: a database import, a scheduled job, whatever happens to be running. That\u0026rsquo;s a maintenance and stability question, separate from anything about power.\nGPU-bound work stayed on the desktop, and that\u0026rsquo;s a separate decision # I did not move everything off the desktop: local LLM inference stayed exactly where it was, running through the existing GPU-arbitration broker. I chose to leave it there instead of moving it along with everything else.\nVRAM, not CPU or system RAM, is the binding constraint for local LLM workloads, and VRAM contention with a running game is the one real risk in sharing a GPU between gaming and inference. Video transcoding and CUDA inference use physically separate silicon on the same card, so they mostly coexist fine.\nMoving LLM inference to its own hardware is a real option. But it\u0026rsquo;s a much bigger, separate spend. A dedicated inference-capable box, something like a Mac Mini M4 Pro with 48GB of unified memory or an AMD Ryzen AI Max+ box with 128GB, runs $600-2000 and only makes financial sense under heavy or continuous inference load.\nBundling that decision in with \u0026ldquo;buy a $300 mini PC for CPU-only services\u0026rdquo; muddies two questions that have different price floors and different payback conditions. I split them on purpose.\nWhat I\u0026rsquo;d actually check before buying # Check your desktop\u0026rsquo;s real uptime pattern before you check mini PC prices. If it\u0026rsquo;s already running 24/7 for reasons unrelated to gaming, buying a low-power box will not lower your electricity bill. Anyone telling you otherwise hasn\u0026rsquo;t looked at your actual load.\nThe purchase can still be worth it, but the reason changes: you\u0026rsquo;re paying for isolation and uptime. Watts stop being the point. I ended up repurposing an old laptop I already owned as the dedicated box, the XPS 17 that now runs Proxmox, rather than buying new hardware, since the reliability case didn\u0026rsquo;t require the cheapest possible idle wattage, just a second machine that wasn\u0026rsquo;t also my gaming rig.\nIf your desktop genuinely goes idle for long stretches, take the wattage math seriously. The payback period is short, and the number is real. Run the arithmetic on your own machine\u0026rsquo;s actual behavior instead of trusting whatever number a random N100 review quotes.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/gaming-desktop-vs-dedicated-compute-box-idle-power/","section":"Blog Posts","summary":"Idle power draw, not the price tag stamped on the mini PC, decides whether a dedicated low-power compute box actually saves you money over a gaming desktop. A gaming desktop idles around 80-200W, depending on the board, the PSU, and how many drives happen to be spinning. A purpose-built low-power box, the N100-class mini PCs and similar, idles at 10-15W.\n","title":"Gaming Desktop or Dedicated Compute Box: Idle Power Decides, Not Sticker Price","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/hardware/","section":"Tags","summary":"","title":"Hardware","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/power-efficiency/","section":"Tags","summary":"","title":"Power Efficiency","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/docker/","section":"Tags","summary":"","title":"Docker","type":"tags"},{"content":"Family-facing and storage-coupled services stay on the NAS. Compute-heavy personal projects move to a separate host with real memory to spare. That\u0026rsquo;s the whole framework, and it took months of pain to earn: a Synology DS1522+ with 8GB of RAM, roughly 35 Docker containers, and a box that kept falling over under memory pressure.\nContainerManager doesn\u0026rsquo;t fail loudly when it runs low on headroom. It just quietly starts murdering things. It stalls. It swaps. Eventually something dies, and figuring out which container actually mattered enough to protect took longer than it should have.\nStorage coupling decides placement # A service that\u0026rsquo;s coupled to storage or answers requests from other people in real time belongs on the NAS regardless of how heavy it is. A photo backup tool sits next to the disks it writes to; someone in the house opens the app, it has to answer, so it stays put.\nA knowledge-graph pipeline or a data-ingestion job is the opposite: it runs on my own schedule, tolerates a restart without anyone noticing, and doesn\u0026rsquo;t need to answer anything at 11pm on a Tuesday. That kind of workload moved to my desktop, which has far more RAM than the NAS and isn\u0026rsquo;t a fragile appliance I need to baby. (The crash saga that proved the NAS couldn\u0026rsquo;t carry the knowledge-graph workload is its own post.)\nThe shift buys headroom on the box that actually has to stay predictable.\nThe placement call itself is a simple branch:\nflowchart TD A[New self-hosted service] --\u003e B{\"Storage-coupled, or answersreal-time requests from people?\"} B --\u003e|Yes| C[Stays on the NAS] B --\u003e|No — tolerates a restart,runs on its own schedule| D[\"Moves to desktop(more RAM headroom)\"] Immich\u0026rsquo;s remote machine-learning support is meant to run alongside the local container, not replace it # Immich, the self-hosted photo app I use for family photo backup, officially supports running its machine-learning container on a separate host from the main server, through the IMMICH_MACHINE_LEARNING_URL setting. That\u0026rsquo;s documented, production-used behavior.\nThe trap is treating it as a full swap: point Immich only at the desktop\u0026rsquo;s ML container, and Smart Search and Face Detection break outright the moment the desktop is off, because my desktop isn\u0026rsquo;t an always-on box the way the NAS is. Immich\u0026rsquo;s own docs are explicit about the right pattern instead:\nKeep the local ML container running as a fallback. Add the remote URL alongside it, not in place of it. Jobs degrade to local processing instead of failing outright. Facial recognition itself talks to the database directly and doesn\u0026rsquo;t care where the ML container lives, so the underlying Postgres database can stay NAS-side no matter what.\nThe ML container ships with no authentication at all. Keep it on the local network and never forward it. SQLite-backed services migrate cheaply; Postgres-backed services need a logical dump # Migrating a stateful service safely comes down to what\u0026rsquo;s storing its state. Anything backed by SQLite in a config directory, which covers most media-automation tools in the *arr family, migrates with a stop-the-container, sync-the-volume, start-on-the-new-host sequence. That\u0026rsquo;s close to zero-risk: the database is just a file sitting still while you copy it.\nPostgres is a different problem. Copying a live data directory risks corruption, so the safe path is:\nTake a logical dump while the source stays running. Transfer that dump to the destination. Restore it, then run a row-count check before you touch the original. I moved a Postgres-backed data pipeline this way and it went cleanly. I\u0026rsquo;d read enough migration horror stories going in that I probably over-prepared for a problem that never showed up.\nA media library mounted at different paths on two hosts needs a one-time remap # One gotcha cost me more time than the actual migration. Media-automation tools store absolute library paths inside their own database, and if the new host mounts the same share at a different path than the old one did, every stored path is now wrong.\nNothing crashes when this happens. Shows just stop being tracked as monitored, and the failure mode looks like a metadata bug instead of a path problem.\nThe fix is a one-time script against the SQLite database that rewrites the stored root-folder paths to match the new mount layout. It\u0026rsquo;s a five-minute job once you know it\u0026rsquo;s coming, and an afternoon of confused debugging if you don\u0026rsquo;t.\nMonitoring belongs on the host that isn\u0026rsquo;t under memory pressure # A watchdog that lives on the same box it\u0026rsquo;s protecting adds to the exact pressure it\u0026rsquo;s supposed to catch.\nI run a lightweight watchdog on the NAS itself, a cron job paired with an ntfy push notification, because that footprint is small enough not to matter. Anything heavier, like Uptime Kuma, I\u0026rsquo;d rather run on the desktop watching the NAS remotely than install directly on the NAS.\nPutting monitoring next to the thing it watches feels natural. On a RAM-constrained box, it\u0026rsquo;s backwards.\nA RAM upgrade is a hedge, not a proven fix # I haven\u0026rsquo;t upgraded the NAS\u0026rsquo;s memory. I genuinely don\u0026rsquo;t know if it would solve the problem I moved workloads to avoid.\nThird-party memory is a real risk on this model specifically: at least one report describes a 16GB module in a DS1522+ registering as only 8GB, so the upgrade can fail silently instead of throwing an obvious error. Even with compatible memory, I couldn\u0026rsquo;t find a solid first-hand account confirming it actually stops the crash pattern rather than just raising the ceiling before it comes back at a higher container count. So it stays on my list as a possible complement to the migration: insurance layered on a split that\u0026rsquo;s already working.\nThe framework holds up months in, but the split isn\u0026rsquo;t finished. Every time a new self-hosted idea shows up, the first question is still which side of the line it belongs on, and I\u0026rsquo;ve gotten that call wrong at least once. A stack I placed on the desktop early has since moved a second time, to the Proxmox box I built from a retired laptop, because \u0026ldquo;more RAM than the NAS\u0026rdquo; turned out not to be the same thing as \u0026ldquo;the right home for this workload.\u0026rdquo;\nThe framework tells you which way to lean. It doesn\u0026rsquo;t promise you\u0026rsquo;ll land a given workload in the right spot on the first try.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/not-every-docker-container-belongs-on-the-nas/","section":"Blog Posts","summary":"Family-facing and storage-coupled services stay on the NAS. Compute-heavy personal projects move to a separate host with real memory to spare. That’s the whole framework, and it took months of pain to earn: a Synology DS1522+ with 8GB of RAM, roughly 35 Docker containers, and a box that kept falling over under memory pressure.\n","title":"Not Every Docker Container Belongs on the NAS","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/synology-nas/","section":"Tags","summary":"","title":"Synology NAS","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/debugging/","section":"Tags","summary":"","title":"Debugging","type":"tags"},{"content":"Fifteen of eighteen root causes I proposed for four firing alerts turned out to be wrong. Four alerts were going off across my home infrastructure at once: a stuck download post-processing backlog, plus three separate automation alerts tied to a video-discovery pipeline.\nMy first instinct on each one: form a theory fast, patch it, watch the alert clear. But I forced myself to do the opposite: generate every plausible root cause I could find, then attack each one before touching anything. Eighteen candidates went in. Three survived. That refutation rate is the actual finding here, more than any single bug I fixed.\nAdversarial verification means trying to kill your own hypothesis # Adversarial verification means treating your own hypothesis as something to disprove. For each candidate root cause, I ran three independent checks against three different failure modes:\nIs the claim actually correct? Is there a more likely alternative explanation for the same symptom? Would acting on this fix cause harm even if the diagnosis were right? Two negative checks out of three killed a finding, and I moved on without touching code.\nI used parallel background agents to run these checks concurrently, one per lens, working off the same evidence but arguing independently. It\u0026rsquo;s the same independence-over-agreement bet behind the dueling-agent review design I sketched elsewhere. The mechanism doesn\u0026rsquo;t matter much: you could run this with three colleagues, or with yourself on three separate days.\nWhat matters is that confirmation and refutation are different jobs. Doing both with the same brain in the same sitting is how bad root causes survive into production.\nHere\u0026rsquo;s how the 18 candidates actually funneled down:\nflowchart TD A[18 candidate root causes] --\u003e B[3 independent adversarial checks per candidate] B --\u003e C{2 of 3 checks negative?} C --\u003e|Yes, 15 candidates| D[Refuted - no action taken] C --\u003e|No majority reached, 1 candidate| E[Left open - reviewers split, no coin flip] C --\u003e|No, holds up, 2 candidates| F[Confirmed - acted on]The whole investigation stayed read-only until every surviving finding cleared verification:\nNo config edits No restarts No \u0026ldquo;let me just try this\u0026rdquo; during the diagnostic pass That discipline is what made the refuted list trustworthy. I never contaminated a measurement by fixing something mid-investigation.\nZero didn\u0026rsquo;t mean what I thought it meant # Earlier in this same session, before I tightened up the process, I had already reported that a download client\u0026rsquo;s bandwidth was pinned at 0 B/s and blamed an empty configuration value colliding with a governor script that writes percentage-based limits. That looked like an obvious bug. It would have been an easy one-line fix: set the missing value.\nIt was wrong, and setting that value would have made things actively worse. I traced the actual code path in this download client\u0026rsquo;s percentage-limit branch.\nA zero limit there means unlimited, not stopped. The log line that reads like a stall is literally the client\u0026rsquo;s own phrasing for \u0026ldquo;no cap applied.\u0026rdquo; I confirmed this three separate ways, including running the branch logic directly inside the container and cross-checking it against a measured throughput number that only made sense if the download was, in fact, running at full speed. Setting the value I\u0026rsquo;d flagged would have flipped the client into a different code branch entirely, one that computes a mismatched percentage on every release cycle and throws a runtime error every time.\nI would have taken a healthy, fast-running download client and broken it myself, on my own advice. It\u0026rsquo;s the same trust-a-single-signal failure that produced my GPU broker\u0026rsquo;s phantom-game bug: one plausible reading of one signal, promoted straight to ground truth.\nThe alert metric was lying about its own units # One of the four original alerts was measuring how long the oldest item had been stuck in the post-processing queue. The number it reported never looked right. It read low even when I could see items sitting untouched for days.\nThe bug turned out to be in how the metric collector seeded its internal clock: it stamped each item\u0026rsquo;s \u0026ldquo;first seen\u0026rdquo; time from the moment the collector itself first observed it, instead of when the item actually entered the queue. Every entry read back the exact same duration, no matter how long it had really been waiting, because the whole gauge was secretly measuring collector uptime.\nThat one survived all three checks cleanly. The alternative-cause reviewer couldn\u0026rsquo;t find a queue-processing explanation that fit the flat, identical readings across separate instances. The fix-safety reviewer confirmed the correct source of truth was already present in the underlying data and just needed to be read instead of guessed.\nAfter I re-seeded the clock from the real timestamp, the two queue instances immediately started reporting different, correct numbers: one nearly four days old, the other over a day and a half. The alert had been reporting a real problem\u0026rsquo;s existence without ever reporting its true severity, for as long as it had been deployed.\nThe budget governor\u0026rsquo;s fix made the problem worse # A budget governor script was supposed to reduce how often a discovery pipeline fired, to stay under a resource cap. Its \u0026ldquo;reduced\u0026rdquo; setting was implemented as a scheduling override applied on top of the baseline schedule. But the override mechanism in the underlying scheduler doesn\u0026rsquo;t replace an existing schedule when you add to it that way. It appends.\nThe lever meant to cut cadence was quietly increasing it: the \u0026ldquo;reduced\u0026rdquo; tier stacked a second firing schedule on top of the baseline instead of replacing it. Separately, a blank scheduling directive left in one code path caused the whole timer unit to fail to load at all, silently, with no warning that it had been disabled rather than paused. Both bugs shipped together and had been live long enough that nobody would have found either by reading the code once and moving on.\nWhat this cost, and what it still couldn\u0026rsquo;t tell me # Running eighteen hypotheses through three-lens verification is not fast. It took a long investigation session, and most of the eighteen candidates burned real analysis time before getting refuted. That\u0026rsquo;s the tax you pay for not shipping a plausible-sounding fix on the first guess.\nI think it was worth it here. Two of the three survivors were actively harmful if left alone, and the one I would have shipped from my earlier, faster pass would have made a healthy system fail on the next release cycle.\nThe process also has a real blind spot. One finding, a file-permission mismatch behind a wave of import errors, split my reviewers: one found evidence the bad files existed for hours before the failures started, another found the same failures beginning within minutes of a container restart despite those files already being in place.\nMajority-refutation needs an actual majority, and a genuine split doesn\u0026rsquo;t produce one. I left that finding open rather than act on a coin flip. That was the right call, but it means adversarial verification didn\u0026rsquo;t resolve it, it just kept me from pretending it had.\nThe backlog itself is also still draining slower than it should, and I haven\u0026rsquo;t traced a single item through the pipeline start to finish to prove why. Eighteen hypotheses in, some things are still genuinely unknown, and the honest move is to say so instead of closing the ticket.\nThe point of this exercise was never about the agents. It was about building a process where a plausible root cause has to survive someone actively trying to kill it before I\u0026rsquo;m allowed to act on it. Fifteen didn\u0026rsquo;t survive. I\u0026rsquo;m glad I found out before I touched anything.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/adversarial-verification-home-lab-alerts/","section":"Blog Posts","summary":"Fifteen of eighteen root causes I proposed for four firing alerts turned out to be wrong. Four alerts were going off across my home infrastructure at once: a stuck download post-processing backlog, plus three separate automation alerts tied to a video-discovery pipeline.\n","title":"Fifteen of Eighteen Root Causes I Was Sure About Were Wrong","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/automation/","section":"Tags","summary":"","title":"Automation","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/code-quality/","section":"Tags","summary":"","title":"Code Quality","type":"tags"},{"content":"Shipping fast is not the same as being done. I learned that the expensive way, on a CLI tool my own agent pipeline built in one afternoon.\nThe pipeline is mine. Feed it a one-line description of what I want, and it runs a fixed sequence:\nWrites a spec Runs that spec through seven parallel agents whose only job is to attack it from different angles (the same independence argument behind the dueling-agent-suites design I sketched separately) Spins up parallel build agents against the hardened spec Runs a full code review pass Smoke-tests the real thing before calling it done For a small outreach-automation CLI (local SQLite state, a human approval gate before anything goes out, a GitHub-facing sourcing loop), that pipeline produced working software in an afternoon. It ran. It did the job I asked for. But it wasn\u0026rsquo;t something I trusted enough to extend to a second platform without checking it first.\nThe pipeline only checks against what the spec asked for # Every phase in that pipeline checks the code against what I asked for. But none of that touches questions I never thought to ask in the spec.\nI hadn\u0026rsquo;t written \u0026ldquo;honor GitHub\u0026rsquo;s rate-limit contract\u0026rdquo; or \u0026ldquo;make sure the SQLite backup survives a write in progress\u0026rdquo; anywhere, so nothing in the pipeline went looking for those gaps. A spec-driven pipeline is only as complete as the spec.\nHere\u0026rsquo;s the shape of both passes, side by side: the pipeline that shipped the CLI, and the separate audit that checked its work.\nflowchart LR A[One-line description] --\u003e B[Spec written] B --\u003e C[7 parallel adversarial challenge agents] C --\u003e D[Hardened spec] D --\u003e E[Parallel build agents] E --\u003e F[Code review pass] F --\u003e G[Smoke test / verify] G --\u003e H[Working CLI, shipped in an afternoon] H -.-\u003e|separate pass, run on purpose| I[External research audit] I --\u003e J[\"4 concrete bugs found:rate limits, WAL backup,approval-log gap, thin lead signal\"] I --\u003e K[\"1 strategic decision:second platform goes draft-only,no automation\"]That something was a separate research pass, run on purpose, to find holes before adding a second platform, since that platform carries a strict terms-of-service posture around automation. It pulled from:\nGitHub\u0026rsquo;s own API documentation SQLite backup literature Comparable open-source tools That platform\u0026rsquo;s actual user agreement The GitHub loops never met GitHub\u0026rsquo;s own rate-limit contract # My tool has three loops that poll and post against GitHub (sourcing, checking, and queue-draining), and none of them honored the limits GitHub documents for its own API. GitHub publishes real numbers:\nA cap on concurrent requests A points-per-minute budget on REST calls A separate, much stricter cap on content-creating requests per minute and per hour GitHub\u0026rsquo;s docs are explicit that repeatedly ignoring rate-limit errors can get an integration banned outright, not just throttled. My loops were calling the API and hoping, with no code anywhere that read a Retry-After header or backed off on a 403.\nThe fix was mechanical once I knew what to build, straight from GitHub\u0026rsquo;s REST API best-practices guide:\nHonor Retry-After and the rate-limit-reset header first Switch polling loops to conditional requests, so unchanged data comes back as a cheap 304 instead of spending budget Space out anything that creates content by at least a second None of that is clever. All of it was missing.\nA raw file copy could have quietly corrupted the backup # The tool\u0026rsquo;s entire state (accounts, drafts, leads) lives in one SQLite file, and the backup routine copied that file directly on a schedule.\nSQLite in its default mode buffers recent writes in a separate write-ahead log file. A plain file copy of the main database while that log holds unflushed writes can capture a database that looks intact and isn\u0026rsquo;t. Testing won\u0026rsquo;t catch this: it only bites the one time you actually need the backup to be good.\nThe fix is a single command swap, from a raw copy to SQLite\u0026rsquo;s own online-backup call that captures a consistent snapshot regardless of what\u0026rsquo;s mid-flight.\nSmall fix. But it was sitting on exactly the failure mode I\u0026rsquo;d never notice until it was too late to matter.\nThe approval gate had no memory of its own decisions # Nothing goes out of this tool without a human approving it first, and that gate is tied to a hash of the exact content being approved, so any edit after approval voids it automatically. That part of the design held up fine under review.\nWhat was missing was history: no log of who approved what, when, or what got rejected and why. If I wanted to know later why a specific piece of content went out, or audit a month of decisions, there was nothing to check against but my own memory of pressing a key.\nCommercial approval-workflow tools keep exactly this kind of log by default. Mine didn\u0026rsquo;t. It\u0026rsquo;s the kind of gap that\u0026rsquo;s invisible right up until you need it, for a reason you never planned for.\nLead-sourcing ran on one thin signal # The tool finds candidates to reach out to using keyword matching against a configured niche list, and that\u0026rsquo;s the whole signal. Comparable tools in this space enrich candidates with graph signals (repository stars, forks, contributor overlap) that catch relevance keyword matching alone misses.\nI haven\u0026rsquo;t fixed this one yet. It\u0026rsquo;s on the list for later, and I\u0026rsquo;m naming it here instead of pretending it\u0026rsquo;s closed, because the rest of this post is about being honest about what \u0026ldquo;done\u0026rdquo; actually took.\nExtending to a second platform meant deciding not to automate it # The biggest finding wasn\u0026rsquo;t a bug. Before writing a single line for the second platform, I checked its user agreement, and it explicitly bans the exact category of automation my GitHub loops already do, spelled out item by item:\nAuto-connecting Auto-posting Auto-commenting Scraping via any bot or script Real ban-rate data on comparable automation tools for that platform backs the terms up. Even the more cautious, cloud-hosted versions of that kind of automation carry meaningful suspension risk. But I also found at least one legitimate, adopted product in that space doing exactly what I was already leaning toward: format drafts for a human to review and post manually, no session automation at all. That\u0026rsquo;s proof draft-only is a real category, not a compromise I was talking myself into.\nSo the second-platform build changed shape entirely. Instead of extending the same auto-post pattern, I\u0026rsquo;m formatting approved drafts with suggested timing for that platform\u0026rsquo;s own native scheduler, and stopping there.\nWhat I\u0026rsquo;m still not sure about # I don\u0026rsquo;t know yet whether a dedicated audit pass like this needs to happen after every run of my build pipeline, or whether this project just happened to be unusual enough (real external APIs, real state that has to survive a backup, a second platform with real legal terms) to need one. Running an audit like this on every small tool I build would be pure overhead for most of them.\nI lean toward doing it whenever a tool talks to another service\u0026rsquo;s API or holds state I\u0026rsquo;d actually miss if it corrupted, and skipping it otherwise. But I\u0026rsquo;ve only tested that rule on one project so far.\nThe build pipeline did exactly what I asked it to do, fast and correctly. \u0026ldquo;What I asked for\u0026rdquo; and \u0026ldquo;what I actually needed before trusting this thing\u0026rdquo; turned out to be two different lists.\nThe second one had a write-ahead log and a rate-limit header on it that the spec never mentioned — finding it took a separate pass I almost skipped.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/auditing-what-an-agent-pipeline-shipped-in-an-afternoon/","section":"Blog Posts","summary":"Shipping fast is not the same as being done. I learned that the expensive way, on a CLI tool my own agent pipeline built in one afternoon.\n","title":"Shipping Fast Isn't the Same as Being Done: Auditing a CLI My Agent Pipeline Built in an Afternoon","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/git/","section":"Tags","summary":"","title":"Git","type":"tags"},{"content":"Three failure modes were hiding behind one name, and only one of them was actually about drift. I run four or five Claude Code agents at once, each in its own repo, and for months every mess that came out of it got filed under the same complaint: things drifting out of state while I wasn\u0026rsquo;t watching. Pulled apart, the three landed in very different places:\nWorktree sprawl: leftover git checkouts an agent session opened and nobody closed. Turned out to be mostly a feature I hadn\u0026rsquo;t turned on, not a missing tool. Resolved. Deploy drift: a running service that no longer matches what an agent thought it built. A problem I\u0026rsquo;d already solved once, for one project, and just needed generalizing. Resolved. Wasted compute: idle capacity across my machines. Still open; nothing solved it. Calling all three \u0026ldquo;state drift\u0026rdquo; was the mistake. That\u0026rsquo;s the reason it took this long to notice only one of them actually was about drift.\nflowchart TD A[\"One complaint: 'state drift'\"] --\u003e B[Worktree sprawl] A --\u003e C[Deploy drift] A --\u003e D[Wasted compute] B --\u003e B1[\"RESOLVED — a feature alreadyshipped, just needed reading the docs\"] C --\u003e C1[\"RESOLVED — pattern I'd alreadybuilt once, generalized to every repo\"] D --\u003e D1[\"OPEN — no real fix found,nothing scheduled to build it yet\"] Worktree sprawl turned out to be a feature nobody had switched on # Worktree sprawl looked like a missing tool, but it wasn\u0026rsquo;t. I run Claude Code as several parallel agents, each working a different repo or branch, and each one needs its own working directory so two agents don\u0026rsquo;t stomp on the same uncommitted edits. (What that style of agent use costs is its own story.) Git\u0026rsquo;s answer to that is a worktree: a second working directory attached to the same repository, checked out on its own branch, addable and removable independently of the main clone.\nThe complaint that started this investigation was plain: I kept finding worktrees on disk that some agent session had opened, and nobody, including me, had closed.\nClaude Code already ships a worktree lifecycle, which the research sweep turned up and I hadn\u0026rsquo;t clocked before. Most of it just needed turning on, not replacing:\nIt auto-sweeps worktrees created for subagents and background sessions once they clear a configurable age, but only if they\u0026rsquo;re clean: no uncommitted changes, no unpushed commits. Anything opened with an explicit --worktree flag is exempt from that sweep entirely; the documentation says directly it never removes a worktree created that way. Worktrees opened mid-session with the EnterWorktree tool only get cleaned up on a clean session exit, so a session that dies partway leaves them behind. That distinction explained most of what I\u0026rsquo;d been seeing. My deliberate multi-agent sessions, opened on purpose rather than the throwaway subagent kind, were never going to get swept. The sweep was never built to touch them.\nDeploy drift isn\u0026rsquo;t a worktree problem: it\u0026rsquo;s a gap between git and live state # Deploy drift means a running service no longer matches what the agent that built it believes is deployed: config edited by hand after the fact, a container that never picked up the latest image, a service pointed at a stale checkout. No worktree cleanup script reaches that gap between git state and live state.\nI\u0026rsquo;d already closed that gap once, for one home-lab service, with a script that checks the deploy target after every push and diffs what\u0026rsquo;s actually running against what git says should be running, backed by a written rule that every service needs the same coverage. The pattern the wider search turned up, a scheduled check that shells out over SSH to compare live state against the repo, was structurally the thing I\u0026rsquo;d already built.\nThe gap wasn\u0026rsquo;t a missing tool. It was that the pattern only ran against one project instead of every project with something deployed.\nHeavier options exist: a continuous-reconciliation controller built for orchestrating containers across a cluster, diffing live state against a git manifest on every change. My footprint is a handful of systemd services and Docker Compose stacks on two machines; adopting that would mean running infrastructure to manage infrastructure I don\u0026rsquo;t have. The actual fix is unglamorous: copy the pattern I already trust to the rest of the repos that deploy something.\nNobody has actually solved wasted compute # Compute utilization across my machines is where the search came back empty-handed. Agents sit idle on one box while the other has spare capacity, and nothing I found actually schedules work across that gap the way a real fleet scheduler would.\nThe closest candidate was a small, early open-source CLI built for exactly this: routing work and judging reliability across agent runtimes.\nIt\u0026rsquo;s unverified, too new and too thin on real adoption to trust with anything that matters. I own that gap. If I want it solved, I have to build a thin version myself, and I haven\u0026rsquo;t started.\nA follow-up audit checked whether the fix actually held # Two weeks after landing on that plan, I went back and checked every repo on both machines against the documented worktree lifecycle instead of taking the research sweep\u0026rsquo;s conclusion on faith, and the audit held up. Claude Code\u0026rsquo;s own EnterWorktree/ExitWorktree lifecycle, the tools for opening and closing a worktree mid-session, works correctly in exactly the one workflow I built for it, and nowhere else yet. That workflow opens a worktree at the start of a run and closes it right after a successful merge.\nEvery other repo on the Mac, roughly two dozen of them, and every repo on the desktop (a deploy target, not somewhere agents run) has never had a worktree at all. There\u0026rsquo;s no adoption gap in those repos: no worktree activity to sweep in the first place.\nTotal inventory across both machines: four worktrees.\nOne was a live session, locked and actively in use, correctly left alone. Two belonged to a separate build-cache tool used by another skill in my pipeline, not Claude Code\u0026rsquo;s own lifecycle: a different kind of accumulation than agent sprawl. One was a genuine dead worktree: a merged, clean, three-day-old checkout that should have been removed and wasn\u0026rsquo;t. That fourth one is the interesting case, because it wasn\u0026rsquo;t a bug. My own workflow documents a fallback rule for exactly this: if a run fails partway through after the merge already succeeded, leave the worktree on disk and say so in the final report instead of silently deleting work mid-failure. The dead worktree on the Mac is that rule firing exactly as designed: some later phase (hardening, review, deploy, or verification) stopped short after the merge had already landed.\nIt did exactly what I told it to do when something breaks downstream: preserve state over convenience. But I still haven\u0026rsquo;t fixed the part where nothing reminds me to go check for it after a run stops early. Right now that\u0026rsquo;s a manual habit I have to remember to do myself.\nOne of the three was already solved by a feature I hadn\u0026rsquo;t read the docs on. One was a pattern I\u0026rsquo;d already proven, just needed pointing at everything. The third still has no real fix. I\u0026rsquo;m not dressing up a thin, unverified GitHub repo and calling it done.\nBut \u0026ldquo;state drift\u0026rdquo; was never a diagnosis. It was a name I gave three problems so I wouldn\u0026rsquo;t have to look at them separately. A stray worktree, a stale container, and an idle box are not the same bug, and treating them as one is what kept me from fixing any of them faster.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/three-failure-modes-one-name-concurrent-claude-code-agents/","section":"Blog Posts","summary":"Three failure modes were hiding behind one name, and only one of them was actually about drift. I run four or five Claude Code agents at once, each in its own repo, and for months every mess that came out of it got filed under the same complaint: things drifting out of state while I wasn’t watching. Pulled apart, the three landed in very different places:\n","title":"Three Failure Modes Wearing One Name: Running Concurrent Claude Code Agents Without State Drift","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/code-review/","section":"Tags","summary":"","title":"Code Review","type":"tags"},{"content":"Run two coding-agent orchestration suites that share nothing between them (no prompts, no config, no instruction derivation) and make them review each other\u0026rsquo;s pull requests, the way two engineers who never compared notes catch each other\u0026rsquo;s mistakes. Suite A opens a PR. Suite B, built from scratch with zero visibility into how A works, reviews it cold. Suite A reads the verdict, decides what\u0026rsquo;s real, fixes what needs fixing, and the loop can run again from there.\nI came up with this on my own, then went looking for it anyway. Checking first is the fastest way to find out whether an idea is obvious or nobody\u0026rsquo;s gotten around to it yet.\nA single agent reviewing its own PR doesn\u0026rsquo;t catch much # A single agent reviewing its own pull request doesn\u0026rsquo;t catch much, and now there\u0026rsquo;s a number attached to it. CodeRabbit, a production code-review tool, cites a study that names the pattern the \u0026ldquo;Homogenization Trap\u0026rdquo;: models trained on overlapping data share the same blind spots, so asking one model to grade its own work just replays the assumptions that produced the bug in the first place.\nThe study CodeRabbit cites found an average failure rate of 64.5 percent when models are asked to correct errors they produced themselves. That\u0026rsquo;s the whole justification for splitting author and reviewer into separate agents. It\u0026rsquo;s also why splitting them into two copies of the same model barely helps.\nI\u0026rsquo;ve watched the same blind spot from the other side already. A separate audit pass on a CLI my own agent pipeline built found four real gaps the pipeline\u0026rsquo;s built-in review never flagged, because the review only checked the code against the spec that shared its assumptions.\nMaking the suites genuinely independent takes three deliberate constraints # The design only works if the two suites are actually independent, not just two separate agent invocations. Three requirements make that real:\nDifferent base models, or at minimum instruction sets and personas built without either side looking at the other\u0026rsquo;s files, the way two engineers who never compared notes end up writing different code for the same ticket. A fresh session every round. When B reviews A\u0026rsquo;s PR, it starts cold instead of carrying context forward. Letting a reviewer hold onto its own earlier verdict is a known way for it to anchor on that verdict instead of actually looking again. A hard round limit, somewhere around three to five exchanges, so the respond-and-re-review cycle can\u0026rsquo;t spin forever on a disagreement neither side will drop. Here\u0026rsquo;s the loop itself:\nflowchart LR A[Suite A opens PR] --\u003e B[\"Suite B reviews cold(fresh session, no shared config)\"] B --\u003e C[Suite A decides what's real, applies fixes] C --\u003e D{\"Round limit reached?(3-5 exchanges)\"} D --\u003e|No| B D --\u003e|Yes| E[Loop ends] Nobody ships this as a preset, but the pieces exist # Nobody ships this exact pattern as a ready preset, but the pieces are scattered across current tools and papers. Academic research on adversarial debate between large language models already studies quality gains when review peers are genuinely different rather than cooperative copies of each other. At least one recent paper formalizes almost the same author-reviewer-critic loop sketched here, adding a third agent that audits the reviewer\u0026rsquo;s own review.\nQodo\u0026rsquo;s second-generation review tool runs several specialized agents in parallel against one PR and posted the best F1 score (a standard accuracy measure combining precision and recall) of eight review tools tested. That\u0026rsquo;s parallel specialist review, though, not an adversarial author-versus-reviewer duel.\nMainstream orchestration frameworks ship a generic writer/reviewer role you can wire up yourself, but none of them package \u0026ldquo;two independently-derived agent suites duel it out\u0026rdquo; as something you install and configure.\nThe market\u0026rsquo;s clearest independent reviewer just lost its independence # The strongest counter-signal I found points the other way. Cursor, one of the more popular AI coding tools, acquired Graphite in December 2025, with a stated plan to combine Graphite\u0026rsquo;s Diamond reviewer with Cursor\u0026rsquo;s own Bugbot. The most notable separate-company code reviewer on the market is now owned by the same vendor that ships the authoring agent.\nIf the industry keeps consolidating that way, buying genuine cross-vendor independence gets harder every year. A dueling-suite design that leans on \u0026ldquo;different vendor, different training run\u0026rdquo; as its independence guarantee is betting against that trend.\nNothing here has actually been built or run # This is a design sketch pulled together from research. I haven\u0026rsquo;t built or run any version of it: no working prototype, no latency or cost numbers from my own attempts. Everything above about round limits and fresh sessions is a plan I haven\u0026rsquo;t tested.\nOne source did report real numbers from someone else\u0026rsquo;s cross-model adversarial review setup.\nEach review pass took 30 to 90 seconds, and a full exchange ran three to five debate rounds with two separate models in play the entire time. Multiply that across a normal-sized PR, and the wait before a suite even finishes disagreeing with itself starts to look expensive for something that might just find the same handful of issues a single well-configured reviewer agent would have caught in one pass.\nThe real question is whether disagreement finds bugs or just makes noise # The real question I can\u0026rsquo;t answer yet is whether independent agent suites disagreeing actually surfaces real bugs, or just generates plausible-sounding noise a human still has to sort through.\nTwo of my sources flagged negation-blindness as a structural weakness independent of which model you pick. A reviewer agent can miss that a fix does the opposite of what\u0026rsquo;s needed, regardless of how independently it was built. If that failure mode shows up in both suites, I end up with two agents that agree with each other and still miss the same bug.\nI don\u0026rsquo;t know yet whether that happens rarely enough to be worth the extra compute and wall-clock time, or often enough that this is just a more expensive way to get the review quality I\u0026rsquo;d already get from one well-configured agent and a human final pass. The only way I\u0026rsquo;ll find out is by building a small version of this against a real repo — probably the old orchestration prototype that\u0026rsquo;s already sitting around gathering dust.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/dueling-agent-orchestration-suites/","section":"Blog Posts","summary":"Run two coding-agent orchestration suites that share nothing between them (no prompts, no config, no instruction derivation) and make them review each other’s pull requests, the way two engineers who never compared notes catch each other’s mistakes. Suite A opens a PR. Suite B, built from scratch with zero visibility into how A works, reviews it cold. Suite A reads the verdict, decides what’s real, fixes what needs fixing, and the loop can run again from there.\n","title":"What If Two Independently-Built Agent Suites Reviewed Each Other's Code?","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/cost-engineering/","section":"Tags","summary":"","title":"Cost Engineering","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/llm-infrastructure/","section":"Tags","summary":"","title":"LLM Infrastructure","type":"tags"},{"content":"You can write great code with agents. I believe that now. But they\u0026rsquo;re pretty bad on their own without manual review at the beginning and end of every initiative. That\u0026rsquo;s what a $364 Claude Code session taught me. I found the number on a quiet Sunday morning, checking on the automated processes I\u0026rsquo;d kicked off the night before.\nFour Numbers Pointed to One Shape: Long, Unattended, Subagent-Heavy Sessions # The week behind that number broke down the same way every time:\n100% of the spend came from sessions that had spawned subagents: the main session delegating to separate Claude instances running in parallel. 99% came from sessions that ran longer than eight hours straight. 90% happened while context sat above 150,000 tokens. 62% of my weekly cap was already burned by the middle of that week, from claude -p jobs firing unattended on my desktop. One shape: long, subagent-heavy, unattended sessions, with no lifecycle boundary at all.\nflowchart TD A[\"$364 session\"] --\u003e B[\"100% of spend: sessions with subagent fan-out\"] A --\u003e C[\"99% of spend: sessions open 8+ hours\"] A --\u003e D[\"90% of spend: context above 150k tokens\"] A --\u003e E[\"62% of weekly cap: unattended claude -p jobs\"] B \u0026 C \u0026 D \u0026 E --\u003e F[\"One shape: long, subagent-heavy,unattended sessions, no lifecycle boundary\"] F --\u003e G[Fix: CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=60] F --\u003e H[Fix: SessionStart recovery hook] F --\u003e I[Fix: --max-turns hard stop] The Actual Stakes: Passive Income and My Own Name on the Code # I don\u0026rsquo;t want to spend a lot of money doing this. The goal is passive income streams for my family, built on the algo work these sessions support. I also care about the codebase and the architecture being as good as possible — I\u0026rsquo;m a software engineer, and this kind of thing matters to me. It bothers me having something run that I don\u0026rsquo;t understand. This sort of stuff is supposed to represent me, since I\u0026rsquo;m staking my identity on being a professional computer toucher.\nSubagents Inherit the Parent\u0026rsquo;s Model Unless Told Otherwise # One gap in the pipeline: subagents inherit the parent session\u0026rsquo;s model by default. A subagent doing mechanical work, checking test coverage, grepping logs, gets billed at the same rate as one doing real design judgment, unless something tells it not to. Claude Code\u0026rsquo;s subagent docs expose three ways to override it: a model: field in the subagent\u0026rsquo;s own frontmatter, an invocation parameter, or a CLAUDE_CODE_SUBAGENT_MODEL environment variable that downgrades every subagent in a session at once. None of my heavier pipelines were using any of the three.\nUnattended Jobs Draw From the Same Cap as My Own Keyboard Time # Programmatic usage draws from the same weekly subscription cap as interactive sessions. On 2026-06-15, Anthropic paused a planned change to Agent SDK billing that would have split headless claude -p and Agent SDK calls onto their own credit pool. The pause settled it the other way. I\u0026rsquo;d been scheduling jobs as if the separation had already happened. It hadn\u0026rsquo;t. The cadence governor I built for those unattended fires exists because of that competition, and because, as I put it after this session, \u0026ldquo;token/performance discipline must be BAKED INTO the workflows, not left to habit.\u0026rdquo;\nThree Fixes Went in the Same Week I Found the Number # Three fixes went in: CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=60, an environment variable that sets the context-fill percentage where auto-compaction fires. Setting it to 60 makes it a rule instead of a habit I have to remember. A SessionStart hook that fires on clear or compact and re-injects the run\u0026rsquo;s on-disk state, so a cleared session recovers instead of losing the thread. And a hard --max-turns stop on every headless invocation, so a misbehaving loop can\u0026rsquo;t run past budget even if the other two are working.\nI Still Don\u0026rsquo;t Have a Grip on the Whole Thing # I still don\u0026rsquo;t really have a good grip on my entire codebase, the work, or the money I\u0026rsquo;m spending on it. I\u0026rsquo;m treating it as a learning process: proposing a hypothesis and collecting data from the experiment. Every positive is an opportunity to refine the pipeline and make it even better.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/what-a-364-dollar-claude-code-session-taught-me-about-agent-hygiene/","section":"Blog Posts","summary":"You can write great code with agents. I believe that now. But they’re pretty bad on their own without manual review at the beginning and end of every initiative. That’s what a $364 Claude Code session taught me. I found the number on a quiet Sunday morning, checking on the automated processes I’d kicked off the night before.\n","title":"What a $364 Claude Code Session Taught Me About Running Agents Unattended","type":"blog"},{"content":"Anthropic will not tell you how many tokens or messages Claude Max 20x actually gives you, and I had to build a throttle for it anyway. I run several personal research projects on background schedules through claude -p — unattended fires that call Claude Code from cron and systemd timers while I\u0026rsquo;m not watching. Those fires draw from the exact same quota as the interactive Claude Code sessions I use to do actual work.\nIf a background job burns the pool at 2pm, my 2:15pm session pays for it. A single $364 session made that competition concrete enough to build against. I wanted those jobs to back off automatically as usage climbed, and hand the room back the moment I sat down to work. Anthropic gives you nothing to calibrate that against.\nAnthropic only publishes a ratio # Anthropic\u0026rsquo;s Max plan documentation defines the 20x tier as \u0026ldquo;20 times more usage per session than the Pro plan,\u0026rdquo; and that\u0026rsquo;s the entire spec. No token count, no message count, no per-window number anywhere in Anthropic\u0026rsquo;s own docs.\nEverything else is qualitative: usage scales with conversation length, model choice, and effort level. I went looking for a hidden number to hardcode against and confirmed there isn\u0026rsquo;t one, at least not one Anthropic publishes.\nThat absence isn\u0026rsquo;t a documentation oversight. It\u0026rsquo;s a load-bearing consequence of the ceiling itself moving. Anthropic doubled the 5-hour rate limit for Claude Code on Pro, Max, and seat-based Enterprise plans on 2026-05-06, and removed a peak-hour reduction that had applied to Pro and Max accounts on the same date.\nAny number I\u0026rsquo;d baked into a governor before that date would have been wrong the moment it shipped, silently, with no changelog entry pointing at my config file. A governor built against a fixed assumed ceiling is a governor built to go stale.\nOne pool, two independent windows, and a hidden sub-cap # The quota itself isn\u0026rsquo;t even one thing to track. Usage across claude.ai, Claude Code, and Claude Desktop draws from a single shared pool. Anthropic states this directly, and it\u0026rsquo;s the fact that makes the whole problem real: a scheduled background fire and an interactive session compete for the same resource, so I can\u0026rsquo;t reason about them as separate budgets.\nOn top of that shared pool sit multiple caps on different clocks:\n5-hour session window: resets from the timestamp of your first prompt, not wall-clock time, so two people starting a session an hour apart are on different reset schedules even on the same day. Weekly cap, all models: sits above the session window. Weekly cap, Sonnet only: narrower, and layered inside the all-models cap. A governor that only watches the 5-hour window runs headlong into the weekly Sonnet cap with no warning. That cap can bind long before the session window ever does. The acceleration limit rules out a hard stop-start throttle # The design constraint that changed my approach most is an acceleration limit: Anthropic\u0026rsquo;s rate limiter applies something like it, where a sharp spike in request volume can trigger a 429 even with headroom remaining in the steady-state window. I found this in a practitioner writeup on Claude Code rate limits; Anthropic\u0026rsquo;s own docs never mention it.\nA background job snapping from idle to full concurrency the instant a window opens looks exactly like the kind of spike that limiter is built to catch, quota headroom or not. That rules out the simplest version of a governor: check remaining budget, run at full speed until the number hits zero, then hard-stop. The governor has to ramp cadence down and back up gradually on both ends.\nWhat I actually built # The governor is a budget-aware layer on top of the timing logic I already had for scheduling background campaigns. It reads from a local SQLite corpus that already tails Claude Code\u0026rsquo;s own transcript files, and from that it computes two rolling figures continuously: weighted token consumption over the trailing 5 hours, and the same over the trailing 7 days, combining interactive and automated usage together since they draw from the same pool.\nAs either figure approaches its ceiling, the governor ramps down the cadence of scheduled claude -p fires, targeting no more than 98% utilization of whatever ceiling it\u0026rsquo;s currently tracking. The fires I run interactively never throttle; only the automated ones do. That reserves roughly 2% of headroom, specifically so an interactive session I start doesn\u0026rsquo;t land on an already-exhausted window.\nAs usage clears on either rolling window, cadence ramps back up on the same gradual curve it ramped down on.\nHere\u0026rsquo;s the loop the governor actually runs:\nflowchart TD A[\"Claude usage: interactive + claude -p, shared pool\"] --\u003e B[Track 5hr rolling window] A --\u003e C[Track 7-day rolling window] B --\u003e D{Approaching ceiling?} C --\u003e D D --\u003e|Yes| E[\"Ramp down claude -p cadence,target 98% utilization\"] D --\u003e|No| F[Ramp cadence back up, gradually] G[429 response received] -.-\u003e|calibrates working ceiling| DThe \u0026ldquo;ceiling it\u0026rsquo;s currently tracking\u0026rdquo; part is the honest workaround for not having a real number. Since Anthropic doesn\u0026rsquo;t publish one, the governor calibrates its threshold from live signals: when a claude -p fire actually gets rate-limited, Claude\u0026rsquo;s own error response carries a reset timestamp, and the governor parses that as ground truth and adjusts its working ceiling estimate from it.\nAbsent a fresh 429 to calibrate against, it falls back to a conservative default. The design works like an adaptive controller: it reacts to real signals because there\u0026rsquo;s no spec sheet to check against.\nI wired the throttle into the two places that actually spend tokens unattended:\nA scheduled research campaign that fires on a timer. A document-ingestion pipeline, where the lever isn\u0026rsquo;t fire frequency but concurrency: how many ingestion workers run in parallel against the shared quota. The governor treats them as two separate levers under the same shared budget. I also checked a third scheduled job that looked like a candidate and found it makes no LLM calls at all: a deterministic RSS collector. It was never competing for the quota, so I left it out of the governor entirely.\nWhere I think this could be wrong # The strongest argument against building any of this is that I might have solved a problem that a much dumber approach handles just as well. A purely reactive design needs a fraction of the code and doesn\u0026rsquo;t require guessing at a ceiling that keeps moving anyway:\nLet jobs run at full speed. Catch the 429 when it happens. Back off with exponential jitter. Retry. I built the proactive version because I wanted to protect interactive sessions from ever seeing a 429 in the first place. Recovering gracefully after the fact wasn\u0026rsquo;t the goal. But I can\u0026rsquo;t prove that protection is worth the complexity it costs — the reactive fallback alone might have covered 90% of the actual harm.\nThe number I\u0026rsquo;m least confident in is the 2% headroom target itself. I picked it because it felt like enough margin without leaving obvious quota on the table. I didn\u0026rsquo;t derive it from anything more rigorous than that. Since Anthropic doesn\u0026rsquo;t publish the real ceiling, I have no way to check that 2% against ground truth.\nI can only watch whether interactive sessions still hit limits in practice and adjust after the fact: the same calibration-from-observed-429s approach the governor itself uses internally. That means the whole system, including the part that\u0026rsquo;s supposed to be doing the calibrating, is tuned against my own incomplete observations. There\u0026rsquo;s still no documented spec to check it against.\nI\u0026rsquo;m comfortable shipping that. I\u0026rsquo;m not comfortable calling it settled.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/self-throttling-claude-max-without-a-published-ceiling/","section":"Blog Posts","summary":"Anthropic will not tell you how many tokens or messages Claude Max 20x actually gives you, and I had to build a throttle for it anyway. I run several personal research projects on background schedules through claude -p — unattended fires that call Claude Code from cron and systemd timers while I’m not watching. Those fires draw from the exact same quota as the interactive Claude Code sessions I use to do actual work.\n","title":"Building a Self-Throttling Governor for Claude Max With No Published Ceiling","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/machine-learning/","section":"Categories","summary":"","title":"Machine Learning","type":"categories"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/runpod/","section":"Tags","summary":"","title":"RunPod","type":"tags"},{"content":"Gemini wins on accuracy. RunPod wins on cost. I run vision-model inference for a personal image pipeline on RunPod GPUs instead of calling Google\u0026rsquo;s Gemini API, and that split is the whole decision, not a verdict on which model is smarter.\nRunPod\u0026rsquo;s low price only holds if something babysits it. The actual engineering problem here wasn\u0026rsquo;t the model weights or the prompt tuning. It was a cron job watching a clock.\nThis is a companion piece to the estate-sale scanner series on this blog, covering one narrow decision about where the vision step runs rather than reworking that whole pipeline.\nGemini 2.5 Flash RunPod dedicated pod (Qwen2.5-VL) Accuracy 0.75 mAP, best score tested Lower raw accuracy, confidence-tagged per field Cost Pay per call, no idle cost ~$0.44/hr, bills whether idle or not Hardware Managed API, no GPU to size 48GB-class GPU Gemini scores higher on accuracy, but my pipeline doesn\u0026rsquo;t need every field to be right # Gemini 2.5 Flash tops a structured-extraction benchmark for vision-language models (VLMs, models that take an image and a text prompt together and return structured output) at 0.75 mAP, the best score of any model tested, self-hosted or managed. Qwen2.5-VL, the self-hosted model I actually run, trails that number on raw accuracy.\nBut Qwen doesn\u0026rsquo;t have a marginal cost per call, and that\u0026rsquo;s the number that shows up on my bill. Every image I send to Gemini costs money no matter what; every image I send to a GPU I already control costs whatever fraction of an hour that request eats off the card.\nThat accuracy gap only matters if the pipeline can tolerate what Qwen actually delivers, and mine can. Every field it extracts carries a confidence tag, and anything low-confidence gets flagged for a human to glance at instead of trusted outright. A task that needs every field right on the first pass shouldn\u0026rsquo;t make this trade at all.\nServerless pricing looked like the whole answer until I read the sizing requirements # RunPod\u0026rsquo;s serverless tier scales to zero between requests, so idle time costs nothing. That\u0026rsquo;s the actual reason serverless looks attractive for a personal project with bursty traffic.\nBut Qwen2.5-VL isn\u0026rsquo;t a drop-in fit on a serverless worker. Community deployment threads spell out what it takes to fit the model weights alongside the KV cache the image tokens generate:\nA 48GB-class card: L40, L40S, or RTX 6000 Ada GPU memory utilization tuned to 0.90 Prefix caching turned on vLLM\u0026rsquo;s own multimodal serving docs require setting --limit-mm-per-prompt explicitly, for example image=1 for a pipeline that sends one photo per request, because the default silently drops image inputs instead of accepting them. The same vLLM community thread that gave me those sizing numbers also flags multi-image batching efficiency as an open problem with no confirmed fix. I don\u0026rsquo;t send multiple images per request today, so that gap doesn\u0026rsquo;t block me, but it\u0026rsquo;s a sign the serverless-vision path is younger than the serverless-text path I already use elsewhere.\nDedicated pods are cheaper per hour, and that\u0026rsquo;s exactly what makes them dangerous # A dedicated RunPod GPU, an A40 with 48GB running a vLLM template, prices out around $0.44 an hour. That\u0026rsquo;s a small fraction of what a bigger card costs me for other GPU work I run at home. At that rate, a dedicated pod running vision inference all day still costs less than a handful of Gemini calls at any real volume.\nBut it bills for every minute it\u0026rsquo;s running, whether or not anything is actually calling it.\nServerless pods scale to zero automatically. Dedicated pods don\u0026rsquo;t. I went looking in RunPod\u0026rsquo;s own docs assuming I\u0026rsquo;d just missed a toggle somewhere.\nThere isn\u0026rsquo;t one. RunPod\u0026rsquo;s GraphQL API documents a podStop mutation, podStop(input: {podId: \u0026quot;ID\u0026quot;}) { id desiredStatus }, which stops a pod and preserves its volume data. But there\u0026rsquo;s no built-in idle timeout anywhere in the dedicated-pod management docs. Idle-auto-stop is a serverless feature. A dedicated pod left running after the last request just keeps billing by the minute until something outside RunPod tells it to stop.\nI built a watchdog because nothing else was going to call podStop for me # Once I confirmed the gap was real and not a documentation oversight, the fix was straightforward: an external watchdog that checks how long the pod has sat idle and calls podStop once that idle window crosses a threshold I set.\nI didn\u0026rsquo;t invent this out of necessity. RunPod\u0026rsquo;s own cost-control guidance recommends exactly this shape: treat the GPU as fully ephemeral, let an external scheduler launch the pod, and have either the job itself or the scheduler call stop once the work is done.\nI\u0026rsquo;d already written a version of this watchdog for a different self-hosted GPU job, so this was mostly reuse.\nHere\u0026rsquo;s what the watchdog does, on a loop:\nflowchart LR A[Watchdog checks pod idle time] --\u003e B{Idle threshold exceeded?} B --\u003e|No| A B --\u003e|Yes| C[Call podStop via RunPod GraphQL API] C --\u003e D[Pod stopped, billing stops,volume data preserved] What I still haven\u0026rsquo;t proven # I\u0026rsquo;ve committed to dedicated-pod-plus-watchdog for now, but I haven\u0026rsquo;t run a real head-to-head between serverless and dedicated at my actual production volume yet.\nThe sizing and batching caveats from the vLLM community are enough to make me wary of trusting serverless vision inference on faith, so a dedicated pod with a watchdog is the safer default while that\u0026rsquo;s unverified. I could end up moving to serverless once I actually benchmark cold-start latency and per-image cost against what the watchdog setup gives me today.\nFor now, the dedicated pod is cheaper and the watchdog keeps it honest, but the whole arrangement still comes down to that same cron job watching the clock. I\u0026rsquo;d rather admit that\u0026rsquo;s a decision I haven\u0026rsquo;t fully stress-tested than pretend the comparison is closed.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/runpod-vs-gemini-vlm-inference-idle-auto-stop-gap/","section":"Blog Posts","summary":"Gemini wins on accuracy. RunPod wins on cost. I run vision-model inference for a personal image pipeline on RunPod GPUs instead of calling Google’s Gemini API, and that split is the whole decision, not a verdict on which model is smarter.\n","title":"RunPod Beats Gemini on Cost for My Vision Pipeline, and the Idle-Stop Feature It's Missing","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/vision-models/","section":"Tags","summary":"","title":"Vision Models","type":"tags"},{"content":"A bulk-reprocess job against one of my LightRAG instances crashed three times in one afternoon, and I shipped eight legitimate fixes before I found the actual cause. That same afternoon I also fixed a false-positive bug in the GPU broker that arbitrates my home GPU between gaming and local inference.\nThe two bugs had nothing to do with each other; they just landed on the same day. The honest version of this story is about the LightRAG crash alone: most of my fixes were correct, and none of them worked.\nThe crash looked like a concurrency problem, and the first fix was one # LightRAG is a knowledge-graph pipeline I run against a local Ollama embedding backend for a personal research project. I\u0026rsquo;d triggered its reprocess_failed endpoint against an 800-document backlog, and it kept dying with the same signature: an httpx.ReadError, then IndexFlushError, then Pipeline halted, cascading the entire in-flight batch to failed.\nA stray backup file on disk showed the cause: an earlier session had quietly raised MAX_ASYNC and MAX_PARALLEL_INSERT from 1 to 4, chasing throughput without realizing it would destabilize a local embedding backend. Community guidance is explicit that parallel-insert should stay well under async concurrency, not equal to it, and that gap matters more against a local model than a cloud API. (The same knobs, tuned against a rate-limited cloud API instead, got a post of their own.)\nI reverted both settings to 1. It was a real bug that had probably been causing failures for a while, but it wasn\u0026rsquo;t the crash.\nReverting concurrency didn\u0026rsquo;t stop the crash, so I chased connections next # The next run survived sixteen minutes instead of failing instantly, then died with a different-looking error: a stale connection reused after going dead. I added explicit idle timeouts on both sides of my broker\u0026rsquo;s HTTP handling.\nAlong the way I found a second real bug: Ollama\u0026rsquo;s embedding model was cold-starting every seven to twelve minutes, because idle gaps between embedding bursts routinely exceeded its five-minute keep-alive default, and every one of those reloads was hitting a missing ROCm library file on my GPU. I set a sixty-minute keep-alive to stop the reload cycling entirely.\nBoth fixes were correct diagnoses of real problems. But the crash came back anyway, at almost the same elapsed time, on a different document.\nThree more fixes landed on real mechanisms with nothing to do with the crash # I kept narrowing, three more fixes deep:\nA retry layer for connection-level failures on the broker\u0026rsquo;s outbound leg. Real hardening, but the retries never fired; the failure wasn\u0026rsquo;t happening on that leg at all. Removing an inbound idle timeout I\u0026rsquo;d added earlier, once I realized it was closing connections during LightRAG\u0026rsquo;s own multi-minute merge phases rather than protecting against staleness. Disabling connection reuse entirely on the broker\u0026rsquo;s batch server, so every request got a fresh TCP connection. Each was a legitimate correction. None changed the outcome. By fix eight I\u0026rsquo;d addressed concurrency, idle timeouts, a GPU driver bug, retry logic, and connection reuse, and the job still died in the same seventeen-to-thirty-seven-minute window every time. That consistency was the actual clue. Something systemic was setting the clock. I kept adjusting the wrong thing.\nHere\u0026rsquo;s the shape of the whole afternoon:\nflowchart TD A[Bulk reprocess job crashes] --\u003e B[Fix 1: revert concurrency 4 to 1] B --\u003e C[Crash persists, 16 min instead of instant] C --\u003e D[Fixes 2-3: idle timeouts, 60min keep-alive] D --\u003e E[Crash persists, same 17-37min window] E --\u003e F[\"Fixes 4-8: retry logic, timeout removal,connection-reuse disabled\"] F --\u003e G[Crash STILL persists, same window every time] G --\u003e H[\"Checked the host directly:NAS at \u0026lt;500MB free, 5GB+ in swap\"] H --\u003e I[\"Real cause: host OOM stalling networkunder memory pressure, not the app\"] I --\u003e J[Real fix: moved the workloadto a host with headroom] The host itself was out of memory # Checking the NAS\u0026rsquo;s own resource state directly settled it.\nThe box had 7.7GB of RAM, roughly 38 Docker containers running on it, and under 500MB genuinely free during a live run, with over 5GB in swap and the kernel\u0026rsquo;s swap-reclaim daemon burning real CPU just to keep everything upright. LightRAG\u0026rsquo;s own footprint was tiny, under 1.5GB, but it didn\u0026rsquo;t need to be large to get caught in the crossfire.\nUnder that kind of sustained memory pressure, the kernel can stall a process\u0026rsquo;s network handling unpredictably, and from either endpoint\u0026rsquo;s perspective that looks exactly like the other side vanished mid-response. No exception in my code, no crash log on Ollama\u0026rsquo;s side, nothing to grep for.\nEvery timing and connection fix I\u0026rsquo;d shipped was chasing a symptom that could show up anywhere the OS decided to stall. The real culprit was never in my code. It was 38 Docker containers fighting over 7.7GB of RAM, and losing.\nMoving the workload off the NAS fixed it # I migrated the LightRAG instance off the NAS onto a desktop machine with far more headroom, keeping every earlier hardening change in place. I hit one more mistake during the move.\nWarning Don\u0026rsquo;t point a migrated container at a loopback address, even when co-locating services on the same host. A container has its own network namespace, so 127.0.0.1 inside it isn\u0026rsquo;t the host\u0026rsquo;s loopback; it won\u0026rsquo;t reach a service the host itself is running. Use the host\u0026rsquo;s real local-network address instead.\nI\u0026rsquo;d reasoned that co-locating services meant loopback would work. It doesn\u0026rsquo;t, for the reason above.\nSwitching to the machine\u0026rsquo;s real local-network address fixed the connection immediately. The reprocess job then ran clean for fifty-two minutes, well past the worst crash point of thirty-seven, with steady progress and zero halts.\nI also owe a correction to my own process here. Partway through this, I declared an earlier fix verified after watching a run for thirty clean minutes, then stopped monitoring it to go write notes. The job crashed seven minutes later.\nThirty minutes of no errors isn\u0026rsquo;t proof of anything if you stop watching before the job finishes. I don\u0026rsquo;t think that mistake changes the eventual diagnosis, but it added a full extra round of debugging that a longer, unattended check would have skipped.\nThe GPU broker bug was a genuinely different problem, same day # The other bug that afternoon lived in a completely separate piece of code: the broker that decides when my shared GPU should yield away from local inference toward gaming or Plex. It was yielding every ten to twenty minutes around the clock, including at 1am, because its detector matched on a process name that Plex also runs for background maintenance work like intro-skip detection, not just during actual playback.\nThe fix was to stop pattern-matching on a process name and start asking Plex\u0026rsquo;s own session API whether anything is actually playing.\nI mention it here only because \u0026ldquo;one bad day\u0026rdquo; is the accurate frame: two real, unrelated bugs, fixed hours apart, that happened to share an afternoon.\nWhat I\u0026rsquo;m not sure about # I\u0026rsquo;ll admit the two bugs aren\u0026rsquo;t fully unrelated in one respect: both started from trusting a single signal without corroborating it, a log line in one case, a process-name match in the other. That\u0026rsquo;s a real pattern in how I was debugging that day, even though the bugs live in different systems.\nI\u0026rsquo;m also not confident I\u0026rsquo;ve found the true floor on the embedding-batch size that caused an earlier, secondary instability risk during that same debugging stretch. I picked two over ten and never bisected further. If that pipeline ever needs more throughput someday, I\u0026rsquo;ll have to go back and find the actual safe threshold properly, instead of just assuming two is magic.\nBut what I am confident about is the general lesson: when a fix addresses a real, verified mechanism and the crash still recurs on the same clock, stop tuning that mechanism and check what the host itself is doing.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/nine-fixes-lightrag-embedding-crash-one-afternoon/","section":"Blog Posts","summary":"A bulk-reprocess job against one of my LightRAG instances crashed three times in one afternoon, and I shipped eight legitimate fixes before I found the actual cause. That same afternoon I also fixed a false-positive bug in the GPU broker that arbitrates my home GPU between gaming and local inference.\n","title":"It Took Nine Fixes to Stop a LightRAG Crash. The First Eight Were All Real Bugs","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/lightrag/","section":"Tags","summary":"","title":"LightRAG","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/ollama/","section":"Tags","summary":"","title":"Ollama","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/rag/","section":"Tags","summary":"","title":"RAG","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/rate-limiting/","section":"Tags","summary":"","title":"Rate Limiting","type":"tags"},{"content":"Feeding a few hundred books into LightRAG through Gemini taught me that concurrency tuning is the wrong first lever, and that the rate-limit table you\u0026rsquo;d normally tune it against doesn\u0026rsquo;t exist anymore anyway. I run a personal knowledge-graph project that ingests close to a thousand book-length documents through LightRAG (HKUDS), using Gemini for entity extraction and embeddings behind a LiteLLM proxy. The corpus is entity-dense enough that the LLM merge phase dominates ingestion time.\nEarly runs kept marking documents FAILED, no obvious cause in the logs, no warning, just gone.\nThis post is what I found chasing that down: the actual concurrency knobs, why Gemini\u0026rsquo;s rate limits are now a moving target, and the one setting that mattered more than any of it.\nA Gemini 429 fails the whole document instead of retrying it # The failure mode is quiet, and that\u0026rsquo;s what makes it dangerous. When a Gemini call returns HTTP 429, LightRAG doesn\u0026rsquo;t queue the document and try again later. It marks the document FAILED and moves on. No crash, no page, nothing. Unless you\u0026rsquo;re watching the per-document status table, you won\u0026rsquo;t notice until the corpus finishes and a chunk of it is just missing from the graph.\nThat\u0026rsquo;s exactly what happened on my first real run against this corpus: documents dropped out of the pipeline looking, from a distance, like success.\nDo LightRAG\u0026rsquo;s concurrency knobs control your Gemini rate limit? # Four environment variables govern ingestion concurrency in LightRAG. I ended up trusting the source over the docs prose to actually understand them:\nMAX_ASYNC_LLM: concurrent LLM calls (extraction, merge, keyword generation, answer synthesis). Default 4. MAX_PARALLEL_INSERT: documents processed in parallel. Default 3; LightRAG\u0026rsquo;s own env.example recommends keeping it near MAX_ASYNC_LLM / 3. EMBEDDING_FUNC_MAX_ASYNC: concurrent embedding calls, on a separate pool from the LLM pool. Default 8. EMBEDDING_BATCH_NUM: chunks bundled into one embedding request. Default 10. The project\u0026rsquo;s documented high-throughput profile is MAX_ASYNC_LLM=8, MAX_PARALLEL_INSERT=3, EMBEDDING_FUNC_MAX_ASYNC=16, EMBEDDING_BATCH_NUM=32, and a real-world test in LightRAG issue #2264 using a similar profile took ingestion from 7 hours 8 minutes down to 1 hour 45 minutes on the same corpus, a legitimate 4x.\nBut that ratio only describes how LightRAG should divide work internally. It says nothing about how much total work your Gemini project is allowed to accept per minute, and that ceiling is the one that actually throws the 429s.\nGemini\u0026rsquo;s rate limit isn\u0026rsquo;t a table you can hardcode anymore # Google stopped publishing a static per-model rate-limit table as of July 2026. The Gemini API rate-limit docs now say limits depend on your project\u0026rsquo;s usage tier and are \u0026ldquo;not guaranteed,\u0026rdquo; which in practice means you read the live number out of AI Studio for your specific project before you tune anything.\nThat was a real adjustment for me: I\u0026rsquo;d been treating rate limits like a spec you design against once. They\u0026rsquo;re now closer to a runtime condition you check on the way in. Free and early-tier flash access is often in the 10-15 RPM range, which makes MAX_ASYNC_LLM=8 from the \u0026ldquo;official\u0026rdquo; profile actively dangerous rather than aspirational.\nThere\u0026rsquo;s also a second, independent limiter on paid tiers: a spend-based burst cap over a rolling 10-minute window, separate from the RPM/TPM ceiling.\nYou can sit well under your requests-per-minute limit and still get 429\u0026rsquo;d by the burst cap. The derivation that actually holds up: set MAX_ASYNC_LLM to roughly your live RPM times average call latency in seconds, divided by 60. Flash\u0026rsquo;s latency runs 1-3 seconds per call, so a 10 RPM tier caps you at 2-4 concurrent calls, while a paid tier with thousands of RPM lets you approach the documented profile.\nInsert parallelism and embedding pool size both derive from that number. They don\u0026rsquo;t set it. Tune to the ratio first and you\u0026rsquo;re tuning against a number that doesn\u0026rsquo;t reflect your actual ceiling.\nHere\u0026rsquo;s the derivation chain end to end, tuning knobs plus the absorb layer:\nflowchart TD A[\"Check live RPM from AI Studio,not a hardcoded table\"] --\u003e B[\"MAX_ASYNC_LLM = live RPM x latency(s) / 60\"] B --\u003e C[MAX_PARALLEL_INSERT derives from ratio] B --\u003e D[EMBEDDING_FUNC_MAX_ASYNC derives from ratio] E[EMBEDDING_BATCH_NUM: fix leftover local-GPU value] --\u003e D B --\u003e F[\"LiteLLM router: rpm/tpm caps + RateLimitErrorRetries\"] F --\u003e G[\"429 becomes a delayed retry,not a FAILED document\"] Fixing the embedding batch size mattered more than any concurrency change # My container had EMBEDDING_BATCH_NUM set to 2, a leftover from an earlier era when embeddings ran on a local GPU model instead of Gemini\u0026rsquo;s hosted embedding API. Against a local model, batch size barely matters; you\u0026rsquo;re not paying per request.\nAgainst a rate-limited cloud API, batch size 2 versus the recommended 32 means sixteen times more embedding requests for the exact same corpus, and sixteen times more pressure on the embedding RPM ceiling for zero benefit. Fixing that one line did more for my 429 rate than any concurrency change did, with no downside: same total work, dramatically fewer requests.\nIf you\u0026rsquo;re moving a LightRAG setup from a local embedder to a cloud one, check this value before you touch anything else.\nLightRAG issue #1648 is a useful reality check here too: someone running a 50,000-document ingest with a conservative embedding concurrency of 5 still hit 429s on the embedding service. Low concurrency lowers the odds of hitting a ceiling. It doesn\u0026rsquo;t eliminate them: a single misconfigured batch size can undo the benefit entirely.\nThe proxy layer needs to absorb overshoot instead of failing documents # Concurrency limits are a best-effort guess at the ceiling, and best-effort guesses are sometimes wrong. The real fix doesn\u0026rsquo;t come from a more precise guess. It comes from making the failure mode survivable when the guess is wrong.\nThe same absorb-don\u0026rsquo;t-fail principle drove the request-parking layer I built for my local embedding broker; here the absorbing layer is the proxy. LiteLLM\u0026rsquo;s router supports rpm and tpm caps per model in its model_list, and if you don\u0026rsquo;t set max_parallel_requests explicitly it derives concurrency from those numbers automatically.\nIt also supports a retry_policy with a dedicated RateLimitErrorRetries count, separate from timeout or server-error retries. That\u0026rsquo;s the setting that actually matters here: a 429 that hits LiteLLM with that policy configured gets retried with backoff instead of surfacing as an error LightRAG has to interpret.\nSet those caps to your project\u0026rsquo;s real live limits, add the retry policy, and a burst that exceeds your ceiling turns into a delayed request instead of a failed document. Skip that layer and every concurrency tweak is a bet that you never overshoot. Eventually you will.\nRunning LiteLLM with multiple worker processes? rpm/tpm counters need to be backed by Redis to be shared across workers; otherwise each worker enforces the cap independently, and your real aggregate concurrency against Gemini is a multiple of what you configured. I haven\u0026rsquo;t needed multi-worker LiteLLM for this corpus size, so I can\u0026rsquo;t speak to how much that matters in practice, but it\u0026rsquo;s a documented gap worth knowing about before you scale up.\nWhat I\u0026rsquo;d push back on in my own conclusion # The uncomfortable part of this whole exercise: the \u0026ldquo;optimal ratio\u0026rdquo; LightRAG documents is close to useless without knowing your live rate limit first, which makes it feel like the wrong place to have started. I could argue I wasted time reading env.example line by line when the actual fix was one line in a docker-compose file.\nI don\u0026rsquo;t think that\u0026rsquo;s quite right, though. The ratio still matters once you know your ceiling: it tells you how to divide a fixed budget of concurrent calls between insertion and embedding, rather than just picking a number.\nWhat I\u0026rsquo;m genuinely unsure about is whether the entity-merge phase\u0026rsquo;s partial serialization (the same GitHub issue that got the 4x speedup also reported the GPU sitting underutilized during ingestion) is a bigger long-term bottleneck than rate limits for a corpus this size. I haven\u0026rsquo;t run the numbers on a from-scratch full reingest with the fixed batch size and proxy guardrails in place. That\u0026rsquo;s a real open question, not a settled one.\nIf you\u0026rsquo;re running LightRAG against any rate-limited cloud LLM, check three things before you touch a single concurrency variable:\nYour embedding batch size Your provider\u0026rsquo;s live rate limit for your actual tier Whether your proxy retries 429s or just lets them through Concurrency tuning is the part that feels like engineering. Getting those three right is the part that actually keeps documents from quietly turning FAILED while you\u0026rsquo;re not looking.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/tuning-lightrag-ingestion-concurrency-against-gemini-rate-limits/","section":"Blog Posts","summary":"Feeding a few hundred books into LightRAG through Gemini taught me that concurrency tuning is the wrong first lever, and that the rate-limit table you’d normally tune it against doesn’t exist anymore anyway. I run a personal knowledge-graph project that ingests close to a thousand book-length documents through LightRAG (HKUDS), using Gemini for entity extraction and embeddings behind a LiteLLM proxy. The corpus is entity-dense enough that the LLM merge phase dominates ingestion time.\n","title":"Tuning LightRAG Ingestion Concurrency Against a Rate-Limited Gemini API","type":"blog"},{"content":"Every embedding server I tested handles a vanished GPU the same way: queue requests until a buffer fills, then reject them. Ollama does this. TEI does this. Infinity and llama.cpp do it too, with different buffer sizes and different error codes but the same outcome. None of them pause a request and wait out a short outage; they drop it the moment the queue overflows or a limit is hit.\nI run one GPU at home across gaming, media transcoding, and every local model behind my personal tools, and a broker process decides who gets the card and when. That gap between reject-fast and wait-it-out is what forced me to build the missing layer myself. Nobody else was going to pause a request while my GPU stepped away to render frames for a game instead.\nThe shared GPU has to change hands, and that\u0026rsquo;s the actual problem # My home GPU juggles three tiers of work:\nInteractive chat: needs an answer in seconds. Batch jobs like embeddings: can tolerate a delay. Long-running jobs: can wait minutes. A broker I run arbitrates between them (the same broker whose phantom-game detection bug got its own post). When gaming or a higher-priority job needs the card, the broker yanks it away from whatever lower-priority work was using it.\nThat yield might last a few seconds, or a couple of minutes. Nothing about the GPU itself failed. It\u0026rsquo;s just busy elsewhere for a bounded window, and any request caught mid-flight has to survive that window instead of dying because of it.\nNo shipping server treats a busy GPU as temporary # I went looking for prior art before writing a line of this. The pattern held across every tool I checked:\nOllama\u0026rsquo;s queue (OLLAMA_MAX_QUEUE, default 512) holds requests FIFO and returns a 503 once it\u0026rsquo;s full. TEI\u0026rsquo;s --max-concurrent-requests flag is explicit reject-fast backpressure by design. Infinity and llama.cpp follow the same logic with their own limits. All of them treat a full queue as a hard stop rather than something to wait out. That\u0026rsquo;s a reasonable default for a public-facing server fielding requests from strangers. It\u0026rsquo;s the wrong default for a private broker that knows exactly why the GPU is unavailable and roughly how long the wait will be.\nLightRAG has no protection of its own, so it has to come from below # I run LightRAG for a knowledge-graph project (the same one whose ingestion concurrency I tuned separately). It talks straight to an embedding backend with no retry logic and no backpressure of its own. The maintainers\u0026rsquo; fix for slow embed calls is to set TIMEOUT=None and disable the timeout entirely, rather than add retries.\nThree separate open issues track embed failures during batch ingest across different backends, and one traces directly to an embed call timing out mid-ingest. None of that gets fixed inside LightRAG. Whatever protection exists has to sit underneath it, in whatever actually talks to the GPU. That\u0026rsquo;s why this logic belongs in the broker instead of waiting on some upstream project to add it.\nlitellm\u0026rsquo;s Router solves a different problem than mine # The closest thing to a real solution I found was litellm\u0026rsquo;s Router, which supports fallback, cooldown, and timeout configuration for embedding calls. It\u0026rsquo;s a useful primitive I\u0026rsquo;d reach for if I ever wanted a second embedding backend to fail over to. But its timeout wraps the entire call including retries, rather than each individual attempt inside it. Backend selection is what it solves. Waiting for one backend to come back online is a different problem.\nI also checked two open-source Ollama proxies: Olla (roughly 260 stars, actively maintained) and ollamaMQ (roughly 114 stars, a fair-share queue proxy written in Rust). Both are solid queueing and failover tools. Neither parks a request through an outage and replays it once the outage ends. That\u0026rsquo;s the specific behavior I needed, and nothing I found already did it.\nThe fix: park requests instead of rejecting them # The fix lives in the fronting proxy inside my broker, one layer above Ollama. When a yield starts, batch-class synchronous requests (in practice, embeddings) get parked instead of bounced:\nHold bound: 600 seconds by default. Parked-queue ceiling: past it, the broker returns a fast 503. That\u0026rsquo;s the same reject-fast principle TEI already applies, just moved up a layer instead of reinvented. Replay: when the yield ends, parked requests replay in FIFO order with a cap on how many go out at once, so the queue doesn\u0026rsquo;t dump a burst back onto Ollama the instant the GPU returns. Metrics: Prometheus gauges for parked depth, time spent parked, and replay outcomes, plus an alert rule. TEI already treats queue depth as worth exposing, so I didn\u0026rsquo;t see a reason to do less. 600 seconds is comfortably under LightRAG\u0026rsquo;s own 1200-second embedding timeout, so a parked request never expires on the caller\u0026rsquo;s side while it\u0026rsquo;s still waiting on mine. The path a request takes once a yield starts:\nflowchart LR A[Embedding request arrives] --\u003e B{GPU yielded tohigher-priority work?} B --\u003e|No| C[Serve immediately] B --\u003e|Yes| D{Parked queue below cap?} D --\u003e|No| E[Fast 503, reject] D --\u003e|Yes| F[Park request, up to 600s] F --\u003e G[Yield ends] G --\u003e H[\"Replay parked requests FIFO,capped rate\"]Whether 600 seconds is the right number, I\u0026rsquo;m honestly not sure. It\u0026rsquo;s tuned to my current yield patterns. If a yield ever runs long for a reason the broker doesn\u0026rsquo;t already know about, that bound will need to move.\nI haven\u0026rsquo;t turned on the CPU fallback I built # There\u0026rsquo;s an obvious alternative to parking: fall back to a CPU-based embedding model during a yield instead of making anything wait. That path is built, but I\u0026rsquo;m leaving it off by default.\nI don\u0026rsquo;t trust CPU fallback as a silent switch. I\u0026rsquo;ve seen it misbehave unpredictably, and a LightRAG issue reports CPU-only embedding backends behaving badly specifically inside LightRAG\u0026rsquo;s pipeline, well beyond just running slow. Before I flip that flag on, I want to smoke-test it through LightRAG\u0026rsquo;s actual embedding function, the real call path it uses during ingest. A prompt-response check alone won\u0026rsquo;t tell me enough. A silent, unverified fallback is worse than an honest wait.\nNext: proving the parking logic survives a real embed burst # The parking logic passes against requests I send it directly, one at a time. What it hasn\u0026rsquo;t seen yet is a forced yield in the middle of a real embed burst. That\u0026rsquo;s the exact failure mode this whole thing exists to survive, and the test is next:\nTrigger a yield artificially while LightRAG is mid-ingest. Confirm zero failures on the caller\u0026rsquo;s side. Add it to the broker\u0026rsquo;s regression suite so it can\u0026rsquo;t quietly break later. Until that runs, this is a design I believe in, not one I\u0026rsquo;ve fully verified under load.\nRunning an embedding server behind a shared GPU at home? Check for this gap yourself. Query your server\u0026rsquo;s own queue limit, and ask what happens to a request sitting in that queue when the GPU it\u0026rsquo;s waiting on disappears for reasons the server itself doesn\u0026rsquo;t control. In every server I checked, the answer was the same: it dies.\nMine doesn\u0026rsquo;t anymore. The GPU still steps away for gaming whenever gaming wins the tiebreak, and that\u0026rsquo;s fine. That\u0026rsquo;s what the burst test above will decide. If it turns up a problem, the 600-second bound moves before I let this run unattended.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/surviving-a-gpu-yield-window-embedding-servers/","section":"Blog Posts","summary":"Every embedding server I tested handles a vanished GPU the same way: queue requests until a buffer fills, then reject them. Ollama does this. TEI does this. Infinity and llama.cpp do it too, with different buffer sizes and different error codes but the same outcome. None of them pause a request and wait out a short outage; they drop it the moment the queue overflows or a limit is hit.\n","title":"No Embedding Server Survives a GPU Yield Gracefully. I Had to Build That Layer Myself","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/go/","section":"Tags","summary":"","title":"Go","type":"tags"},{"content":"My GPU broker kept canceling live inference jobs over games that weren\u0026rsquo;t running. Most of the time, nothing had launched at all.\nThe service is a Go broker I run at home that arbitrates my desktop\u0026rsquo;s single GPU between gaming, Plex transcoding, and local inference through Ollama; the same broker later grew a parking layer for embedding requests caught mid-yield. When it detects gaming or Plex activity, it force-cancels whatever inference is running and unloads the model from VRAM, no exceptions.\nIn my house, whoever\u0026rsquo;s playing a game or watching something wins that argument. That priority order is correct. The detector deciding when to enforce it was not.\nI found the bug while chasing a different crash: the LightRAG embedding crash that took nine fixes to actually stop. A bulk ingestion job that leans on the broker for embeddings kept dying partway through with a read error on the Ollama calls, which cascaded into a full pipeline halt. Nothing in the job\u0026rsquo;s own code looked wrong.\nChecking the broker\u0026rsquo;s logs during the failure windows turned up the real problem: it kept flipping into a \u0026ldquo;yielding\u0026rdquo; state with nothing running.\nThe broker flipped to yielding roughly every 10 to 20 minutes, around the clock, including the 1am to 6am stretch when nobody in this house was playing anything. ps aux during one of those windows showed exactly one candidate: Steam\u0026rsquo;s idle background client, doing nothing more incriminating than existing in the process table.\nA single matching process was enough to cancel a running job # One matching line in /proc was enough to kill a running job. The detector polls /proc every three seconds for command-line substrings:\nPlex Transcoder Steam\u0026rsquo;s launch marker Heroic\u0026rsquo;s and Lutris\u0026rsquo;s runner patterns a bare wine .exe The moment any one poll matched, the controller flipped to yielding and canceled whatever inference was in flight.\nThere was no debounce (the industry term for waiting out a signal before trusting it) and no second signal to corroborate the first. One sample counted as ground truth. That design wasn\u0026rsquo;t an oversight so much as an unexamined assumption: I\u0026rsquo;d built the hard-cancel policy deliberately, then never asked whether the thing triggering it deserved that much trust.\nPlex\u0026rsquo;s own maintenance jobs look identical to real playback # Plex\u0026rsquo;s own support documentation confirms that Skip Intro and Credits detection, along with chapter-thumbnail generation, run as scheduled server maintenance through the same Plex Transcoder binary that handles real playback, on a cadence that has nothing to do with anyone pressing play. My detector grepped for that process name, so a 3am maintenance pass looked exactly like me starting a movie.\nNo amount of debounce timing fixes this: the false match isn\u0026rsquo;t a brief blip, it can run for several minutes at a stretch. Tautulli, a widely used third-party Plex monitoring tool, sidesteps the problem by reading Plex\u0026rsquo;s /status/sessions API instead of the process table, since that endpoint only reports sessions that are actually \u0026ldquo;now playing.\u0026rdquo; The real fix for the Plex side: stop grepping for the binary and ask Plex what\u0026rsquo;s actually playing.\nNo game launcher exposes a real \u0026ldquo;foreground game\u0026rdquo; signal # The gaming side is a different problem: I can\u0026rsquo;t fix it by finding a better API, because none exists. Steam\u0026rsquo;s overlay APIs report whether the overlay is active, not whether a game is running in the foreground. Heroic and Lutris expose no equivalent signal at all.\nProcess-name matching is the only practical option left for gaming detection. The logs showed those false matches clustering in three-to-six-second blips, much shorter than Plex\u0026rsquo;s multi-minute stretches: different noise shape, different fix.\nConfirmation only gates the cancel # The fix makes the broker demand confirmation before it cancels a job, but not before it recovers from one. Here\u0026rsquo;s the actual change, before and after:\nflowchart LR subgraph Before[\"Before: single-poll trigger\"] A1[Poll /proc every 3s] --\u003e A2{Any match?} A2 --\u003e|1 match| A3[Cancel inference immediately] end subgraph After[\"After: debounced trigger\"] B1[Poll /proc every 3s] --\u003e B2{Match?} B2 --\u003e|1st match| B3[Wait for confirmation] B3 --\u003e B4{2-3 consecutive matches?} B4 --\u003e|Yes| B5[Cancel inference] B4 --\u003e|No, false blip| B6[Ignore, keep running] endFor the gaming side, the fix is the debounce pattern I should have had from the start: require several consecutive positive polls before flipping to yielding, instead of trusting a single one. I set the default at two or three consecutive matches.\nRecovery, the transition back out of yielding, stays instant and undebounced. Delaying it only costs a few extra seconds of inference downtime, and never risks letting inference run over an actual game. That asymmetry is deliberate: the two directions carry different failure costs. A genuine game launch now takes a few seconds longer for the GPU to free up, a small price against jobs dying for no reason.\nI\u0026rsquo;ve only shipped half of this fix. The poll-confirmation gate is small, self-contained, and went in first. The Plex session-API swap hasn\u0026rsquo;t happened yet. It needs a token Plex issues locally, and I haven\u0026rsquo;t wired that up. Until I do, a multi-minute Plex maintenance run will still trip the broker no matter how high I set the confirm-poll count: debounce only filters single-sample noise, and does nothing against a signal that stays true for five straight minutes.\nI\u0026rsquo;m also not confident two or three polls is the right number for every workload this machine runs. I picked it from a general flapping-detection convention rather than from measurement on my own logs. I won\u0026rsquo;t know if it\u0026rsquo;s wrong until the false positives either stop or don\u0026rsquo;t.\nHard-canceling instead of throttling is a defensible but costly choice # Hard-canceling instead of throttling is the right call for my house, and it\u0026rsquo;s also why this bug turned into a real pipeline outage instead of a minor annoyance. My broker treats every real contention event as a hard stop:\ncancel the inference request unload the model hand the GPU over completely Process Lasso does something closer to priority scheduling instead, deprioritizing background compute rather than killing it outright when a game starts. That approach would have made this whole bug far less painful: a false positive would have meant a slower inference request instead of a canceled one.\nI built it as a hard cutover on purpose. I wanted a guarantee that the GPU comes back completely clean the moment someone in this house wants to play, and priority-based throttling can\u0026rsquo;t promise that as cleanly. I still think that tradeoff was right for a shared family machine.\nThe debounce fix is live; the Plex fix isn\u0026rsquo;t. I\u0026rsquo;ll find out whether either was tuned right the next time this job runs unattended overnight, and either survives or it doesn\u0026rsquo;t.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/debugging-false-positive-gpu-contention-detection/","section":"Blog Posts","summary":"My GPU broker kept canceling live inference jobs over games that weren’t running. Most of the time, nothing had launched at all.\n","title":"My GPU Broker Kept Killing Inference Jobs for Games That Weren't Running","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/categories/case-study/","section":"Categories","summary":"","title":"Case Study","type":"categories"},{"content":"My resale-clothing monitor\u0026rsquo;s hardest problem isn\u0026rsquo;t finding new listings. It\u0026rsquo;s deciding which ones fit my taste well enough to interrupt me over, and the design leans hard toward false positives over false negatives, a call I can defend today but haven\u0026rsquo;t actually stress-tested.\nThree posts make up this series: part 1 covers the shared architecture, part 2 covers a sibling project, an estate-sale scanner, running on the same foundation, and this is the third.\nA free rules layer rejects most listings before any model sees them # The monitor watches several secondhand clothing marketplaces. Before any model ever sees a listing, a pre-filter of deterministic rules runs first:\nA fast-fashion brand blocklist. A per-brand minimum plausible price (a \u0026ldquo;designer\u0026rdquo; item priced far below that floor is usually a knockoff). A price ceiling by category. Those three rules eliminate roughly 40 to 60 percent of raw listings for free, before any model call. Size is never a hard-reject rule: sizing across resale platforms is too unreliable to gate on mechanically.\nBrands run differently. Cuts vary. Sellers mislabel. So instead of a brittle \u0026ldquo;reject anything not size L\u0026rdquo; rule, the raw size text and any stated measurements get passed to the model as a soft signal. Measurements in the description always override the label.\nScoring is two passes, and only the ambiguous cases get the expensive one # Every new listing goes through a local model first, batched 15 to 20 listings per call. Each listing carries:\nTitle Brand A truncated description Price Condition Size That batch range matters: fewer wastes the fixed cost of the system prompt, and past roughly 30 the model\u0026rsquo;s attention starts to degrade.\nThe model returns a verdict, YES, MAYBE, or NO, across three independent dimensions (quality, value, aesthetic), plus a separate size read.\nOnly listings that come back MAYBE and have a usable image go to a second pass with a vision-capable model. But a MAYBE with no resolvable image isn\u0026rsquo;t dropped: it stays a MAYBE and surfaces at lower confidence. A parse error or malformed model output defaults the same way.\nThe provider for each pass, local, cloud, or a hybrid, sits behind one interface, so which backend runs a given scoring pass is a config change, not a code change.\nOnce a listing has a real score, it\u0026rsquo;s never re-scored. That alone is the single biggest cost reduction in the pipeline, ahead of anything model-related.\nHere\u0026rsquo;s the scoring pipeline a listing moves through:\nflowchart TD A[New listing] --\u003e B{\"Rules pre-filter:brand blocklist, price floor/ceiling\"} B --\u003e|Rejected, 40-60%| C[Discarded, free] B --\u003e|Passed| D[Local model, batched 15-20/call] D --\u003e E{Verdict per dimension} E --\u003e|NO| C E --\u003e|YES| F[Surfaced as alert] E --\u003e|MAYBE + usable image| G[Vision model, second pass] E --\u003e|MAYBE, no image| H[Surfaced at lower confidence] G --\u003e F The bias toward false positives has no counterweight yet # Missing a genuinely good item is worse than one extra alert I dismiss in two seconds. For a system with one user and nothing riding on a bad alert, I still think that\u0026rsquo;s the right call.\nBut it doesn\u0026rsquo;t push back against alert volume creeping up as more edge cases land in MAYBE instead of NO over time, and nothing in the current design notices that drift or corrects for it. If this ever had to serve more than one household, that gap would be the first thing I\u0026rsquo;d have to actually solve instead of shrug at.\nFeedback splits into two tiers with different lifespans # Every run, the system prompt gets appended with a rotating set of my most recent thumbs-up and thumbs-down reactions to past alerts. The effect is staged:\nUnder 10 feedback events: nothing measurable. 10 to 25: a noticeable improvement. 25 to 50: strong calibration. Past 50: the oldest examples age out in favor of recent ones. Separately, I can hand-write known-good and known-bad example items directly into config. Those never rotate out, and exist so the system has some ground truth before a single real alert has fired.\nOne migration broke three things at once, silently # The incident I\u0026rsquo;d flag hardest here broke silently across three places at once. The monitor\u0026rsquo;s alerts and feedback originally rode the same channel: a chat bot where a thumbs-up or thumbs-down tap wrote straight back to the feedback table, no extra infrastructure needed.\nWhen the design changed to stop depending on that bot\u0026rsquo;s webhook, the alert transport got swapped to a self-hosted push service in one large change. But the feedback-ingestion path got gutted down to a disabled stub with no replacement wired up yet, while the docs of record still described the old bot.\nFor a stretch, the code, the deployment config, and the docs each told a different story about how alerts and feedback worked, and the system\u0026rsquo;s only learning mechanism sat fully severed with no error to say so. The fix: a proper API endpoint on the dashboard, no more depending on a chat bot\u0026rsquo;s callback.\nWhere both projects\u0026rsquo; open questions actually meet # Both this project\u0026rsquo;s MAYBE-drift and the estate scanner\u0026rsquo;s cascade-complexity doubt (in part 2) share a shape I didn\u0026rsquo;t notice until writing all three of these posts back to back: every incident across both systems announced itself eventually, through a dashboard that looked stale or a log that looked suspiciously clean. Neither open question has that kind of tell.\nTwo things wouldn\u0026rsquo;t announce themselves at all:\nAlert volume creeping up over months as more edge cases land in MAYBE instead of NO. A feedback loop gradually reinforcing a preference I don\u0026rsquo;t actually hold anymore. I\u0026rsquo;d have to notice it myself, on some Saturday, looking at a list that feels a little worse than it used to for reasons I can\u0026rsquo;t immediately name. I haven\u0026rsquo;t built anything that would catch it sooner than that, and I don\u0026rsquo;t have a good reason why not beyond not having hit it yet.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/deciding-what-fits-resale-clothing-monitor/","section":"Blog Posts","summary":"My resale-clothing monitor’s hardest problem isn’t finding new listings. It’s deciding which ones fit my taste well enough to interrupt me over, and the design leans hard toward false positives over false negatives, a call I can defend today but haven’t actually stress-tested.\n","title":"Deciding What Fits: Inside My Resale-Clothing Monitor","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/local-llm/","section":"Tags","summary":"","title":"Local LLM","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/typescript/","section":"Tags","summary":"","title":"TypeScript","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/computer-vision/","section":"Tags","summary":"","title":"Computer Vision","type":"tags"},{"content":"My estate-sale scanner has one job: decide which of a week\u0026rsquo;s new listings deserve an actual Saturday. The scraping is boring. What\u0026rsquo;s interesting is how the system scores photos nobody\u0026rsquo;s labeled, and an asymmetric feedback loop that treats a good sale and a bad sale as completely different kinds of evidence. This is part 2 of a series; part 1 covers the shared architecture and GPU constraints behind this project and a second one.\nEvery photo runs through four cheap gates before any paid model call # The scanner pulls new listings from a regional aggregator once a week, then runs each photo through a pipeline in order:\nPerceptual-hash dedup. Catches the same photo re-uploaded across listings. A quality gate. Brightness and blur checks, cheap and CPU-only, drop photos too dark or blurry to read. A free local pre-filter. A small local Ollama call answers PASS or SKIP on things like empty rooms, driveways, or cardboard boxes, before any money gets spent on a stronger model. It\u0026rsquo;s fail-open: if the model call errors, the photo passes through anyway. An outage never suppresses a real find, it just costs more that week. Full vision analysis. Either a local Ollama model or, for volume, a hosted GPU endpoint running a larger vision-language model. The pipeline as an actual flow:\nflowchart TD A[New listing photo] --\u003e B[Perceptual-hash dedup] B --\u003e C[Quality gate: brightness/blur] C --\u003e D[\"Free local pre-filter (fail-open)\"] D --\u003e|PASS or error| E[Full vision analysis] D --\u003e|SKIP| F[Discarded] E --\u003e G[Item list: maker, era, materials, condition, confidence]I tell the model what I collect: quality furniture and antiques, kitsch and camp collectibles, vintage electronics. It lists each item with a maker guess, era, materials, condition, and a confidence tag:\nDanish teak side table, likely 1960s, veneer chip on one corner [high] Chalkware TV lamp, mid-century, black light wear on base [medium] NOTHING Plain text, not JSON. An internal comparison found the plain-text format caught meaningfully more real items than forcing the same model into strict JSON output.\nTwo separate scores exist because they answer two separate questions # One score decides whether the rest of a sale\u0026rsquo;s photos are worth analyzing at all. It\u0026rsquo;s pure cost control: it decides how many model calls a sale gets, not whether any single item is good. The scanner processes the first quarter of a sale\u0026rsquo;s photos, then branches:\nStrong results → run the rest of the sale. Empty results → spot-check a handful from later in the listing before giving up on the sale entirely. A second, separate display score is what the dashboard actually sorts by, built from three inputs: a curated brand list, era keywords, and the model\u0026rsquo;s own confidence tag. The budget heuristic optimizes for not wasting calls on a dead sale; the display score optimizes for what to look at first. Conflate the two, and cheap sales start looking worse than they actually are.\nA \u0026ldquo;waste\u0026rdquo; outcome teaches the system more than a \u0026ldquo;good\u0026rdquo; one does # After visiting a sale, I log an outcome: good, meh, or waste. That single decision is the anti-overfit design for this whole project, and it\u0026rsquo;s deliberately lopsided.\nA \u0026ldquo;waste\u0026rdquo; outcome propagates in bulk. Every photo from that sale becomes a confirmed-negative training example, because \u0026ldquo;the whole sale was junk\u0026rdquo; is a clean, complete signal.\nA \u0026ldquo;good\u0026rdquo; outcome doesn\u0026rsquo;t get the same treatment. It only proves something there was worth it, not which item — auto-labeling the whole sale would teach a future ranker that the box of tube socks next to the good chair was also desirable. So positive labels only get created when I tap the specific item that earned the trip. It\u0026rsquo;s slower to build a clean positive set this way, but a small, correct one beats a large, contradictory one.\nThere\u0026rsquo;s also a real ground-truth run behind the scenes. Occasionally I run every surviving photo from a batch of sales through the strongest model available, no sampling, no budget limit, and treat that as the reference answer. Comparing a cheaper run\u0026rsquo;s recall against that reference is how I picked a monthly spend target instead of guessing at one.\nThe tiered cascade\u0026rsquo;s complexity is the part I\u0026rsquo;m least sure about # The reference-pass math tells me the cheap tiers catch most of what the expensive tier would have found, which is the number I actually wanted. It doesn\u0026rsquo;t tell me whether a dumber two-tier version, a quality gate plus one model call, would have caught nearly as much for a lot less engineering. I never built that version to find out.\nThe tiered design looks rigorous because I can point at a recall number that justifies it. I\u0026rsquo;d be lying if I said that number wasn\u0026rsquo;t also the thing that let me stop second-guessing myself and ship it.\nThree failures that never threw an error # Every incident here shares one shape: nothing crashed, nothing logged an error, and the system kept looking healthy from the outside while quietly doing the wrong thing.\nThe free pre-filter asks a model to answer with exactly one word, PASS or SKIP, within a small token budget. The model in use is reasoning-tuned: it spends part of that budget thinking before it answers, and at the original budget it never got past its own reasoning. Every single call came back with an empty response.\nBecause the fail-open logic treats anything that isn\u0026rsquo;t literally SKIP as a pass, the gate silently passed everything, every time, for an unknown stretch. Fixed by raising the token budget and telling the model explicitly to skip its reasoning step. It\u0026rsquo;s now the first thing checked whenever this project swaps in a new model.\nWorse, a run could fail completely and still report success. Per-image failures were counted internally but never surfaced anywhere or reflected in the run\u0026rsquo;s exit status.\nA week where every single paid vision call failed still logged \u0026ldquo;scan complete, 0 findings\u0026rdquo; and exited clean — indistinguishable from a genuinely quiet week, despite real money spent on every failed call. The fix split \u0026ldquo;found nothing\u0026rdquo; into three honest, differently-alarmed outcomes:\nGenuinely nothing found. The source site\u0026rsquo;s page structure likely changed. The vision backend failed enough calls that the count can\u0026rsquo;t be trusted. For a period, the scan ran on one machine and served the dashboard from a different one, each with its own separate copy of the same SQLite file. The dashboard was quietly showing stale results relative to what the last real scan had actually found, with nothing anywhere to flag that the two had diverged.\nFixed by consolidating both onto a single always-on host — the kind of bug that\u0026rsquo;s obvious in hindsight and invisible while it\u0026rsquo;s happening.\nI don\u0026rsquo;t have a general fix for this class of bug beyond looking harder at exactly the places I\u0026rsquo;m most tempted to assume are fine, and I\u0026rsquo;m not confident I\u0026rsquo;ve caught the last one.\nPart 3 covers the resale-clothing monitor, its own scoring problem, and where the two projects\u0026rsquo; open questions actually converge.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/deciding-whats-worth-a-saturday-estate-sale-scanner/","section":"Blog Posts","summary":"My estate-sale scanner has one job: decide which of a week’s new listings deserve an actual Saturday. The scraping is boring. What’s interesting is how the system scores photos nobody’s labeled, and an asymmetric feedback loop that treats a good sale and a bad sale as completely different kinds of evidence. This is part 2 of a series; part 1 covers the shared architecture and GPU constraints behind this project and a second one.\n","title":"Deciding What's Worth a Saturday: Inside My Estate-Sale Scanner","type":"blog"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/sqlite/","section":"Tags","summary":"","title":"SQLite","type":"tags"},{"content":"Two personal tools I\u0026rsquo;ve built, an estate-sale scanner and a resale-clothing monitor, run on the exact same architecture: scrape listings, score every photo with a local vision model, surface only the ones worth my attention. Same four stages, same database, same GPU, underneath two very different projects.\nThis post covers that shared foundation and the two decisions in it I\u0026rsquo;m still not fully sure were right. Two follow-ups go deep on how each project decides what actually counts as a match: Deciding what\u0026rsquo;s worth a Saturday for the estate-sale scanner, and Deciding what fits for the resale monitor.\nOne pipeline, two projects, no message queue # Both projects are the same four stages, talking to each other through a single SQLite database instead of a queue:\nflowchart LR A[Listings site] --\u003e B[\"Scrape(new items)\"] B --\u003e C[\"Prefilter(free, no LLM cost)\"] C --\u003e D[\"LLM / Vision Score(Ollama + cloud escalationfor hard cases)\"] D --\u003e E[\"Alertdashboard / push\"] B -.-\u003e S[(\"SQLitesingle writer per stage\")] C -.-\u003e S D -.-\u003e S E -.-\u003e SA run is one process that walks through the stages in order and writes its results to disk as it goes. The next stage reads whatever the last one left behind. I\u0026rsquo;d defend that against anyone who reflexively reaches for a queue on a hobby project this size: at dozens to low hundreds of listings per run, a queue buys nothing and costs a service to operate and monitor.\nThe choice isn\u0026rsquo;t free, though. The first time I want two scrapers writing to the same SQLite file at once, or want one stage to retry independently of the one before it, this is the design that starts to hurt. I haven\u0026rsquo;t hit that yet. I expect I will.\nWhat differs between the two projects is entirely inside the middle two boxes: what gets filtered out before it costs anything, and what the model actually gets asked to judge. That\u0026rsquo;s what the next two posts cover.\nOne shared GPU forces the same cost tradeoff on both projects # Both pipelines lean on the same home-lab constraint: one GPU, shared with everything else that machine does, including gaming and media transcoding. That constraint shapes the architecture more than almost anything else.\nNeither obvious option worked alone:\nCloud, unthrottled: the strongest available vision model, run on every image, priced out at ten to twenty times a reasonable monthly budget. Local, unbounded: running everything on the home GPU worked, but a full pass over a week\u0026rsquo;s photos took on the order of a day — on a machine other people in the house wanted to use in the meantime. That\u0026rsquo;s why both projects ended up with a tiered cascade instead of calling the best model on everything:\nCheap local checks run first. A stronger model runs only on what survives. An optional even-stronger model handles genuinely ambiguous cases. Fitting a large vision model onto a consumer-class GPU brought its own failure mode. The full-precision checkpoint of one candidate model didn\u0026rsquo;t fit, and it left the worker in a permanently unhealthy state until I switched to an FP8-quantized build of the same model, which loaded cleanly.\nServerless GPU workers scale to zero when idle, which is exactly what keeps cost near zero between runs, but they carry a real cold-start cost too. One backend took roughly eight minutes to spin up from cold, against a hardcoded two-minute timeout on the client side.\nThat mismatch was a guaranteed failure on the first image of every single run. It stayed that way until I timed the cold start myself, instead of assuming a fixed timeout was generous enough. Neither Ollama instance gets addressed directly by IP in either codebase anymore. (Where the stronger vision tier runs, and what it costs to keep a cloud GPU honest, became its own decision.) Both pipelines read a plain environment variable for wherever inference happens to be running. That decision paid off the first time I moved the GPU host.\nThe scoring logic is where the two projects genuinely diverge # The scoring logic is different enough between the two projects that it doesn\u0026rsquo;t fit here. Part 2 covers the estate-sale scanner\u0026rsquo;s asymmetric feedback loop, where a bad sale and a good sale teach the system very different things. Part 3 covers the resale monitor\u0026rsquo;s two-pass scoring and a false-positive bias I haven\u0026rsquo;t fully stress-tested.\n","date":"10 August 2026","externalUrl":null,"permalink":"/blog/scrape-score-alert-resale-hunting-pipelines-local-vision-models/","section":"Blog Posts","summary":"Two personal tools I’ve built, an estate-sale scanner and a resale-clothing monitor, run on the exact same architecture: scrape listings, score every photo with a local vision model, surface only the ones worth my attention. Same four stages, same database, same GPU, underneath two very different projects.\n","title":"Scrape, Score, Alert: The Pattern Behind Two Home-Lab Vision Pipelines","type":"blog"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/categories/cloud-computing/","section":"Categories","summary":"","title":"Cloud Computing","type":"categories"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/categories/containerization/","section":"Categories","summary":"","title":"Containerization","type":"categories"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/tags/containerization/","section":"Tags","summary":"","title":"Containerization","type":"tags"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/categories/cybersecurity/","section":"Categories","summary":"","title":"Cybersecurity","type":"categories"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/tags/devops/","section":"Tags","summary":"","title":"DevOps","type":"tags"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/tags/docker-compose/","section":"Tags","summary":"","title":"Docker Compose","type":"tags"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/categories/docker-tutorials/","section":"Categories","summary":"","title":"Docker Tutorials","type":"categories"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/categories/it-infrastructure/","section":"Categories","summary":"","title":"IT Infrastructure","type":"categories"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/tags/openvpn/","section":"Tags","summary":"","title":"OpenVPN","type":"tags"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/categories/software-development/","section":"Categories","summary":"","title":"Software Development","type":"categories"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/tags/step-by-step-guide/","section":"Tags","summary":"","title":"Step-by-Step Guide","type":"tags"},{"content":" Docker Containers Inherit Your Connection\u0026rsquo;s Exposure by Default # Docker containers share the host\u0026rsquo;s network stack by default.\nThat means any service you run is exactly as exposed to the outside world as the raw connection it rides on, with nothing standing between your traffic and anyone watching that connection.\nRoute it through a VPN container instead, and requests leave through the VPN, not your raw connection — the container\u0026rsquo;s real IP disappears.\nSkip it, and your home IP does the talking. This guide builds a Docker Compose file that puts one or more services behind a VPN container using network_mode: service:vpn. You\u0026rsquo;ll:\nSet up the VPN container Wire dependent services to route through it Verify traffic actually goes through the VPN once everything\u0026rsquo;s running (Which host those containers should even run on is a separate question — see Not every Docker container belongs on the NAS.)\nBasic Docker familiarity helps but isn\u0026rsquo;t required — the official Docker documentation covers anything unfamiliar here.\nDocker Compose Handles Orchestration; a VPN Container Handles Privacy # A diagram of docker compose with a vpn Docker Compose Replaces a Pile of docker run Flags with One File # Docker Compose defines your services, networks, and volumes in one YAML file instead of a pile of docker run commands. A multi-container setup that would otherwise take a dozen flags to launch comes up with one.\nDocker Compose Keeps Multi-Container Environments Consistent # Simplifies multi-container deployments Ensures consistency across development, testing, and production environments Streamlines application scaling and maintenance Docker Compose Shows Up Most in Microservices, Dev, and CI/CD Work # Microservices architecture Development environments Continuous integration and continuous deployment (CI/CD) pipelines Why Use a VPN with Docker Services? # A VPN encrypts a container\u0026rsquo;s outbound traffic and hides its real IP behind the VPN provider\u0026rsquo;s. That matters most for services that talk to external networks or handle data you don\u0026rsquo;t want tied back to your home connection.\nRouting Through a VPN Also Protects Data in Transit # Securing communications between distributed services Protecting data in transit from eavesdropping Ensuring privacy for services that need to access external resources Using a VPN allows for more secure communication across your Docker services. Without a VPN, Containers Inherit the Host\u0026rsquo;s Full Exposure # A container with no VPN in front of it sends traffic exactly the way the host would: same IP, same exposure to anything watching the host\u0026rsquo;s connection.\nRouting a service through a VPN container fixes this at the network layer, instead of trusting each service to handle it individually.\nNetwork Isolation Introduces Its Own Management Problems # Potential exposure of sensitive data Difficulty in managing network policies Ensuring consistent VPN connections for all services Route it through the VPN container instead, and the outside world sees the VPN\u0026rsquo;s exit node instead of your router blinking away in the closet.\nGet Docker and Docker Compose Installed Before Configuring the VPN # Docker Installs via apt; Docker Compose Installs via a Direct Download # Steps to Install Docker: # Update Your Package Database: # Ensure your system\u0026rsquo;s package database is up-to-date\nsudo apt update Install Prerequisite Packages # Install packages that allow apt to use repositories over HTTPS\nsudo apt install apt-transport-https ca-certificates curl software-properties-common Add Docker\u0026rsquo;s Official GPG Key: # Add Docker\u0026rsquo;s GPG key to verify the integrity of the software.\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - Add Docker Repository: # Add Docker\u0026rsquo;s official repository to your sources list.\nsudo add-apt-repository \u0026#34;deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable\u0026#34; Install Docker: # Update the package database again and install Docker.\nsudo apt update sudo apt install docker-ce Verify Docker Installation: # Confirm Docker is installed correctly by running:\nsudo docker --version Steps to Install Docker Compose # Download the Latest Version: # Download the Docker Compose from its official Github repository.\nsudo curl -L \u0026#34;https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)\u0026#34; -o /usr/local/bin/docker-compose Apply Executable Permissions: # Make the downloaded file executable.\nsudo chmod +x /usr/local/bin/docker-compose Verify Docker Compose Installation: # Check the version to ensure Docker Compose is installed.\ndocker-compose --version Creating a Docker Compose File # Basic Structure of a docker-compose.yml File: # A docker-compose.yml file defines the services, network, and volumes used in your application. Here is the basic structure:\nversion: \u0026#39;3.8\u0026#39; services: # Define your services here networks: # Define custom networks if needed volumes: # Define named volumes if needed Explanation of Key Directives: # version: Specifies the version of the Docker Compose file format.\nservices: Defines the containers to be run as the part of the application.\nimage: Specifies the Docker image to use. build: Allows specifying a build context and Dockerfile. ports: Maps container ports to host ports. volumes: Mounts host paths or named volumes. networks: Connects services to specific networks. networks: Customized networking configurations for services.\nvolumes: Manages data persistence using named volumes.\nExample: Basic Docker Compose File # Here\u0026rsquo;s a simple example with two services: a web server and a database.\nversion: \u0026#39;3.8\u0026#39; services: web: image: nginx:latest ports: - \u0026#34;80:80\u0026#34; networks: - webnet database: image: postgres:latest environment: POSTGRES_USER: exampleuser POSTGRES_PASSWORD: examplepass POSTGRES_DB: exampledb volumes: - db-data:/var/lib/postgresql/data networks: - webnet networks: webnet: volumes: db-data: That\u0026rsquo;s the whole shape of a Compose file: services, networks, volumes. Everything from here is just filling in services: correctly for a VPN-routed setup.\nConfiguring Each Service to Use the VPN # Not Every VPN Provider Works Cleanly Inside a Container # A few factors decide whether it will:\nKey Factors to Consider: # Reliablity: Choose a provider with a reputation for uptime and reliability. Security Features: Ensure the provider offers strong encryption and no-log policies. Compatibility: Verify that the VPN service is compatible with Docker and can be used within containers. Performance: Consider the speed and latency, especially if your servicers require high bandwidth. Support: Look for providers that offer good customer support and detailed documentation. OpenVPN Is the Flexible Default; WireGuard Is the Faster Alternative # OpenVPN is the flexible, open-source default here. WireGuard is the other real option — simpler, faster, less config surface. Either works fine inside Docker.\nOpenVPN is a popular choice. Setting Up the VPN Container # Pulling a VPN Container Image (e.g., OpenVPN): # Pull the OpenVPN image from Docker Hub first:\ndocker pull kylemanna/openvpn That pulls the image you\u0026rsquo;ll configure next.\nConfiguring the VPN Container: # Initialize the OpenVPN Configuration: Create a directory to store the OpenVPN configuration and initialize it: mkdir -p /path/to/your/config docker run -v /path/to/your/config:/etc/openvpn kylemanna/openvpn ovpn_genconfig -u udp://YOUR_VPN_SERVER Generate the Certificates: Generate the necessary certificates and keys: docker run -v /path/to/your/config:/etc/openvpn -it kylemanna/openvpn ovpn_initpki This initializes the PKI (Public Key Infrastructure) that generates OpenVPN\u0026rsquo;s certificates and keys.\nStart the OpenVPN Container: Start the container with the generated configuration: docker run -v /path/to/your/config:/etc/openvpn -d -p 1194:1194/udp --cap-add=NET_ADMIN kylemanna/openvpn This runs the OpenVPN container in detached mode, maps the port, and grants the network administration capability it needs.\nModifying the Docker Compose File # Adding the VPN Container to the docker-compose.yml File: # Add the VPN container to your docker-compose.yml, then point your other services at it.\nConfiguring Services to Route Traffic Through the VPN: # Set each dependent service\u0026rsquo;s network_mode to the VPN service\u0026rsquo;s name, and its traffic routes through the VPN container automatically.\nExample: Updated Docker Compose File with VPN: # Here\u0026rsquo;s a step-by-step example:\nversion: \u0026#39;3.8\u0026#39; services: vpn: image: kylemanna/openvpn cap_add: - NET_ADMIN ports: - \u0026#34;1194:1194/udp\u0026#34; volumes: - /path/to/your/config:/etc/openvpn environment: - OPENVPN_PROVIDER=YourProvider - OPENVPN_CONFIG=YourConfig networks: - vpn_net web: image: nginx:latest depends_on: - vpn network_mode: service:vpn ports: - \u0026#34;80:80\u0026#34; volumes: - ./web:/usr/share/nginx/html environment: - VIRTUAL_HOST=yourdomain.com database: image: postgres:latest depends_on: - vpn network_mode: service:vpn environment: POSTGRES_USER: exampleuser POSTGRES_PASSWORD: examplepass POSTGRES_DB: exampledb volumes: - db-data:/var/lib/postgresql/data networks: vpn_net: volumes: db-data: In this example:\nThe vpn service pulls up OpenVPN and does the actual connecting. web and database both set network_mode: service:vpn, so they share the VPN container\u0026rsquo;s network stack instead of getting one of their own. Every request either service makes now leaves through that shared network stack, so it exits through the VPN automatically. That\u0026rsquo;s the whole pattern: define the VPN service, then set network_mode: service:vpn on anything that needs to ride behind it.\nVerify the VPN Connection Before You Trust It # Don\u0026rsquo;t Trust the Compose File Without Checking the Exit IP # Verifying the VPN Connection: # A few checks confirm the VPN connection is actually working:\nCheck the VPN Container Logs: Inspect the logs of the VPN container to confirm it has started correctly and is connected.\ndocker logs \u0026lt;vpn-container-name\u0026gt; Test the VPN Connection: Run curl or wget from inside a container on the VPN and check the external IP. It should differ from your local IP and match the VPN server\u0026rsquo;s.\ndocker exec -it \u0026lt;container-name\u0026gt; curl ifconfig.me Ensuring Services are Behind the VPN: # Same check, service-side: access the service and look at its outgoing IP.\nCheck Service IP: From within the service container, use the following command:\ndocker exec -it \u0026lt;service-container-name\u0026gt; curl ifconfig.me If that IP matches the VPN\u0026rsquo;s, the service is routing through the VPN correctly.\nCommon Issues and Solutions # Network Connectivity Issues: # Issue: Services cannot connect to the internet. Solution: Double-check the VPN container configuration, including the network mode setting in the docker.compose.yml file. VPN Container Fails to Start: # Issue: The VPN container doesn\u0026rsquo;t start / keeps restarting. Solution: Check the logs for any errors, and check that the configuration files and credentials you provided are correct. Make sure that the required ports are not bloced by a firewall. Services Not Routing Through the VPN: # Issue: Services bypass the VPN and use the host network. Solution: Verify the network_mode: service:vpn setting in the docker-compose.yml file. Verify that the dependent services start after the VPN container. This is the failure mode that matters most: a service can run fine while silently leaking your real IP. Tips for Troubleshooting # Useful Commands and Logs to Check: # View Container Logs: Check the logs for the VPN container and the services for any error messages.\ndocker logs \u0026lt;container-name\u0026gt; Inspect Network Settings: Verify that the network settings of your containers are properly configured.\ndocker network inspect \u0026lt;network-name\u0026gt; Check IP Routes: Check the containers\u0026rsquo; IP routing tables to confirm traffic routes through the VPN.\ndocker exec -it \u0026lt;container-name\u0026gt; ip route Community and Support Resources: # Docker Documentation: The official Docker documentation is the defacto resource for troubleshooting and best practices when using Docker.\nOpenVPN Documentation: The OpenVPN documentation will help you in determining specific configurations and in general troubleshooting.\nCommunity Forums: Search your issue on community forums such as Stack Overflow, Docker Community Forums, and Reddit.\nThe network_mode: service:vpn Line Does the Real Work # That one setting forces a dependent service to share the VPN container\u0026rsquo;s network namespace instead of the host\u0026rsquo;s. Everything else in this guide (provider choice, the OpenVPN setup, the verification commands) just gets you to a Compose file where that line does its job correctly.\nIf curl ifconfig.me from inside a dependent container returns the VPN\u0026rsquo;s IP instead of your own, it\u0026rsquo;s working.\n","date":"1 July 2024","externalUrl":null,"permalink":"/blog/secure-services-docker-compose-and-nordvpn/","section":"Blog Posts","summary":"Docker Containers Inherit Your Connection’s Exposure by Default # Docker containers share the host’s network stack by default.\n","title":"Step-by-Step Guide to Creating a Secure Docker Compose Script with VPN Integration","type":"blog"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/categories/tech-how-tos/","section":"Categories","summary":"","title":"Tech How-Tos","type":"categories"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/tags/tutorial/","section":"Tags","summary":"","title":"Tutorial","type":"tags"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/tags/vpn/","section":"Tags","summary":"","title":"VPN","type":"tags"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/categories/vpn-and-security/","section":"Categories","summary":"","title":"VPN and Security","type":"categories"},{"content":"I\u0026rsquo;m Preston Bernstein, a full-stack developer based in Atlanta, GA, with over ten years of experience across front-end and back-end work. My professional focus is building and optimizing web applications; my off-hours focus is the infrastructure underneath them.\nWhat I work on # My home lab is the test bed for most of what appears on this blog:\nInfrastructure and networking. A Synology NAS, a desktop workstation, and a retired Dell XPS 17 running Proxmox VE, behind a UniFi gateway and switch with Pi-hole DNS filtering. Self-hosted services. Media automation, family photo backup with Immich, and a placement framework for deciding which containers run where, with Prometheus and Grafana observability shared across all of it. AI and agent engineering. Claude Code agent pipelines that spec, build, review, and deploy software; knowledge-graph retrieval built on LightRAG; local LLM and vision-model inference through Ollama on a single shared GPU; and the cost-control tooling that keeps unattended agents inside a budget. About this blog # Every post here comes from a real build or debugging session, written up with the numbers and the failure modes intact — including what stayed broken or unproven. If a post says something was fixed, it was deployed; if it wasn\u0026rsquo;t verified, the post says so.\nContact # The best ways to reach me:\nContact form on this site GitHub LinkedIn ","externalUrl":null,"permalink":"/about/","section":"Pages","summary":"I’m Preston Bernstein, a full-stack developer based in Atlanta, GA, with over ten years of experience across front-end and back-end work. My professional focus is building and optimizing web applications; my off-hours focus is the infrastructure underneath them.\n","title":"About","type":"pages"},{"content":"","externalUrl":null,"permalink":"/contact/","section":"Contact Me","summary":"","title":"Contact Me","type":"contact"},{"content":"","externalUrl":null,"permalink":"/pages/","section":"Pages","summary":"","title":"Pages","type":"pages"},{"content":" Responsibility of Contributors # Lorem ipsum dolor sit amet, consectetur adipiscing elit. Purus, donec nunc eros, ullamcorper id feugiat quisque aliquam sagittis. Sem turpis sed viverra massa gravida pharetra. Non dui dolor potenti eu dignissim fusce. Ultrices amet, in curabitur a arcu a lectus morbi id. Iaculis erat sagittis in tortor cursus. Molestie urna eu tortor, erat scelerisque eget. Nunc hendrerit sed interdum lacus. Lorem quis viverra sed\npretium, aliquam sit. Praesent elementum magna amet, tincidunt eros, nibh in leo. Malesuada purus, lacus, at aliquam suspendisse tempus. Quis tempus amet, velit nascetur sollicitudin. At sollicitudin eget amet in. Eu velit nascetur sollicitudin erhdfvssfvrgss eget viverra nec elementum. Lacus, facilisis tristique lectus in.\nGathering of Personal Information # Lorem ipsum dolor sit amet, consectetur adipiscing elit. Purus, donec nunc eros, ullamcorper id feugiat quisque aliquam sagittis. Sem turpis sed viverra massa gravida pharetra. Non dui dolor potenti eu dignissim fusce. Ultrices amet, in curabitur a arcu a lectus morbi id. Iaculis erat sagittis in tortor cursus. Molestie urna eu tortor, erat scelerisque eget. Nunc hendrerit sed interdum lacus. Lorem quis viverra sed\nProtection of Personal- Information # Lorem ipsum dolor sit amet, consectetur adipiscing elit. Purus, donec nunc eros, ullamcorper id feugiat quisque aliquam sagittis. Sem turpis sed viverra massa gravida pharetra. Non dui dolor potenti eu dignissim fusce. Ultrices amet, in curabitur a arcu a lectus morbi id. Iaculis erat sagittis in tortor cursus.\nMolestie urna eu tortor, erat scelerisque eget. Nunc hendrerit sed interdum lacus. Lorem quis viverra sed Lorem ipsum dolor sit amet, consectetur adipiscing elit. Purus, donec nunc eros, ullamcorper id feugiat\nPrivacy Policy Changes # Sll the Themefisher items are designed to be with the latest , We check all comments that threaten or harm the reputation of any person or organization personal information including, but limited to, email addresses, telephone numbers Any Update come in The technology Customer will get automatic Notification. ","externalUrl":null,"permalink":"/privacy-policy/","section":"Pages","summary":"Responsibility of Contributors # Lorem ipsum dolor sit amet, consectetur adipiscing elit. Purus, donec nunc eros, ullamcorper id feugiat quisque aliquam sagittis. Sem turpis sed viverra massa gravida pharetra. Non dui dolor potenti eu dignissim fusce. Ultrices amet, in curabitur a arcu a lectus morbi id. Iaculis erat sagittis in tortor cursus. Molestie urna eu tortor, erat scelerisque eget. Nunc hendrerit sed interdum lacus. Lorem quis viverra sed\n","title":"Privacy","type":"pages"}]