If your AI stack dies when the internet does, it was never yours. Everything on this page keeps working with the network cable pulled.
00Why offline-first
Most "AI engineering" today is API calls to someone else's computer. I deliberately build the other way: models arrive as files, dependencies are pinned, telemetry is off, and there is no cloud fallback to hide behind. When something breaks, the fix has to come from documentation, logs, and understanding — not a status page.
Those are exactly the habits isolated and air-gapped networks demand: install from artifacts you carried in, configure from files you can audit, troubleshoot with what's on the box, and document well enough that the next operator can rebuild the stack without you.
01The fleet
Two machines, two operating models — one portable powerhouse I tune by hand, one headless Linux server I administer the way production is administered.
128 GB unified-memory workstation
Ryzen AI Max+ 395 with 128 GB of unified memory — up to 96 GB allocatable to the GPU. Runs ROCm 7 acceleration on native Windows (a genuinely rare configuration, tuned by hand) alongside llama.cpp (Vulkan + ROCm), Ollama, and LM Studio. Big enough to load 100B-class mixture-of-experts models entirely locally.
role: quantization testing · model gauntlets · speculative decoding experiments · daily local inference
Headless NVIDIA DGX Spark
An ARM Linux GPU server administered entirely over SSH — no desktop, no shortcuts. It runs four LLM inference services concurrently plus a routing gateway that fronts them as one OpenAI-compatible API, with a large mixture-of-experts model hot-swapped into a slot when a job needs it. Every session appends to a machine-local ops log; the runbooks live on the box they describe.
role: multi-model serving · service management · the "production" discipline half of the lab
02The deployment work
Not "I tried a model once" — the recurring, unglamorous work of making local inference reliable:
| Discipline | What it looks like here |
|---|---|
| Serving | llama.cpp server and Ollama behind OpenAI-compatible endpoints, with Open WebUI (Docker) as a chat front end wired to the local backend; several models resident simultaneously under an explicit memory budget per service. |
| Quantization | GGUF quant selection (Q2→Q8) to fit 100B+ parameter models into unified memory — with the quality cost measured by evaluation gauntlets, never guessed. |
| Performance tuning | Got multi-token-prediction speculative decoding working for a 26B model through llama.cpp when the GUI tooling architecturally couldn't — a measured ~1.35× generation speedup. |
| Routing | A gateway service that presents N models as one API surface, so clients don't care which box or backend answers. |
| Config as files | Launch scripts, YAML configuration, pinned versions. The test: could someone rebuild the stack from the docs alone? On the Linux box, the answer is yes — the runbook lives on the machine. |
| Monitoring | System performance counters, a self-built hardware flight recorder for crash forensics, and an append-only ops log updated every working session. |
03Troubleshooting war stories
Anyone can follow an install guide. The job is what happens when the install guide runs out. Three real ones from this lab — symptom, investigation, root cause, runbook:
case · gpu memory forensicsThe phantom OOM that survived a reboot
After force-killing a stuck inference process, every subsequent model load had mysteriously less GPU memory to work with — a decaying ceiling that made a 128 GB machine act like a much smaller one. The obvious fix, rebooting, did not clear it.
Working the problem systematically — isolating variables, testing after each change instead of assuming — pointed away from the model runtime and into GPU driver state. The only reliable fix turned out to be a privileged device-level restart of the GPU itself (pnputil /restart-device), not a system reboot.
Lesson: the "obvious" remediation is a hypothesis like any other. Test it, and when it fails, say so — then keep digging until the fix is reproducible enough to write down. It's a one-line runbook entry now.
case · driver-level isolationThe 64 GB wall on a 128 GB machine
Large single allocations kept failing at a suspiciously round threshold — about 64 GB — on hardware with double that available. Application settings, quant sizes, and backend flags changed nothing.
Systematic elimination (same model, different backends; same backend, different sizes) isolated the ceiling below every layer I controlled: it was a driver-level limit. I documented the boundary, sized deployments under it, and verified the fix the day a driver update finally lifted it — rather than assuming the release notes were true.
Lesson: know which layer of the stack owns your problem. Config, runtime, driver, and hardware fail differently — and "vendor says fixed" is a claim to verify, not a fact.
case · capacity planningThe carve-out that could crash the box
Unified memory means the GPU's 96 GB carve-out and the operating system share one physical pool — allocate against the GPU's theoretical maximum and the OS is left starved, which on Windows ends in a bugcheck, not a polite error.
The durable fix wasn't a setting; it was a sizing rule: budget model loads against measured available host memory, not the accelerator's advertised capacity, with headroom margins written into the lab's standing rules.
Lesson: capacity planning for LLM serving is a systems problem, not a model problem. The number on the spec sheet is where the analysis starts, not where it ends.
04Evaluation & red-teaming
A model isn't "good" because its demo went well. Before any model gets real work in this lab, it runs a gauntlet — a falsification-first benchmark suite built to make it fail: decoy controls, contradiction checks, and tasks with verifiable right answers.
- Honest negative results count. A heavily-pruned 100B-class model passed casual use but failed the gauntlet's knowledge probes — verdict recorded: not a daily driver. The gauntlet exists precisely to catch what a demo hides.
- Decoy controls beat vibes. Wrapping a 458M-parameter biomedical foundation model as a local tool server, I ran decoy compounds through its drug-binding predictions — and the controls disproved its confident answers. What survived testing was kept; the rest was labeled untrusted.
- Integration is measurable. Getting small local models to use tools reliably was a measured engineering problem, not a prompting vibe: naming tools by capability took one harness from 1/9 → 9/9 successful calls; keeping tool exchanges visible in context, 3/8 → 8/8; simplifying tool results to a single value, 4/12 → 12/12. Structural fixes beat prompt fixes — I know because I counted.
This is the same discipline as MAVL, pointed at models instead of code: every capability claim gets a test that could falsify it, and the test result — not the claim — is what ships.
05The Operator — a personal AI operating system
The lab's control plane is a system I built called The Operator: a self-hosted platform that manages a working fleet of AI "seats" — local models served from my own hardware alongside frontier-model CLIs — and enforces the lab's rules in software instead of relying on discipline.
| Subsystem | What it does |
|---|---|
| Seat tiers | Seven seats across capability tiers — local models for fast, private work; frontier CLIs in a trusted lane for the hard problems. Each seat is registered with its capabilities, limits, and data policy. |
| Proving Ground | No seat gets real work on reputation. New models run benchmark tasks first, and a "Measured" panel shows actual recorded scores next to each seat — capability as data, not marketing. |
| Data governance | Fail-closed gates: a seat whose provider trains on submitted data is blocked from sensitive lanes automatically. Routing is explicit and auditable — the guard is the gate, not a weaker model. |
| Nightly verification | Overnight, one agent proposes verifications of the day's work, an independent model reviews them adversarially, and nothing is marked verified without human sign-off — MAVL, running unattended, every night. |
| Council | For judgment calls, multiple model families answer the same question blind, then a synthesis maps agreements and dissents. Every council session lands in an append-only ledger. |
| Knowledge graph | My tooling turns codebases and documents into a queryable knowledge graph — a few thousand nodes and growing — rendered as the interactive 3D constellation on the home page. |
| Session audits | Long AI sessions degrade ("context rot"). The Operator audits session transcripts for it, so degraded runs get flagged instead of trusted. |
live example · adversarial adjudicationTwo frontier models disagreed about my own notes. Good.
A recent morning's queue, from the 900+ sessions this system has processed: an automated verification pass claimed a research note was source-checked. One frontier model reviewed the claim and passed it. An independent model from a different family reviewed the same claim and returned fail. The Operator's response is the whole point: it refuses to auto-resolve. There is deliberately no one-click action on a contested verification. The disagreement is surfaced with both reviews attached, and a human adjudicates.
Wargamers will recognize the shape: a side proposes a move, a red team attacks it, and a white cell adjudicates the result and records the dissent. MAVL is that structure applied to engineering and research claims, running nightly, unattended, on my own work. The system watches itself with the same discipline: the morning I wrote this section, it had detected one of its own gateway processes down and ranked that finding above everything else in the queue.
Design rule: agreement between models is a signal, not proof, and a surfaced disagreement is the most valuable output the system produces. Nothing gets marked verified without human sign-off.
The point isn't any one feature — no single piece here was hard to build. The rare part was deciding it needed to exist: treating AI output as untrustworthy enough to deserve adversarial infrastructure, then living inside that decision every day until it accumulated a track record. Orchestrating multiple AI systems safely is an engineering problem: registration, measurement, policy gates, audit trails, and adversarial review, all enforced by software. It's also the discipline joint wargaming already runs on: propose, attack, adjudicate, record the dissent. That's what integrating AI into an environment that matters actually takes.
06Working isolated, on purpose
Some of the lab's lanes are deliberately fenced. One coding environment runs a dedicated agent runtime that is firewall-fenced to a single local model endpoint — telemetry off, cloud egress blocked at the network layer, every dependency pinned. It's a working proof that a capable AI-assisted workflow can run with zero outside connectivity — because the fence, not good intentions, guarantees it.
The same instinct runs through the rest of the stack: a hardware flight recorder for post-crash forensics on the rig, append-only logs for anything that makes decisions, and documentation written for the operator who comes after me.
07What this adds up to
Install and configure LLM stacks where the cloud can't follow. Squeeze real performance out of fixed hardware. Troubleshoot down to the driver when it breaks. Measure models instead of believing them. Wire multiple AI systems together with governance and audit trails. Write it all down well enough that someone else could run it.
That's not a skills list assembled for a page — it's what this lab does every week, and it's all verifiable: the systems are running, the logs exist, and the methodology is written up.