The WOPR manuscript#
A journey log of building WOPR, told as a story rather than a build log: why second-hand enterprise iron, what the interim lab taught, and the day the storage came together. It is unfinished, and the chapters that exist are early drafts.
Redaction level: strict
This page is redacted more heavily: second-order details are replaced as well.
Chapter 1 — The Platform and the Iron#
Timeline: July 2025 — 23 May 2026. Platform decision, July 2025. Board delivered, 24 July 2025. Coolers, RAM batch 1, and boot drives land May 2026. First POST on the benchtop, 23 May 2026.
The only winning move#
The machine is called WOPR, and if you grew up when I did you don't need the reference explained: the war-room computer from WarGames, the one that learned. Naming a homelab server after it is either hubris or a mission statement, and I'd argue it's the second. The point of this machine was never to host a media library — though it does — and never to be a NAS — though it is one. The point was to build a thing big enough to learn on: a platform for a degree I was finally starting, for local AI models I wanted to run on my own iron, and for two decades of accumulated wanting-to-do-this-properly.
The platform decision came in July 2025, and it was the least glamorous decision of the whole project, which is exactly why I'm proud of it. No RGB, no consumer flagship board, no this-year's-anything. A Supermicro X11DPH-i — a dual-socket server motherboard — paired with two Intel Xeon Platinum 8164s. Second-hand enterprise kit, a generation past its corporate service life, at the point on the depreciation curve where the price has collapsed and the capability hasn't.
The rationale was written down at the time and it held for the entire build: PCIe capacity, bifurcation support, and headroom. Not clock speed. Not single-core benchmarks. Lanes. A dual-socket Xeon platform has PCIe lanes the way consumer boards have marketing, and everything this machine was ever going to become — banks of NVMe drives on passthrough controllers, one GPU and then two — depended on being able to feed cards without compromise. One quietly modern touch settled early: Windows 11 VMs would get a virtual TPM from the hypervisor rather than a physical module on a header, which is the sort of decision you can only make once you've accepted that the hypervisor, not the metal, is where the machine's identity lives.
And one structural decision mattered more than any component: the budget cadence. Monthly staged purchases, not a lump sum. Set from the very start, and never broken. It meant the build would take the better part of a year. It also meant every price could be watched, every market wobble waited out, and no single month ever had to absorb the whole cost. Patience was a component. It got installed first.
The board itself arrived on 24 July 2025 — £350 from a UK eBay seller specialising in decommissioned trade kit. It then did something almost no consumer purchase ever does: it waited. Ten months on a shelf while the rest of the machine assembled around it, one budget cycle at a time. It was finally confirmed working in May 2026, and the gap between those two dates is the honest shape of this whole project.
Choosing the iron#
The RAM campaign#
The memory target was set early and it was not modest: 1TB, via sixteen 64GB LRDIMM ECC modules — Samsung DDR4-2666, the exact part number recorded in the notes so that every future purchase would match the first.
Then the market made it interesting. By May 2026 the LRDIMM market was active and volatile, with prices elevated by hyperscale AI demand — the big model-training buildouts were eating the same second-hand server memory that homelabbers live on. So buying RAM stopped being shopping and became a small intelligence operation. The notes from that period read like a trading desk's: a target price band of £2.34–£3.76 per gigabyte depending on lot size; a compatible Kingston alternative (Cisco-certified, slightly cheaper, not Supermicro-validated) evaluated and held in reserve rather than bought; two sellers cultivated deliberately — a high-volume trader with tens of thousands of sales as the preferred bulk source, the relationship kept warm for the big batches, and a smaller outfit with multi-buy pricing for the small lots.
The population plan was equally unsentimental. The X11DPH-i's manual dictates which slots to fill in which order for two CPUs, and the staging plan followed it exactly: four modules, then eight, then twelve, then sixteen, each batch landing in its designated slots. Batch 1 — four modules, the first real outlay — arrived around 22 May 2026. 256GB. The BIOS would shortly agree.
That's a quarter of the target, and the plan says so without embarrassment: batches 2–4 staged as finances allow. The discipline isn't having the money. The discipline is the sequence.
The PSU, and a decision that lived in chat#
The power supply story is worth telling precisely, because it's the first appearance of a theme this book keeps returning to: the gap between what happened and what got written down.
The plan, for most of the project, said SilverStone HELA 2050R. The machine contains a Seasonic 2200W ATX 3.1. There's no drama in the switch itself — a couple of months before the purchase, the Seasonic simply won on value, and the decision was made in a chat thread, casually, the way real decisions actually get made. What's instructive is that the switch never reached the project notes. The documentation went on saying HELA while the desk research had already moved on, and months later the record flatly contradicted itself. Nobody lied. The notes just weren't in the room when it happened. Hold onto that; it becomes a discipline later.
The Seasonic itself contributed two early lessons. First: during the benchtop build it fell off the desk and damaged an 8-pin CPU cable — recoverable only because a fully modular PSU ships spare cables in the box, which is an argument for modular PSUs that no spec sheet makes. Second, and better: on the first POST attempt the PSU appeared completely dead — no fan, no sign of life — and cost real diagnostic time before the answer surfaced. It has a hybrid fan mode. At low load, the fan simply doesn't spin. The unit was working perfectly. The lesson filed itself away in negative form — no lights doesn't mean nobody's home — and its more famous sibling would be coined later, the hard way.
Three graphics cards, two of which were never bought#
The GPU plan went through three distinct phases, and precision matters here because two of the candidates have nearly identical names and are entirely different products.
Phase one was the obvious one: an RTX 3090, the default homelab inference card of its era. Abandoned. Phase two, by the April planning notes, was the Intel Arc B770 — a consumer card. Phase three, from May onward, was the Intel Arc Pro B70 — a workstation card. B770, then Pro B70: names close enough to blur in conversation, yet a different die, a different market, a different machine's worth of consequences. The Pro B70 is the one that stuck: 32GB of ECC GDDR6 on a 256-bit bus, launched March 2026, workstation- and AI-focused, with full IPEX-LLM support on Linux — and available in the UK as a blower-design card at £1,199.99, the blower being exactly what you want exhausting heat from a server chassis rather than recirculating it.
The evaluation around it shows the method. The cheaper Arc Pro B60 (24GB, roughly half the price) was properly considered and rejected on grounds that had nothing to do with this year and everything to do with the endgame: it's different silicon, mixing the two is inadvisable, and two B60s cap out at 48GB where two B70s reach 64GB of ECC VRAM — the difference between hosting mid-size models and hosting serious ones. A PCIe Gen 5 card on this Gen 4 board was checked and confirmed a non-issue for LLM inference. Two units planned; one first, the second deferred.
And then the most important GPU decision of all, which was a RAM decision: RAM before GPU, confirmed priority. The glamorous purchase waits. A notify-list registration at the retailer is as close as the GPU got, and as close as it needed to get.
Quiet iron#
The chassis is a Thermaltake CTE C750 — a big E-ATX tower, around £140, chosen for airflow and for its support of large slow fans: 200mm and 140mm sizes throughout. That preference isn't aesthetic. This machine lives in a home, not a rack room, and running big fans slowly instead of small fans desperately is the founding move of a noise philosophy that shapes decisions right to the end of this story. As of the May 2026 snapshot the fan armoury stood at four 200mm PWM intakes and three 140mm magnetic daisy-chain fans in hand, temporarily ganged onto two motherboard headers via Y-splitters — with proper routing across the board's nine fan headers deferred, correctly, until the board actually lived in the chassis.
CPU cooling is a pair of Noctua NH-D9 DX-3647 4U towers — the narrow-ILM variant, confirmed against the board's socket bracket before ordering, which is a five-minute check that prevents a fortnight of RMA. Their installation produced the kind of small hard-won knowledge this book exists to pass on: the tower sections must be disassembled to reach the mounting bracket at all; the paste is factory-applied and adding more is a mistake; and the printed manual in the box was inadequate to the point of comedy while the proper English manual sat online. Both coolers mounted, both confirmed working.
The storage fleet, assembled before its pool existed#
By May 2026 the storage hardware was largely in hand, and its shape is worth recording because everything in Act 2 happens on it.
For boot: two Crucial P310 1TB NVMe drives, installed in the board's two onboard M.2 slots, destined for a mirrored pair. They're QLC drives, and the notes are honest about what that means: fine for an OS volume that's read-dominant after setup, categorically not suitable for write-hammered roles like a ZFS log device. Right drive, right job.
For the main pool: seven Crucial P3 Plus 4TB NVMe drives targeted, four in hand — with a complication that deserves its own foreshadowing. Three of those drives were full. They held partition-recovery data from a previous disk failure — original, clone, and recovered copies — and could not be wiped until somewhere safe existed to put that data. So the pool plan became a dependency chain: acquire a cold-storage HDD pair, offload the recovery data, wipe all seven drives, and only then build the full array in one operation. The recovery data itself has a story, and it is not a happy one, and it's told in full when we get to the storage day.
Feeding those drives takes controllers, and here the board's PCIe lanes pay their first dividend: three Supermicro dual-M.2 carriers and one ASUS quad-M.2 carrier — ten NVMe slots across four controllers, with ZFS spanning drives across controllers natively, no RAID cards, no per-controller constraints. Behind that, the plan sketched a SATA tier — sixteen hot-swap 2.5" bays via two drive cages, fed by onboard SATA and a £29 second-hand LSI controller in IT mode, with an eighteen-drive job lot of used Samsung 850 EVOs identified on eBay and the seller already talking — and a mirrored pair of large refurbished HDDs as the cold archive, flagged priority purchase because the whole NVMe consolidation waited on it.
Notice the register of all of this: identified, watchlisted, seller cooperative, not yet ordered. The used market rewards the buyer who has already done the homework and can wait. Most of this chapter is a man waiting, on purpose.
First light#
On 23 May 2026, on a benchtop — not in the chassis; the tower stood empty across the room — the accumulated iron was assembled into a machine for the first time. Two coolers torqued down onto two Xeons. Four LRDIMMs in their designated slots. Two boot drives in the onboard M.2 slots. The PSU that had already survived a fall off the desk, connected and silently pretending to be off.
It POSTed. The BIOS reported 262144MB — 256GB on the nose, all four modules recognised. And rather than savour the moment, the same day saw Proxmox VE 9.2 installed across the two boot drives as a ZFS mirror.
One shadow attended the celebration. The board's management controller — the BMC, the thing that lets you run a server headless — was locked with a previous owner's password, and clearing the CMOS didn't touch it, because BMC credentials live in their own non-volatile storage and survive tricks that reset everything else. Second-hand enterprise kit sells at that price for a reason, and the reason is homework: the fix was identified and queued rather than improvised.
The build-gate list stood at five items ticked — coolers, RAM, boot drives, first POST, hypervisor — and ten items open, starting with the most physical one: getting the board off the bench and into the tower. The plan had survived contact with the parts. Contact with reality was scheduled next.
Chapter 2 — The Architecture on Paper#
Timeline: July 2025 — 23 May 2026. The architecture decision predates the hardware's arrival; the plan consolidated across the winter; Proxmox VE 9.2 went onto the metal on 23 May 2026, and this chapter describes the plan as it stood in that moment — complete on paper, almost entirely unbuilt.
One sentence, load-bearing#
Strip away every service list and every deferred item, and the entire software architecture is one sentence:
Proxmox VE on bare metal, with TrueNAS SCALE running as a full virtual machine holding direct PCIe passthrough of the storage controllers.
Every word of that sentence is doing work, and the plan's own notes are unusually clear about why. TrueNAS is never to run in Docker. Not as a preference — as a doctrine. A storage appliance needs full OS control: it loads ZFS kernel modules, it talks SMART to the physical drives, it serves iSCSI and NFS, it manages snapshots. Containerise it and every one of those becomes a lie told through an abstraction layer. Virtualising it is legitimate on exactly one condition: the storage controllers are handed to the VM whole, by PCIe passthrough, so that TrueNAS owns its disks at the hardware level and the hypervisor never touches them.
What you buy with that condition is a clean split of sovereignty. Proxmox owns the machine: CPU, RAM, PCIe topology, every future GPU. TrueNAS owns the data: the drives behind the passed-through controllers, the pools, the exports. Each layer does the one thing it's best at, and — this mattered to a machine designed around expansion headroom — the hypervisor keeps full flexibility to add GPUs and controllers without the storage layer ever being party to the negotiation.
It's worth pausing on the direction of causality here. The iron in chapter 1 was procured to fit the architecture, not the other way round — the platform decision's own stated rationale was PCIe capacity and headroom for NVMe controllers and GPUs, which is to say: the lane budget, the four NVMe carriers, the controller count were all downstream of a virtualisation-with-passthrough design that existed on paper first.
What was actually running in May 2026#
Against that grand design, the reality on 23 May 2026 was modest and correct: Proxmox VE 9.2, freshly installed on the benchtop machine, mirrored across the two boot NVMe drives, reachable on the LAN by its web interface, holding 256GB of RAM and precisely zero guests.
Two details of that install deserve the record.
First, the boot mirror was itself a plan revision. The original design had the two 1TB drives doing different jobs — one carrying the Proxmox OS, the other passed to TrueNAS for cache duty. That got revised before installation: both drives into a single ZFS mirror for the OS, redundancy over cleverness, with TrueNAS's cache needs to be met later from the main pool's own drives. On paper, the sober choice — a mirrored boot volume is the difference between a dead drive being an incident and being an afternoon. Keep an eye on that mirror, though. It has a part to play later, and the record of how it played it has a hole in it that this book will not be papering over.
Second, the installer capped ZFS's ARC — its in-RAM read cache — at 16GB, a default the notes flagged immediately for revisiting once the RAM staging advanced. Noted and parked: the plan's first entry in a long ledger of correct settings for a machine that doesn't exist yet.
Around the install sat a short, sequenced post-install list — bring up the management controller, then firmware, then repositories, then updates, then the chassis move, then fans — each step gated on the one before. The same discipline as the hardware chapter, transposed into software.
Three pools, three temperatures#
The storage architecture was finalised on paper before a single pool existed, and its organising principle is temperature: three independent ZFS pools, one hot, one warm, one cold, each of which can be created, expanded, or torn down without consulting the others. Independence between pools is itself an architectural decision — no pool is load-bearing for another; failure domains stay separated; and ZFS's own indifference to controller boundaries (it spans drives across HBAs natively) means the pools are shaped by role, not by wiring.
Pool 1 — hot. Seven 4TB NVMe drives in a single RAIDz2 vdev: roughly 20TB usable with any two drives sacrificable, by the planning arithmetic — the built pool would eventually measure a more honest 16.83 TiB, planning numbers being what they are. This is where everything alive would sit: VM disks, application data, LLM model storage, the cloud-replacement files. The build plan was explicitly staged, and explicitly not an expansion: start a small pool on the four free drives, wait for cold storage, offload the recovery data holding the other three hostage, then destroy and rebuild the full seven-drive RAIDz2 in one operation. The notes are blunt on the point — "not an online expansion, full rebuild" — a constraint the plan absorbed instead of fighting.
Pool 2 — warm. Eight used 1TB SATA SSDs in four mirrored pairs — mirrors rather than RAIDz, chosen deliberately for the better random I/O of a scratch tier: download staging, media pipeline processing, transcode cache, AI model staging. Everything on this pool is by definition re-obtainable; the pool's job is to be fast and hot-swappable, not precious.
Pool 3 — cold. Two large refurbished HDDs, mirrored, on the board's powered SATA ports: archives, old-mailbox migration data, ISOs, long-term backups. Cold in every sense — big, slow, boring, and marked priority purchase anyway, because of the dependency chain from chapter 1: nothing about the hot pool could be finished until the cold pool existed to receive the recovery data. The least exciting pool unlocked the most exciting one. Scheduling insight of the whole build, in one line.
The list that became an order#
Every homelab starts as a wish-list. The transformation worth documenting is the one the planning notes performed across the winter: from list of things wanted to sequence of things earned, eleven steps deep, each unlocked by the one before.
The shape of the sequence tells you the priorities. Hypervisor first — done, 23 May. Then firmware and hardening: the machine made trustworthy before it's made useful. Then TrueNAS and the pools: storage before any service, because every service stores something. Then the container runtime, then the first working loads — the local-AI stack among them, not as a stretch goal but as core services, which is what they were to this build's purpose. Then, pointedly, a security layer as its own numbered phase — reverse proxy, single-sign-on, certificates, tunnel — before the crowd-pleasers. Mail. Media. Productivity. Dev tools, last.
And beneath the ordered list, its enforcement arm: the deferred-items table. Every postponed ambition written down with its reason attached — this one waits on the chassis, that one's infrastructure-heavy, Kubernetes is flatly "overkill for current scale," the Windows VM stack waits for a GPU that hasn't been bought. Deferral-with-reason is a different mental act from forgetting, and the table is what makes the wish-list safe to walk away from: nothing is lost, everything is parked, and each item names the event that un-parks it. One entry sits in the table like a pebble in a shoe — a small security item, deferred longer than the plan itself was comfortable with, and every reader who has ever kept their own list knows exactly which entry theirs is.
Two service threads deserve a note each, because both pay off much later. The AI stack's goal was written down with unusual restraint: local inference for code work — a specific list of model families to try — with no model training planned; the interim lab had already proven the tooling worked on CPU alone, at speeds best described as devotional, and everything waited on the GPU that RAM outranked. And around naming, a small deferred decision sat unresolved: three domains owned, three possible identities for the machine and its owner — infrastructure, vanity, personal — with the choice consciously postponed. The machine had a name long before it had an address.
The bridge#
The plan ends with a migration manifest: the interim lab — the nested, triple-virtualised laptop rig whose pain justified this entire build — was to stay operational through commissioning, then hand over its services one by one. Nextcloud. Jellyfin. The arr download pipeline, with a note attached that reads like a sigh of relief: on the new NFS fabric, hardlinks will finally work* — a one-line epitaph for an entire category of interim-era suffering. And Ollama with its tooling, off the laptop's CPU and onto real iron at last.
That was the architecture on paper as of late May 2026: one load-bearing sentence, three temperatures of storage, eleven ordered steps, a table of honest deferrals, and a bridge back to the old world with the traffic direction already agreed.
Paper is where every architecture is invincible. The next chapter is about what happened when this one met the machine.
Chapter 3 — The Cardboard Castle#
Timeline: April — 23 May 2026. The interim lab's consolidated notes date from April 2026; the lab kept running right through the benchtop POST on 23 May, and beyond.
Three layers deep#
While the iron accumulated on its shelf, a laptop spent the spring pretending to be a datacentre.
The interim lab was a Dell Precision 5540 — a machine built to run CAD software on trains — hosting, in order: Kubuntu on the metal; a virtual machine manager running QEMU/KVM; Proxmox VE nested inside that VM, behind NAT; and eleven LXC containers nested inside Proxmox. Three layers of virtualisation before a single service did any work. Every packet in or out of a container traversed a NAT boundary that existed only because the hypervisor was itself a guest. If you want to understand why the rest of this book exists, that topology is a good place to start.
And yet — and this is the part that matters — it worked. Eleven containers, each with one job: a media server; the request frontend that lets you ask for things politely instead of touching the pipeline; the three-part acquisition pipeline (an indexer proxy and two library managers) with a download client behind them; a tunnel egress container; a VPN egress container; a full Nextcloud; and an AI pair — an inference engine and an agent framework — that will get their own reckoning below. A real stack, doing real work, for real users. In cardboard.
What the castle taught#
The filesystem sin#
The lab's storage was the laptop's own home directory, exported to the nested Proxmox over sshfs, then bind-mounted into the containers. It worked, in the sense that files moved. But sshfs is a filesystem impersonation, and the impersonation has a gap with real consequences: hardlinks don't work across it. The media pipeline is built around hardlinking — a completed download becomes a library entry by adding a second name to the same data, instantly, for free. Over sshfs, every import silently degraded to a copy: same file, twice the space, until seeding finished. On a laptop with one SSD.
This is the single most instructive failure of the interim era, because of what it did to the bare-metal plan: it turned "NFS from TrueNAS" from a nice-to-have into a requirement with a grudge. Chapter 2's migration manifest carried a note that hardlinks would finally work on the new fabric. This is where that note earned its place. The lesson generalises: you find out what a storage abstraction actually preserves by running a workload that depends on it — no spec sheet volunteers this.
The cloud, replaced properly#
The Nextcloud container was the interim lab's quiet triumph. An Alpine build, deliberately lean, later given more disk, memory, and cores as adoption grew — the resize commands a one-liner from the hypervisor, which is LXC's whole sales pitch. Onto it moved calendars, contacts, tasks, notes sync, and file sync, wired to every device through the DAV protocols. And the wiring carried a small discipline worth recording: every client got its own app password — each an individually named, individually revocable credential, no shared secret anywhere. When one device goes missing, you revoke one password. Security that costs thirty seconds at setup and nothing thereafter.
By the end of April the migration had teeth: my task list formally moved off a certain productivity giant's to-do app, and the notes record the retirement with something close to ceremony. Self-hosting stops being a hobby the day the canonical copy of something you rely on lives on your own hardware. That day was in April.
The pipeline, and routing done at container granularity#
The acquisition stack taught its own networking lesson. The download client's traffic had no business leaving through the house connection, so the lab grew a dedicated VPN egress container: a tiny Alpine LXC running WireGuard to a commercial provider, with kernel IP forwarding and NAT masquerade turned on — a router, in 256MB of RAM. The download client's default route was then pointed at that container, persistently, via a systemd unit. One container's traffic, and only that container's, exits through the tunnel.
That's the pattern worth keeping: routing policy applied at container granularity, with the VPN as infrastructure rather than as an app on the endpoint. The setup notes record thirteen alternative endpoint configs sitting ready for failover — and record, honestly, that automatic failover never got built. The TODO list remembers.
Seven minutes per reply#
The AI pair deserves its numbers preserved. Getting the inference engine installed at all required first growing the nested hypervisor's virtual disk — the original allocation had no room for model weights, and the fix ran down through every layer of the nesting: resize the disk image on the host, then grow the partition, the physical volume, and the logical volume inside the guest. Four commands, four layers, one afternoon. The models themselves taught the next lesson: the small model that fit comfortably didn't support tool calling — useless to an agent framework — and the model that did support it needed double the container's originally allocated memory. Capability, not size, was the selection criterion; the spec that mattered wasn't on the model card's front page.
The agent framework on top contributed a catalogue of container-era gotchas that the notes preserve like pressed flowers: the health-check command that must run before onboarding because it applies the database migrations; the binary that never lands in PATH; the OS keychain that cannot work in a container because there's no desktop session bus to host it, forcing a workaround for secret storage — a compromise, noted as one.
And then the headline figure. With everything wired correctly — agent, engine, tool-calling model — a single reply took around seven minutes on the laptop's CPU, triple-nested. The notes call this "acceptable for proof-of-concept," which is the correct engineering judgment and also a very funny sentence. The PoC proved two things simultaneously: it works, and it is unusable here. Both facts were the point. Nobody buys a GPU on faith; the lab converted faith into a requirements document.
The break-in, from inside#
One service failure from the era earns its story. An update left the request frontend's single-sign-on incompatible with the media server — the login page simply stopped accepting anyone. The recovery was pleasingly burglar-shaped, from the inside: switch the account off SSO and back to local auth at the application's own data store — thirty seconds of duct tape once you know where to look. The full walkthrough lives in the documentation pool, where procedures belong; two of its lessons belong here. First: the update path for that container creates a brand-new container unless told otherwise — an update script that quietly provisions a second instance is a gotcha worth a scar. Second, and better: the notes immediately append "do not expose publicly until SSO is restored." A service running on duct-tape auth stays inside the walls. The perimeter instinct predates the perimeter; Act 3 will formalise it.
A laptop, taught to be a server#
Beneath everything, the host itself needed continuous persuasion to stop behaving like a laptop. An OS upgrade mid-era brought its own fallout — a file-sharing daemon that now demanded its legacy-protocol opt-out live in the right config section, desktop apps migrated between packaging formats with their data coaxed across — and the config that says the most about the whole era is three lines in the login manager: ignore the lid switch, ignore it on external power, never idle-sleep. A machine designed to nap the moment you look away, ordered to stay awake because eleven containers and a household depended on it.
That's the interim lab in one image: hardware fighting its own nature, competently, under protest.
The seam#
Read the era's TODO list today and it sorts itself into two piles with almost no remainder. One pile is housekeeping. The other is a single repeated sentence wearing different clothes: bare metal phase. Bare metal phase. Resolves with TrueNAS NFS on bare metal. Resolved with GPU on bare metal. The lab had stopped being a project and become a waiting room — and among the entries, still unticked, the same pebble from chapter 2's shoe. Some debts survive every migration.
The waiting ended on 23 May 2026, on the benchtop across the room, the way chapter 1 told it. First POST.
On the forums where I misspent an earlier decade, first post was the emptiest thing you could claim — being first to say nothing under someone else's work. The server world means something older and stricter by it: Power-On Self-Test, the machine's own audit of whether it deserves to boot. It took ten months of staged patience to earn the second meaning, and the first one hovers over it anyway, grinning, because this build was always also a story being written for a thread not yet opened. Both readings are true. Neither needs the other's permission.
The cardboard castle didn't come down that day. It kept serving — the media, the calendars, the seven-minute oracle — with no idea that across the room, on a bare bench, its replacement had just opened its eyes. Every service it ran was now a migration waiting for a date.
The machine POSTed. Act 1 ends here.
Interlude — First POST#
This part of the story, from the benchtop into the chassis, is still to be written. The manuscript resumes on 9 June 2026.
Chapter 4 — The Storage Day#
Timeline: 9 June 2026. One session, roughly eight hours. The weeks between First POST and this day — the chassis move, the drives that kept arriving, and reckonings of their own — are a scene still to be written (a chapter still to be written). This day stands alone, and earned it.
Three mysteries and a hostage#
Two and a half weeks after First POST, the machine was upright, in its chassis, and haunted.
Three problems had accumulated since the build began, and on the morning of 9 June all three were still open. One of the four 64GB DIMMs would intermittently report 0 MiB — some boots the machine had 256GB, other boots 188GB, and the difference correlated with nothing obvious. Of the eight 4TB NVMe drives that should have been visible across the controllers, only three or four ever appeared. And one more 4TB drive sat apart from all of this, physically isolated, holding 2.6TB of recovered data from an earlier disk accident — carved back from the dead by recovery tooling, stranded for two and a half months because nowhere safe enough to put it had existed yet. That drive was the hostage. The session's one headline objective: build the storage pool, and bring the data home.
Eight hours later all three mysteries were closed, the pool was alive, and the copy was running. What follows is how — told at the pace it happened, because the method is the thing this chapter exists to hand over. The session's own log states the operating principle up front, and it governs everything below: change one variable at a time, verify, then proceed. Hardware debugging falls apart the moment two things change between observations, because you can no longer attribute cause.
The three views of a drive#
The single most useful thing the day produced is not a fix but a frame: a drive can be "present" at one layer of the system and "absent" at another, and which layer it's missing from is the diagnosis.
Three commands give the three views. lspci lists everything electrically present on the PCI Express bus — if a drive appears here, the slot, the card, and the device are physically fine. lsblk lists what the kernel has turned into usable block devices — things you can partition, format, mount. And nvme list gives the NVMe-specific view with the one detail that actually identifies a physical object: the serial number. Device names like nvme0n1 are assigned by enumeration order and change between boots — today's nvme0n1 is tomorrow's nvme3n1. The serial number never changes. Identify drives by serial, always; a device name is a nickname the kernel gives out and takes back.
The frame, then. A drive in lspci but missing from lsblk: hardware fine, problem lives in driver or power state. Missing from lspci too: the problem is physical — seating, slot, or bifurcation. Every storage mystery this machine ever presented resolves into one of those two branches, and the day would visit both.
The DIMM that moved when the case did#
The memory fault first. On bad boots, POST flagged the same DIMM slot and the OS saw 188GB; on good boots, all four modules appeared and the OS reported 251GB. Both numbers deserve a note. The drop was one whole module going absent. But 251 rather than 256 on healthy boots is not a fault at all — ECC memory reserves capacity for the parity data it uses to detect and correct bit errors, so ~251GB is the all-DIMMs-present figure. Know your healthy number, or you'll chase phantoms.
The breakthrough on the intermittency wasn't a command — it was a correlation. The fault tracked physical disturbance of the chassis. Move cards around: fails. Cold boot after a long rest: sometimes recovers. That pattern points away from the module and toward mechanics, and the working theory fit perfectly: a motherboard standoff slightly out of position, flexing the board when the case stands upright, breaking marginal contact on that one slot. The DIMM was never the suspect; the chassis was.
Intermittent faults that correlate with physical disturbance are almost always mechanical — seating, standoffs, cable tension, board flex — and almost never the component itself. The diagnostic signal is the correlation with movement, not the text of the error message.
By day's end, after eight hours of cards in and out, all four DIMMs held at 251GB — the fault massaged into remission by the day's incidental reseating. The honest ledger entry: the proper fix, pulling the motherboard to correct the standoff, remained on the follow-up list. Remission is not cure, and the notes say so.
Bifurcation, and the cost of not knowing where your card is#
The most instructive problem of the day was the quad-drive carrier card showing exactly one of its four drives.
The mechanism first, because everything follows from it. A PCIe x16 slot is sixteen lanes. A GPU consumes all sixteen as one device. But the ASUS quad-M.2 carrier is a passive card — no PCIe switch chip of its own, a fact confirmed against the vendor's documentation mid-session. It depends entirely on the motherboard to split the slot into four independent x4 links: bifurcation, written x4x4x4x4. If the board presents the slot as a single x16, a passive carrier shows you exactly one drive — which is precisely what was on the screen.
On this dual-Xeon board, the BIOS groups its PCIe slots under IOUs — I/O Units — each with its own bifurcation setting. The fix should have been one setting. It took hours, because of a two-part mistake the log records with commendable honesty: the bifurcation was set on the wrong IOU, because the card was assumed to be in one physical slot when it was actually in its neighbour. Set the IOU that actually controls the card's real slot to x4x4x4x4, and all four drives materialised on the bus at once.
A passive multi-drive carrier is only as good as the bifurcation on the slot it occupies — and that requires two facts, not one: the right IOU mode, and certainty about which physical slot the card is in. Verifying slot identity before touching BIOS settings would have saved hours. Look before you touch.
The final BIOS layout was then chosen deliberately, with the future in the room: one IOU split four ways for the carrier, and both x16 groups left whole — reserved for the two GPUs that don't exist yet. The lane budget from chapter 1, paying out exactly as designed: no compromise between storage and compute, because the platform was bought so there'd never have to be one.
The drives that refused to wake#
Even with the bus enumeration solved, drives kept appearing in lspci and refusing to become disks — the first branch of the three-views frame. The kernel log named the culprit in one line: a device unable to change power state, reported by the passthrough driver as it tried to move a drive between D0 — fully awake — and D3, deep sleep. Some NVMe drives, particularly DRAM-less budget models like these, mishandle that transition: they go down into D3 and don't come back.
The fix is a kernel module option that forbids the nvme driver from ever letting the drives idle into D3 — one line in a modprobe conf file. And one non-negotiable step after it: rebuild the initramfs. The option must be baked into the early-boot environment so it applies before the drives are first touched; writing the file without rebuilding changes nothing, which is the kind of nothing that eats an afternoon.
Then the subtlety the day paid for in full. Even with the fix in place, drives that had been through a hard reset — power button held down — sometimes stayed wedged in a bad power state that survived soft reboots. The only reliable cure was a full shutdown and a thirty-second rest, long enough for the board's capacitors to drain and the drives' internal state to truly die.
Hard resets leave NVMe drives in undefined power states that soft reboots inherit. Prefer a graceful shutdown always; and when a drive misbehaves after a hard reset, power fully off and count thirty seconds. Patience at the power button saves hours at the keyboard.
One owner per device#
The last enumeration mystery wasn't a fault at all — it was the system working exactly as designed, unrecognised. Drives kept "vanishing" from the host. The cause: they were bound to vfio-pci, the passthrough driver, whose entire job is to reserve a device for a virtual machine — which makes it invisible as a block device on the host, on purpose.
The diagnostic is one readlink against the device's driver symlink: if it points at vfio-pci, a VM owns the drive; if at nvme, the host does. Moving a device between owners is two echo commands — unbind from one driver, bind to the other. Simple, once you hold the model:
In a virtualisation host, a physical device has exactly one owner at a time — host kernel or passthrough driver, never both. "The drive vanished" almost always means "the drive is doing its job for a VM." Check the symlink before suspecting the hardware.
This is also where chapter 2's doctrine stopped being prose and became practice. TrueNAS runs as a full VM because it needs this — real drives, real SMART, real sector access, none of which a container sharing the host kernel can be given. The handover is explicit: each physical drive's PCI address named in the VM config, one line per drive, with a flag presenting it to the guest as native PCIe (kinder to those same power states). And because device names shuffle at every boot, the name-to-address mapping was re-derived fresh from the live system rather than trusted from an old note. Passthrough is a deliberate, named, per-device handoff — the architecture's one condition, honoured literally.
Tank#
With seven drives finally visible, awake, and owned by the right operating system, the pool went up. A single RAIDz2 vdev across all seven: five drives of data, two of parity, any two drives simultaneously sacrificable. The planning arithmetic in chapter 2 said "roughly 20TB"; the built pool said 16.83TiB usable, and the built pool wins — that's the number this book uses from here on.
The reasoning was recorded, not just the choice. Mirrors: fastest, simplest recovery, but half your capacity gone — wasteful at seven drives. RAIDz1: one parity drive, which at 4TB per drive means the entire rebuild window after a failure is spent one hiccup from total loss. Rebuild windows on large drives are long; RAIDz2 exists precisely so that a second failure during the rebuild is an inconvenience instead of an obituary.
Choose redundancy by rebuild risk, not by capacity arithmetic. The question is never "how many drives can fail?" — it's "what happens if one more thing goes wrong while I'm recovering from the first?"
The pool's name is tank — plain ZFS convention, and a deliberate refusal to encode drive counts or hardware into a name that's nearly impossible to change later. Topology goes in documentation; the pool just gets a name. Two properties were set in the same breath as creation: autoexpand, on now because it only helps if it's on before drives grow; and auto-TRIM, so an all-flash pool keeps telling its SSDs which blocks are garbage — write performance and drive longevity, maintained by default rather than remembered occasionally.
And then, within hours, the new pool staged a small emergency as a final exam. After a passthrough change, the interface showed DEGRADED — one member listed as unavailable, its slot held by a long numeric ID where a device name should be. The correct reading of that screen: a RAIDz2 pool with one absent member is running with reduced redundancy and zero data loss — every byte still available, exactly the scenario two parity drives exist for. The number is just ZFS's internal ID for the member it expects back. A clean VM restart let it re-find the drive, and the "rebuild" took one second — an empty pool has only metadata to reconcile.
DEGRADED means "act soon," not "data lost." The redundancy you paid for is doing its job at that exact moment. Panic destroys more pools than drive failures do.
Bringing the data home#
Which left the hostage.
After a day of reboots and passthrough juggling, even finding the recovery drive required discipline — device names had shuffled all day. The decisive move was one command against the sole unassigned drive: blkid, which reported ext4. That single word settled everything, and understanding why is the day's last teaching point. Nearly every other drive in the machine was now a RAIDz2 member — and one drive from a RAIDz2 set carries striped, parity-encoded fragments that are useless in isolation; no tool reads files off one-seventh of an array. But ext4 is a complete, standalone filesystem. One drive, whole data, directly mountable.
Filesystem type answers "can I get data off this drive by itself?" A standalone filesystem — ext4, NTFS, exFAT — yes, mount it. A lone member of any striped array — no; the data exists only as the ensemble.
blkidis the one-command triage.
Mounted, the drive showed hundreds of recup_dir.* folders — the unmistakable handwriting of PhotoRec, the file-carver that had pulled 2.6TB back from the earlier accident. (How that accident happened, and what carved output does to your sense of order — eight hundred folders of anonymous, extensionless salvage — is its own story, told when the archive gets properly sorted.) The copy itself was almost an anticlimax, and deserved to be: with both the ext4 drive and tank visible inside the same VM, one rsync -av moved the archive pool-ward with no network hop at all. 2.6TB, flowing home after two and a half months in limbo.
What the day was actually about#
The session log closes by stepping back from the specifics, and the chapter should too, because the six habits it names are the method this whole build runs on from here: change one variable at a time. Verify at every layer rather than assuming any. Identify hardware by serial, never by nickname. When identity is uncertain, physically isolate — pull the drive and remove all doubt; certainty beats cleverness. Shut down gracefully and let hardware rest. And don't panic at warnings that are, in fact, the system's designed-for states doing their jobs.
The log adds a seventh, about itself: document the reasoning, not just the commands. A command without its rationale is a spell; a command with its rationale is a skill. That sentence — coined on this day, in this document's margins — is why this chapter reads the way it does, and fair warning: it's how the narrator thinks for the rest of the book.
The final state table that evening: 251GB of RAM holding steady on a standoff theory. All four carrier-card drives enumerated. tank ONLINE, 16.83TiB, copying. The follow-up list, honest as ever: let the copy finish and verify it; wipe the freed recovery drive and grow the pool to eight; cold storage still to buy; the motherboard still to pull for the permanent standoff fix; and — quietly humane among the hardware items — make the management-controller login easier on the hands; an awkward password is a small daily tax when dexterity is part of the spec sheet.
And one line in that table, easy to read past, that this book refuses to read past. Boot: Proxmox on one P310, single drive — "mirror pending." Chapter 2 installed that mirror. It was confirmed healthy on 25 May. On 9 June the machine's own log describes a single boot drive as if the second had never existed — and no document in the entire record captures what happened in between. No failure event, no removal, no decision. The mirror that chapter 2 told you to keep an eye on is simply gone from the story, and the honest telling is exactly that: a hole where an incident should be. The notes were getting better by June — this same day's log is the best document the project had yet produced — but they were not yet good enough, and this gap is the proof. What the project does about that, and what it costs first, belongs to the chapters ahead.
; sc.6 (pending) for the disaster that taught verification; sc.8 (pending) for tank growing its spine.
More chapters are still to be written.