Self-hosting has a marketing problem. The pitch is the first afternoon: a compose file, a container starts, you open a web page served from your own hardware. The actual work is everything after that afternoon. Knowing the thing is still doing its job a month later. Knowing the backup restores. Knowing the update you applied on Wednesday did not quietly break something you will not use until Saturday.
My homelab runs my media library, my photos, my passwords, my files, and the sync between my devices. It has gone through three shapes in a few months, and most of what I learned is about one failure mode: systems that report success while doing nothing.
Shape one: a rented Kubernetes node#
The first version was a single-node k3s cluster on a rented cloud VPS. Nothing was published to the internet. Every service was reachable only over Tailscale, and the only inbound firewall rule was SSH, pinned to one source address.
Kubernetes on one node sounds like overkill, and for scale it is. It earned its place for three other reasons.
A Pod is already a shared network namespace. One of my services needs to reach the internet only through its own VPN tunnel. In Compose that takes network_mode: service:... plus explicit resets on inherited ports, networks and sysctls. In a Pod, the app container has no interface except the tunnel's. It is not blocked from the internet; it has no route to it. If the tunnel container dies, the pod's network dies with it.
Ingress and TLS come free. The Tailscale operator turns each Ingress into a tailnet device with a real Let's Encrypt certificate. That replaced an ingress controller, cert-manager, DNS records and open ports with an annotation.
State is declarative. kubectl apply -k against an overlay converges. The Compose version needed a deploy script whose whole job was remembering which override files to pass, in which order.
The most useful number from that setup came from measuring: the entire media stack used about 1.5 GiB of memory and 18 millicores. A game server on the same node used more than all of it combined.
In August 2026 I tore the node down. It was a cost decision taken while it was healthy, not a failure. Everything needed to rebuild it is still in its repo.
Shape two: Compose on a PC at home#
At home, the same services ran under Docker Compose on a small-form-factor PC under WSL2, alongside the ones I would never put on rented hardware: photos (Immich), files (Nextcloud), passwords (Vaultwarden) and device sync (Syncthing).
This is where most of the silent failures lived.
Up with no network#
After I moved the WSL distro to a different disk, docker compose start brought every container back reporting Up. They were attached to no Docker network at all: no IP, no published port.
It never presented as a network problem. One container crash-looped complaining about a missing default route. One logged DNS failures. Two just returned 502 through the reverse proxy, with nothing in their own logs. docker restart did not help, because it reuses the same broken namespace. Only recreating the container did.
The check is one line, docker inspect on the container's networks, where an empty object means detached. That line now runs across every container from a script, because "the container is healthy" and "the container can reach anything" turned out to be independent facts.
The watcher that watches nothing#
My music library lives on a Windows drive mounted into WSL over 9p. The music server's file watcher started cleanly and logged that it was watching the library. Nothing was wrong in any log.
I tested it with three probe files. A write from inside another container fired the watcher. A write from WSL did not. A write from Windows did not. inotify_add_watch() on that mount succeeds and returns a valid watch descriptor, then delivers zero events, forever.
The two writes that failed are the only two ways music actually arrives on that box. The one that worked is exactly the test you would run if you were being careless, which is how this nearly got missed. The fix is a scheduled scan every hour, and that one config line is the only reason new music ever shows up.
MTU, three tunnels deep#
The network path for one container was three tunnels stacked: the router's WireGuard, then a Docker bridge network sitting inside that, then the container's own WireGuard inside the bridge. Each layer subtracts header space from the one outside it.
The container's default MTU was too large for the innermost layer. The symptom was "TLS handshake: context deadline exceeded", which reads as a dead server or bad credentials. It was neither. Small packets flowed and large ones silently vanished, so ping worked and the handshake did not. Before I pinned the bridge's MTU, the same class of bug produced every "flaky network" symptom I had: corrupted image pulls, truncated downloads, intermittent DNS failures, and a git clone ending in "early EOF".
The diagnostic is boring and decisive: send a do-not-fragment ping at increasing sizes until it fails, and you have the real MTU.
A check that cannot fail is not a check#
The lesson that changed how I write every monitor came from a leak checker.
I wrote a script to confirm that traffic from one segment of the network only left through its tunnel. It compared IPv6 addresses as strings. One tool prints IPv6 fully expanded; another prints it compressed. Whole-address string equality never matched, so the logic was wrong in both directions, and the output was still a reassuring "CLEAN".
The rewrite tests the dangerous condition directly (did this flow leave carrying a non-tunnel address?) instead of trying to enumerate every valid address. Then I proved it could fail: I ran it with the wrong prefix on purpose and confirmed it flagged essentially every flow.
That is now a rule for everything I build. A monitor that only ever prints green proves nothing until you have watched it print red. It applies equally to the futility monitors on my ML research loops, which is a different post.
Auth that disappears behind a proxy#
Several media-management apps have a setting called something like "authentication disabled for local addresses". It sounds reasonable on a home network.
Behind a reverse proxy or Docker's port proxy, every request arrives from a private address. So that setting means no authentication at all, for anyone who can reach the proxy. When I probed it, two of those apps served their full admin API key to an anonymous request on the LAN path. The tailnet path was clean, which is exactly why it went unnoticed.
The fix had two parts. Every app reachable from the LAN now requires a login regardless of source address. And the access contract is a generated test matrix: every service, every way of reaching it, the expected status code, and whether an anonymous request must be challenged. Probes run nightly from several vantage points, and a green morning is silent while a changed row is one line in the report.
The design rule underneath is binding follows auth. A service publishes on the LAN only if it has a real login of its own. Everything else binds to loopback and is reached through Tailscale, so the tailnet ACL is its access control.
A backup is a restore#
I used to think of backups as files that exist. Now I think of them as restores that have happened.
Each app's own consistent backup runs before anything is archived, because a tar of a live SQLite file can capture a torn write that restores into a database that will not open. Photos are mirrored to a second physical disk, deliberately without propagating deletes: a deletion spreading out of a corrupted library would erase the copy that exists to survive it. Orphans on the backup disk are cheap; a mirrored delete is not. Then the backups go off-site.
None of that counts until it is restored. A weekly drill takes the newest password-vault dump and one random original photo, restores them to a temporary directory, and compares hashes. The password vault was not opened to real accounts until its restore drill passed. Everything cheap to check about a backup file is equally true of one that will not restore, so the restore is the only check that means anything.
Shape three: a NAS in the closet#
The PC is a fine compute box and a poor appliance. It is also my desktop, and it has had hardware faults. So the media services are moving to a four-bay NAS.
The order of operations matters more than the tooling.
Burn-in before data. The drive is refurbished, so nothing was copied until a full SMART extended self-test passed, with reallocated sector counts checked before and after.
Replaceable data first. Until a second drive makes the pool a mirror, the NAS holds only the media library, which I could rebuild. Photos, files and passwords stay where they are, with their backups.
Waves, not a big bang. Media moves first; books and music follow after a RAM upgrade. Each wave is a runbook: a seed copy, a freeze, a final copy, a verify step that must pass, deploy, then prove every login with anonymous probes before the reverse proxy is switched over. If an app fails its auth check, the proxy does not switch for that app.
Keep a rollback copy. The source copy stays untouched and read-only for 14 days after cut-over. Rollback is three steps. The one thing it loses is anything written on the NAS after cut-over, and the thing people actually notice is watch history.
Updates that roll themselves back#
The first home stack ran :latest images, so any pull could silently change nine applications at once, including database migrations that cannot be undone. On the NAS, every image is pinned to a tag plus its SHA-256 digest, and nothing updates by itself.
An update tool checks the registries and applies patch and minor updates one service at a time, on a dated branch. It pulls, starts the container, waits for healthy, then holds for a minute watching the restart count and start time, because a crash-looping container can read "running" at every single poll. On any failure it restores the old image line, brings the old version back up, and records the candidate as held, so one bad release cannot block the rest of the queue. Major versions are reported and never applied automatically. It does not undo a database migration the new version already ran, and it says so.
Auto-heal on a short leash#
When the morning health check reports a problem, a heal script may start containers that are exited or missing from an expected list, through the same guarded deploy script I use by hand. It never touches a running container, even an unhealthy one. It never runs down, restart, pull or remove. A start only counts as healed if the container is still up, not unhealthy, and has not restarted 30 seconds later. After two failed attempts it stops and escalates to me.
The point of the leash is that automated repair is itself a source of silent failure. A healer that restarts a crash-looping container forever produces a system that always looks like it is recovering.
What I would tell someone starting#
- Measure what each service actually uses before buying hardware for it.
- Assume every "healthy" signal is lying until you have checked the thing it is supposed to imply.
- Write every monitor so you can force it red, and do so once.
- Treat "disable auth for local addresses" as "disable auth".
- A backup is a restore that has happened.
- Migrate with the old copy still there, and know exactly what a rollback loses.
The first afternoon is still the fun part. It is just a small fraction of the work.