Marek Tran

A film night is a perfectly reasonable excuse to build a distributed system. At least, that is how I ended up with one Jellyfin, a k3s cluster spread across friends' homes and a cloud VM, PostgreSQL, and a filesystem called SeaweedFS. I wanted to find out whether the spare machines and disks we already had could behave like one resilient media service. The most useful result was discovering which parts should not fail over.

This is the first entry in my homelab journal, and a pilot rather than a production recipe. I like self-hosting partly because it lets me follow a question all the way down: from “can we watch this?” to database consistency, WAN latency and an Intel encoder setting. Sometimes the answer is a nicer architecture. Sometimes it is a very well-justified teardown.

Where this ended: both experimental stacks were torn down by 13 October. The manifests are parked, not running. What follows reconstructs the experiments from my notes and Git history; the simpler design at the end is the proposed next build, not a deployment announcement.

One server, several homes

The idea was not to keep a complete copy of everybody's library everywhere. Each owner would contribute storage, and their media would stay on their own node. Jellyfin would present those separate libraries through one service.

That gives two different problems which are easy to blur together:

• Service recovery: can Jellyfin and its database recover when a machine disappears?
• Media availability: can the replacement still read the particular film somebody wants?

Kubernetes can move a pod. It cannot move a disconnected house's disks into reach.

Nor was this an active-active Jellyfin design. The application had one active writer. “Failover” meant restarting that writer somewhere else, not running a fleet of interchangeable Jellyfin instances against shared state.

The route was not a straight line

The September experiments started with SQLite, Litestream, backups and explicit fencing. The interesting question was not merely whether another pod would start, but whether the isolated original could keep writing. In the earlier tests, a lease-based sidecar stopped the isolated writer within about 20 seconds, and the pre-isolation state remained canonical after reconnection.

Optional media mounts were another trap. An unavailable NFS server could block a replacement pod before Jellyfin even started. Host automounts and a watchdog improved that, but meant carrying a node-level mounting layer alongside the Kubernetes configuration. The service was distributed; so was the list of things I had to remember.

In early October I moved the database work toward PostgreSQL and CloudNativePG. Then I rebuilt the storage experiment around SeaweedFS on 12–13 October, measured the result, and tore it down. Keeping that sequence matters: the last pilot was a deliberately small testbed, not every earlier HA mechanism combined into one finished system.

PostgreSQL, with a very important credit

The PostgreSQL route was possible because of JPVenson's Jellyfin.Pgsql. I used the custom image ghcr.io/jpvenson/jellyfin.pgsql:12.1-2; this was not stock Jellyfin with a magic environment variable, and it was not my database-provider implementation.

The upstream project explicitly calls itself highly experimental and not intended for production. That warning belongs next to the exciting part, not hidden below it. In my pilot, scanning, direct streaming, transcoding and resume worked. My notes also record a PK_UserData insert failure at playback start: playback positions saved, but play count and last-played did not. That is an observation about the tested image, not a claim about every version of the project.

CloudNativePG addresses a different layer: managing PostgreSQL instances and recovery. It was part of the earlier HA iteration and remains the choice in the proposed rebuild. Its primary/replica model and rewind path are a much better fit for the intended single canonical database than inventing more application-specific SQLite fencing.

The 2 October notes do record a healthy three-instance CNPG cluster, a planned switchover in about nine seconds, and the old primary rejoining as a replica within a minute. That was useful progress, but a planned switchover is not a node-loss or network-partition test; those remained follow-up work in that entry.

The distinction: the final SeaweedFS pilot used one plain PostgreSQL StatefulSet, not a replicated CNPG cluster. Its successful playback is not evidence of CNPG failover. Rejoining an old primary also needs the recovery procedure to be right; “discard divergent offline writes” is the intended policy, not permission to assume partitions are harmless.

Postgres did not make Jellyfin stateless either. Library definitions, server identity and XML configuration still lived in /config. Pointing an empty config directory at the same database did not recreate the libraries. A recovery plan has to account for both.

The primary's location mattered, too. My notes recorded several dependent database round trips per client call: a six-call sequence took around 0.2 seconds locally and 1.3 seconds with about 35 ms between application and database. Connection pooling cannot erase serial network round trips. The next design keeps the primary beside Jellyfin.

SeaweedFS: distribution is not redundancy

SeaweedFS gave me a filer namespace, volume servers on the storage nodes, and a FUSE mount for Jellyfin. Path-placement rules directed each owner's shard to its designated node. The pilot demonstrated that placement and playback from every shard worked.

There are three different ideas hiding behind the word “sharding” here:

1. Library sharding: divide the media collection by owner or location. A whole film belongs to one logical shard.
2. File chunking: SeaweedFS can represent a large file as smaller stored chunks, with metadata describing how they form the file. A mounted client still sees a normal file. Chunking is an internal storage representation, not a promise that each chunk has a spare copy.
3. Replication or erasure coding: add redundant information so storage can survive failures. Replication stores additional copies; erasure coding uses data and parity fragments. Neither follows automatically from splitting a file into chunks.

My placement rules used replication=000: one copy, pinned to the shard's owner node. The chunks belonging to that shard were not deliberately striped across friends' homes for resilience. No erasure-coding setup or test was part of this pilot. Losing contact with the owner meant losing access to its media; losing its only disk would be a separate durability problem. Sharding is not a backup.

SeaweedFS itself has replication and HA options, but the pilot did not exercise a highly available metadata tier either: it had one master and one filer with embedded metadata storage. A common mount endpoint is a useful interface, not proof that the endpoint survives failure.

One hoped-for benefit was a cleaner distinction between an unreachable shard and deleted media. I had not yet tested a dead volume server or measured time-to-error, so I cannot claim that failure behavior was solved.

A library scan can be destructive without deleting a file

One of the best lessons came from testing Jellyfin 10.11's scan behavior. A missing or entirely empty library folder was skipped. A reachable folder with only some files missing was different: those missing items were removed from the catalogue.

When a file returned at the same path, its path-derived ID and watch history returned after a scan, but manual metadata, collections and playlists did not necessarily survive that round trip.

That pushed me toward one library folder per shard, stable opaque shard names, and no merged filesystem view that could quietly present a partial library as complete. Scanning is an operation with failure semantics, not just housekeeping. FUSE also meant no usable inotify-based realtime monitoring in this setup, so scans had to be triggered.

The numbers that changed the goal

The fun part was seeing playback actually work across the shards. The sobering part was following where the bytes had to travel.

A remote media read is limited by the media owner's upload. A viewer outside Jellyfin's site also consumes that site's outbound bandwidth. Moving Jellyfin does not remove either physical constraint; it may simply put the application farther from the useful storage.

GPU placement was similarly concrete. In the recorded single-stream, 20-second 4K HEVC HDR-to-SDR tests, the N97 went from about 0.80× realtime with the regular HEVC encoder to 2.58× with the low-power encoder. The i5-12400 reference went from 1.04× to 3.09×. That checkbox mattered more than another layer of orchestration. These were short runs on one clip, not proof of sustained or concurrent capacity.

Then there was the outage the design was supposed to help with: roughly three minutes of lost connectivity at the main site. With about 45 seconds before NotReady and a 300-second eviction toleration, automatic relocation would not even begin before that outage ended. Shortening timers would not bring the main site's media back online.

The four-voter etcd layout also had a bad property: two votes at the main site and one at each of two other sites. Splitting the main site away left neither side with the three votes needed for quorum. More nodes had not produced the failure tolerance I actually wanted.

The sentence I kept was: whatever holds the content is the service.

The next build is less clever

The proposed next version keeps a hub with Jellyfin, its config, the PostgreSQL primary and the most-used media, especially 4K. Remote nodes contribute long-tail storage, an off-site database copy and sensibly distributed quorum. CNPG manages the database layer; a long outage gets a manual recovery runbook, not an automatic promise that an unreachable library will become reachable somewhere else.

I still want three appropriately placed etcd voters, an actual simulated hub outage, a dead-volume-server test, and two concurrent transcodes on the chosen hub hardware. The latest pilot did not validate those. The GPU measurements used browser-oriented test conditions, one file per class and short runs; a TV that direct-plays HDR can have a very different workload.

None of that makes the experiment a disappointment. Getting the distributed version working was enjoyable. Finding the boundary where distribution stopped helping was the better result. That is the homelab part I want to keep writing about: not a rack that looks finished, but the questions, the odd failures, and the occasional simpler design earned the hard way.

---

Based on my September–October 2026 experiment history and the conclusion recorded on 13 October. Historical measurements above were not rerun for this post. Public starting points: Jellyfin.Pgsql and its safety warning, CloudNativePG documentation, and SeaweedFS's architecture overview, replication and erasure-coding documentation.