Engineering

Why we rebuilt Sproobo on Docker-everything

July 11, 2026 · 5 min read

Every recurring outage in Sproobo v2 traced back to the same root cause: production behavior tied to whatever package manager, PPA, or PATH happened to be on the host that week. v6 deletes that failure class by running everything — apps and the databases behind them — as pinned containers, and keeping only the data that actually needs to survive on the host at all.

The outage postmortem that never ended

v2 provisioned software the way most of the industry still does: native packages, installed straight onto the host, managed by whatever tool distributes them. That worked fine as engineering, and failed constantly as operations. Four incident classes kept recurring, and every one traces back to the same design decision.

NodeSource versus the distro's own libnode-dev. Two competing package sources for the same runtime, and a dpkg conflict waiting for the day an unrelated apt upgrade reshuffled dependencies.

The ondrej/php PPA lagging new Ubuntu codenames. A new release would ship, and the PPA serving PHP builds wouldn't have packages for it yet — a routine OS upgrade quietly became a broken PHP install.

nvm and PM2 PATH fragility. Version managers and process managers each maintain their own idea of PATH and shell profile; a cron job or a systemd unit invoked outside an interactive shell would resolve the wrong binary, or none at all.

A foreign container seizing :80/:443. Nothing stopped some other container on the box from binding the public ports before Sproobo's own proxy did, silently serving the wrong certificate to real traffic.

None of these are bugs in the ordinary sense. They're the cost of coupling something that should be a versioned, swappable dependency to the mutable, shared state of a live production host.

The pattern underneath the outages

Every one of those incidents has the same shape: something Sproobo depends on — a runtime, a PPA-fed package set, a PATH entry, a listening port — lives in shared, mutable host state that anything else on the box can also touch, upgrade, or race for. Fixing any single instance only patches that one failure. Pin the Node version here, vendor the PHP build there, and the class stays open, because the dependency still lives on the host.

The fix isn't a better installer. It's removing the host as a place where dependencies live at all.

Docker for everything, not just the app

v6's answer is to stop treating containers as something only the application gets. Every workload — the app and every backing service it depends on: PostgreSQL, MySQL, MariaDB, Redis — runs as a container with a version you choose and pin yourself. A database upgrade happens on your schedule, never when an apt repository or a PPA decides for you — and it's never a bare in-place tag flip either: upgrades run through a backup-gated recipe into a fresh instance, with the old data directory left intact as a rollback. See how that catalog is put together on Databases & services.

The same logic extends to how images get built in the first place. Nothing gets compiled on a customer's box: every image is built off-box, in a capped control-plane builder, and the agent only ever pulls a finished, tagged image from Sproobo's own registry. A runaway build competing with a live database for CPU, memory, or disk was just another shape of the same v2 problem — one workload able to starve its neighbors because they all shared a host. It can't happen when the build never runs on the server at all.

Durable data still has to survive — bind mounts, not volumes

Making the runtime disposable only works if the data isn't disposable along with it. v6 keeps anything durable — database files, app uploads, anything a restart can't lose — on bind-mounted host directories under /var/lib/sproobo/..., never in a Docker named volume:

$ docker compose down -v
$ ls /var/lib/sproobo/services/pg-8f2a/data
base  pg_wal  postgresql.conf  PG_VERSION

A bind mount isn't tied to the container's lifecycle the way a named volume is — down -v and docker volume prune can't touch it, because Docker never owned it in the first place. It's inspectable and recoverable with plain tools — ls, tar, rsync — no Sproobo client required. Read the rest of that data-ownership model on Security model.

Logic lives in the control plane, not on the fleet

The other half of the fix is where behavior lives, not just where it runs. v2's provisioning logic — and its bugs — shipped as scripts executed directly on customer hosts. In v6, the control plane compiles desired state into a declarative plan, and the on-server agent — a single static Go binary — only ever executes a small, generic set of typed steps: bring up an image, ensure a container, ensure a mount, ensure a proxy route, ensure a certificate, ensure a firewall rule. Deploy and provisioning behavior changes ship centrally, in the control plane's compiler, instead of as a new agent binary rolled out across every server one host at a time. Walk the full enroll-to-deploy path on Getting started, and see what that blue-green pipeline looks like once it's running on Apps & deploys.

A strangler migration, not a rewrite gamble

None of this shipped as a big-bang cutover. v6 runs alongside v2 today, and servers move over one at a time, by re-enrolling — so the new architecture proves itself against real traffic before it has to carry all of it. That's slower than declaring victory on day one. It's also the only way to bet on an architecture change without betting the whole fleet on it at once.

← Back to the blog

Get started

Run this on a server you own.