`compose up -d` decides whether to replace a container by comparing its configuration, and the configuration is the string `runway:latest` whether or not the image behind that tag has moved. It never moves. So every deploy since this script was written swapped the frontend and left the backend exactly where it was, and the first one to matter was the one just now: the sidebar's new editor asks `/api/calendars` for `own_color`, and the backend answering it was built three days before that field existed. This is the third variation on one theme and the worst of them. The first failed loudly, the second failed quietly and exited 1; this one reports success, and what it leaves behind is precisely the state the single-image design exists to prevent -- a frontend calling an API the running backend does not have. `--force-recreate` is the fix. The check after it is the part worth keeping. Nothing above it could have caught this, because nothing failed: podman was asked to bring a service up, it was already up, and it said so. So the last thing the script does before recording a deploy is compare what is actually running against what it just pulled, and refuse to write the stamp if they disagree. A run that half-happens now leaves no record, which is what makes the next tick try again.
Deploying Runway
The app is one image holding both halves — the server binary and the built
frontend — pushed by CI on every merge to main and picked up by a timer on
the server. Nothing is copied from a laptop, and there is no step that can
deploy one half without the other. That was the audit's one "do it differently"
(F76): v1's frontend went up by rsync over SSH, by hand, separately from its
backend's CI.
What runs where
| Image | git.rcjohnstone.com/connor/runway:latest, and :<commit> |
| Built by | .gitea/workflows/release.yml, on push to main |
| Deployed by | deploy/runway-update, from a systemd system timer, as root |
| Backend | runway-backend in ~/compose.yml, on the internal network, in root podman |
| Frontend | ~/data/runway/web/dist, served by the root Caddy from /srv/runway/dist |
| Database | ~/data/runway/db, a volume on the backend container |
| TLS and auth | The root Caddy, which already fronts runway.rcjohnstone.com behind Authelia |
The frontend asks for /api/... relative to wherever it is served, so the same
image is correct in every environment and nothing is compiled in per host. v1
baked the API's absolute URL into the WASM, which is why its frontend could not
be deployed by the same pipeline as its backend.
Setting it up
Six things, of which five are one-time.
1. The image. Set on the Gitea repository, under Settings → Actions:
- variable
REGISTRY=git.rcjohnstone.com - variable
USERNAME=connor - secret
DOCKER_PASSWORD= a token with package write
These are the names v1's workflow used, so they may already exist at the organisation level.
2. The signing key. Generate it once, on the server, and keep it:
podman run --rm git.rcjohnstone.com/connor/runway:latest runway-server genkey
Put the output in ~/.env as RUNWAY_SECRET_KEY. It encrypts the stored
CalDAV passwords, so if it changes, every saved credential becomes
unreadable and everybody has to sign in again. It is not derived from
anything and cannot be recovered; back it up with the rest of ~/.env.
3. The compose service. Replace the existing runway-backend block:
runway-backend:
image: git.rcjohnstone.com/connor/runway:latest
restart: unless-stopped
networks:
- internal
environment:
# Absent on purpose: RUNWAY_INSECURE_COOKIES. Caddy terminates TLS, so
# the session cookie is Secure, and setting this would quietly stop it
# being so.
- RUNWAY_SECRET_KEY=${RUNWAY_SECRET_KEY}
- RUNWAY_DATABASE_URL=sqlite:///db/runway.db
- TZ=America/Louisville
volumes:
- ./data/runway/db:/db
~/data/runway/db is root-owned from v1 and the container still runs as root,
so it keeps working untouched. To run it unprivileged instead, chown that
directory and add a user: line.
The database is runway.db, not v1's calendar.db, so this starts empty —
which is what you want. There is nothing to migrate across: the schemas share
no ancestry, and v1's stored passwords were encrypted with a key this app does
not have.
4. Caddy. The proxy half needs no change — the existing
runway.rcjohnstone.com block already sends /api/* to runway-backend:3000
and serves the rest with an index.html fallback. The mount does need one.
Caddy has to mount the directory that holds dist, not dist itself:
volumes:
- ./data/runway/web:/srv/runway:ro # was ./data/runway/dist
root * /srv/runway/dist # was /srv/runway
A bind mount resolves to an inode when the container starts. A deploy replaces
the frontend by renaming one directory over another, which leaves the old inode
exactly where Caddy is still looking — so mounting dist means Caddy serves the
previous frontend until something unrelated happens to restart it, and the
rollback below silently does nothing too. Mounting the holder makes the rename
happen inside what Caddy can see, which is the whole point of doing it as a
rename.
web/ exists so that the holder contains only things Caddy may serve. Mounting
~/data/runway directly would put the database one root directive away from
being public.
5. The frontend directory. It has to exist before the first deploy — the
updater replaces things inside it and refuses to build the tree itself, because
the case where it would is sudo handing it $HOME=/root and a deploy into a
directory nobody serves:
mkdir -p ~/data/runway/web
The updater runs as root, so it can write there whatever the ownership. It
still checks, and still prints the chown to fix it, for the case where
somebody runs it as themselves.
~/data/runway/db stays where it is, outside web/ and so outside Caddy's
mount, which is the point of the extra directory.
6. The timer. A system timer, not a user one:
sudo ln -sf /home/connor/docs/projects/runway/deploy/runway-update.service /etc/systemd/system/
sudo ln -sf /home/connor/docs/projects/runway/deploy/runway-update.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now runway-update.timer
As root, because that is the podman the rest of this machine runs in. Run as
your own account it deploys a backend into your rootless namespace instead:
the container starts, the script reports success, and Caddy — which is in root
podman — cannot resolve runway-backend to it at all. That failure is
invisible from inside the script, which is why the choice is written down here
rather than left to whichever podman is first on $PATH.
The unit sets RUNWAY_DIST and RUNWAY_COMPOSE_DIR explicitly, since the
script's defaults hang off $HOME and root's is /root.
Deploying by hand
The timer runs the same script, so this is what it does, and it is also the answer to "I do not want to wait ten minutes":
sudo systemctl start runway-update.service
journalctl -u runway-update.service -n 20 --no-pager
Through the unit rather than by running the script, so that it gets the same
environment the timer gives it. sudo deploy/runway-update is not the same
command: $HOME is /root under sudo, so the paths differ. The script stops
rather than guessing, but the tidier habit is to not ask it to.
It exits without touching anything if the image it pulls is already the one
deployed — which it knows from ~/data/runway/web/dist.deployed, written at
the end of a deploy that finished. Anything that fails before that point leaves
no record, so the next run does the whole thing again rather than mistaking a
pulled image for a deployed one.
Rolling back
The frontend of the previous deploy is kept:
cd ~/data/runway/web
sudo mv dist dist.bad && sudo mv dist.previous dist
It takes effect immediately, with no restart, because the rename happens inside the directory Caddy mounts.
The timer will leave it alone rather than undoing it on the next tick:
web/dist.deployed still names the image you rolled away from, so the updater
sees nothing to do until a genuinely new image is pushed. Which is the
behaviour you want at the moment you are rolling something back, and the reason
to fix forward rather than sit on a rollback.
The backend is a tag. Pin the service's image: to
git.rcjohnstone.com/connor/runway:<commit> and bring it up; the timer will
leave a pinned tag alone, because it only ever pulls what the service names.
Rolling back across a migration is the case to think about before doing it: migrations only go forwards, so an older binary meeting a newer database is not something to try casually. Roll the frontend back first — that is usually the half that is wrong, and it is reversible in a way the schema is not.