Files

T

connor e1e253319e Stop the deploy succeeding without deploying

The chown from the last commit unblocked the updater, and the updater then went
on not deploying -- quietly, and now with exit 0. Three separate reasons, none
of which the script could see from where it stood.

It compared the image it had just pulled against the image it had before, which
answers "is there anything new?" rather than "is what is running current". The
failed run had already pulled, so the next run saw nothing new and stopped. It
now records the image in `dist.deployed` at the end of a run that finished, and
compares against that; a run that dies half way leaves no record and the next
one does the whole thing again.

It deployed into the wrong podman. The stack on this machine -- Caddy, Authelia
and forty others -- is root podman, and the timer was a user unit, so `podman
compose up` built a perfectly healthy backend in a rootless namespace that
Caddy has no route to, and said so approvingly. The unit is now a system unit.
That failure is worth naming because nothing local to the script can detect it:
the container starts, the command succeeds, and the only symptom is somewhere
else entirely.

And Caddy could not see the frontend anyway. A bind mount holds an inode, so
renaming `dist` out from under it leaves Caddy serving the directory that was
there when it started -- which is also why rolling back by `mv`, as the README
promised, would have done nothing at all. Caddy now mounts `web/`, the holder,
so the rename happens inside what it can see. The database stays outside that
directory, which is the reason for the extra level rather than mounting
`data/runway` itself.

One more, found by fixing the second: a container started from a `oneshot` unit
inherits that unit's cgroup, so systemd tore down the backend the moment
ExecStart returned. `restart: unless-stopped` put it back, which is precisely
what made it hard to see -- the site was fine and the unit was red.
`KillMode=process` leaves the container where podman put it.

2026-08-28 16:02:00 -04:00

7.6 KiB

Raw Permalink Blame History

Deploying Runway

The app is one image holding both halves — the server binary and the built frontend — pushed by CI on every merge to main and picked up by a timer on the server. Nothing is copied from a laptop, and there is no step that can deploy one half without the other. That was the audit's one "do it differently" (F76): v1's frontend went up by rsync over SSH, by hand, separately from its backend's CI.

What runs where


Image	`git.rcjohnstone.com/connor/runway:latest`, and `:<commit>`
Built by	`.gitea/workflows/release.yml`, on push to `main`
Deployed by	`deploy/runway-update`, from a systemd system timer, as root
Backend	`runway-backend` in `~/compose.yml`, on the `internal` network, in root podman
Frontend	`~/data/runway/web/dist`, served by the root Caddy from `/srv/runway/dist`
Database	`~/data/runway/db`, a volume on the backend container
TLS and auth	The root Caddy, which already fronts `runway.rcjohnstone.com` behind Authelia

The frontend asks for /api/... relative to wherever it is served, so the same image is correct in every environment and nothing is compiled in per host. v1 baked the API's absolute URL into the WASM, which is why its frontend could not be deployed by the same pipeline as its backend.

Setting it up

Six things, of which five are one-time.

1. The image. Set on the Gitea repository, under Settings → Actions:

variable REGISTRY = git.rcjohnstone.com
variable USERNAME = connor
secret DOCKER_PASSWORD = a token with package write

These are the names v1's workflow used, so they may already exist at the organisation level.

2. The signing key. Generate it once, on the server, and keep it:

podman run --rm git.rcjohnstone.com/connor/runway:latest runway-server genkey

Put the output in ~/.env as RUNWAY_SECRET_KEY. It encrypts the stored CalDAV passwords, so if it changes, every saved credential becomes unreadable and everybody has to sign in again. It is not derived from anything and cannot be recovered; back it up with the rest of ~/.env.

3. The compose service. Replace the existing runway-backend block:

  runway-backend:
    image: git.rcjohnstone.com/connor/runway:latest
    restart: unless-stopped
    networks:
      - internal
    environment:
      # Absent on purpose: RUNWAY_INSECURE_COOKIES. Caddy terminates TLS, so
      # the session cookie is Secure, and setting this would quietly stop it
      # being so.
      - RUNWAY_SECRET_KEY=${RUNWAY_SECRET_KEY}
      - RUNWAY_DATABASE_URL=sqlite:///db/runway.db
      - TZ=America/Louisville
    volumes:
      - ./data/runway/db:/db

~/data/runway/db is root-owned from v1 and the container still runs as root, so it keeps working untouched. To run it unprivileged instead, chown that directory and add a user: line.

The database is runway.db, not v1's calendar.db, so this starts empty — which is what you want. There is nothing to migrate across: the schemas share no ancestry, and v1's stored passwords were encrypted with a key this app does not have.

4. Caddy. The proxy half needs no change — the existing runway.rcjohnstone.com block already sends /api/* to runway-backend:3000 and serves the rest with an index.html fallback. The mount does need one.

Caddy has to mount the directory that holds dist, not dist itself:

    volumes:
      - ./data/runway/web:/srv/runway:ro    # was ./data/runway/dist

        root * /srv/runway/dist             # was /srv/runway

A bind mount resolves to an inode when the container starts. A deploy replaces the frontend by renaming one directory over another, which leaves the old inode exactly where Caddy is still looking — so mounting dist means Caddy serves the previous frontend until something unrelated happens to restart it, and the rollback below silently does nothing too. Mounting the holder makes the rename happen inside what Caddy can see, which is the whole point of doing it as a rename.

web/ exists so that the holder contains only things Caddy may serve. Mounting ~/data/runway directly would put the database one root directive away from being public.

5. The frontend directory. It has to exist before the first deploy — the updater replaces things inside it and refuses to build the tree itself, because the case where it would is sudo handing it $HOME=/root and a deploy into a directory nobody serves:

mkdir -p ~/data/runway/web

The updater runs as root, so it can write there whatever the ownership. It still checks, and still prints the chown to fix it, for the case where somebody runs it as themselves.

~/data/runway/db stays where it is, outside web/ and so outside Caddy's mount, which is the point of the extra directory.

6. The timer. A system timer, not a user one:

sudo ln -sf /home/connor/docs/projects/runway/deploy/runway-update.service /etc/systemd/system/
sudo ln -sf /home/connor/docs/projects/runway/deploy/runway-update.timer   /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now runway-update.timer

As root, because that is the podman the rest of this machine runs in. Run as your own account it deploys a backend into your rootless namespace instead: the container starts, the script reports success, and Caddy — which is in root podman — cannot resolve runway-backend to it at all. That failure is invisible from inside the script, which is why the choice is written down here rather than left to whichever podman is first on $PATH.

The unit sets RUNWAY_DIST and RUNWAY_COMPOSE_DIR explicitly, since the script's defaults hang off $HOME and root's is /root.

Deploying by hand

The timer runs the same script, so this is what it does, and it is also the answer to "I do not want to wait ten minutes":

sudo systemctl start runway-update.service
journalctl -u runway-update.service -n 20 --no-pager

Through the unit rather than by running the script, so that it gets the same environment the timer gives it. sudo deploy/runway-update is not the same command: $HOME is /root under sudo, so the paths differ. The script stops rather than guessing, but the tidier habit is to not ask it to.

It exits without touching anything if the image it pulls is already the one deployed — which it knows from ~/data/runway/web/dist.deployed, written at the end of a deploy that finished. Anything that fails before that point leaves no record, so the next run does the whole thing again rather than mistaking a pulled image for a deployed one.

Rolling back

The frontend of the previous deploy is kept:

cd ~/data/runway/web
sudo mv dist dist.bad && sudo mv dist.previous dist

It takes effect immediately, with no restart, because the rename happens inside the directory Caddy mounts.

The timer will leave it alone rather than undoing it on the next tick: web/dist.deployed still names the image you rolled away from, so the updater sees nothing to do until a genuinely new image is pushed. Which is the behaviour you want at the moment you are rolling something back, and the reason to fix forward rather than sit on a rollback.

The backend is a tag. Pin the service's image: to git.rcjohnstone.com/connor/runway:<commit> and bring it up; the timer will leave a pinned tag alone, because it only ever pulls what the service names.

Rolling back across a migration is the case to think about before doing it: migrations only go forwards, so an older binary meeting a newer database is not something to try casually. Roll the frontend back first — that is usually the half that is wrong, and it is reversible in a way the schema is not.

7.6 KiB Raw Permalink Blame History