Files
connor 43b94489f5
Check / guardrails (push) Successful in 1m43s
Check / bundle (push) Successful in 1m15s
Check / check (push) Successful in 2m42s
Image / image (push) Failing after 20s
v0.2 wave 1: merge-rule-v2 clustering with LLM-candidate gray zone, 0.2.0 version plumbing, deploy verification docs
2026-09-03 16:46:12 -04:00
..

Deploying news-triage

The app is one image holding both halves: the server binary and the built frontend, pushed by CI on every merge to main and picked up by a timer on the server. Nothing is copied from a laptop, and there is no step that can deploy one half without the other. That matches the lesson from runway's v1: a frontend calling an endpoint the backend does not have is what happens when the two halves travel separately.

What runs where

Image git.rcjohnstone.com/connor/news:latest, and :<commit>
Built by .gitea/workflows/release.yml, on push to main
Deployed by deploy/news-update, from a systemd system timer, as root
Backend news-backend in ~/compose.yml, on the internal network, in root podman
Frontend ~/data/news/web/dist, served by the root Caddy from /srv/news/dist
Database ~/data/news/db, a volume on the backend container
Config ~/config/news/config.toml, mounted read-only at /config/config.toml
TLS and auth The root Caddy, on news.rcjohnstone.com, LAN/VPN-only behind the internal snippet

The frontend asks for /api/... relative to wherever it is served, so the same image is correct in every environment and nothing is compiled in per host.

Setting it up

Eight things, all of them one-time.

1. The image. Set on the Gitea repository, under Settings → Actions:

  • variable REGISTRY = git.rcjohnstone.com
  • variable USERNAME = connor
  • secret DOCKER_PASSWORD = a token with package write

These are the same names runway's workflow uses.

2. The config file. Copy the example from the repo and edit it:

mkdir -p ~/config/news
cp /home/connor/docs/projects/news/config.example.toml \
   ~/config/news/config.toml.example

The real file is ./config/news/config.toml. The binary loads it from /config/config.toml inside the container and reloads it on SIGHUP. The schema is in crates/news-server/src/config.rs: [[sources]], [notify], and [topics] are all required. config.example.toml in the repo root is a working starting point.

3. The compose service. Add a news-backend block to ~/compose.yml:

  news-backend:
    image: git.rcjohnstone.com/connor/news:latest
    restart: unless-stopped
    networks:
      - internal
    environment:
      - NEWS_DATABASE_URL=sqlite:///db/news.db
      - TZ=America/Louisville
      - NEWS_API_BASE_URL=https://news.rcjohnstone.com
      - NTFY_URL=${NTFY_URL}
      - NTFY_USER=${NTFY_USER}
      - NTFY_PASS=${NTFY_PASS}
    volumes:
      - ./data/news/db:/db
      - ./config/news:/config:ro
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 5s

NEWS_API_BASE_URL is what the notification feedback buttons (Interested / Not interested) POST to, and ntfy runs those http actions on the phone — so it has to be the name Caddy serves, never localhost. When it is unset the buttons are omitted from notifications and the server logs a warning.

Put NTFY_URL, NTFY_USER, and NTFY_PASS in ~/.env, alongside the other container secrets. The database is news.db at ./data/news/db; it starts empty and the migrations run at container startup.

4. Caddy. Add a news.rcjohnstone.com block to ~/Caddyfile. It is LAN/VPN-only, like grafana and lazylibrarian: internal CA, source-IP gating, and no # ddns: public marker.

news.rcjohnstone.com {
	# Internal CA and no `# ddns: public` marker. Publishing this name would
	# put it in Certificate Transparency permanently and make a personal
	# dashboard reachable from the internet. Source-IP gating is the control;
	# the missing public A record is what keeps the name off CT for good.
	tls internal
	import internal
	import secure_headers
	import accesslog
	@backend {
		path /api /api/*
	}
	handle @backend {
		reverse_proxy news-backend:3000
	}
	handle {
		root * /srv/news/dist
		file_server
		try_files {path} /index.html
	}
}

Caddy has to mount the directory that holds dist, not dist itself:

    volumes:
      - ./data/news/web:/srv/news:ro    # was ./data/news/dist
        root * /srv/news/dist             # was /srv/news

A bind mount resolves to an inode when the container starts. A deploy replaces the frontend by renaming one directory over another, which leaves the old inode exactly where Caddy is still looking, so mounting dist means Caddy serves the previous frontend until something unrelated happens to restart it, and the rollback below silently does nothing too. Mounting the holder makes the rename happen inside what Caddy can see, which is the whole point of doing it as a rename.

web/ exists so that the holder contains only things Caddy may serve. Mounting ~/data/news directly would put the database one root directive away from being public.

5. The frontend directory. It has to exist before the first deploy. The updater replaces things inside it and refuses to build the tree itself, because the case where it would is sudo handing it $HOME=/root and a deploy into a directory nobody serves:

mkdir -p ~/data/news/web

The updater runs as root, so it can write there whatever the ownership. It still checks, and still prints the chown to fix it, for the case where somebody runs it as themselves.

~/data/news/db stays where it is, outside web/ and so outside Caddy's mount, which is the point of the extra directory.

6. ntfy access. The ntfy server is configured deny-by-default:

# /home/connor/data/ntfy/etc/server.yml
auth-default-access: "deny-all"

So the publishing user must be created and granted read/write access to the news-triage topic before news-server can publish notifications. Run from the host that owns the ntfy container:

# Create the publishing user (interactive password prompt).
sudo podman exec -it ntfy ntfy user add news-bot

# Grant read/write access to the news-triage topic.
sudo podman exec ntfy ntfy access news-bot news-triage rw

Store the resulting password in ~/.env as NTFY_USER and NTFY_PASS.

7. Prometheus. Add a scrape job to ./config/prometheus/prometheus.yml:

  - job_name: news-triage
    static_configs: [{ targets: ['news-backend:3000'] }]
    metrics_path: /metrics

The file is mounted read-only into the prometheus container, so this is a one-time edit.

8. The timer. A system timer, not a user one:

sudo ln -sf /home/connor/docs/projects/news/deploy/news-update.service /etc/systemd/system/
sudo ln -sf /home/connor/docs/projects/news/deploy/news-update.timer   /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now news-update.timer

As root, because that is the podman the rest of this machine runs in. Run as your own account it deploys a backend into your rootless namespace instead: the container starts, the script reports success, and Caddy, which is in root podman, cannot resolve news-backend to it at all. That failure is invisible from inside the script, which is why the choice is written down here rather than left to whichever podman is first on $PATH.

The unit sets NEWS_DIST and NEWS_COMPOSE_DIR explicitly, since the script's defaults hang off $HOME and root's is /root.

Deploying by hand

The timer runs the same script, so this is what it does, and it is also the answer to "I do not want to wait ten minutes":

sudo systemctl start news-update.service
journalctl -u news-update.service -n 20 --no-pager

Through the unit rather than by running the script, so that it gets the same environment the timer gives it. sudo deploy/news-update is not the same command: $HOME is /root under sudo, so the paths differ. The script stops rather than guessing, but the tidier habit is to not ask it to.

It exits without touching anything if the image it pulls is already the one deployed, which it knows from ~/data/news/web/dist.deployed, written at the end of a deploy that finished. Anything that fails before that point leaves no record, so the next run does the whole thing again rather than mistaking a pulled image for a deployed one.

Verifying a deployment

The version shows up in three places, and after a deploy they should all tell the same story.

The web footer shows v{version} · {short sha}, fetched from the API when the page loads; if that fetch fails, no footer renders at all. The same endpoint, asked directly:

curl https://news.rcjohnstone.com/api/version
# {"version":"0.2.0","git_sha":"…","build_time":"…"}

The image carries the same facts as OCI labels, and the tags show what points where:

sudo podman inspect git.rcjohnstone.com/connor/news:latest \
  --format '{{index .Config.Labels "org.opencontainers.image.version"}} {{index .Config.Labels "org.opencontainers.image.revision"}}'
sudo podman images git.rcjohnstone.com/connor/news

CI tags a single image id as latest, the short commit sha, and the release tag (v0.2.0), so podman images lists three rows with the same image id — and the labels are the durable record of which version and which commit an image was actually built from.

The updater keeps its own trail: web/dist.deployed holds the image id of the last deploy that got all the way through, and the service journal holds every run, including the ones that changed nothing:

cat ~/data/news/web/dist.deployed
journalctl -u news-update.service -n 50 --no-pager

systemctl list-timers news-update.timer says when the next check fires.

None of that changes what the updater does: it still pulls :latest, compares the id against dist.deployed, and exits when they match. The vX.Y.Z tags are for people, not for the updater — a way to pin the compose file's image: or to roll back to a build you can name by version instead of a commit hash you have to go and look up.

Rolling back

The frontend of the previous deploy is kept:

cd ~/data/news/web
sudo mv dist dist.bad && sudo mv dist.previous dist

It takes effect immediately, with no restart, because the rename happens inside the directory Caddy mounts.

The timer will leave it alone rather than undoing it on the next tick: web/dist.deployed still names the image you rolled away from, so the updater sees nothing to do until a genuinely new image is pushed. Which is the behaviour you want at the moment you are rolling something back, and the reason to fix forward rather than sit on a rollback.

The backend is a tag. Pin the service's image: to git.rcjohnstone.com/connor/news:<commit> and bring it up; the timer will leave a pinned tag alone, because it only ever pulls what the service names.

Rolling back across a migration is the case to think about before doing it: migrations only go forwards, so an older binary meeting a newer database is not something to try casually. Roll the frontend back first. That is usually the half that is wrong, and it is reversible in a way the schema is not.