Deploying news-triage
The app is one image holding both halves: the server binary and the built
frontend, pushed by CI on every merge to main and picked up by a timer on
the server. Nothing is copied from a laptop, and there is no step that can
deploy one half without the other. That matches the lesson from runway's v1:
a frontend calling an endpoint the backend does not have is what happens when
the two halves travel separately.
What runs where
| Image | git.rcjohnstone.com/connor/news:latest, and :<commit> |
| Built by | .gitea/workflows/release.yml, on push to main |
| Deployed by | deploy/news-update, from a systemd system timer, as root |
| Backend | news-backend in ~/compose.yml, on the internal network, in root podman |
| Frontend | ~/data/news/web/dist, served by the root Caddy from /srv/news/dist |
| Database | ~/data/news/db, a volume on the backend container |
| Config | ~/config/news/config.toml, mounted read-only at /config/config.toml |
| TLS and auth | The root Caddy, on news.rcjohnstone.com, LAN/VPN-only behind the internal snippet |
The frontend asks for /api/... relative to wherever it is served, so the same
image is correct in every environment and nothing is compiled in per host.
Setting it up
Eight things, all of them one-time.
1. The image. Set on the Gitea repository, under Settings → Actions:
- variable
REGISTRY=git.rcjohnstone.com - variable
USERNAME=connor - secret
DOCKER_PASSWORD= a token with package write
These are the same names runway's workflow uses.
2. The config file. Copy the example from the repo and edit it:
mkdir -p ~/config/news
cp /home/connor/docs/projects/news/config.example.toml \
~/config/news/config.toml.example
The real file is ./config/news/config.toml. The binary loads it from
/config/config.toml inside the container and reloads it on SIGHUP. The
schema is in crates/news-server/src/config.rs: [[sources]], [notify],
and [topics] are all required. config.example.toml in the repo root is a
working starting point.
3. The compose service. Add a news-backend block to ~/compose.yml:
news-backend:
image: git.rcjohnstone.com/connor/news:latest
restart: unless-stopped
networks:
- internal
environment:
- NEWS_DATABASE_URL=sqlite:///db/news.db
- TZ=America/Louisville
- NEWS_API_BASE_URL=https://news.rcjohnstone.com
- NTFY_URL=${NTFY_URL}
- NTFY_USER=${NTFY_USER}
- NTFY_PASS=${NTFY_PASS}
volumes:
- ./data/news/db:/db
- ./config/news:/config:ro
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3000/healthz"]
interval: 30s
timeout: 5s
retries: 3
start_period: 5s
NEWS_API_BASE_URL is what the notification feedback buttons (Interested /
Not interested) POST to, and ntfy runs those http actions on the phone —
so it has to be the name Caddy serves, never localhost. When it is unset
the buttons are omitted from notifications and the server logs a warning.
Put NTFY_URL, NTFY_USER, and NTFY_PASS in ~/.env, alongside the other
container secrets. The database is news.db at ./data/news/db; it starts
empty and the migrations run at container startup.
4. Caddy. Add a news.rcjohnstone.com block to ~/Caddyfile. It is
LAN/VPN-only, like grafana and lazylibrarian: internal CA, source-IP gating,
and no # ddns: public marker.
news.rcjohnstone.com {
# Internal CA and no `# ddns: public` marker. Publishing this name would
# put it in Certificate Transparency permanently and make a personal
# dashboard reachable from the internet. Source-IP gating is the control;
# the missing public A record is what keeps the name off CT for good.
tls internal
import internal
import secure_headers
import accesslog
@backend {
path /api /api/*
}
handle @backend {
reverse_proxy news-backend:3000
}
handle {
root * /srv/news/dist
file_server
try_files {path} /index.html
}
}
Caddy has to mount the directory that holds dist, not dist itself:
volumes:
- ./data/news/web:/srv/news:ro # was ./data/news/dist
root * /srv/news/dist # was /srv/news
A bind mount resolves to an inode when the container starts. A deploy replaces
the frontend by renaming one directory over another, which leaves the old inode
exactly where Caddy is still looking, so mounting dist means Caddy serves the
previous frontend until something unrelated happens to restart it, and the
rollback below silently does nothing too. Mounting the holder makes the rename
happen inside what Caddy can see, which is the whole point of doing it as a
rename.
web/ exists so that the holder contains only things Caddy may serve. Mounting
~/data/news directly would put the database one root directive away from
being public.
5. The frontend directory. It has to exist before the first deploy. The
updater replaces things inside it and refuses to build the tree itself, because
the case where it would is sudo handing it $HOME=/root and a deploy into a
directory nobody serves:
mkdir -p ~/data/news/web
The updater runs as root, so it can write there whatever the ownership. It
still checks, and still prints the chown to fix it, for the case where
somebody runs it as themselves.
~/data/news/db stays where it is, outside web/ and so outside Caddy's
mount, which is the point of the extra directory.
6. ntfy access. The ntfy server is configured deny-by-default:
# /home/connor/data/ntfy/etc/server.yml
auth-default-access: "deny-all"
So the publishing user must be created and granted read/write access to the
news-triage topic before news-server can publish notifications. Run from the
host that owns the ntfy container:
# Create the publishing user (interactive password prompt).
sudo podman exec -it ntfy ntfy user add news-bot
# Grant read/write access to the news-triage topic.
sudo podman exec ntfy ntfy access news-bot news-triage rw
Store the resulting password in ~/.env as NTFY_USER and NTFY_PASS.
7. Prometheus. Add a scrape job to ./config/prometheus/prometheus.yml:
- job_name: news-triage
static_configs: [{ targets: ['news-backend:3000'] }]
metrics_path: /metrics
The file is mounted read-only into the prometheus container, so this is a one-time edit.
8. The timer. A system timer, not a user one:
sudo ln -sf /home/connor/docs/projects/news/deploy/news-update.service /etc/systemd/system/
sudo ln -sf /home/connor/docs/projects/news/deploy/news-update.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now news-update.timer
As root, because that is the podman the rest of this machine runs in. Run as
your own account it deploys a backend into your rootless namespace instead:
the container starts, the script reports success, and Caddy, which is in root
podman, cannot resolve news-backend to it at all. That failure is invisible
from inside the script, which is why the choice is written down here rather
than left to whichever podman is first on $PATH.
The unit sets NEWS_DIST and NEWS_COMPOSE_DIR explicitly, since the script's
defaults hang off $HOME and root's is /root.
Deploying by hand
The timer runs the same script, so this is what it does, and it is also the answer to "I do not want to wait ten minutes":
sudo systemctl start news-update.service
journalctl -u news-update.service -n 20 --no-pager
Through the unit rather than by running the script, so that it gets the same
environment the timer gives it. sudo deploy/news-update is not the same
command: $HOME is /root under sudo, so the paths differ. The script stops
rather than guessing, but the tidier habit is to not ask it to.
It exits without touching anything if the image it pulls is already the one
deployed, which it knows from ~/data/news/web/dist.deployed, written at the
end of a deploy that finished. Anything that fails before that point leaves no
record, so the next run does the whole thing again rather than mistaking a
pulled image for a deployed one.
Verifying a deployment
The version shows up in three places, and after a deploy they should all tell the same story.
The web footer shows v{version} · {short sha}, fetched from the API when the
page loads; if that fetch fails, no footer renders at all. The same endpoint,
asked directly:
curl https://news.rcjohnstone.com/api/version
# {"version":"0.2.0","git_sha":"…","build_time":"…"}
The image carries the same facts as OCI labels, and the tags show what points where:
sudo podman inspect git.rcjohnstone.com/connor/news:latest \
--format '{{index .Config.Labels "org.opencontainers.image.version"}} {{index .Config.Labels "org.opencontainers.image.revision"}}'
sudo podman images git.rcjohnstone.com/connor/news
CI tags a single image id as latest, the short commit sha, and the release
tag (v0.2.0), so podman images lists three rows with the same image id —
and the labels are the durable record of which version and which commit an
image was actually built from.
The updater keeps its own trail: web/dist.deployed holds the image id of the
last deploy that got all the way through, and the service journal holds every
run, including the ones that changed nothing:
cat ~/data/news/web/dist.deployed
journalctl -u news-update.service -n 50 --no-pager
systemctl list-timers news-update.timer says when the next check fires.
None of that changes what the updater does: it still pulls :latest, compares
the id against dist.deployed, and exits when they match. The vX.Y.Z tags
are for people, not for the updater — a way to pin the compose file's image:
or to roll back to a build you can name by version instead of a commit hash
you have to go and look up.
Rolling back
The frontend of the previous deploy is kept:
cd ~/data/news/web
sudo mv dist dist.bad && sudo mv dist.previous dist
It takes effect immediately, with no restart, because the rename happens inside the directory Caddy mounts.
The timer will leave it alone rather than undoing it on the next tick:
web/dist.deployed still names the image you rolled away from, so the updater
sees nothing to do until a genuinely new image is pushed. Which is the
behaviour you want at the moment you are rolling something back, and the reason
to fix forward rather than sit on a rollback.
The backend is a tag. Pin the service's image: to
git.rcjohnstone.com/connor/news:<commit> and bring it up; the timer will
leave a pinned tag alone, because it only ever pulls what the service names.
Rolling back across a migration is the case to think about before doing it: migrations only go forwards, so an older binary meeting a newer database is not something to try casually. Roll the frontend back first. That is usually the half that is wrong, and it is reversible in a way the schema is not.