CLAUDE.md: podman ps saying running is not evidence a container is alive

Measured 2026-09-19. After a reboot, eleven containers reported `Up 9 hours`
with a State.Pid that no longer existed -- ntfy, bitwarden, authelia, gitea,
grafana, prometheus among them. ntfy's healthcheck had failed 539 consecutive
times while podman ps called it up, and it stayed that way overnight because
every ordinary check said yes.

podman-restart.service is Type=oneshot and forks a conmon per container into
its own cgroup. One container failing to create fails the unit, and systemd's
default KillMode=control-group then SIGKILLs every conmon it just started.
conmon is what updates container state, so killed it updates nothing.

Fix applied as ~/config/host/podman-restart-killmode.sh (KillMode=process),
which also carries a --check mode for finding zombies.

⚠ Also records that `kill -0 <pid>` is the wrong liveness test here: the
containers run as root, so as a normal user it fails with EPERM rather than
ESRCH and reports every container dead. That misdiagnosis was made during
this very investigation and read as a total outage for a minute.
This commit is contained in:
connor committed 2026-09-19 10:57:02 -04:00
1 parent fecda07e5c
commit 24dd2bf59f
1 file changed
+24
+24
View File
@@ -31,6 +31,30 @@ The five things most likely to waste an hour if you don't know them:
`sudo bash ~/config/host/podman-docker-forward.sh`. Don't remove Docker — the
Gitea Actions runners need it.
3a. **`podman ps` saying `running` is not evidence a container is alive.**
After a reboot on 2026-09-19, eleven containers — ntfy, bitwarden,
authelia, gitea, grafana, prometheus among them — reported `Up 9 hours`
with a `State.Pid` that no longer existed. ntfy's healthcheck had failed
539 consecutive times while `podman ps` called it up. Cause:
`podman-restart.service` is `Type=oneshot` and forks a conmon per
container into its own cgroup; ONE container failing to create fails the
unit, and systemd's default `KillMode=control-group` then SIGKILLs every
conmon it just started. conmon is what updates container state, so killed
it updates nothing and the state stays `running` forever.
Check with `bash ~/config/host/podman-restart-killmode.sh --check`, which
compares each running container's pid against `/proc`. That script also
applies the fix (`KillMode=process`), applied here 2026-09-19 — it takes
effect on the next boot, so a reboot before then can still produce this.
Recover one with `sudo podman restart -t 5 <name> && sudo podman start
<name>`; the restart is EXPECTED to error with `conmon process killed`,
and the start afterwards is what works.
⚠ **Do not test liveness with `kill -0 <pid>`.** The containers run as
root, so as a normal user it fails with EPERM, not ESRCH — it reports
every container dead and reads as a total outage. That misdiagnosis was
made once already. `/proc/<pid>` needs no privilege.
4. **HTTP 200 with empty content is not necessarily a bug.** Three causes: an
Anthropic refusal, a reasoning model eating its whole `max_tokens` budget
before emitting text, or a dropped parameter. Local models do it too. Give