CLAUDE.md: podman ps saying running is not evidence a container is alive
Measured 2026-09-19. After a reboot, eleven containers reported `Up 9 hours` with a State.Pid that no longer existed -- ntfy, bitwarden, authelia, gitea, grafana, prometheus among them. ntfy's healthcheck had failed 539 consecutive times while podman ps called it up, and it stayed that way overnight because every ordinary check said yes. podman-restart.service is Type=oneshot and forks a conmon per container into its own cgroup. One container failing to create fails the unit, and systemd's default KillMode=control-group then SIGKILLs every conmon it just started. conmon is what updates container state, so killed it updates nothing. Fix applied as ~/config/host/podman-restart-killmode.sh (KillMode=process), which also carries a --check mode for finding zombies. ⚠ Also records that `kill -0 <pid>` is the wrong liveness test here: the containers run as root, so as a normal user it fails with EPERM rather than ESRCH and reports every container dead. That misdiagnosis was made during this very investigation and read as a total outage for a minute.
This commit is contained in:
1 parent
fecda07e5c
commit
24dd2bf59f
1 file changed
+24
@@ -31,6 +31,30 @@ The five things most likely to waste an hour if you don't know them:
|
||||
`sudo bash ~/config/host/podman-docker-forward.sh`. Don't remove Docker — the
|
||||
Gitea Actions runners need it.
|
||||
|
||||
3a. **`podman ps` saying `running` is not evidence a container is alive.**
|
||||
After a reboot on 2026-09-19, eleven containers — ntfy, bitwarden,
|
||||
authelia, gitea, grafana, prometheus among them — reported `Up 9 hours`
|
||||
with a `State.Pid` that no longer existed. ntfy's healthcheck had failed
|
||||
539 consecutive times while `podman ps` called it up. Cause:
|
||||
`podman-restart.service` is `Type=oneshot` and forks a conmon per
|
||||
container into its own cgroup; ONE container failing to create fails the
|
||||
unit, and systemd's default `KillMode=control-group` then SIGKILLs every
|
||||
conmon it just started. conmon is what updates container state, so killed
|
||||
it updates nothing and the state stays `running` forever.
|
||||
|
||||
Check with `bash ~/config/host/podman-restart-killmode.sh --check`, which
|
||||
compares each running container's pid against `/proc`. That script also
|
||||
applies the fix (`KillMode=process`), applied here 2026-09-19 — it takes
|
||||
effect on the next boot, so a reboot before then can still produce this.
|
||||
Recover one with `sudo podman restart -t 5 <name> && sudo podman start
|
||||
<name>`; the restart is EXPECTED to error with `conmon process killed`,
|
||||
and the start afterwards is what works.
|
||||
|
||||
⚠ **Do not test liveness with `kill -0 <pid>`.** The containers run as
|
||||
root, so as a normal user it fails with EPERM, not ESRCH — it reports
|
||||
every container dead and reads as a total outage. That misdiagnosis was
|
||||
made once already. `/proc/<pid>` needs no privilege.
|
||||
|
||||
4. **HTTP 200 with empty content is not necessarily a bug.** Three causes: an
|
||||
Anthropic refusal, a reasoning model eating its whole `max_tokens` budget
|
||||
before emitting text, or a dropped parameter. Local models do it too. Give
|
||||
|
||||
Reference in new issue
Block a user