slug: news-triage status: approved intent: clear review_required: false pending-action: write .omo/plans/news-triage.md approach: Rust workspace, containerized in podman-compose behind Caddy, with a Leptos/WASM control UI. NO LLM and NO GPU: classical IR only - SQLite FTS5/BM25 relevance + Naive-Bayes-on-feedback + cross-source corroboration for breaking detection + adaptive percentile thresholds under a token-bucket budget, publishing to the existing ntfy instance. TDD.

Draft: news-triage

Components (topology ledger)

id	outcome (one line)	status	evidence path
C1-ingest	Poll heterogeneous sources (RSS, news-sitemap) into a local store, deduped, with polite conditional GET	active	verified: aljazeera all.xml 200, bbc world rss 200, apnews news-sitemap-content.xml 200
C2-cluster	Collapse syndicated/duplicate copies of one event into a single story cluster; count independent sources	active	AP copy syndicated widely; AJ feed repeats same event twice (Al-Aqsa)
C3-score	Two-axis scoring with NO LLM: prominence/corroboration (importance) + BM25 & Bayes (relevance)	active	sqlite.org/fts5.html s5.1.1 bm25(); ORDER BY rank
C4-gate	Notification budget: token bucket + quiet hours + breaking-news bypass lane	active	user requirement: ~2/day avg, immediate on big news
C5-notify	Publish to ntfy with title/priority/click/markdown + feedback action buttons	active	/home/connor/.local/bin/movie_recs_notify:40-55; data/ntfy/etc/server.yml
C6-operate	Containerized deploy, tiny web UI, failure notifications, metrics	active	/home/connor/compose.yml; runway/shanty conventions
C7-digest	Once-daily low-priority ntfy digest of sub-threshold-but-interesting stories, outside the notification budget	active	resolves Metis gap #18
C8-replay	Offline replay harness + live dry-run mode for tuning thresholds without sending notifications	active	resolves Metis gap #18; required by TDD/offline-tuning workflow

Open assumptions (announced defaults)

assumption	adopted default	rationale	reversible?
Language/runtime	Rust, edition 2024, Cargo workspace: `news-core` (domain model + scoring math, pure/no I/O), `news-ingest` (feed polling, conditional GET, parsing), `news-store` (sqlx/SQLite repositories + migrations), `news-server` (axum backend + notification gate + scheduler), `news-web` (Leptos CSR frontend), `news-cli` (replay/dry-run/admin tool)	Both deployed real projects are exactly this shape (runway/Cargo.toml:1-9,3-9 names 5 analogous crates: core/caldav/server/web/cli; shanty/CLAUDE.md:174-181); resolves Metis gap #5	no (structural)
Backend framework	axum + tokio + sqlx(sqlite) + thiserror + tracing	runway/Cargo.toml:50-63; shanty uses Actix+Sea-ORM, runway is the newer pattern	yes
Frontend	Leptos 0.8 CSR + Trunk + Tailwind v4; image build LIFTS `dist` OUT to a host directory (`./data/news/web/dist`); Caddy serves it as static files; the backend never serves the SPA, only `/api/*`	runway/Dockerfile:9-16,86-88 (explicit: "not served from here"); runway/deploy/README.md:18,79-104 (Caddy mounts the holder dir, not `dist` itself, so a deploy-time rename is atomic)	yes
Frontend addressing	Frontend calls the backend via relative `/api/...`, same-origin behind Caddy; never a baked-in absolute URL	runway/Dockerfile:13-16 (named as the specific lesson v1 got wrong)	no
Store	SQLite via sqlx, migrations compiled in via sqlx::migrate! and run at startup	runway/Dockerfile:90-99	yes
Lints	dead_code/unused_must_use deny, unsafe forbid, unwrap_used/expect_used deny	runway/Cargo.toml:17-26	no
CI	.gitea/workflows: fmt --check, clippy -D warnings, cargo test, cargo-machete, cargo-deny, wasm check	runway/.gitea/workflows/ci.yml:38-51, 66-74	yes
Deploy	Gitea CI builds one image -> `git.rcjohnstone.com/connor/news:latest` (+ `:<commit>`); root systemd system timer pulls and runs `deploy/news-update` (extracts frontend dist to host, restarts container); service block `news-backend` in ~/compose.yml on `internal` network, root podman	runway/deploy/README.md:10-20,106-140	yes
Exposure posture	LAN/VPN-only: `news.rcjohnstone.com` in Caddy, internal CA cert, NO `# ddns: public` marker (so no public DNS record, no CT log entry), source-IP gating via Caddy `@internal` matcher; no Authelia (single-user, low-value target, matches grafana/lazylibrarian posture, not runway's public+Authelia posture)	compose.yml:1016-1018 (grafana "no ddns: public marker" comment); Caddyfile:389-395 (llama-swap: same internal-CA + no-public-DNS pattern with source-IP gating)	yes
Frontend/backend split	ONE image holds both halves so they cannot deploy out of step; frontend extracted to `~/data/news/web/dist` at deploy time, NOT served by the backend	runway/Dockerfile:3-16 (explicit lesson from v1); runway/deploy/README.md:106-121 (frontend dir must pre-exist; atomic rename)	no
Secrets	`${VAR}` interpolation in the `news-backend` compose block, sourced from `~/.env` (rbw as source of truth) - matches every containerized service (runway-backend, litellm), NOT the bare-script `movie_recs_notify` `~/.config/<app>/*.env` pattern, which does not apply to a container	compose.yml:303-317 (`runway-backend` reads `${RUNWAY_SECRET_KEY}` etc. from env, no mounted secrets file)	yes
ntfy ACL provisioning	Implementer emits a documented `ntfy access` grant command as an operator-applied snippet (per Scope-OUT's protected-file rule, extended explicitly to `data/ntfy/etc/server.yml`); confirmed `auth-default-access: "deny-all"` at plan-write time	verified directly: `grep auth-default-access ~/data/ntfy/etc/server.yml` -> `"deny-all"` (2026-08-31)	yes
Metrics wiring	Implementer exposes `/metrics` (Prometheus text format) only; operator applies a scrape-config snippet to `./config/prometheus/prometheus.yml` (mounted read-only into the prometheus container)	compose.yml:1053,1064	yes
Notification budget	token bucket: 2/day refill, burst 3, replenish continuously (not a hard midnight reset); quiet hours 22:00-07:00 America/Louisville	user stated ~2/day; TZ from compose.yml	yes
Breaking bypass	Separate bypass ceiling: max 4 bypass-lane notifications per rolling 24h, independent of and in addition to the normal budget, so a chaotic day cannot fire unboundedly. Fires when corroboration >= 2-of-N enabled sources (not a fixed 3, since only 3 sources are confirmed live in v1) within 45 min, OR importance percentile >= 99th. Bypasses quiet hours but NOT the bypass ceiling itself.	objective signal, no LLM drift; scaled to the real v1 source count (AJ, BBC, AP-sitemap; Reuters excluded from v1, see Sources decision below)	yes
Adaptive percentile threshold	Separate from the breaking-bypass importance percentile above. Definition: the notify-eligible cut is the 90th percentile of RELEVANCE scores computed over the trailing 7-day window of all ingested (non-opinion) stories, recomputed once per poll cycle from a materialized rolling stats table. The breaking-bypass importance percentile (99th, above) uses the IMPORTANCE score's own trailing-7-day distribution, computed the same way. Two distinct distributions (importance vs relevance), same recompute mechanism.	resolves Metis gap #9: previously undefined and conflated with the bypass threshold	yes
Sources (v1)	Enabled: Al Jazeera `https://www.aljazeera.com/xml/rss/all.xml`, BBC World `https://feeds.bbci.co.uk/news/world/rss.xml`, AP news-sitemap `https://apnews.com/news-sitemap-content.xml`. Reuters explicitly OUT of v1 (no verified free feed); config schema supports adding it later without a code change.	verified live 2026-08-31; resolves Metis gaps #8, #21	yes
Poll interval	Per-source, honoring each feed's own declared cache lifetime where present, capped at a 5-minute floor for breaking-news latency: BBC declares `ttl 15` (poll every 15 min); AJ and AP-sitemap have no declared ttl, default to 5 min. Conditional GET (ETag/If-Modified-Since) always used.	resolves Metis gap #20; honors BBC's stated preference instead of uniformly overriding it	yes
Relevance learning	BM25 from FTS5 plus Naive Bayes feedback model; blocklist is a hard veto. Cold start (0 feedback samples): relevance score = BM25-only component, Bayes term contributes exactly 0 (neutral, not suppressive) until >= 20 feedback samples exist.	deterministic, explainable, trains from explicit feedback; resolves Metis gap #17	yes
Op-eds	ingested but flagged `kind=opinion`, never notify-eligible or counted in the percentile distributions unless `notify_opinions = true` in config	user called it only somewhat interesting	yes
Config file	`./config/news/config.toml`, mounted read-only into the container at `/config/config.toml`, hot-reloaded on SIGHUP. Keys: `[[sources]]` (url, kind, weight, enabled), `[notify]` (notify_opinions: bool, quiet_hours_start/end, budget_refill_per_day, budget_burst, bypass_ceiling_per_day), `[topics]` (interests: [string], blocklist: [string]).	resolves Metis gap #6; path matches estate's `./config/<svc>` convention (compose.yml)	yes
Data retention	Raw ingested items older than 90 days pruned by a startup+daily job; story clusters and feedback rows retained indefinitely (small, needed for the Bayes model and digest history). SQLite file included in no automatic backup (single-user, rebuildable from feeds); documented as an accepted risk, not a gap, since re-ingestion loses no unrecoverable data.	resolves Metis gap #10	yes
Healthcheck	`GET /healthz` returns `200 {"status":"ok","db":"ok"}` after a `SELECT 1` round-trip; compose `healthcheck:` block: `test: ["CMD", "curl", "-f", "http://localhost:3000/healthz"]`, `interval: 30s`	resolves Metis gap #12	yes
WASM bundle budget	CI `bundle` job enforces a 1.8MB budget (same number as runway, since the news UI is comparably small: story list, config editor, feedback buttons)	runway/.gitea/workflows/ci.yml:112-121; resolves Metis gap #13	yes
Rust toolchain pin	`rust-toolchain.toml` added at repo root, pinned to the same channel runway uses (stable, verify exact version at implementation time via `rustc --version` on the dev box)	runway convention (README: "toolchain is pinned in rust-toolchain.toml"); resolves Metis gap #15	yes
Daily digest (C7)	Sent once/day at 07:00 America/Louisville via ntfy (separate low-priority message, does not draw from the notification token bucket), listing stories that scored above a lower secondary threshold (75th percentile) but below the notify-eligible 90th percentile in the prior 24h. Entry point: same scheduler tick as the poller, gated on time-of-day.	resolves Metis gap #18 (was unconstrained scope creep); promoted to its own component below	yes
Replay/dry-run harness (C8)	`cargo run -p news-cli -- replay <fixture-dir>` ingests a directory of saved feed XML snapshots through the full scoring+gate pipeline and prints (never sends) what would have been notified, with scores and reasons. `--dry-run` flag on the main server does the same against live feeds without calling ntfy.	resolves Metis gap #18; promoted to its own component below	yes

Findings (cited - path:lines)

Repo is empty except .git -> greenfield: /home/connor/docs/projects/news.
ntfy is already self-hosted at ntfy.rcjohnstone.com; auth-default-access: "deny-all" re-verified directly via grep on 2026-08-31, so a new topic requires an explicit access grant. Evidence: /home/connor/compose.yml, /home/connor/Caddyfile, /home/connor/data/ntfy/etc/server.yml:10.
Estate conventions are podman-compose, restart: unless-stopped, TZ=America/Louisville, ${VAR} secrets from env (never per-app secret files for containerized services), and per-service ./config/<svc> + ./data/<svc> directories. Evidence: /home/connor/compose.yml.
Real deployed project conventions in this workspace are Rust workspace + Dockerfile + Gitea workflows, with one image holding both backend and frontend halves; the frontend is explicitly NOT served by the backend, it is extracted to a host directory Caddy serves directly. Evidence: /home/connor/docs/projects/runway/Cargo.toml, /home/connor/docs/projects/runway/Dockerfile:3-16,86-88, /home/connor/docs/projects/runway/deploy/README.md:10-20,79-104,106-140, /home/connor/docs/projects/runway/.gitea/workflows/ci.yml, /home/connor/docs/projects/shanty/CLAUDE.md.
Two distinct internal-service exposure postures exist in the estate: public-hostname-plus-Authelia (runway) and LAN/VPN-only-with-no-public-DNS-marker (grafana, llama-swap). Evidence: /home/connor/compose.yml:1016-1018, /home/connor/Caddyfile:389-395.
SQLite FTS5 supplies built-in relevance ranking via bm25() and supports tokenizer configuration, including porter stemming. Evidence: https://sqlite.org/fts5.html.
Verified live sources this session: Al Jazeera RSS 200, BBC World RSS 200, AP RSS path 401 but AP news sitemap 200 and advertised in robots.txt. Reuters free RSS remains unverified and is excluded from the v1 source list. Evidence: live fetches recorded in previous exploration.
Shared estate files (~/compose.yml, ~/Caddyfile, ~/data/ntfy/etc/server.yml, ~/config/prometheus/prometheus.yml) are not edited by the implementer; the plan emits operator-applied snippets for all four.
Metis gap analysis (2026-08-31) surfaced 21 findings across contradictions, missing constraints, missing acceptance criteria, scope creep, and unvalidated assumptions; all are resolved in the Open assumptions table above and folded into Scope IN/Decisions below. None were disputed; all were accepted as valid.

Decisions (with rationale)

D1: NO LLM anywhere in the ingestion, scoring, or notification path. This avoids GPU contention, removes model nondeterminism, and makes notifications explainable.
D2: Relevance uses a hybrid of operator-configured topics, SQLite FTS5 BM25, and a Naive Bayes feedback model, with a defined cold-start fallback (BM25-only until 20 feedback samples exist). This gives a clean cold start and improves with use.
D3: Importance is arithmetic, not learned: source prominence, feed position, source trust weight, and cross-source corroboration inside a recency window.
D4: Breaking news detection is objective and separately capped: corroboration/velocity thresholds trigger a bypass lane that has its own 24h ceiling, independent of the normal notification budget, so no single day can produce unbounded notifications.
D5: The product is a real service, not a cron script: one container image (frontend extracted to a host directory at deploy time, never served by the backend), tiny web UI, CI with lint/test/bundle-size gates, healthcheck, and Prometheus metrics.
D6: Suppression lives in the database: one story cluster should not notify twice unless a material-update rule fires.
D7: TDD is the delivery mode: scorer, bucket, clustering, and threshold logic all land behind tests first, plus an offline replay harness (C8) that doubles as the tuning tool.
D8: Exposure is LAN/VPN-only with no public DNS record (matches grafana/llama-swap posture), not runway's public-plus-Authelia posture, because this is a single-user tool with no external audience.
D9: All shared-estate file changes (compose, Caddy, ntfy ACL, Prometheus scrape config) are delivered as documented operator snippets; the implementer never edits those four files directly.

Scope IN

Source registry: Al Jazeera RSS, BBC World RSS, AP news-sitemap enabled in v1 (Reuters config-ready but disabled), per-source weight/enable, per-source poll interval honoring declared feed TTL with a 5-min floor, conditional GET, honest User-Agent, backoff on 429/5xx.
Story normalization, dedupe, clustering, and cross-source corroboration counting (threshold scaled to actual enabled source count).
Two-axis non-LLM scoring: importance (arithmetic: prominence/trust/corroboration) and relevance (BM25 + Naive Bayes with defined cold-start fallback) computed separately, then combined at gate time via two independently-tracked trailing-7-day percentile distributions.
Operator control surface: ./config/news/config.toml (sources, notify settings, topics/blocklist, hot-reload on SIGHUP) plus a tiny local Leptos web UI for reviewing stories, suppression reasons, and feedback.
Notification gate: token bucket (continuous refill), quiet hours, adaptive percentile threshold, breaking-news bypass lane with its own independent 24h ceiling, and duplicate suppression.
ntfy publishing: title, markdown body, priority, click-through link, and feedback action buttons; documented ntfy access grant snippet for the operator.
Feedback capture persisted into the relevance model.
C7 daily digest for sub-threshold but still-interesting stories, delivered outside the notification budget.
C8 replay harness (news-cli replay <fixtures>) and a --dry-run server flag, both printing decisions without calling ntfy.
Containerized deployment: one image, frontend extracted to a host directory at deploy time (never served by the backend), operator snippets for compose/Caddy/ntfy-ACL/Prometheus-scrape, /healthz endpoint, /metrics endpoint, data-retention pruning job, and failure-notification surfacing.
rust-toolchain.toml pin and a CI bundle job enforcing a WASM size budget, alongside fmt/clippy/test/machete/deny.
Tests and agent-executed QA, including dry-run and replay modes as the TDD/tuning workflow.

Scope OUT (Must NOT have)

No LLM, LiteLLM, llama.cpp, or GPU dependency.
No full-article scraping, paywall circumvention, or republishing article text.
No paid API and no hosted AI service.
No new Postgres, Redis, Kafka, or Kubernetes.
No public consumer-facing product, mobile app, or new public DNS requirement; no Authelia gate (LAN/VPN-only posture).
No implementer edits to shared-estate files (~/compose.yml, ~/Caddyfile, ~/data/ntfy/etc/server.yml, ~/config/prometheus/prometheus.yml); only documented snippets for the operator.
No unbounded breaking-news notifications: the bypass lane is capped independently of the normal budget.
No serving the frontend SPA from the backend process; the built image only lifts dist out for Caddy to serve.
No invented MVP/phase reduction.

Open questions

None. User approved the non-LLM + podman-compose + tiny web UI + TDD direction.

Approval gate

status: plan-written Plan written to .omo/plans/news-triage.md (21 implementation todos, 4 final-verification-wave reviewers, Metis gap analysis folded in). Execution remains separate via $start-work news-triage.

18 KiB Raw Blame History