Decisions — news-triage

Architectural choices and rationales discovered during work on this plan.

Auto-scaffolded by /start-work. Append new entries below - never overwrite.

Todo 2 — serde discriminant wire format

All enum discriminants use #[serde(rename_all = "snake_case")], not derive-default PascalCase. Applies to SourceKind (rss_atom, news_sitemap), ItemKind (news, opinion), FeedbackKind (interested, not_interested). This is deliberate and locked by a test (opinion_item_serializes_kind_as_snake_case_discriminant in news-core/src/model.rs) — the plan's QA section explicitly requires kind: ItemKind::Opinion to serialize as "opinion".
Downstream todos matching on these discriminants in SQL or JSON must use the snake_case string form: "rss_atom", "news_sitemap", "news", "opinion", "interested", "not_interested". This affects todo 8 (source registry storage), any SQLite column that stores kind as TEXT, and any frontend/JSON consumer matching on the discriminant.

Todo 4 — NaiveBayesModel Laplace-smoothing formula (locked, downstream contract)

Exact formula used by NaiveBayesModel::score(tokens) in crates/news-core/src/scoring/relevance.rs (per-token, summed across tokens):
```
score(tokens) = sum over token in tokens of:
    ln((count_interested(token) + 1) / (total_interested + 2))
  - ln((count_not_interested(token) + 1) / (total_not_interested + 2))
```
Additive (Laplace) smoothing with +1 on counts and +2 on the class totals avoids ln(0). A token never seen in training still contributes a nonzero term derived purely from total_interested/total_not_interested (not 0.0) — only the cold-start gate in relevance_score produces an exact 0.0 Bayes contribution.
Cold-start gate: relevance_score(bm25, bayes, tokens) returns bm25 bit-identically (Bayes term is exactly 0.0, not merely small) whenever bayes.total_interested + bayes.total_not_interested < 20; at >= 20 total samples it returns bm25 + bayes.score(tokens). Locked by cold_start_boundary_at_20_samples (19 vs 20 exact boundary).
score() CAN return negative values — this happens when a token's training history skews toward NotInterested (see score_is_negative_when_token_skews_not_interested). This is a legitimate, finite relevance signal, distinct from and must NEVER be confused with f64::NEG_INFINITY, which todo 10's blocklist-veto logic reserves as a separate hard-suppression sentinel. Callers combining the two (todo 10, todo 13) must check for the veto sentinel explicitly (e.g. is_infinite() && is_sign_negative()) rather than assuming "any negative relevance score" means "blocked".
No persistence in news-core: NaiveBayesModel's fields (feature_counts, total_interested, total_not_interested) are private; news-store (a later todo) owns serializing/loading this state to/from the bayes_model table (see plan todo 5).

Todo 3 — importance scoring formula

Formula (additive, unweighted sum of three terms), implemented in crates/news-core/src/scoring/importance.rs:

importance_score(prominence_rank, source_trust_weight, corroboration_count, _window)
    = prominence_term + trust_term + corroboration_term
where
    prominence_term    = 1.0 / (1.0 + prominence_rank)
    trust_term          = source_trust_weight                     (assumed in [0.0, 1.0])
    corroboration_term = 1.0 - 1.0 / (1.0 + corroboration_count)  (saturating, 0 -> 0.0, asymptotic -> 1.0)

Worked example (asserted by test worked_example_matches_doc_comment): importance_score(0, 0.8, 3, 45) == 2.55 (1.0 + 0.8 + 0.75).

Numeric range: with source_trust_weight in [0.0, 1.0] (the assumed/documented range for Source.weight), the score ranges from 0.0 (rank -> infinity, trust 0, zero corroboration) up toward but never reaching 3.0 (rank 0 gives prominence_term 1.0, trust 1.0, corroboration approaching but never reaching 1.0 as corroboration_count grows). Todo 12 (notification gate, importance percentile >= 99th) should treat this as an open-ended-but-bounded-near-3.0 range, not a normalized [0,1] score — percentile ranking (not a fixed numeric threshold) is what the plan specifies, so the exact ceiling doesn't need to be a round number.
corroboration_window_minutes is accepted but NOT used in the formula — it's a pure-function parameter for API completeness/future use. Window enforcement (deciding which raw items count toward corroboration_count in the first place, e.g. "sources within 45 minutes") is the CALLER's responsibility in news-store's ClusterRepo (todo 9). importance_score trusts the corroboration_count it's given and does not re-derive timing. Todo 11 (importance scorer wiring) must compute corroboration_count correctly at the call site before invoking this function — passing an un-windowed raw source count would be wrong, but that bug would be in the caller, not in this pure function.
Saturation curve is 1.0 - 1.0/(1.0 + n) (same family as the prominence term, inverted) rather than e.g. log(1+n) or tanh(n) — chosen for symmetry with the prominence term and because it has a clean worked-example fraction (3 sources -> 0.75, easy to hand-verify). Marginal gain shrinks monotonically: 0->1 adds 0.5, 1->2 adds ~0.167, 2->3 adds 0.083, 9->10 adds ~0.009 — confirmed by test corroboration_saturates_so_tenth_source_adds_less_than_third.
Six tests total (4 required by the plan + 2 extra: the worked-example doc-comment assertion and an explicit saturation-margin test), all in crates/news-core/src/scoring/importance.rs's inline #[cfg(test)] mod tests, using plain assert!/assert_eq! (no .unwrap()/.expect()), per todo 2's established pattern.
Ran into a transient race with the parallel todo 4 agent: it added pub mod relevance; to crates/news-core/src/scoring/mod.rs (shared file) slightly before creating scoring/relevance.rs, causing a momentary E0583 file not found for module compile error. Resolved itself ~15s later once the other agent finished writing relevance.rs — not a bug in this todo's work, just a heads-up for future same-file parallel edits.

Todo 5 — news-store SQLite schema (locked, downstream contract for 6/8/9/11)

All ids are TEXT primary keys storing Uuid::to_string() (hyphenated lowercase form, sqlx's default Display for uuid::Uuid). No BLOB/16-byte-uuid storage — plain text, matching runway's convention (see stamp/parse_stamp note below) and keeping the schema greppable with the sqlite3 CLI.
All timestamps are TEXT storing RFC3339 (DateTime<Utc>::to_rfc3339() or equivalent). No SQLite DATETIME/INTEGER epoch columns — every *_at column (published_at, first_seen_at, last_seen_at, created_at, computed_at) uses this same TEXT/RFC3339 representation. Todo 6/8/9's repositories must parse with DateTime::parse_from_rfc3339 (mirrors runway's db::parse_stamp helper) rather than inventing a second timestamp format.
Enum-like columns (kind on sources/raw_items, lane on notification_log, kind on feedback) are TEXT storing the snake_case discriminant strings locked by todo 2's decision above ("rss_atom", "news_sitemap", "news", "opinion", "interested", "not_interested") — no separate CHECK (kind IN (...)) constraint was added in this todo (kept out of scope; a later todo can add one without a schema-shape change since the column stays TEXT).
Booleans are INTEGER (sources.enabled, notification_log.dry_run) — SQLite has no native boolean type; sqlx maps bool to INTEGER (0/1) transparently via the sqlite feature, so repositories bind/read plain bool values without manual i64 conversion.
bayes_model is a singleton row (id INTEGER PRIMARY KEY CHECK (id = 1), one column serialized_json TEXT) holding the entire NaiveBayesModel (todo 4) as one JSON blob via serde_json. Todo 4's feature_counts/total_interested/total_not_interested fields are private but the type derives Serialize/Deserialize per todo 4's notepad — todo 6/11's bayes-model repo should UPSERT (INSERT ... ON CONFLICT(id) DO UPDATE) this single row rather than delete-then-insert.
StoreError added an InsufficientData { needed: u32, available: u32 } variant now (forward-looking, unused until the percentile tracker todo), documented in its doc comment as unused — todo 11 should reuse this variant rather than adding a second "not enough data" error type.
sqlx pinned to the 0.8 line, not runway's 0.9: this workspace's rust-toolchain.toml is pinned to 1.89.0, but sqlx 0.9.0 requires rustc 1.94.0 (confirmed via cargo test failing with an explicit MSRV error naming sqlx@0.9.0/sqlx-core@0.9.0/etc.). sqlx = "0.8" (resolved to 0.8.6) has no such floor and builds cleanly under 1.89.0. Any future todo bumping sqlx must re-check this MSRV constraint against rust-toolchain.toml before bumping the major/minor version — do not blindly copy runway's 0.9 pin.

Todo 6 — SourceRepo fixed seed ids and weights (locked, reusable by todo 9+)

Fixed, stable UUIDs for the 3 v1 seed sources, defined as constants in crates/news-store/src/sources.rs. Any later todo (e.g. todo 9's clustering tests) referencing "the BBC source" by id should reuse these literals rather than re-deriving or re-seeding:
- Al Jazeera: 6f6a0e3e-9d1a-4b7a-8b8a-000000000001
- BBC World: 6f6a0e3e-9d1a-4b7a-8b8a-000000000002
- AP: 6f6a0e3e-9d1a-4b7a-8b8a-000000000003
Weights (in [0.0, 1.0], per todo 3's assumed Source.weight range): Al Jazeera 0.7, BBC World 0.9, AP 0.9. Rationale: BBC and AP get the higher trust weight as established wire-service/broadcast sources with long corroboration track records; Al Jazeera is reputable but set slightly lower to reflect more contested editorial framing on some regional coverage. These are subjective starting defaults, not a derived formula — a future todo could recalibrate them from feedback data without a schema change (the column is a plain REAL). Source.weight values are chosen defaults, not derived from any formula in this todo.
upsert is the sole idempotency mechanism for seed_default_sources: INSERT ... ON CONFLICT(id) DO UPDATE SET ... on the fixed ids above, so calling it any number of times leaves exactly 3 rows. No separate "does this row exist" pre-check was added.
Reuters is deliberately absent from this crate's code (no seeded row, enabled or disabled) — per the plan, it belongs only in the config template (todo 18/21) as a disabled, commented-out example.
kind TEXT values match todo 2's snake_case convention exactly: "rss_atom" / "news_sitemap", via an explicit match in kind_to_str/kind_from_str (no serde_json round-trip through the DB column — a direct hand-written match keeps the DB layer decoupled from news_core's serde attribute and makes the two directions symmetric and easy to audit).

F2/F3 final-wave fixes — cold-start percentile rule

Cold-start rule for trailing-window percentile ranks: the draft specifies a cold-start fallback only for the relevance score (BM25-only until 20 feedback samples; .omo/drafts/news-triage.md:45), not for percentile ranks. When PercentileTracker::percentile_rank returns StoreError::InsufficientData because the trailing 7-day window is empty, the scheduler maps it to 0.0 with a one-line tracing::info! log.
Rationale: conservative gate behavior. With percentile ranks at 0.0, no story can meet the notify threshold (relevance @ p90) or digest band [75, 90) until the window has data, preventing bogus batch-local percentiles from driving decisions. The breaking-news bypass lane (source_count >= 2 or importance @ p99) is unaffected: corroboration bypass is percentile-independent, and the importance percentile uses its own distribution.
JSON boundary for blocklist veto: StoryRow.relevance is Option<f64>, mapped with relevance.is_finite().then_some(relevance). This keeps the f64::NEG_INFINITY sentinel inside the backend scorer while serializing honestly as JSON null at the API boundary. importance remains f64 because it is finite by construction (multi-hot arithmetic, no veto path).

12 KiB Raw Blame History