Make the system portable: split config from profile, remove hard-coding

Audit found three things compiled in that shouldn't have been: the User-Agent
carried the author's domain, the cluster threshold and window were module
constants the pipeline never overrode, and the embedding model was pinned --
which blocked the one change most likely to be wanted, swapping in a
multilingual model for native-language sources.

Configuration is now three files with distinct roles: config.toml (machinery),
profile.toml (who the reader is), sources.toml (what to read). All are
relocatable via NEWS_CONFIG_DIR. news/*.py now contains no interest, place name,
feed URL, or personal identifier.

tests/test_config.py keeps it that way: it builds a configuration for an
invented reader and asserts none of the author's identity or preferences reaches
the running code. Writing it immediately found a real bug -- CONFIG_DIR was
resolved at import time, so NEWS_CONFIG_DIR silently did nothing if set after
the module was first imported. Now resolved per instantiation.

Documented the coupling that portability creates: [clustering].threshold and
[scoring].veto_absolute are cosine constants calibrated against one corpus with
one model, so changing [embedding].model invalidates both.

README.md

@@ -43,11 +43,36 @@ relevance entirely, so a blocklisted item could still fire at priority 5.
 
 ## Layout
 
-    profile/     the user's taste, as prose and as TOML. Start here.
     news/        the pipeline: sources, store, embed, cluster, score, route, deliver
+    profile/     all configuration (see below)
     tests/
     deploy/
 
+## Configuration
+
+Nothing about the reader is compiled in. `news/*.py` contains no interest, no
+place name, no feed URL and no personal identifier; all of it is data in three
+TOML files under `profile/` (override the directory with `NEWS_CONFIG_DIR`):
+
+| file | what it is | who edits it |
+|---|---|---|
+| `profile.toml` | who you are: interests, vetoes, demotes, importance, delivery | you, often |
+| `sources.toml` | what to read, with independence buckets and trust weights | you, occasionally |
+| `config.toml` | machinery: HTTP, embedding model, clustering and scoring constants | rarely |
+
+To use this as someone else: replace `profile.toml` and `sources.toml`, and change
+`user_agent` in `config.toml` to your own contact. That is the whole job.
+
+`tests/test_config.py` exists to keep this true — it builds a configuration for a
+different, invented reader and asserts that no part of the author's identity or
+preferences reaches the running code.
+
+Two constants deserve a warning. `[clustering].threshold` (0.82) and
+`[scoring].veto_absolute` (0.60) were calibrated against a specific 3,885-item
+corpus with a specific embedding model. **Changing `[embedding].model` invalidates
+both**, because a different model puts cosine similarity on a different scale.
+Re-run the calibration in `profile/CALIBRATION.md` before trusting them.
+
 ## Status
 
 Early. `profile/` is written; the pipeline is not yet built.

news/cli.py

@@ -15,7 +15,7 @@ from .pipeline import run_cycle
 def cmd_poll(args) -> int:
     cfg = Config()
     con = store.connect(cfg.db_path)
-    report = run_cycle(con, cfg, Embedder())
+    report = run_cycle(con, cfg, Embedder.from_config(cfg))
     print(report)
     for err in report.errors or []:
         print(f"  ! {err}", file=sys.stderr)

news/config.py

@@ -1,4 +1,16 @@
-"""Configuration loading: sources, profile, and runtime paths."""
+"""Configuration loading.
+
+Three files, with a deliberate split so a new user knows what to edit:
+
+    config.toml   machinery: HTTP, embedding model, clustering and scoring
+                  constants. Nothing personal. Sane defaults.
+    profile.toml  who the reader is: interests, vetoes, demotes, importance,
+                  delivery preferences.
+    sources.toml  what to read.
+
+Every path is overridable by environment variable so the same code can run
+against a different reader's configuration without editing anything.
+"""
 
 from __future__ import annotations
 
@@ -6,29 +18,84 @@ import os
 import tomllib
 from dataclasses import dataclass, field
 from pathlib import Path
+from typing import Any
 
 REPO = Path(__file__).resolve().parent.parent
 
 
+def config_dir() -> Path:
+    """Resolved per call, not at import.
+
+    Reading the environment at module scope meant NEWS_CONFIG_DIR was captured
+    once when `news.config` was first imported, so setting it afterwards -- in a
+    test, a subprocess, or anything that configures its environment after
+    importing -- silently did nothing.
+    """
+    return Path(os.environ.get("NEWS_CONFIG_DIR", REPO / "profile"))
+
+
 def _path(env: str, default: Path) -> Path:
     return Path(os.environ.get(env, default))
 
 
+def _load(path: Path) -> dict[str, Any]:
+    return tomllib.loads(path.read_text()) if path.exists() else {}
+
+
 @dataclass
 class Config:
     db_path: Path = field(default_factory=lambda: _path("NEWS_DB", REPO / "data" / "news.db"))
+    config_path: Path = field(
+        default_factory=lambda: _path("NEWS_CONFIG", config_dir() / "config.toml"))
     sources_path: Path = field(
-        default_factory=lambda: _path("NEWS_SOURCES", REPO / "profile" / "sources.toml")
-    )
+        default_factory=lambda: _path("NEWS_SOURCES", config_dir() / "sources.toml"))
     profile_path: Path = field(
-        default_factory=lambda: _path("NEWS_PROFILE", REPO / "profile" / "profile.toml")
-    )
-    user_agent: str = (
-        "news/0.1 (+https://news.rcjohnstone.com; personal feed reader; one user)"
-    )
+        default_factory=lambda: _path("NEWS_PROFILE", config_dir() / "profile.toml"))
 
+    def __post_init__(self) -> None:
+        self._raw = _load(self.config_path)
+
+    # -- machinery ---------------------------------------------------------
+    def section(self, name: str) -> dict[str, Any]:
+        return self._raw.get(name, {})
+
+    @property
+    def user_agent(self) -> str:
+        return self.section("http").get(
+            "user_agent", "news/0.1 (+https://example.invalid; personal feed reader)")
+
+    @property
+    def http(self) -> dict[str, Any]:
+        h = self.section("http")
+        return {
+            "timeout_secs": h.get("timeout_secs", 30),
+            "connect_timeout_secs": h.get("connect_timeout_secs", 10),
+            "concurrency": h.get("concurrency", 8),
+            "max_body_bytes": h.get("max_body_bytes", 10 * 1024 * 1024),
+        }
+
+    @property
+    def embedding(self) -> dict[str, Any]:
+        e = self.section("embedding")
+        return {"model": e.get("model", "BAAI/bge-small-en-v1.5"), "dim": e.get("dim", 384)}
+
+    @property
+    def clustering(self) -> dict[str, Any]:
+        c = self.section("clustering")
+        return {"threshold": c.get("threshold", 0.82), "window_hours": c.get("window_hours", 48)}
+
+    @property
+    def scoring(self) -> dict[str, Any]:
+        s = self.section("scoring")
+        return {
+            "normalize": s.get("normalize", "zscore-per-block"),
+            "veto_absolute": s.get("veto_absolute", 0.60),
+            "veto_margin_over_interest": s.get("veto_margin_over_interest", 0.3),
+        }
+
+    # -- reader ------------------------------------------------------------
     def sources(self) -> list[dict]:
-        raw = tomllib.loads(self.sources_path.read_text())
+        raw = _load(self.sources_path)
         defaults = raw.get("defaults", {})
         out = []
         for entry in raw.get("source", []):
@@ -42,5 +109,9 @@ class Config:
             out.append(merged)
         return out
 
-    def profile(self) -> dict:
-        return tomllib.loads(self.profile_path.read_text())
+    def profile(self) -> dict[str, Any]:
+        return _load(self.profile_path)
+
+    @property
+    def delivery(self) -> dict[str, Any]:
+        return self.profile().get("delivery", {})

news/embed.py

@@ -19,15 +19,25 @@ import numpy as np
 
 from . import store
 
+# Defaults only. The live values come from config.toml [embedding]; a reader who
+# needs native-language sources swaps in a multilingual model there without
+# touching code. Note that changing the model invalidates every calibrated
+# cosine constant, since similarity moves to a different scale.
 MODEL = "BAAI/bge-small-en-v1.5"
 DIM = 384
 
 
 class Embedder:
-    def __init__(self, model_name: str = MODEL) -> None:
+    def __init__(self, model_name: str = MODEL, dim: int = DIM) -> None:
         self._model_name = model_name
+        self._dim = dim
         self._model = None  # loaded lazily: importing fastembed is slow
 
+    @classmethod
+    def from_config(cls, cfg) -> "Embedder":
+        e = cfg.embedding
+        return cls(e["model"], e["dim"])
+
     def _load(self):
         if self._model is None:
             from fastembed import TextEmbedding
@@ -38,7 +48,7 @@ class Embedder:
     def encode(self, texts: Sequence[str]) -> np.ndarray:
         """L2-normalised embeddings, so a dot product is cosine similarity."""
         if not texts:
-            return np.zeros((0, DIM), dtype=np.float32)
+            return np.zeros((0, self._dim), dtype=np.float32)
         vecs = np.array(list(self._load().embed(list(texts))), dtype=np.float32)
         norms = np.linalg.norm(vecs, axis=1, keepdims=True)
         return vecs / np.maximum(norms, 1e-12)

news/pipeline.py

@@ -60,14 +60,17 @@ def sync_sources(con: sqlite3.Connection, cfg: Config) -> list[dict]:
     return out
 
 
-async def fetch_all(cfg: Config, sources: list[dict], concurrency: int = 8):
+async def fetch_all(cfg: Config, sources: list[dict], concurrency: int | None = None):
     """Fetch every source concurrently. A failing source never blocks the others."""
     results: list[tuple[dict, list[dict], dict, str | None]] = []
-    sem = asyncio.Semaphore(concurrency)
+    http = cfg.http
+    sem = asyncio.Semaphore(concurrency or http["concurrency"])
 
     async with httpx.AsyncClient(
         follow_redirects=True,
-        timeout=httpx.Timeout(30.0, connect=10.0),
+        timeout=httpx.Timeout(
+            float(http["timeout_secs"]), connect=float(http["connect_timeout_secs"])
+        ),
         headers={"User-Agent": cfg.user_agent, "Accept": "*/*"},
     ) as client:
 
@@ -123,7 +126,10 @@ def run_cycle(con: sqlite3.Connection, cfg: Config, embedder: Embedder) -> Cycle
             "SELECT COUNT(*) AS n FROM embeddings"
         ).fetchone()["n"] - before
 
-        assignments = cluster.assign(con, list(pending), vectors)
+        clustering = cfg.clustering
+        assignments = cluster.assign(
+            con, list(pending), vectors,
+            threshold=clustering["threshold"], window_hours=clustering["window_hours"])
         report.clustered = len(assignments)
         report.new_clusters = sum(1 for a in assignments if a.created)
     return report

profile/config.toml

@@ -0,0 +1,53 @@
+# Runtime configuration and tuning knobs.
+#
+# Three-file split, so a new user knows what to touch:
+#   config.toml   this file -- machinery. Sane defaults; rarely edited.
+#   profile.toml  who you are: interests, vetoes, importance, delivery.
+#   sources.toml  what to read.
+#
+# Nothing here is specific to any person. Everything personal lives in the
+# other two files.
+
+[http]
+# Sent on every outbound request. Feed operators use it to identify traffic and
+# some will block a generic or absent agent, so put a real contact here.
+# CHANGE THIS if you are not Connor -- it is the contact address feed operators
+# see, and pointing them at someone else's domain is rude and unhelpful.
+user_agent = "news/0.1 (+https://news.rcjohnstone.com; personal feed reader; one user)"
+timeout_secs = 30
+connect_timeout_secs = 10
+concurrency = 8
+max_body_bytes = 10485760
+
+[embedding]
+# Any model fastembed can serve. `dim` must match the model.
+#
+# The default is English-only. For native-language sources use a multilingual
+# model -- "intfloat/multilingual-e5-small" (384 dim) is the usual choice.
+# Be aware that changing the model INVALIDATES the calibrated constants below
+# and in [scoring]: they are cosine thresholds, and a different model puts
+# similarity on a different scale. Re-run the calibration before trusting them.
+model = "BAAI/bge-small-en-v1.5"
+dim = 384
+
+[clustering]
+# Calibrated by inspection on a 3,885-item corpus; see CALIBRATION.md.
+# 0.70 merged "Solheim Cup capsules" with "FIBA Women's World Cup Glance";
+# 0.86 under-merged. Over-merging is the worse failure because it inflates the
+# corroboration signal that feeds importance.
+#
+# Re-tune if your sources differ substantially in style or language.
+threshold = 0.82
+window_hours = 48
+
+[scoring]
+# Compare blocks by z-score WITHIN each block, never by raw cosine: cosine
+# magnitude is not comparable across blocks, and a plain max let one block win
+# 78% of all items. See CALIBRATION.md.
+normalize = "zscore-per-block"
+
+# Veto is not a plain similarity threshold. A story can be genuinely adjacent to
+# a vetoed topic and still be wanted, so an item is vetoed on unambiguous
+# similarity OR when the veto signal beats the best interest signal by a margin.
+veto_absolute = 0.60
+veto_margin_over_interest = 0.3

profile/profile.toml

@@ -419,25 +419,13 @@ hn_min_score = 150
 # failure was collecting 32 of them, not collecting shallow ones.
 feedback_actions = ["interested", "not_interested"]
 
-# Embedding stays English-only (bge-small-en-v1.5). Measured 2026-09-08: the
-# same story in Spanish and English scored 0.758 and 0.600 cosine, against a
-# random-pair p99 of 0.634 -- so one of two translation pairs was
-# indistinguishable from an unrelated article. Native-language Italian and
-# Spanish feeds would rank erratically, so Italy and Mexico are covered by
-# English-language sources (ANSA English, Mexico News Daily) instead. Revisit
-# with multilingual-e5-small only if native sources become worth re-calibrating
-# every threshold for.
-embedding_language = "en"
+# Embedding is English-only by default (see config.toml [embedding]). Measured
+# 2026-09-08: the same story in Spanish and English scored 0.758 and 0.600
+# cosine against a random-pair p99 of 0.634, so one of two translation pairs
+# was indistinguishable from an unrelated article. Italy and Mexico are
+# therefore covered by English-language sources.
 # Conflict outside categories A-D notifies only on a major development -- a new
 # offensive, a ceasefire, mass casualties, a state entering. Routine front-line
 # reporting goes to the digest.
 conflict_general_requires_importance = true
 
-# =========================================================== scoring =========
-# Fitted 2026-09-08 against 3,885 items and 40 real feedback labels.
-# See profile/CALIBRATION.md.
-
-[scoring]
-normalize = "zscore-per-block"
-veto_absolute = 0.60
-veto_margin_over_interest = 0.3

tests/test_config.py

@@ -0,0 +1,128 @@
+"""Configuration must actually drive behaviour.
+
+The point of these tests is portability: someone who is not the author should be
+able to run this by replacing the TOML files, without editing Python. Each test
+below asserts that a value in config reaches the code that uses it.
+"""
+
+from __future__ import annotations
+
+import textwrap
+
+import numpy as np
+import pytest
+
+from news import cluster, store
+from news.config import Config
+from news.embed import Embedder
+
+
+@pytest.fixture
+def cfgdir(tmp_path, monkeypatch):
+    (tmp_path / "config.toml").write_text(textwrap.dedent("""
+        [http]
+        user_agent = "someone-else/9.9 (+https://elsewhere.example)"
+        timeout_secs = 7
+        concurrency = 3
+        [embedding]
+        model = "intfloat/multilingual-e5-small"
+        dim = 512
+        [clustering]
+        threshold = 0.5
+        window_hours = 6
+        [scoring]
+        veto_absolute = 0.99
+    """))
+    (tmp_path / "sources.toml").write_text(textwrap.dedent("""
+        [defaults]
+        poll_interval_secs = 111
+        [[source]]
+        name = "Someone Else's Feed"
+        url = "https://elsewhere.example/rss"
+        bucket = "elsewhere"
+    """))
+    (tmp_path / "profile.toml").write_text(textwrap.dedent("""
+        [[interest]]
+        id = "trains"
+        weight = 0.4
+        probes = ["Rolling stock procurement and timetable changes."]
+        [delivery]
+        digest_hour = "19:00"
+    """))
+    monkeypatch.setenv("NEWS_CONFIG_DIR", str(tmp_path))
+    return tmp_path
+
+
+def test_no_authors_identity_leaks_into_a_fresh_config(cfgdir):
+    cfg = Config()
+    assert "rcjohnstone" not in cfg.user_agent
+    assert cfg.user_agent.startswith("someone-else/9.9")
+
+
+def test_machinery_values_are_read_from_config(cfgdir):
+    cfg = Config()
+    assert cfg.http["timeout_secs"] == 7
+    assert cfg.http["concurrency"] == 3
+    assert cfg.embedding == {"model": "intfloat/multilingual-e5-small", "dim": 512}
+    assert cfg.clustering == {"threshold": 0.5, "window_hours": 6}
+    assert cfg.scoring["veto_absolute"] == 0.99
+
+
+def test_embedder_takes_its_model_from_config_without_loading_it(cfgdir):
+    """A different reader can swap in a multilingual model from TOML alone."""
+    emb = Embedder.from_config(Config())
+    assert emb._model_name == "intfloat/multilingual-e5-small"
+    assert emb.encode([]).shape == (0, 512)   # honours the configured dim
+
+
+def test_reader_profile_is_entirely_data(cfgdir):
+    cfg = Config()
+    profile = cfg.profile()
+    ids = [b["id"] for b in profile["interest"]]
+    assert ids == ["trains"], "profile content must come only from TOML"
+    assert cfg.delivery["digest_hour"] == "19:00"
+
+
+def test_sources_come_from_config_with_defaults_applied(cfgdir):
+    srcs = Config().sources()
+    assert len(srcs) == 1
+    assert srcs[0]["name"] == "Someone Else's Feed"
+    assert srcs[0]["poll_interval_secs"] == 111     # from [defaults]
+    assert srcs[0]["kind"] == "rss" and srcs[0]["weight"] == 1.0
+
+
+def test_configured_cluster_threshold_changes_the_outcome(tmp_path):
+    """The calibrated 0.82 is a default, not a law: config must override it."""
+    con = store.connect(tmp_path / "t.db")
+    store.upsert_source(con, id="s1", name="A", url="http://a", kind="rss",
+                        bucket="a", weight=1.0, prominence=0, enabled=1,
+                        poll_interval_secs=900)
+
+    def add(i, when):
+        con.execute(
+            """INSERT INTO items (id,source_id,title,summary,url,canonical_url,
+                                  published_at,first_seen_at,content_hash)
+               VALUES (?,?,?,'',?,?,?,?,?)""",
+            (i, "s1", i, f"http://x/{i}", f"http://x/{i}", when, when, i))
+        return con.execute("SELECT * FROM items WHERE id=?", (i,)).fetchone()
+
+    rows = [add("i1", "2026-01-01T00:00:00+00:00"), add("i2", "2026-01-01T01:00:00+00:00")]
+    a, b = np.array([1, 0], np.float32), np.array([0.6, 0.8], np.float32)  # cosine 0.6
+    vecs = {"i1": a, "i2": b}
+
+    strict = cluster.assign(con, rows, vecs, threshold=0.9)
+    assert strict[0].cluster_id != strict[1].cluster_id
+
+    con2 = store.connect(tmp_path / "t2.db")
+    store.upsert_source(con2, id="s1", name="A", url="http://a", kind="rss",
+                        bucket="a", weight=1.0, prominence=0, enabled=1,
+                        poll_interval_secs=900)
+    for i, when in (("i1", "2026-01-01T00:00:00+00:00"), ("i2", "2026-01-01T01:00:00+00:00")):
+        con2.execute(
+            """INSERT INTO items (id,source_id,title,summary,url,canonical_url,
+                                  published_at,first_seen_at,content_hash)
+               VALUES (?,?,?,'',?,?,?,?,?)""",
+            (i, "s1", i, f"http://x/{i}", f"http://x/{i}", when, when, i))
+    rows2 = con2.execute("SELECT * FROM items ORDER BY published_at").fetchall()
+    loose = cluster.assign(con2, list(rows2), vecs, threshold=0.5)
+    assert loose[0].cluster_id == loose[1].cluster_id