Seed the rewrite: profile, calibration, and repo scaffolding
Ground-up replacement for news_triage, whose signal quality failed because it had no representation of what its user likes -- three keywords fed to BM25 plus a two-term substring blocklist. profile/ holds the taste model as prose and TOML: 11 interest blocks, 3 veto blocks, 5 importance blocks, each a set of probe sentences to be embedded. The [scoring] constants are fitted, not guessed. CALIBRATION.md records the measurements against all 3,885 items in the v0.2 database and 40 real feedback labels, including two findings that changed the design: - compare blocks by z-score within block, never raw cosine; raw cosine let geopolitics win 78% of items and buried known-liked items at median rank 1246 of 3885 (293 after normalizing) - veto needs a margin over the interest signal, not a threshold; a plain 0.55 cutoff falsely killed "Belgian mayor blocks club from hosting Israeli opponent in Europa League", which the old substring blocklist also killed
This commit is contained in:
commit
c89a723603
6 files changed
+551
No files matched your search
@@ -0,0 +1,106 @@
|
||||
# Calibration
|
||||
|
||||
Measured 2026-09-08 against the full v0.2 corpus (3,885 items) and Connor's 40
|
||||
real feedback labels (18 interested / 22 not_interested). Model:
|
||||
`BAAI/bge-small-en-v1.5` via fastembed, CPU, quantized ONNX.
|
||||
|
||||
Everything here is a measurement, not a design intention. Re-run it before
|
||||
changing any constant in `profile.toml`'s `[scoring]` block.
|
||||
|
||||
## Does the embedding model separate signal from noise?
|
||||
|
||||
Yes, decisively — this was the premise the whole rewrite rests on, so it was
|
||||
tested before any pipeline code was written.
|
||||
|
||||
| metric | value |
|
||||
|---|---|
|
||||
| mean interest score, items Connor liked | 0.552 |
|
||||
| mean interest score, items Connor rejected | 0.465 |
|
||||
| liked items falsely vetoed | **0 of 18** |
|
||||
| rejected items caught by veto | 17 of 22 |
|
||||
|
||||
The old system could not do this at all: its `blocklist = ["sports"]` matched
|
||||
none of the baseball recaps Connor spent 18 clicks rejecting.
|
||||
|
||||
## Finding 1 — compare blocks by z-score, never by raw cosine
|
||||
|
||||
Raw cosine magnitude is not comparable across blocks. Mean similarity to the
|
||||
whole corpus, per block:
|
||||
|
||||
geopolitics 0.501 us-national 0.477
|
||||
italy-mexico 0.494 labor-inequality 0.475
|
||||
global-south 0.490 software-systems 0.469
|
||||
energy-climate 0.455 tech-science-policy 0.468
|
||||
cincinnati 0.464 urbanism 0.448
|
||||
kentucky-louisville 0.441
|
||||
|
||||
Taking a plain `max` across blocks therefore hands almost everything to
|
||||
`geopolitics`. Normalizing within each block first fixes it:
|
||||
|
||||
| ranking | liked items' median rank | in top 200 | geopolitics' share of items |
|
||||
|---|---|---|---|
|
||||
| raw cosine | 1246 / 3885 | 4/18 | 78% |
|
||||
| **z-score per block** | **293 / 3885** | **7/18** | **19%** |
|
||||
| percentile rank | 1504 / 3885 | 4/18 | 34% |
|
||||
|
||||
Percentile ranking is *worse than raw* because converting to ranks discards
|
||||
magnitude. Z-score wins clearly.
|
||||
|
||||
Under z-scoring the top of the feed becomes topically varied — AI regulation,
|
||||
the Google ads ruling, a Kenyan airport strike, the shutdown vote, India's grid
|
||||
crunch, mail voting at the Supreme Court, childcare waitlists — instead of
|
||||
twenty consecutive Lebanon headlines.
|
||||
|
||||
## Finding 2 — veto needs a margin rule, not a threshold
|
||||
|
||||
A pure threshold at 0.55 caught 20/22 rejections but falsely vetoed real news
|
||||
that is merely sports-*adjacent*:
|
||||
|
||||
- "Belgian mayor blocks club from hosting Israeli opponent in Europa League" (0.579)
|
||||
- "Shein trades down after Hong Kong debut" (0.562) — matched on "trades"
|
||||
- "California lawmakers pull deal on wildfire victim compensation" (0.569)
|
||||
- "Kushner dice que lamenta su participación en el plan de venta del Mundial" (0.562)
|
||||
|
||||
The first of those is notable: the substring blocklist killed it too, via
|
||||
`"la liga"` matching "la **Liga** Europa". Two entirely different mechanisms,
|
||||
same wrong answer, because both ask "is this about sport?" when the question is
|
||||
"is this *more* about sport than about anything Connor wants?"
|
||||
|
||||
Hence the hybrid rule. Sweep, with canary stories tracked:
|
||||
|
||||
| absolute | margin | rejections caught | likes lost | corpus vetoed | false vetoes |
|
||||
|---|---|---|---|---|---|
|
||||
| threshold only 0.55 | — | 20/22 | 0/18 | 28.6% | **4 known** |
|
||||
| **0.60** | **0.3** | **17/22** | **0/18** | **20.7%** | **0** |
|
||||
| 0.60 | 0.5 | 16/22 | 0/18 | 17.6% | 0 |
|
||||
| 0.62 | 0.5 | 13/22 | 0/18 | 15.5% | 0 |
|
||||
|
||||
Chosen: `veto_absolute = 0.60`, `veto_margin_over_interest = 0.3`.
|
||||
|
||||
The one rejection that still escapes is "Umpire leaves game between Red Sox and
|
||||
Mariners after ball catches him under the chin", which genuinely reads as
|
||||
human-interest oddity rather than sport. Acceptable.
|
||||
|
||||
## Finding 3 — throughput is fine, but far slower than assumed
|
||||
|
||||
Embedding 3,885 items took **358 s — about 11 items/sec** with default fastembed
|
||||
settings on this box (quantized bge-small, ~15 cores busy, 2.2 GB RSS). The plan
|
||||
had claimed "thousands/sec"; that was wrong and is corrected here.
|
||||
|
||||
It does not matter operationally — a realistic day of ~2,000 items embeds in
|
||||
about three minutes, and embeddings are cached by content hash so each item is
|
||||
embedded once. But it is worth tuning batch size and thread count before
|
||||
assuming headroom, and worth re-measuring if the corpus grows.
|
||||
|
||||
Critically, it ran entirely on CPU: no GPU, no llama-swap contention, nothing
|
||||
competing with the OMO stack.
|
||||
|
||||
## Caveats
|
||||
|
||||
- 40 labels is a small evaluation set. These numbers are directional, not proof.
|
||||
- The labels are biased: 18 of 22 negatives are AP sports, because AP was 52% of
|
||||
the corpus. Veto performance is therefore better characterized than interest
|
||||
ranking.
|
||||
- Z-score needs a population to normalize over. In production this must be a
|
||||
rolling per-block mean and standard deviation over a trailing window, not a
|
||||
one-shot pass over a static corpus.
|
||||
Reference in new issue
Block a user