connor
c89a723603
Ground-up replacement for news_triage, whose signal quality failed because it had no representation of what its user likes -- three keywords fed to BM25 plus a two-term substring blocklist. profile/ holds the taste model as prose and TOML: 11 interest blocks, 3 veto blocks, 5 importance blocks, each a set of probe sentences to be embedded. The [scoring] constants are fitted, not guessed. CALIBRATION.md records the measurements against all 3,885 items in the v0.2 database and 40 real feedback labels, including two findings that changed the design: - compare blocks by z-score within block, never raw cosine; raw cosine let geopolitics win 78% of items and buried known-liked items at median rank 1246 of 3885 (293 after normalizing) - veto needs a margin over the interest signal, not a threshold; a plain 0.55 cutoff falsely killed "Belgian mayor blocks club from hosting Israeli opponent in Europa League", which the old substring blocklist also killed
107 lines
4.7 KiB
Markdown
107 lines
4.7 KiB
Markdown
# Calibration
|
|
|
|
Measured 2026-09-08 against the full v0.2 corpus (3,885 items) and Connor's 40
|
|
real feedback labels (18 interested / 22 not_interested). Model:
|
|
`BAAI/bge-small-en-v1.5` via fastembed, CPU, quantized ONNX.
|
|
|
|
Everything here is a measurement, not a design intention. Re-run it before
|
|
changing any constant in `profile.toml`'s `[scoring]` block.
|
|
|
|
## Does the embedding model separate signal from noise?
|
|
|
|
Yes, decisively — this was the premise the whole rewrite rests on, so it was
|
|
tested before any pipeline code was written.
|
|
|
|
| metric | value |
|
|
|---|---|
|
|
| mean interest score, items Connor liked | 0.552 |
|
|
| mean interest score, items Connor rejected | 0.465 |
|
|
| liked items falsely vetoed | **0 of 18** |
|
|
| rejected items caught by veto | 17 of 22 |
|
|
|
|
The old system could not do this at all: its `blocklist = ["sports"]` matched
|
|
none of the baseball recaps Connor spent 18 clicks rejecting.
|
|
|
|
## Finding 1 — compare blocks by z-score, never by raw cosine
|
|
|
|
Raw cosine magnitude is not comparable across blocks. Mean similarity to the
|
|
whole corpus, per block:
|
|
|
|
geopolitics 0.501 us-national 0.477
|
|
italy-mexico 0.494 labor-inequality 0.475
|
|
global-south 0.490 software-systems 0.469
|
|
energy-climate 0.455 tech-science-policy 0.468
|
|
cincinnati 0.464 urbanism 0.448
|
|
kentucky-louisville 0.441
|
|
|
|
Taking a plain `max` across blocks therefore hands almost everything to
|
|
`geopolitics`. Normalizing within each block first fixes it:
|
|
|
|
| ranking | liked items' median rank | in top 200 | geopolitics' share of items |
|
|
|---|---|---|---|
|
|
| raw cosine | 1246 / 3885 | 4/18 | 78% |
|
|
| **z-score per block** | **293 / 3885** | **7/18** | **19%** |
|
|
| percentile rank | 1504 / 3885 | 4/18 | 34% |
|
|
|
|
Percentile ranking is *worse than raw* because converting to ranks discards
|
|
magnitude. Z-score wins clearly.
|
|
|
|
Under z-scoring the top of the feed becomes topically varied — AI regulation,
|
|
the Google ads ruling, a Kenyan airport strike, the shutdown vote, India's grid
|
|
crunch, mail voting at the Supreme Court, childcare waitlists — instead of
|
|
twenty consecutive Lebanon headlines.
|
|
|
|
## Finding 2 — veto needs a margin rule, not a threshold
|
|
|
|
A pure threshold at 0.55 caught 20/22 rejections but falsely vetoed real news
|
|
that is merely sports-*adjacent*:
|
|
|
|
- "Belgian mayor blocks club from hosting Israeli opponent in Europa League" (0.579)
|
|
- "Shein trades down after Hong Kong debut" (0.562) — matched on "trades"
|
|
- "California lawmakers pull deal on wildfire victim compensation" (0.569)
|
|
- "Kushner dice que lamenta su participación en el plan de venta del Mundial" (0.562)
|
|
|
|
The first of those is notable: the substring blocklist killed it too, via
|
|
`"la liga"` matching "la **Liga** Europa". Two entirely different mechanisms,
|
|
same wrong answer, because both ask "is this about sport?" when the question is
|
|
"is this *more* about sport than about anything Connor wants?"
|
|
|
|
Hence the hybrid rule. Sweep, with canary stories tracked:
|
|
|
|
| absolute | margin | rejections caught | likes lost | corpus vetoed | false vetoes |
|
|
|---|---|---|---|---|---|
|
|
| threshold only 0.55 | — | 20/22 | 0/18 | 28.6% | **4 known** |
|
|
| **0.60** | **0.3** | **17/22** | **0/18** | **20.7%** | **0** |
|
|
| 0.60 | 0.5 | 16/22 | 0/18 | 17.6% | 0 |
|
|
| 0.62 | 0.5 | 13/22 | 0/18 | 15.5% | 0 |
|
|
|
|
Chosen: `veto_absolute = 0.60`, `veto_margin_over_interest = 0.3`.
|
|
|
|
The one rejection that still escapes is "Umpire leaves game between Red Sox and
|
|
Mariners after ball catches him under the chin", which genuinely reads as
|
|
human-interest oddity rather than sport. Acceptable.
|
|
|
|
## Finding 3 — throughput is fine, but far slower than assumed
|
|
|
|
Embedding 3,885 items took **358 s — about 11 items/sec** with default fastembed
|
|
settings on this box (quantized bge-small, ~15 cores busy, 2.2 GB RSS). The plan
|
|
had claimed "thousands/sec"; that was wrong and is corrected here.
|
|
|
|
It does not matter operationally — a realistic day of ~2,000 items embeds in
|
|
about three minutes, and embeddings are cached by content hash so each item is
|
|
embedded once. But it is worth tuning batch size and thread count before
|
|
assuming headroom, and worth re-measuring if the corpus grows.
|
|
|
|
Critically, it ran entirely on CPU: no GPU, no llama-swap contention, nothing
|
|
competing with the OMO stack.
|
|
|
|
## Caveats
|
|
|
|
- 40 labels is a small evaluation set. These numbers are directional, not proof.
|
|
- The labels are biased: 18 of 22 negatives are AP sports, because AP was 52% of
|
|
the corpus. Veto performance is therefore better characterized than interest
|
|
ranking.
|
|
- Z-score needs a population to normalize over. In production this must be a
|
|
rolling per-block mean and standard deviation over a trailing window, not a
|
|
one-shot pass over a static corpus.
|