news/profile/CALIBRATION.md

Calibration

Measured 2026-09-08 against the full v0.2 corpus (3,885 items) and Connor's 40 real feedback labels (18 interested / 22 not_interested). Model: BAAI/bge-small-en-v1.5 via fastembed, CPU, quantized ONNX.

Everything here is a measurement, not a design intention. Re-run it before changing any constant in profile.toml's [scoring] block.

Does the embedding model separate signal from noise?

Yes, decisively — this was the premise the whole rewrite rests on, so it was tested before any pipeline code was written.

metric	value
mean interest score, items Connor liked	0.552
mean interest score, items Connor rejected	0.465
liked items falsely vetoed	0 of 18
rejected items caught by veto	17 of 22

The old system could not do this at all: its blocklist = ["sports"] matched none of the baseball recaps Connor spent 18 clicks rejecting.

Finding 1 — compare blocks by z-score, never by raw cosine

Raw cosine magnitude is not comparable across blocks. Mean similarity to the whole corpus, per block:

geopolitics          0.501      us-national          0.477
italy-mexico         0.494      labor-inequality     0.475
global-south         0.490      software-systems     0.469
energy-climate       0.455      tech-science-policy  0.468
cincinnati           0.464      urbanism             0.448
kentucky-louisville  0.441

Taking a plain max across blocks therefore hands almost everything to geopolitics. Normalizing within each block first fixes it:

ranking	liked items' median rank	in top 200	geopolitics' share of items
raw cosine	1246 / 3885	4/18	78%
z-score per block	293 / 3885	7/18	19%
percentile rank	1504 / 3885	4/18	34%

Percentile ranking is worse than raw because converting to ranks discards magnitude. Z-score wins clearly.

Under z-scoring the top of the feed becomes topically varied — AI regulation, the Google ads ruling, a Kenyan airport strike, the shutdown vote, India's grid crunch, mail voting at the Supreme Court, childcare waitlists — instead of twenty consecutive Lebanon headlines.

Finding 2 — veto needs a margin rule, not a threshold

A pure threshold at 0.55 caught 20/22 rejections but falsely vetoed real news that is merely sports-adjacent:

• "Belgian mayor blocks club from hosting Israeli opponent in Europa League" (0.579)
• "Shein trades down after Hong Kong debut" (0.562) — matched on "trades"
• "California lawmakers pull deal on wildfire victim compensation" (0.569)
• "Kushner dice que lamenta su participación en el plan de venta del Mundial" (0.562)

The first of those is notable: the substring blocklist killed it too, via "la liga" matching "la Liga Europa". Two entirely different mechanisms, same wrong answer, because both ask "is this about sport?" when the question is "is this more about sport than about anything Connor wants?"

Hence the hybrid rule. Sweep, with canary stories tracked:

absolute	margin	rejections caught	likes lost	corpus vetoed	false vetoes
threshold only 0.55	—	20/22	0/18	28.6%	4 known
0.60	0.3	17/22	0/18	20.7%	0
0.60	0.5	16/22	0/18	17.6%	0
0.62	0.5	13/22	0/18	15.5%	0

Chosen: veto_absolute = 0.60, veto_margin_over_interest = 0.3.

The one rejection that still escapes is "Umpire leaves game between Red Sox and Mariners after ball catches him under the chin", which genuinely reads as human-interest oddity rather than sport. Acceptable.

Finding 3 — throughput is fine, but far slower than assumed

Embedding 3,885 items took 358 s — about 11 items/sec with default fastembed settings on this box (quantized bge-small, ~15 cores busy, 2.2 GB RSS). The plan had claimed "thousands/sec"; that was wrong and is corrected here.

It does not matter operationally — a realistic day of ~2,000 items embeds in about three minutes, and embeddings are cached by content hash so each item is embedded once. But it is worth tuning batch size and thread count before assuming headroom, and worth re-measuring if the corpus grows.

Critically, it ran entirely on CPU: no GPU, no llama-swap contention, nothing competing with the OMO stack.

Caveats

• 40 labels is a small evaluation set. These numbers are directional, not proof.
• The labels are biased: 18 of 22 negatives are AP sports, because AP was 52% of the corpus. Veto performance is therefore better characterized than interest ranking.
• Z-score needs a population to normalize over. In production this must be a rolling per-block mean and standard deviation over a trailing window, not a one-shot pass over a static corpus.

---

Normalizer switch: z-score -> top-anchored

Measured 2026-09-17 on 1,400 live clusters, nine days into production.

Why

Z-scoring subtracts each block's mean, which encodes how much of the corpus that block matches. A broad block is compressed, a narrow one stretched. The same raw cosine of 0.67 scored:

block	z
software-systems (broad)	1.28
space-aerospace	~3.0
cincinnati	4.10
conflict-undercovered (narrow)	4.49

Every block's own 99th percentile landed at z 2.2-2.9, yet the threshold was a flat 2.0 and blocks were ranked against each other on that scale. A subject was penalised for being well covered in the reader's own source list.

Recalibrated thresholds

Each chosen to admit the same share of the corpus its predecessor did. Measured, not converted by arithmetic.

knob	z	share	top-anchored
interest_threshold	2.0	8.3%	0.70
important_threshold	2.0	21.5%	1.00
digest_floor	0.5	47.9%	-0.55
serendipity_max_interest	0.5	47.9%	-0.55
veto_margin_over_interest	0.3	22.0% vetoed	0.12 (gives 22.1%)

veto_absolute and demote_absolute are raw cosine and do not move.

Score distribution under top-anchored: interest p50 -0.59, p95 0.79, p99 1.06, max 1.58. Importance p50 0.60, p95 1.54, p99 2.12, max 2.86. Half of all clusters score below zero by construction -- the scale is anchored on the median and the 99th percentile.

Effect

Digest world-politics concentration fell: diplomacy-foreign-policy plus far-right-watch from 10 of 28 slots to 6.

It did not put software-systems or space-aerospace into the digest. Their best clusters score 0.63 and 0.91 against an interest cut of roughly 0.7 for a 28-slot digest -- genuinely just below, not blocked by any rule.

Two hypotheses tested and rejected as the cause of that:

• aggregator-immature is what excludes them. Making those items eligible moves space 0 -> 1 and software 0 -> 0. It is not the binding constraint for the digest (it remains a real defect for other reasons).
• The normalizer alone would fix it. It does not; see above.

What does work is weight, which top-anchored restores as a meaningful knob: software 0.9 -> 1.15 and space 0.95 -> 1.25 yields one digest slot each. That is a taste decision, not a bug fix, and is left to the reader.