# Calibration

Measured 2026-09-08 against the full v0.2 corpus (3,885 items) and Connor's 40
real feedback labels (18 interested / 22 not_interested). Model:
`BAAI/bge-small-en-v1.5` via fastembed, CPU, quantized ONNX.

Everything here is a measurement, not a design intention. Re-run it before
changing any constant in `profile.toml`'s `[scoring]` block.

## Does the embedding model separate signal from noise?

Yes, decisively — this was the premise the whole rewrite rests on, so it was
tested before any pipeline code was written.

| metric | value |
|---|---|
| mean interest score, items Connor liked | 0.552 |
| mean interest score, items Connor rejected | 0.465 |
| liked items falsely vetoed | **0 of 18** |
| rejected items caught by veto | 17 of 22 |

The old system could not do this at all: its `blocklist = ["sports"]` matched
none of the baseball recaps Connor spent 18 clicks rejecting.

## Finding 1 — compare blocks by z-score, never by raw cosine

Raw cosine magnitude is not comparable across blocks. Mean similarity to the
whole corpus, per block:

    geopolitics          0.501      us-national          0.477
    italy-mexico         0.494      labor-inequality     0.475
    global-south         0.490      software-systems     0.469
    energy-climate       0.455      tech-science-policy  0.468
    cincinnati           0.464      urbanism             0.448
    kentucky-louisville  0.441

Taking a plain `max` across blocks therefore hands almost everything to
`geopolitics`. Normalizing within each block first fixes it:

| ranking | liked items' median rank | in top 200 | geopolitics' share of items |
|---|---|---|---|
| raw cosine | 1246 / 3885 | 4/18 | 78% |
| **z-score per block** | **293 / 3885** | **7/18** | **19%** |
| percentile rank | 1504 / 3885 | 4/18 | 34% |

Percentile ranking is *worse than raw* because converting to ranks discards
magnitude. Z-score wins clearly.

Under z-scoring the top of the feed becomes topically varied — AI regulation,
the Google ads ruling, a Kenyan airport strike, the shutdown vote, India's grid
crunch, mail voting at the Supreme Court, childcare waitlists — instead of
twenty consecutive Lebanon headlines.

## Finding 2 — veto needs a margin rule, not a threshold

A pure threshold at 0.55 caught 20/22 rejections but falsely vetoed real news
that is merely sports-*adjacent*:

- "Belgian mayor blocks club from hosting Israeli opponent in Europa League" (0.579)
- "Shein trades down after Hong Kong debut" (0.562) — matched on "trades"
- "California lawmakers pull deal on wildfire victim compensation" (0.569)
- "Kushner dice que lamenta su participación en el plan de venta del Mundial" (0.562)

The first of those is notable: the substring blocklist killed it too, via
`"la liga"` matching "la **Liga** Europa". Two entirely different mechanisms,
same wrong answer, because both ask "is this about sport?" when the question is
"is this *more* about sport than about anything Connor wants?"

Hence the hybrid rule. Sweep, with canary stories tracked:

| absolute | margin | rejections caught | likes lost | corpus vetoed | false vetoes |
|---|---|---|---|---|---|
| threshold only 0.55 | — | 20/22 | 0/18 | 28.6% | **4 known** |
| **0.60** | **0.3** | **17/22** | **0/18** | **20.7%** | **0** |
| 0.60 | 0.5 | 16/22 | 0/18 | 17.6% | 0 |
| 0.62 | 0.5 | 13/22 | 0/18 | 15.5% | 0 |

Chosen: `veto_absolute = 0.60`, `veto_margin_over_interest = 0.3`.

The one rejection that still escapes is "Umpire leaves game between Red Sox and
Mariners after ball catches him under the chin", which genuinely reads as
human-interest oddity rather than sport. Acceptable.

## Finding 3 — throughput is fine, but far slower than assumed

Embedding 3,885 items took **358 s — about 11 items/sec** with default fastembed
settings on this box (quantized bge-small, ~15 cores busy, 2.2 GB RSS). The plan
had claimed "thousands/sec"; that was wrong and is corrected here.

It does not matter operationally — a realistic day of ~2,000 items embeds in
about three minutes, and embeddings are cached by content hash so each item is
embedded once. But it is worth tuning batch size and thread count before
assuming headroom, and worth re-measuring if the corpus grows.

Critically, it ran entirely on CPU: no GPU, no llama-swap contention, nothing
competing with the OMO stack.

## Caveats

- 40 labels is a small evaluation set. These numbers are directional, not proof.
- The labels are biased: 18 of 22 negatives are AP sports, because AP was 52% of
  the corpus. Veto performance is therefore better characterized than interest
  ranking.
- Z-score needs a population to normalize over. In production this must be a
  rolling per-block mean and standard deviation over a trailing window, not a
  one-shot pass over a static corpus.