Z-scoring subtracts each block's mean, which encodes how much of the corpus that
block matches: broad blocks compressed, narrow ones stretched. The same raw
cosine of 0.67 scored z 1.28 in `software-systems` and z 4.49 in
`conflict-undercovered`. Every block's own 99th percentile landed at z 2.2-2.9,
yet the threshold was a flat 2.0 and blocks were ranked against each other on
that scale -- so a subject was penalised for being well covered in the reader's
own source list. That is the mechanism behind a routine story scoring 5.22 on
war-escalation.
top-anchored puts each block's own 99th percentile at 1.0, which removes the
distortion by construction and leaves `weight` to express preference instead of
fighting corpus composition.
Every threshold in normalised units had to move at the same time, or the
interest lane goes silent. Each was recalibrated to admit the same share of the
corpus its predecessor did, measured on 1,400 live clusters rather than
converted by arithmetic:
interest_threshold 2.0 -> 0.70 (top 8.3%)
important_threshold 2.0 -> 1.00 (top 21.5%)
digest_floor 0.5 -> -0.55 (top 47.9%)
serendipity_max_interest 0.5 -> -0.55
veto_margin_over_interest 0.3 -> 0.12 (22.1% vetoed, against 22.0%)
`veto_absolute` and `demote_absolute` are raw cosine and do not move. The
negative values are not a mistake: the scale is anchored on the median and the
99th percentile, so half of all clusters score below zero by construction.
Effect: digest world-politics concentration falls from 10 of 28 slots to 6.
It does NOT by itself put software or space into the digest, and two hypotheses
for that were tested and rejected -- `aggregator-immature` is not the binding
constraint there (making those items eligible moves space 0->1, software 0->0),
and the normalizer alone is not enough. Their best clusters score 0.63 and 0.91
against a cut around 0.70; they are just below it, not blocked. Raising the two
weights is what works, and that is a taste decision left to the reader.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015xPLgmcXCc9JjVwmQF63eM
Ground-up replacement for news_triage, whose signal quality failed because it
had no representation of what its user likes -- three keywords fed to BM25 plus
a two-term substring blocklist.
profile/ holds the taste model as prose and TOML: 11 interest blocks, 3 veto
blocks, 5 importance blocks, each a set of probe sentences to be embedded.
The [scoring] constants are fitted, not guessed. CALIBRATION.md records the
measurements against all 3,885 items in the v0.2 database and 40 real feedback
labels, including two findings that changed the design:
- compare blocks by z-score within block, never raw cosine; raw cosine let
geopolitics win 78% of items and buried known-liked items at median rank 1246
of 3885 (293 after normalizing)
- veto needs a margin over the interest signal, not a threshold; a plain 0.55
cutoff falsely killed "Belgian mayor blocks club from hosting Israeli opponent
in Europa League", which the old substring blocklist also killed
connorandClaude Opus 5