Connor noticed 5 of 5 alerts were Middle East. Measuring showed that was not
proportionate: Middle East blocks are ~11% of the eligible corpus while
software-systems alone is 25%, yet Middle East took 35% of the top 20 and
software took none.
Two mechanisms, both measured:
1. Five conflict-* blocks could each claim their own alert slot. Interest blocks
now carry a `group`, and routing enforces max_alerts_per_group_per_day, so
related blocks share one quota. This alone moved the mix from 5/5 Middle East
to three distinct groups.
2. Z-scoring subtracts each block's MEAN, which silently encodes how much of the
corpus that block matches. software-systems has the HIGHEST mean raw
similarity of any block (0.537) and the LOWEST max z (2.51);
conflict-mideast-israel has a lower mean (0.460) and reaches 3.47. A topic is
penalised for being well covered in your own feed list.
Added `normalize = "top-anchored"` as an alternative that rescales each block by
its own upper tail instead of its mean, leaving `weight` to express preference
rather than corpus composition.
Also ruled out a plausible-sounding cause rather than assuming it: HN and
Lobsters items are title-only and much shorter than news articles, so terse
titles seemed likely to match prose probes poorly. Measured the opposite --
title-only HN items score 0.618 against the software block, ABOVE Phoronix/LWN
with summaries (0.610) and Ars Technica (0.540). Short text is not the problem;
prevalence is.
connor
d4631ec13d