Rebalance the digest: rank by interest, cap per group, fix HN maturation

Connor: too much world news, very little software/aerospace/urbanism/Cincinnati.
Measuring found three separate causes.

1. The digest ranked by max(interest, importance), and importance is largely a
   "big world news" detector -- 22 of 28 slots were entered on the importance
   axis. A personal reading list ordered by global importance is the wrong
   objective. The bulk of the digest is now ranked by INTEREST, with a bounded
   allowance (digest_importance_slots = 6) so a major event still makes the
   record on a busy day.

2. No per-group cap, so conflict and diplomacy took 18 of 28 slots.
   digest_max_per_group = 4, with unused capacity flowing back to the best
   remaining items so a quiet day still fills.

3. HN maturation could never happen. Items came only from `search_by_date`,
   which returns posts at 2-3 points; once one ages out of the newest window its
   score is never refetched, so it can never clear the bar and HN's best content
   was structurally invisible. 45 software clusters were sitting in
   "aggregator-immature". Added the front page via the `search` endpoint, which
   returns the crowd's verdict (measured: median 117 points, max 912) and whose
   re-polling is what actually supplies maturation. hn_min_score 150 -> 100,
   since a higher bar rejects most of the front page.

Digest went from 8 groups (conflict 10, diplomacy 8, software 0) to 13 groups
with a far flatter distribution, and the software slot is now
"Benchmarking Qwen3.8 27B quantizations" -- the local-inference material that
was asked for.

Process note: the first two attempts at this silently did nothing. I used
str.replace() without asserting the anchor matched, so a stale anchor wrote the
file back unchanged while reporting success, and I measured "no change" twice
before checking the file. Edits here now assert before writing.
This commit is contained in:
connor committed 2026-09-08 17:46:48 -04:00
1 parent 83bc36aa79
commit 7f402f22fe
5 files changed
+91 -21

No files matched your search

+11 -1
View File
@@ -97,12 +97,22 @@ digest_floor = 0.5
# A floor alone yielded 198 items a day. The digest is capped by count and fed
# strongest-first, so the cap is always spent on the best candidates.
digest_max_items = 25
# Max items per topic group in the digest, so no single subject crowds it out.
# Measured before this existed: conflict and diplomacy took 18 of 28 slots while
# software-systems took 0, despite being 141 of the eligible clusters.
digest_max_per_group = 4
# Slots reserved for the day's most important stories regardless of topic, so a
# major event stays in the record even when your own interests are busy. The
# remaining slots are ranked by interest.
digest_importance_slots = 6
# Serendipity slots are filled from items BELOW this interest score, so they are
# genuinely outside the profile rather than near-misses.
serendipity_max_interest = 0.5
# Aggregator maturation: an HN post at 3 points is noise, at 300 it is news.
hn_min_score = 150
# 100, not 150: the HN front page has a median of ~117 points, so a higher bar
# would reject most of what the crowd has actually promoted.
hn_min_score = 100
# Nothing older than this may generate a notification. Feeds return large
# archives on first fetch -- one run pulled 88 items going back five months --
+14
View File
@@ -118,6 +118,20 @@ kind = "hn"
bucket = "hacker-news"
poll_interval_secs = 1800
# The front page, which is the only way HN scores ever mature. Items fetched by
# `search_by_date` arrive at 2-3 points and, once they age out of that window,
# are never refetched -- so on their own they can never clear the maturation
# bar and HN's best content stays permanently invisible. Re-polling the front
# page refreshes the score of whatever the crowd has actually promoted.
[[source]]
name = "Hacker News front page"
url = "https://hn.algolia.com/api/v1/search"
kind = "hn"
query = "front_page"
bucket = "hacker-news"
page_size = 50
poll_interval_secs = 1800
[[source]]
name = "Lobsters"
url = "https://lobste.rs/rss"