Profile v2: split conflict from statecraft, add space, expand vetoes
Second elicitation round. The significant correction is Connor's own framing:
"my actual war interest is low, but unfortunately all wars are important." The
click history looked war-heavy only because world-news feeds are war-heavy, so
geopolitics-at-weight-1.0 was reading the corpus, not the reader. Statecraft is
now its own block and armed conflict is five weighted bands (anti-imperial,
US-involved, Mideast/Israel, undercovered, general) routed mainly by importance.
New: space-aerospace at 0.95, the highest-weighted block -- Connor is an
aerospace engineer. Plus science-discovery, environment-ecology,
ohio-state-politics, left-movements, anticolonial-sovereignty, far-right-watch,
political-theory. Linux/ricing/neovim folded into software-systems.
Introduces `demote` blocks, distinct from hard vetoes, for categories adjacent to
real interests: crypto/AI hype sits next to AI regulation, culture-war outrage
next to far-right-watch, true crime next to criminal-justice policy. Hard-vetoing
those would take real stories with them -- the lesson from the Europa League
story that both a substring blocklist and a cosine threshold wrongly killed.
Sources 39 -> 56, all fetched and verified: space (SpaceNews, Spaceflight Now,
NASASpaceflight, NASA, ESA, Phys.org), Linux (Phoronix, LWN, It's FOSS,
r/unixporn), left/anticolonial (Intercept, Drop Site, Tribune, African
Arguments). Rejected on inspection: nature.rss is journal metadata ("Author
Correction: ...") and sciencedaily is a press-release mill.
Embedding stays English-only, measured rather than assumed: the same story in
Spanish and English scored 0.758 and 0.600 cosine against a random-pair p99 of
0.634, so one pair was indistinguishable from an unrelated article. Italy and
Mexico are covered by English-language sources instead.
Also fixes clustering performance: centroids are held in memory across a batch
instead of one SELECT plus a numpy rebuild per item, which had taken ~15 minutes
for 1,258 items.
This commit is contained in:
1 parent
64690ecfe0
commit
6dc3694260
4 files changed
+572
-193
No files matched your search
+111
-80
@@ -1,108 +1,139 @@
|
||||
# Connor's news profile
|
||||
|
||||
The thing the previous system never had. `news_triage` v0.2's entire model of taste was
|
||||
three keywords and two blocked substrings; everything else was ranking machinery with
|
||||
nothing meaningful to rank. This file is the replacement, and `profile.toml` is its
|
||||
machine-readable form.
|
||||
The thing the previous system never had. `news_triage` v0.2's entire model of taste
|
||||
was three keywords and two blocked substrings; everything else was ranking
|
||||
machinery with nothing meaningful to rank. This file is the replacement, and
|
||||
`profile.toml` is its machine-readable form.
|
||||
|
||||
Written 2026-09-08, from a structured interview plus evidence from 32 real feedback
|
||||
clicks in the v0.2 database.
|
||||
Written 2026-09-08 from a structured interview plus evidence from 40 real feedback
|
||||
clicks in the v0.2 database. Schema v2.
|
||||
|
||||
## How this is used
|
||||
|
||||
Every `description` and `probe` below is embedded (bge-small-en-v1.5, CPU) into a vector.
|
||||
Item relevance is the **max** cosine similarity across probes, never the mean — averaging
|
||||
"Middle East geopolitics" with "grid-scale energy storage" produces a vector near neither.
|
||||
Feedback then moves these vectors over time (Rocchio); the written text is the cold-start
|
||||
seed so the system is useful on day one instead of after 200 clicks.
|
||||
Every `description` and `probe` is embedded (bge-small-en-v1.5, CPU) into a vector.
|
||||
An item's score for a block is the **max** cosine across that block's probes, never
|
||||
the mean — averaging "Middle East geopolitics" with "grid-scale energy storage"
|
||||
gives a vector near neither. Blocks are then compared by **z-score within block**,
|
||||
because raw cosine is not comparable across blocks; see `CALIBRATION.md`.
|
||||
|
||||
Vetoes work the same way, as *regions* rather than strings. This is the whole point: the old
|
||||
system could not express "sports" as a concept, only as the literal characters `s-p-o-r-t-s`,
|
||||
which is why 18 of 32 feedback clicks were Connor hand-rejecting AP baseball recaps that a
|
||||
`blocklist = ["sports"]` never matched.
|
||||
Feedback moves these vectors over time. The written text is the cold-start seed, so
|
||||
the system is useful on day one rather than after 200 clicks.
|
||||
|
||||
## Interests
|
||||
|
||||
Ordered roughly by strength. Weights are multipliers on the similarity score.
|
||||
**Foreign affairs is split in two.** Connor's own framing: *"my actual war interest
|
||||
is low, but unfortunately all wars are important."* The click history looked
|
||||
war-heavy only because world-news feeds are war-heavy. So statecraft —
|
||||
`diplomacy-foreign-policy`, weight 0.9 — is the standing interest, while armed
|
||||
conflict is weighted lower and routed mainly through importance.
|
||||
|
||||
1. **Geopolitics and conflict** (1.0) — the strongest and most consistent signal. Middle East
|
||||
above all, plus great-power dynamics: China's expanding global reach, US foreign policy,
|
||||
war and escalation, sanctions, the UN.
|
||||
2. **Tech and science policy** (0.9) — AI regulation, platform governance, surveillance and
|
||||
privacy law, research funding. Policy *about* technology.
|
||||
3. **Labor, inequality and social policy** (0.9) — the "democratic socialism" of the old
|
||||
config, read properly: unions, strikes, wages, housing, healthcare, education, welfare
|
||||
states.
|
||||
4. **Software and systems, deep** (0.9) — Connor runs a self-hosted inference stack and a
|
||||
substantial homelab. Rust and WASM, systems programming, self-hosting, local model
|
||||
inference, and release notes for things he actually runs. **Eligible for push**, not
|
||||
digest-only — this was an explicit choice.
|
||||
5. **Urbanism and city planning** (0.8) — transit, zoning, housing density, street design,
|
||||
public space. New in this profile; entirely absent from v0.2.
|
||||
6. **Energy, climate and infrastructure** (0.6) — grid buildout, storage, renewables, climate
|
||||
policy. Real but explicitly weaker than the others.
|
||||
7. **Global South** (0.6) — Africa, South Asia, Latin America. Not selected in the interview,
|
||||
but three of the fourteen starred items were Uber exiting Nigeria and Uganda, India's grid
|
||||
and storage crunch, and explosions at a Bolivian military base. Carried at a modest weight
|
||||
on that evidence. **Flagged for review** — drop it if it turns out to be noise.
|
||||
Within conflict, his stated ordering became five weighted blocks:
|
||||
|
||||
### Geography
|
||||
| block | weight | |
|
||||
|---|---|---|
|
||||
| `conflict-anti-imperial` | 0.90 | anti-colonial and socialist-revolutionary wars |
|
||||
| `conflict-us-involved` | 0.85 | wars the US is actively engaged in |
|
||||
| `conflict-mideast-israel` | 0.80 | Middle East, especially engagements with Israel |
|
||||
| `conflict-undercovered` | 0.75 | Sudan, DRC, Myanmar, the Sahel, Yemen |
|
||||
| `conflict-general` | 0.45 | everything else — low interest, still important |
|
||||
|
||||
- **Cincinnati** — where he lives. Local politics and city stories, at a much lower
|
||||
importance bar than world news.
|
||||
- **Louisville / Kentucky** — where he is from. Still wanted, weaker than Cincinnati.
|
||||
- **Ohio** — state politics.
|
||||
- **US national politics** — federal policy, courts, elections, legislation.
|
||||
- **Italy** and **Mexico** — specific standing country interests.
|
||||
**Space and aerospace** (0.95) is the single highest-weighted block: Connor is an
|
||||
aerospace engineer. He asked for launches and missions, deep technical coverage
|
||||
(propulsion, materials, orbital mechanics, failure post-mortems), and
|
||||
astronomy/planetary science — deliberately *not* the industry-and-contracts beat,
|
||||
which his policy blocks already catch.
|
||||
|
||||
## Vetoes
|
||||
**Software and systems** (0.9, push-eligible) covers local model inference, Rust and
|
||||
WebAssembly, self-hosting and infrastructure, and — in his words — *"Linux news in
|
||||
general. The ricing world/neovim configs/etc."* Notably he did **not** pick security
|
||||
and vulnerabilities, so that is absent rather than assumed.
|
||||
|
||||
- **US big-three sports** — professional and college basketball, American football, baseball,
|
||||
and hockey. Game recaps, scores, trades, drafts, salary-cap news, betting lines.
|
||||
- **Royal and celebrity gossip** — the British royal family, red carpets, box office,
|
||||
influencer drama.
|
||||
- **Horoscopes, listicles, product-roundup content.**
|
||||
**Political economy** spans `labor-inequality` (0.85), `left-movements` (0.85),
|
||||
`anticolonial-sovereignty` (0.8), `far-right-watch` (0.75, tracked as a threat
|
||||
rather than out of sympathy), and `political-theory` (0.7).
|
||||
|
||||
### Explicitly NOT vetoed
|
||||
**Place**: `cincinnati` (0.9 — where he lives, so a lower bar), `ohio-state-politics`
|
||||
(0.75), `kentucky-louisville` (0.6 — where he's from), `us-national` (0.8), and
|
||||
`italy-mexico` (0.75).
|
||||
|
||||
Connor is not anti-sport; he is anti *US-big-three-wire-recap*. These stay:
|
||||
Italy and Mexico are a special case. He follows both for politics but also because
|
||||
he is learning the languages and expects to travel there, so **culture and popular
|
||||
culture count as signal in this block, not noise** — the one place where lifestyle
|
||||
content is wanted. The `consumer-lifestyle` veto must not swallow it.
|
||||
|
||||
- **Rock climbing, chess, the Olympics, oddball sports** — actively wanted.
|
||||
- **Soccer** — he is trying to get into it. Match previews and live threads are noise, but
|
||||
the sport is not blocked as a concept. This is the first thing to loosen if soccer feels
|
||||
under-covered.
|
||||
- **Sports governance and politics** — FIFA/UEFA corruption, Olympic host-city displacement,
|
||||
Qatar migrant labour. These read as politics, not sport, and the old blocklist's
|
||||
collisions with them were a bug.
|
||||
Also carried: `science-discovery` (0.8), `environment-ecology` (0.75),
|
||||
`tech-science-policy` (0.9), `urbanism` (0.8), `global-south` (0.7),
|
||||
`energy-climate` (0.6).
|
||||
|
||||
## Vetoes, and why some things are demotes instead
|
||||
|
||||
Hard vetoes are reserved for categories that never overlap a real interest: US
|
||||
big-three sports recaps, royal and celebrity gossip, consumer/lifestyle filler.
|
||||
|
||||
Anything that *might* overlap is a **demote** — a penalty on the score rather than
|
||||
exclusion. This distinction exists because a hard veto on a sports-adjacent region
|
||||
killed *"Belgian mayor blocks club from hosting Israeli opponent in Europa League"*,
|
||||
which is Middle East politics. Overlap is the norm, not the exception:
|
||||
|
||||
- `crypto-ai-hype` sits next to AI regulation and local-inference news
|
||||
- `culture-war-outrage` sits next to `far-right-watch`
|
||||
- `true-crime-blotter` sits next to criminal-justice policy
|
||||
|
||||
### Explicitly not vetoed
|
||||
|
||||
Connor is not anti-sport; he is anti *US-big-three-wire-recap*. **Rock climbing,
|
||||
chess, the Olympics and oddball sports are actively wanted.** Soccer is not blocked
|
||||
as a concept — he is trying to get into it — though match previews and live threads
|
||||
are noise. Sports *governance* (FIFA/UEFA corruption, Olympic host-city displacement,
|
||||
Qatar migrant labour) reads as politics and stays.
|
||||
|
||||
## The importance lane
|
||||
|
||||
Independent of interest. This is the "I need to know this even though it isn't my topic"
|
||||
channel, and it is half the reason the project exists. Its own small daily budget.
|
||||
Independent of interest, and half the reason the project exists: war and escalation,
|
||||
mass-casualty events, US federal shocks, economic breaks, and major US political
|
||||
events (decisive elections, deaths of political figures).
|
||||
|
||||
- War and escalation — new military action between states, major escalation in an ongoing
|
||||
conflict, nuclear or WMD events.
|
||||
- Mass-casualty events and disasters — earthquakes, major attacks, industrial and transport
|
||||
catastrophes.
|
||||
- US federal shocks — constitutional crises, major Supreme Court rulings, shutdowns, sudden
|
||||
sweeping policy moves.
|
||||
- Economic breaks — market crashes, bank failures, currency crises, central bank surprises.
|
||||
- Major US political events — significant election results, deaths of political figures.
|
||||
|
||||
**Vetoes still apply on this lane.** In v0.2 the bypass lane ignored relevance entirely
|
||||
(`gate.rs:302`), so a blocklisted item could still fire at priority 5. That must not recur.
|
||||
**Vetoes still apply on this lane.** In v0.2 the bypass lane ignored relevance
|
||||
entirely (`gate.rs:302`), so a blocklisted item could still fire at priority 5.
|
||||
|
||||
## Delivery
|
||||
|
||||
- 2–5 pushes per day total, across both lanes. High bar.
|
||||
- The daily digest is a **web page**, not a push payload. v0.2 sent a long markdown blob that
|
||||
ntfy turned into a `.md` attachment, which is miserable to read on a phone. The push is
|
||||
three lines and a link.
|
||||
- ntfy feedback buttons **must work**. They never have — `NEWS_API_BASE_URL` was unset, so
|
||||
they were silently dropped at boot. That is the entire training signal for everything above.
|
||||
- Digest at **08:45**, quiet hours **22:00–07:00**. The importance lane **may** wake
|
||||
him; the interest lane never may.
|
||||
- 2–5 pushes/day across both lanes.
|
||||
- Conflict outside categories A–D notifies only on a major development — a new
|
||||
offensive, a ceasefire, mass casualties, a state entering. Routine front-line
|
||||
reporting goes to the digest.
|
||||
- HN items are ignored below **150 points**: the crowd does the first filtering pass.
|
||||
- The digest is a **complete record** of the day, including already-notified items,
|
||||
and the weekend edition runs longer and leans longform.
|
||||
- **Three serendipity slots** in every digest for items that scored well on quality
|
||||
and importance but poorly on the profile, labelled "outside your usual". A system
|
||||
that only shows you what you already like tightens its own bubble as it learns.
|
||||
- No format filter: he reads analysis, wire copy, investigations *and* opinion, which
|
||||
also confirms the old config's `notify_opinions = true` was deliberate.
|
||||
- Feedback stays **binary**. The old system's failure was collecting 32 labels, not
|
||||
collecting shallow ones.
|
||||
|
||||
## Source trust
|
||||
|
||||
`weight` in `sources.toml` is editorial trust, carried over from the old config where
|
||||
Connor had deliberately set Al Jazeera highest (1.2) and NYT lowest (0.8), and
|
||||
extended to weight independent outlets (The Intercept, Drop Site, Tribune, African
|
||||
Arguments, Middle East Eye, +972) at 1.1. It affects importance and tie-breaking,
|
||||
never whether an item is visible.
|
||||
|
||||
## Language
|
||||
|
||||
Embedding is English-only. Measured: the same story in Spanish and English scored
|
||||
0.758 and 0.600 cosine against a random-pair p99 of 0.634 — one of two translation
|
||||
pairs was indistinguishable from an unrelated article. Native-language feeds would
|
||||
rank erratically, so Italy and Mexico are covered by English-language sources.
|
||||
Revisit with a multilingual model only if that becomes worth re-calibrating every
|
||||
threshold for.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Digest hour, and the exact quiet-hours window.
|
||||
- Whether the importance lane may ever override quiet hours, and at what score.
|
||||
- Whether "Global South" is a real standing interest or an artifact of a small sample.
|
||||
- Whether `global-south` is a genuine standing interest or an artifact of a small
|
||||
sample. Confirmed wanted by Connor, but worth re-checking against real feedback.
|
||||
- Exact score at which the importance lane may override quiet hours.
|
||||
Reference in new issue
Block a user