Profile v2: split conflict from statecraft, add space, expand vetoes

Second elicitation round. The significant correction is Connor's own framing:
"my actual war interest is low, but unfortunately all wars are important." The
click history looked war-heavy only because world-news feeds are war-heavy, so
geopolitics-at-weight-1.0 was reading the corpus, not the reader. Statecraft is
now its own block and armed conflict is five weighted bands (anti-imperial,
US-involved, Mideast/Israel, undercovered, general) routed mainly by importance.

New: space-aerospace at 0.95, the highest-weighted block -- Connor is an
aerospace engineer. Plus science-discovery, environment-ecology,
ohio-state-politics, left-movements, anticolonial-sovereignty, far-right-watch,
political-theory. Linux/ricing/neovim folded into software-systems.

Introduces `demote` blocks, distinct from hard vetoes, for categories adjacent to
real interests: crypto/AI hype sits next to AI regulation, culture-war outrage
next to far-right-watch, true crime next to criminal-justice policy. Hard-vetoing
those would take real stories with them -- the lesson from the Europa League
story that both a substring blocklist and a cosine threshold wrongly killed.

Sources 39 -> 56, all fetched and verified: space (SpaceNews, Spaceflight Now,
NASASpaceflight, NASA, ESA, Phys.org), Linux (Phoronix, LWN, It's FOSS,
r/unixporn), left/anticolonial (Intercept, Drop Site, Tribune, African
Arguments). Rejected on inspection: nature.rss is journal metadata ("Author
Correction: ...") and sciencedaily is a press-release mill.

Embedding stays English-only, measured rather than assumed: the same story in
Spanish and English scored 0.758 and 0.600 cosine against a random-pair p99 of
0.634, so one pair was indistinguishable from an unrelated article. Italy and
Mexico are covered by English-language sources instead.

Also fixes clustering performance: centroids are held in memory across a batch
instead of one SELECT plus a numpy rebuild per item, which had taken ~15 minutes
for 1,258 items.
This commit is contained in:
connor committed 2026-09-08 15:25:56 -04:00
1 parent 64690ecfe0
commit 6dc3694260
4 files changed
+572 -193

No files matched your search

+111 -80
View File
@@ -1,108 +1,139 @@
# Connor's news profile
The thing the previous system never had. `news_triage` v0.2's entire model of taste was
three keywords and two blocked substrings; everything else was ranking machinery with
nothing meaningful to rank. This file is the replacement, and `profile.toml` is its
machine-readable form.
The thing the previous system never had. `news_triage` v0.2's entire model of taste
was three keywords and two blocked substrings; everything else was ranking
machinery with nothing meaningful to rank. This file is the replacement, and
`profile.toml` is its machine-readable form.
Written 2026-09-08, from a structured interview plus evidence from 32 real feedback
clicks in the v0.2 database.
Written 2026-09-08 from a structured interview plus evidence from 40 real feedback
clicks in the v0.2 database. Schema v2.
## How this is used
Every `description` and `probe` below is embedded (bge-small-en-v1.5, CPU) into a vector.
Item relevance is the **max** cosine similarity across probes, never the mean — averaging
"Middle East geopolitics" with "grid-scale energy storage" produces a vector near neither.
Feedback then moves these vectors over time (Rocchio); the written text is the cold-start
seed so the system is useful on day one instead of after 200 clicks.
Every `description` and `probe` is embedded (bge-small-en-v1.5, CPU) into a vector.
An item's score for a block is the **max** cosine across that block's probes, never
the mean — averaging "Middle East geopolitics" with "grid-scale energy storage"
gives a vector near neither. Blocks are then compared by **z-score within block**,
because raw cosine is not comparable across blocks; see `CALIBRATION.md`.
Vetoes work the same way, as *regions* rather than strings. This is the whole point: the old
system could not express "sports" as a concept, only as the literal characters `s-p-o-r-t-s`,
which is why 18 of 32 feedback clicks were Connor hand-rejecting AP baseball recaps that a
`blocklist = ["sports"]` never matched.
Feedback moves these vectors over time. The written text is the cold-start seed, so
the system is useful on day one rather than after 200 clicks.
## Interests
Ordered roughly by strength. Weights are multipliers on the similarity score.
**Foreign affairs is split in two.** Connor's own framing: *"my actual war interest
is low, but unfortunately all wars are important."* The click history looked
war-heavy only because world-news feeds are war-heavy. So statecraft —
`diplomacy-foreign-policy`, weight 0.9 — is the standing interest, while armed
conflict is weighted lower and routed mainly through importance.
1. **Geopolitics and conflict** (1.0) — the strongest and most consistent signal. Middle East
above all, plus great-power dynamics: China's expanding global reach, US foreign policy,
war and escalation, sanctions, the UN.
2. **Tech and science policy** (0.9) — AI regulation, platform governance, surveillance and
privacy law, research funding. Policy *about* technology.
3. **Labor, inequality and social policy** (0.9) — the "democratic socialism" of the old
config, read properly: unions, strikes, wages, housing, healthcare, education, welfare
states.
4. **Software and systems, deep** (0.9) — Connor runs a self-hosted inference stack and a
substantial homelab. Rust and WASM, systems programming, self-hosting, local model
inference, and release notes for things he actually runs. **Eligible for push**, not
digest-only — this was an explicit choice.
5. **Urbanism and city planning** (0.8) — transit, zoning, housing density, street design,
public space. New in this profile; entirely absent from v0.2.
6. **Energy, climate and infrastructure** (0.6) — grid buildout, storage, renewables, climate
policy. Real but explicitly weaker than the others.
7. **Global South** (0.6) — Africa, South Asia, Latin America. Not selected in the interview,
but three of the fourteen starred items were Uber exiting Nigeria and Uganda, India's grid
and storage crunch, and explosions at a Bolivian military base. Carried at a modest weight
on that evidence. **Flagged for review** — drop it if it turns out to be noise.
Within conflict, his stated ordering became five weighted blocks:
### Geography
| block | weight | |
|---|---|---|
| `conflict-anti-imperial` | 0.90 | anti-colonial and socialist-revolutionary wars |
| `conflict-us-involved` | 0.85 | wars the US is actively engaged in |
| `conflict-mideast-israel` | 0.80 | Middle East, especially engagements with Israel |
| `conflict-undercovered` | 0.75 | Sudan, DRC, Myanmar, the Sahel, Yemen |
| `conflict-general` | 0.45 | everything else — low interest, still important |
- **Cincinnati** — where he lives. Local politics and city stories, at a much lower
importance bar than world news.
- **Louisville / Kentucky** — where he is from. Still wanted, weaker than Cincinnati.
- **Ohio** — state politics.
- **US national politics** — federal policy, courts, elections, legislation.
- **Italy** and **Mexico** — specific standing country interests.
**Space and aerospace** (0.95) is the single highest-weighted block: Connor is an
aerospace engineer. He asked for launches and missions, deep technical coverage
(propulsion, materials, orbital mechanics, failure post-mortems), and
astronomy/planetary science — deliberately *not* the industry-and-contracts beat,
which his policy blocks already catch.
## Vetoes
**Software and systems** (0.9, push-eligible) covers local model inference, Rust and
WebAssembly, self-hosting and infrastructure, and — in his words — *"Linux news in
general. The ricing world/neovim configs/etc."* Notably he did **not** pick security
and vulnerabilities, so that is absent rather than assumed.
- **US big-three sports** — professional and college basketball, American football, baseball,
and hockey. Game recaps, scores, trades, drafts, salary-cap news, betting lines.
- **Royal and celebrity gossip** — the British royal family, red carpets, box office,
influencer drama.
- **Horoscopes, listicles, product-roundup content.**
**Political economy** spans `labor-inequality` (0.85), `left-movements` (0.85),
`anticolonial-sovereignty` (0.8), `far-right-watch` (0.75, tracked as a threat
rather than out of sympathy), and `political-theory` (0.7).
### Explicitly NOT vetoed
**Place**: `cincinnati` (0.9 — where he lives, so a lower bar), `ohio-state-politics`
(0.75), `kentucky-louisville` (0.6 — where he's from), `us-national` (0.8), and
`italy-mexico` (0.75).
Connor is not anti-sport; he is anti *US-big-three-wire-recap*. These stay:
Italy and Mexico are a special case. He follows both for politics but also because
he is learning the languages and expects to travel there, so **culture and popular
culture count as signal in this block, not noise** — the one place where lifestyle
content is wanted. The `consumer-lifestyle` veto must not swallow it.
- **Rock climbing, chess, the Olympics, oddball sports** — actively wanted.
- **Soccer** — he is trying to get into it. Match previews and live threads are noise, but
the sport is not blocked as a concept. This is the first thing to loosen if soccer feels
under-covered.
- **Sports governance and politics** — FIFA/UEFA corruption, Olympic host-city displacement,
Qatar migrant labour. These read as politics, not sport, and the old blocklist's
collisions with them were a bug.
Also carried: `science-discovery` (0.8), `environment-ecology` (0.75),
`tech-science-policy` (0.9), `urbanism` (0.8), `global-south` (0.7),
`energy-climate` (0.6).
## Vetoes, and why some things are demotes instead
Hard vetoes are reserved for categories that never overlap a real interest: US
big-three sports recaps, royal and celebrity gossip, consumer/lifestyle filler.
Anything that *might* overlap is a **demote** — a penalty on the score rather than
exclusion. This distinction exists because a hard veto on a sports-adjacent region
killed *"Belgian mayor blocks club from hosting Israeli opponent in Europa League"*,
which is Middle East politics. Overlap is the norm, not the exception:
- `crypto-ai-hype` sits next to AI regulation and local-inference news
- `culture-war-outrage` sits next to `far-right-watch`
- `true-crime-blotter` sits next to criminal-justice policy
### Explicitly not vetoed
Connor is not anti-sport; he is anti *US-big-three-wire-recap*. **Rock climbing,
chess, the Olympics and oddball sports are actively wanted.** Soccer is not blocked
as a concept — he is trying to get into it — though match previews and live threads
are noise. Sports *governance* (FIFA/UEFA corruption, Olympic host-city displacement,
Qatar migrant labour) reads as politics and stays.
## The importance lane
Independent of interest. This is the "I need to know this even though it isn't my topic"
channel, and it is half the reason the project exists. Its own small daily budget.
Independent of interest, and half the reason the project exists: war and escalation,
mass-casualty events, US federal shocks, economic breaks, and major US political
events (decisive elections, deaths of political figures).
- War and escalation — new military action between states, major escalation in an ongoing
conflict, nuclear or WMD events.
- Mass-casualty events and disasters — earthquakes, major attacks, industrial and transport
catastrophes.
- US federal shocks — constitutional crises, major Supreme Court rulings, shutdowns, sudden
sweeping policy moves.
- Economic breaks — market crashes, bank failures, currency crises, central bank surprises.
- Major US political events — significant election results, deaths of political figures.
**Vetoes still apply on this lane.** In v0.2 the bypass lane ignored relevance entirely
(`gate.rs:302`), so a blocklisted item could still fire at priority 5. That must not recur.
**Vetoes still apply on this lane.** In v0.2 the bypass lane ignored relevance
entirely (`gate.rs:302`), so a blocklisted item could still fire at priority 5.
## Delivery
- 2–5 pushes per day total, across both lanes. High bar.
- The daily digest is a **web page**, not a push payload. v0.2 sent a long markdown blob that
ntfy turned into a `.md` attachment, which is miserable to read on a phone. The push is
three lines and a link.
- ntfy feedback buttons **must work**. They never have — `NEWS_API_BASE_URL` was unset, so
they were silently dropped at boot. That is the entire training signal for everything above.
- Digest at **08:45**, quiet hours **22:00–07:00**. The importance lane **may** wake
him; the interest lane never may.
- 2–5 pushes/day across both lanes.
- Conflict outside categories A–D notifies only on a major development — a new
offensive, a ceasefire, mass casualties, a state entering. Routine front-line
reporting goes to the digest.
- HN items are ignored below **150 points**: the crowd does the first filtering pass.
- The digest is a **complete record** of the day, including already-notified items,
and the weekend edition runs longer and leans longform.
- **Three serendipity slots** in every digest for items that scored well on quality
and importance but poorly on the profile, labelled "outside your usual". A system
that only shows you what you already like tightens its own bubble as it learns.
- No format filter: he reads analysis, wire copy, investigations *and* opinion, which
also confirms the old config's `notify_opinions = true` was deliberate.
- Feedback stays **binary**. The old system's failure was collecting 32 labels, not
collecting shallow ones.
## Source trust
`weight` in `sources.toml` is editorial trust, carried over from the old config where
Connor had deliberately set Al Jazeera highest (1.2) and NYT lowest (0.8), and
extended to weight independent outlets (The Intercept, Drop Site, Tribune, African
Arguments, Middle East Eye, +972) at 1.1. It affects importance and tie-breaking,
never whether an item is visible.
## Language
Embedding is English-only. Measured: the same story in Spanish and English scored
0.758 and 0.600 cosine against a random-pair p99 of 0.634 — one of two translation
pairs was indistinguishable from an unrelated article. Native-language feeds would
rank erratically, so Italy and Mexico are covered by English-language sources.
Revisit with a multilingual model only if that becomes worth re-calibrating every
threshold for.
## Open questions
- Digest hour, and the exact quiet-hours window.
- Whether the importance lane may ever override quiet hours, and at what score.
- Whether "Global South" is a real standing interest or an artifact of a small sample.
- Whether `global-south` is a genuine standing interest or an artifact of a small
sample. Confirmed wanted by Connor, but worth re-checking against real feedback.
- Exact score at which the importance lane may override quiet hours.