← All projects
50 stories/day → 5 · LLM-as-judge eval gate active

News

A daily AI and EdTech news filter that cuts fifty stories down to five, paired with a golden-set evaluation pipeline to measure whether the scorer still agrees with my own judgment.

News is a personalized, curated news service. I started it because I was pissed off that Feedly wouldn’t offer APIs to paying subscribers. Sidenote: there’s a question around what “moats” are post-LLM. Services should offer a headless experience by default.

News fetches roughly fifty daily stories and narrows them to at most five worth reading. The harder part was deciding whether its judgment matched mine. The project pairs the digest with an evaluation loop so a prompt change can be checked against examples I’ve already labeled.

How it works

Three scheduled jobs fetch stories, score them, and assemble a daily digest. They share a database and run in parallel.

Fetcher Poll RSS, Hacker News, GitHub Trending, Google News Deduplicate by URL hash, write unscored items EVERY 6H unscored items in DynamoDB Scorer Pull up to 80 unscored items, oldest first Apply 4-criterion rubric — does ANY criterion clear the bar? Write verdict, criterion, confidence, reason EVERY 6H + 15M scored items, clears_bar=true|false Digest Builder Filter clears_bar=true AND confidence=high AND undelivered Rank by criterion priority, cap at 5 items Write today's digest row (or has_items=false) DAILY 7AM ET At most 5 items, served via REST + MCP

The fetcher polls RSS, Hacker News, GitHub Trending, and narrow Google News queries. A hash of each URL prevents the same link being stored twice.

Haiku-level worker evaluates each item against four criteria. The digest takes up to five high-confidence items.

"Anthropic releases Claude 4.7" title + summary + source Haiku reads it, tests against 4 criteria CLAUDE HAIKU 4.5 passes if ANY one criterion clears the bar CRITERION 1 Capability model capability, architectural pattern, or tool that changes design decisions CRITERION 2 Community emerging practice stress-tested by practitioners (HN, trending repos) CRITERION 3 K12 Policy regulatory or structural shift in K12 education CRITERION 4 Frontier advances what's known about frontier lab capabilities Strict JSON verdict { "clears_bar": true, "criterion": 1, "confidence": "high", "reason": "new model with..." } Eligible for digest Cut (still queryable)

Measuring judgment

I manually label a golden set of stories, replay the scorer against it, and compare the verdicts and reasons. A larger model reviews the reasoning. A regression gate checks proposed changes before they replace the current prompt.

Golden set 75 hand-labeled items verdict + criterion "what I would say" Replay the scorer Same prompt, same model, on each item in the set HAIKU 4.5 Compare verdicts, compute metrics precision · recall · F1 · criterion agreement confidence calibration · run-to-run consistency Judge the reasoning Did the scorer's stated reason actually justify the verdict? Specificity · faithfulness to the source text SONNET 4.6 Regression gate F1 ≥ 0.75 · reason quality ≥ 80% · consistency ≥ 85% If any threshold fails, the build fails Ship the change Block, investigate

Stack

PartTool
Jobs and scheduleAWS Lambda and EventBridge
Stories, verdicts and digestsDynamoDB
Source listS3
ScorerClaude Haiku
Evaluation judgeClaude Sonnet
AccessMCP and REST