Scoopkit collapses duplicate AI industry headlines into one structured event. When 3 outlets cover the same model release or earnings report, Scoopkit merges the coverage into a single event record instead of returning 3 near-identical articles.
The merge runs on 2 similarity thresholds, an entity check, and an LLM tie-break for anything that lands in between. A nightly pass re-checks recent events and fixes wrong calls on its own. The whole pipeline costs about $40 a month to run at current volume.
Pipeline architecture locked 2026-07-16. The numbers below come from a live sample of 600 events pulled from the public API on 2026-08-27, covering July 22 to August 27, 2026.
Facts
| Stage | What it checks | Real number |
|---|---|---|
| Embedding similarity, high end | Are 2 articles almost certainly the same event | 0.85+ cosine similarity merges automatically |
| Embedding similarity, low end | Are 2 articles almost certainly different events | 0.65 or below skips the merge |
| Entity overlap gate | Do the organizations mentioned in both actually match | Checked on every borderline pair before the tie-break |
| Tie-break | Same real event or not, for anything between 0.65 and 0.85 | Free-model cascade first, falls back to Claude Haiku 4.5 |
| Nightly merge pass | Re-checks recent events for missed matches | Runs once a day, up to 20 pairs at a time |
The two-threshold model
One similarity number can't carry the whole decision. Above 0.85 cosine similarity, 2 articles are close enough that they're almost always covering the same real-world event, so Scoopkit merges them without asking anything else. Below 0.65, they're almost always different, so the merge stops there too.
The gap between those 2 numbers is where the hard cases live. A funding announcement and an unrelated funding announcement, written in similar corporate language, can land in the same similarity band as a genuine duplicate. That band gets a second check instead of a single unclear guess.
The entity overlap gate
Similarity alone can be fooled by 2 articles that just sound alike. Before anything is allowed to merge, Scoopkit checks whether the organizations mentioned in both pieces actually overlap, pulled from a list of about 30 known AI labs and tech companies.
If neither article mentions a known org, the gate passes automatically. There's nothing to check, so it doesn't block the merge on a technicality.
The tie-break and the nightly pass
Borderline pairs get one direct question sent to an LLM: is this the same real-world event as the one already in the database. Scoopkit tries a free model first and falls back to Claude Haiku 4.5 only when the free cascade doesn't return a usable answer.
That single call isn't the end of the process. A nightly merge pass separately re-examines recent event pairs, up to 20 at a time, and re-runs the check. If an article got a wrong "no" earlier in the day, the nightly pass catches it and merges the events after the fact. The pipeline corrects itself instead of leaving a bad call in place until someone notices.
A real merge: Nvidia's Q2 earnings
On August 26, 2026, Techmeme aggregated 3 separate articles covering Nvidia's Q2 FY2027 earnings. Scoopkit merged all 3 into one event (ev_3b1d8159b2144540) instead of leaving them as 3 duplicate headlines.
The merged event carries structured numbers pulled from the coverage: revenue of $96.2 billion, up 106% year over year, data center revenue of $89 billion, up 117%, and net income of $59.7 billion, up 126%. A feed built on raw articles would show that earnings report 3 times. A Scoopkit query returns it once, with the numbers already extracted.
You can see the same kind of merged event live by getting a free API key and querying /v1/events.
Across the 600-event sample, the average event merges 1.27 articles, and 17.2% of events merge 2 or more. The widest single merge in that sample pulled 9 separate articles into one event. Over the 37 days sampled, Scoopkit produced roughly 16.7 events a day.
FAQ
Does Scoopkit ever miss a duplicate?
Yes, sometimes, at first. If an article lands just outside the merge thresholds or the tie-break call goes the wrong way, it can start as its own event. The nightly merge pass exists for exactly this: it re-checks recent events every day and merges the ones that got a wrong "no" earlier.
Why not just dedupe by headline text similarity?
Different outlets rarely write the same headline for the same event. Comparing raw text catches the easy cases and misses the rest, which is why the pipeline compares embeddings and checks entity overlap instead of matching words.
How much does this cost to run?
About $40 a month at current volume. That covers the embedding calls, the LLM tie-break, and the nightly merge pass.