Scoopkit is not a real-time wire, and the delay is deliberate rather than incidental. An event whose cluster has 2 or more articles publishes as soon as the second one lands. An event with only 1 article waits 2 hours to see whether a second shows up. Since 83.1% of events never get a second article, most events wait out the full hold, which puts the expected median at roughly 2 hours from the moment we first see an article. If you need sub-minute alerting, this is the wrong tool, and that is worth knowing before you build on it rather than after.
Config values read from the live production service on 2026-09-10. The distribution figures below are derived from the gate rules plus two measured inputs, not from a fresh percentile pull: the hold was cut from 6 hours to 2 on 2026-09-10, so the archive currently mixes events published under both settings and any percentile taken today would describe neither.
The gate
Two conditions, checked every cycle. The first one that passes publishes the event.
| Condition | Effect |
|---|---|
article_count >= 2 | Publish now |
| Oldest article in the cluster is 2+ hours old | Publish now, however many articles |
| Neither | Keep waiting, stay unpublished |
That is the whole rule. PUBLISH_MIN_ARTICLES is 2 and PUBLISH_AGE_HOURS is 2 in production.
The reason for holding at all is that clustering cannot deduplicate coverage that has not arrived yet. Publish a single-article event the instant it appears and you have committed to it being its own event. When the second and third outlets cover the same story an hour later, you either merge after the fact and change an already-published record, or you leave three events describing one story.
What the wait looks like
Two measured inputs decide the shape. 16.9% of events end up with more than one article. Of those second articles, 39.6% arrive within 2 hours of the first.
| Outcome | Share of events | Lag |
|---|---|---|
| Second article lands inside the hold | 6.7% | under 2 hours, often minutes |
| Everything else | 93.3% | about 2 hours |
So the distribution is not a spread, it is a spike just past the 2-hour mark with a small fast tail below it. If you chart your events by lag you should see that shape, and seeing it means the gate is working rather than that something is backed up.
The 6.7% figure is 16.9% multiplied by 39.6%. It is small because second articles are slower than intuition suggests: the median gap between the first and second article on the same story is 3.5 hours, and the 90th percentile is 43.7 hours.
Why it used to be 6 hours, and why that changed
The hold was 6 hours until 2026-09-10. Measuring it produced an uncomfortable result. A 6-hour hold caught 57.6% of second articles before publication. A 2-hour hold catches 39.6%. So four extra hours of delay on every single event bought 18 percentage points of extra deduplication, applied to the 16.9% of events that have a second article at all: about 3% of events.
That is a bad trade, and it was only defensible while the thing meant to catch the rest was not running. A merge pass re-examines recent events and folds duplicates together after publication. It exists, and it turned out never to have run in production, which meant the gate was doing the whole job alone. Once the merge pass ran on every cycle, holding everything for 6 hours stopped earning its keep.
The honest framing of the current tradeoff: at 2 hours, roughly 3% more of our events may briefly appear as two events rather than one, until the merge pass catches them on a subsequent cycle.
Where the other minutes go
Everything before the gate is fast enough not to matter, but it is worth knowing the shape.
A cron cycle runs every 10 minutes and does five things in sequence: queue any source that is due, drain the fetch queue, run the pipeline over up to 16 events, check ingestion freshness, then run the merge pass. Sources are polled on their own schedules rather than all at once, currently 4 every 15 minutes, 16 every 30 minutes, 12 hourly, and 3 daily, across 35 configured sources (34 RSS feeds and 1 scraper, since Anthropic publishes no feed at all).
The --limit 16 per cycle exists because embeddings run against Voyage's free tier and the rate limit is real. A burst of news drains across several cycles rather than one.
Then each event gets embedded, clustered against the last 4 days of events at a 0.85 similarity threshold with an entity-overlap check, and classified in a single model call. That call cascades across free models first and falls back to a paid one only when every free candidate is exhausted.
All of that is seconds to minutes. The hours are the gate.
If you need it faster
Use webhooks rather than polling. They fire on publication, so you learn about an event as soon as it clears the gate instead of on your next poll interval. That removes your own polling delay, which for an hourly job is now the larger contributor. Webhooks are a Pro feature.
Do not build a trading signal on this. Two hours is still a deliberate accuracy-over-speed trade. Anything where minutes matter needs a different kind of product, and being honest about that is more useful to you than a caveat buried in the docs.
Expect a small amount of visible duplication. At a 2-hour hold, some stories publish as one event and gain their second article afterwards. Deduplicating on event_id alone is not quite enough if you keep a local copy; see the reconciliation note below.
FAQ
Why 2 hours specifically?
Because it is roughly where the tradeoff stops being lopsided. Going to 1 hour would catch only 28.9% of second articles pre-publication instead of 39.6%, and going to 3 would catch 46.2%. None of these has been tuned against a labelled set, so treat 2 as a measured judgement rather than a proven optimum.
Does an event change after it publishes?
Yes, and more often now than when the hold was longer. The merge pass can fold an event into another one when later coverage makes the match clear. The absorbed event's ID keeps working: merges record an alias, so a request for the old ID returns the surviving event rather than a 404. Verified on 2026-09-10 against real merges. The one caveat is that an alias is subject to the same tier rules as any other ID, so if the surviving event has aged past your archive window you get a 404 for that reason, not because the alias is missing.
How should I reconcile that?
Reconcile against GET /v1/events?since=... rather than treating each webhook delivery as final. That catches post-publication merges, which webhooks do not currently fire on.
How fresh is the underlying source polling?
Between 15 minutes and a day per source, depending on how often it publishes anything worth fetching. High-volume sources get the tighter intervals.
How would I know if ingestion stopped?
We would, and quickly. Every cron cycle checks the age of the newest published event and pages if it exceeds 150 minutes. You can see the same signal yourself in /health, which reports ingestion and last_event_age_s.