Scoopkit Now Publishes Events Immediately and Corrects Them Afterwards

Until 2026-09-21, an event with only one article waited 2 hours before publishing, to see whether a second one arrived. That hold is now zero. An event publishes on a following pipeline cycle instead of waiting out a timer, which moved the measured median from 2.04 hours to 4.4 minutes. What changed is not that Scoopkit got faster at fetching. It is that publishing early stopped costing anything.

Config values read from the live production service on 2026-09-21, and the first measurement is below. It is a small sample taken the same evening, so treat it as a first reading rather than a settled distribution.

What the wait looks like now

Measured over the 64 events whose first article arrived after the transition backlog had cleared, covering about 20 hours including one full morning:

Against a median of 2.04 hours under the 2-hour hold, the median is roughly 28x faster and the 90th percentile is still about 6x faster.

The first version of this table, published a few hours after the change, read 2.5 minutes at the median and 5.2 at the 90th from a sample of 11 events on a quiet evening. It carried a warning that a busy morning would push the upper percentiles out. It did, and by a lot: the 90th percentile roughly quadrupled. The median moved much less. That shape is the real behaviour, and it is worth understanding rather than averaging away.

The reason is that latency is now bounded by throughput rather than by a timer. The pipeline classifies each event with a model call, processes up to 16 events per 10-minute cycle, and falls back through rate-limited free models when they return 429. Under light load an event publishes on the next cycle. Under a burst it queues behind other events, and the tail stretches. So the median tells you the common case and the 90th tells you what a news spike costs you.

Still owed, and stated so it is checkable: a full week of measurement. Two days is not enough to see a weekend, and the tail is the part most likely to move.

Worth recording what the changeover itself looked like, since it is the part a status page would have hidden. For the first couple of hours after the hold was removed, latency was worse than the hold it replaced: every waiting event became eligible at once, the queue drains oldest first, and those events published having already sat out most of their old 2-hour wait, at 183 to 190 minutes. The backlog peaked at 29 events and cleared in about two hours.

The gate now

Two conditions, checked every cycle. The first that passes publishes the event.

PUBLISH_MIN_ARTICLES is still 2 and PUBLISH_AGE_HOURS is now 0. In practice the age gate always passes, so an event publishes as soon as the pipeline reaches it. The article_count >= 2 branch still exists and still fires first when a cluster already has corroboration, which matters only for logging: the detail on a publish tells you which branch let it through.

Why the hold existed, and why that reason is gone

This is worth stating precisely, because the obvious explanation was wrong and this page said so before the hold was removed.

The hold was never about deduplication. Duplicates merge perfectly well after publication. The clustering candidate query deliberately does not filter on whether an event has published, so coverage arriving hours later attaches to an event that already went out, and a separate merge pass folds in pairs that clustering missed entirely.

The hold was about classification evidence. An event was classified exactly once, at publish, from whatever articles existed at that moment. When a late article attached, the event's embedding and article_count updated and its category did not. Publishing a single-article event immediately meant classifying it from one article, and one terse article is where the classifier is weakest. A headline like shot-scraper 1.12 carries very little to classify from. Three articles about the same release carry a lot.

That one-shot constraint no longer holds. Classification now re-runs when an event gains articles, the change is recorded in correction_history, and a webhook announces it. So the question stopped being "how long should we wait for evidence before our single attempt" and became "publish now, improve when better evidence arrives", which is roughly what a wire service does.

The correction path is not theoretical. In its first five days it corrected 421 events, and the corrections are not cosmetic: events move between categories, not just reword. An event can start in RESEARCH_AND_PAPERS and end in MODEL_RELEASE once more coverage lands.

What this costs you

Being honest about the trade, because there is one.

More events will be reclassified after you first see them. Previously most events had 2 hours of evidence before their only classification. Now single-article events are classified from a single article, which is exactly the weak case described above. The difference is that the classification is no longer final.

A story may briefly appear as two events. It always could, since only 17.5% of published events ever gain a second article (641 of 3,664, measured 2026-09-21) and the merge pass has always run after publication. Removing the hold widens that window somewhat.

If you keep a local copy, reconcile. Deduplicating on event_id alone was never quite enough and is now less so. Two mechanisms make this tractable, and both are described below.

How to stay correct

Subscribe to update and merge webhooks. As of 2026-09-21 webhooks fire on three things, not one: event.published, event.updated, and event.merged, carried in the X-Scoopkit-Event-Type header and in meta.delivery_type. An event.updated delivery carries meta.changed_fields with the exact {field: {from, to}} delta, so you can apply the correction without diffing against what you stored. An event.merged delivery carries the surviving event plus meta.merged_from, naming the ids it absorbed. Webhooks are a Pro feature.

Or reconcile against GET /v1/events?since=.... Still the right backstop if your receiver has real downtime, and the reason the archive window exists.

Absorbed ids keep working. A merge records an alias, so a request for an absorbed id returns the surviving event rather than a 404. The one caveat is that an alias obeys the same tier rules as any other id, so if the surviving event has aged past your archive window you get a 404 for that reason, not because the alias is missing.

Read correction_history. Every correction is appended there, capped at the most recent 20 entries, with the reason and the changed fields. It is a paid field.

The history, since the reasoning is the useful part

The hold was 6 hours until 2026-09-10, then 2 hours until 2026-09-21, then zero.

Measuring the 6-hour version produced an uncomfortable result. A 6-hour hold caught 57.6% of second articles before publication; a 2-hour hold caught 39.6%. Four extra hours of delay on every event bought 18 percentage points of extra deduplication applied to the small minority of events that get a second article at all, or roughly 3% of events. That was a bad trade, and it was only defensible while the merge pass meant to catch the rest was not running. It turned out never to have run in production, which meant the gate was doing the whole job alone.

Under the 2-hour hold, the measured distribution was a spike rather than a spread: 25th percentile 2.01 hours, median 2.04, 75th 2.16, 90th 2.18. Only 8.6% of events published early on a second article. The reason so few cleared early is that second articles are slower than intuition suggests: the median gap between the first and second article on one story is 3.5 hours, and the 90th percentile is 43.7 hours. Most of the duplicate coverage a longer hold would catch arrives well after any hold worth having, which is the observation that eventually made the hold itself look unnecessary.

Where the minutes go now

A cron cycle runs every 10 minutes and does several things in sequence: queue any source that is due, drain the fetch queue, run the pipeline over up to 16 events, check ingestion freshness, run the merge pass, then reclassify events that have grown.

The --limit 16 per cycle exists because embeddings run against Voyage's free tier and the rate limit is real. A burst of news drains across several cycles rather than one, so the cycle interval and classification throughput are now the dominant terms in your latency, not a gate. Each event is committed as it publishes, so a cycle that runs out of time keeps the events it got through rather than discarding the batch.

Each event gets embedded, clustered against the last 4 days of events at a 0.85 similarity threshold with an entity-overlap check, and classified in a single model call that cascades across free models before falling back to a paid one.

If you need it faster still

Use webhooks rather than polling. They fire on publication, so you learn about an event as soon as it clears the pipeline rather than on your next poll interval. With the hold gone, your own polling interval is now the largest remaining delay for most consumers.

Do not build a trading signal on this. Removing the hold cut hours, but the floor is still source polling, which runs between every 15 minutes and once a day per source. The minutes measured above are from the moment Scoopkit first sees an article, not from the moment the news happened. Anything where seconds matter needs a different kind of product, and saying so plainly is more useful than burying it.

FAQ

Is there any delay attached to a plan?

No. There never has been. Every tier sees an event the moment it publishes. The hold that used to exist applied equally to everyone and is now gone entirely.

Why keep PUBLISH_MIN_ARTICLES at 2 if the age gate always passes?

Because it still determines which branch publishes the event, and that shows up in the logs. It also means the setting is there if the hold ever needs to come back.

Does an event change after it publishes?

Yes, and now by design rather than as an exception. It can be reclassified when it gains articles, and it can be merged into another event. Both are announced by webhook and both are recorded.

Will the hold come back?

Only if the correction path turns out to be worse than waiting, which would show up as a high rate of category churn on events that never gain articles. That is worth measuring and is not measured yet.

How would I know if ingestion stopped?

Every cron cycle checks the age of the newest published event and pages if it exceeds 150 minutes. You can see the same signal in /health, which reports ingestion and last_event_age_s.