ScoopkitBot is the crawler behind Scoopkit, an API that collapses duplicate AI-industry headlines into single structured events. If you found this page from a user-agent string in your access logs, this is what it was doing.
It identifies itself as:
ScoopkitBot/0.1 (+https://scoopkit.dev/bot)What it fetches
RSS and Atom feeds, plus one XML sitemap. It does not crawl your site, follow links, execute JavaScript, or fetch images, stylesheets, or scripts. It reads the feed URL you already publish for feed readers, and nothing else.
From each feed entry it keeps the title, the link, the publication date, and the summary your feed itself provides. It does not scrape article bodies.
How often
Between every 15 minutes and once a day per source, depending on how often that source actually publishes. Most sources are on 30 or 60 minutes. Requests are conditional where the server supports it, so a feed that has not changed returns 304 and costs you almost nothing.
One request per poll. No parallel fetching of the same host, no retry storms.
How to block it
Add this to your robots.txt:
User-agent: ScoopkitBot
Disallow: /Or email keys@scoopkit.dev and the source gets removed from the list. A person reads that address, and removal does not require a form or an account.
Why you might not want to
Scoopkit links back to the original article as the event's source_url, and does not republish full text. Coverage flows outward rather than being captured.
Questions, or something the bot did that it should not have: get in touch.