DOD Bot

Designing Open Democracy runs automated scripts to monitor the organisations listed in its Democracy Landscape. This page describes what those scripts do, how often they run, and how to opt out.

What the bot does

The bot performs read-only checks against publicly accessible URLs. It does not create accounts, submit forms, or interact with authenticated content.

Runs automatically, weekly (GitHub Actions, Fridays 03:00 UTC):

Script	Purpose
`check_rss.py`	Probes for RSS/Atom feeds and sitemaps; records the latest post date to show activity status
`scrape_news.py`	Reads news/blog index pages for orgs that lack a machine-readable feed; extracts dates from structured markup only (JSON-LD, OpenGraph, `<time>` tags)
`check_fragments.py`	Re-fetches pages cited as evidence for an org's timeline events and prose citations, to confirm the quoted text is still there
`check_event_urls.py`	Checks that URLs cited as event evidence are still live (not 404/redirected)

Run manually by a human maintainer, roughly quarterly, not on a fixed schedule:

Script	Purpose
`check_urls.py`	Verifies that `website:` URLs in the landscape are still reachable
`check_wikipedia.py`	Checks that Wikipedia links in org pages resolve correctly (queries Wikipedia's own REST API, not third-party sites)
`check_contact.py` / `check_contact_deep.py`	Looks for publicly published contact info (email/phone/form) on an org's own site
`check_logo.py`	Looks for a usable logo image on an org's own site
`manual_check_worklist.py`	Generates a checklist of citations the automated checks couldn't resolve, for a human to verify by hand in a real browser
`import_manual_dump.py`	Imports a human-saved browser snapshot for a citation URL that's blocked to every automated path (see CLAUDE.md's "manual-dump" runbook)

Frequency

The automated pass above runs weekly. The manually-run scripts run whenever a human maintainer does a maintenance pass — in practice more like quarterly. None of this is a high-frequency or continuous crawl.

robots.txt

Every script listed above checks robots.txt before fetching a page and skips it if disallowed — util/robots_check.py is the single shared implementation all of them use, so this can't quietly drift out of sync script by script. The one exception is check_wikipedia.py, which only ever queries Wikipedia's own REST API rather than a third-party site, and Wikipedia's API is designed for exactly this kind of programmatic access (same reasoning as check_fragments.py not gating its own Wikipedia lookups). An unreachable robots.txt is treated as "allow everything," not "block everything" — a transient failure to fetch robots.txt shouldn't silently stop a legitimate check.

User-Agent string

All requests identify as:

DOD-Bot/1.0 (+https://www.designingopendemocracy.com/bot/)

Making your site bot-friendly

The bot works best when your site publishes machine-readable signals. In priority order:

1. Publish an RSS or Atom feed This is the most reliable signal. The bot probes 23 common feed paths automatically — no configuration needed on your end if your CMS already generates one. WordPress, Ghost, Substack, and most modern platforms do this by default.

2. Add structured markup to your news/blog pages If you don't have a feed, the bot falls back to scraping your news page. It reads dates only from machine-readable markup — not from visible text. Any of these work:

JSON-LD — "datePublished" or "dateModified" in a <script type="application/ld+json"> block
OpenGraph — <meta property="article:published_time"> or article:modified_time
HTML time element — <time datetime="2026-05-01"> on article listings

3. Publish a sitemap A sitemap.xml with <lastmod> dates is used as a last-resort activity signal when no feed or structured news page is available.

4. Explicitly allow the bot in robots.txt If your site uses aggressive bot-blocking, add an explicit allow:

User-agent: DOD-Bot
Allow: /

Opting out

If you would prefer your site not be checked, add the following to your robots.txt:

User-agent: DOD-Bot
Disallow: /

Every script above honors this (see robots.txt section above for the one exception, which never queries your site at all). Alternatively, contact us and we will remove your organisation from automated checks.

Citation data

Every event and prose-footnote quote check_fragments.py verifies (see above) is exported as a machine-readable CSL-JSON file: /data/citations.json. Each entry carries the standard bibliographic fields (id, type, URL, title, archive/archive_location for a Wayback Machine snapshot when one exists) plus a DOD extension evidence array recording, per quote, its last verification result (MATCH/MISMATCH), when it was last checked, and whether the check was automated or a human-confirmed browser snapshot. This is the same evidence check_fragments.py itself checks — the file is a read-only projection of it, regenerated fresh on every build, so it's never stale relative to what the bot last confirmed.

Individual citations are also directly discoverable from the pages that cite them: a quoted org-timeline event or footnote embeds a COinS <span class="Z3988"> next to its citation link — the same mechanism Wikipedia uses, and the one Zotero, EndNote, and RefWorks already scan any webpage for — so a citation can be imported straight into a reference manager without knowing citations.json exists at all.

Source code

The scripts are open source: util/

Our own automation

DOD also runs a periodic maintenance pass over its own Democracy Landscape, authored by Claude Code and reviewed by a human before merging. The log of these runs — landscape stats, orgs verified, structural fixes — is kept separate from the main blog at /heartbeat, with its own RSS feed, so it doesn't mix with human-written posts.