Skip to content
Status Active
Contributors BrianKhuu

Democracy Landscape Maintenance

Internal project covering the technical infrastructure that keeps the Democracy Landscape directory healthy — activity tracking, data quality checks, and automated maintenance scripts.

Scope

  • Activity tracking — automated scripts probe org websites for RSS feeds, sitemaps, and structured news content to keep "last active" dates current
  • Data quality — URL reachability checks, Wikipedia link verification, frontmatter consistency
  • Data exports — CSV, JSON, GeoJSON, and KML snapshots generated at build time from org frontmatter
  • Bot infrastructure — unified crawler identity, robots.txt compliance via a shared util/robots_check.py implementation (see the bot page for the one deliberate exception), public bot page

Tooling

Utility scripts live under util/ and are documented in three places depending on kind: util/SOUL.md for the core org-maintenance workflow (staleness, structural lint, stamp.py/check_orgs.py/etc.), CLAUDE.md's "Utility scripts" section for citation/event-sourcing verification tooling, and the bot page for exactly which scripts run automatically vs. by hand, at what frequency, and their robots.txt behavior — that table isn't repeated here to avoid the two drifting apart.

Recurring tasks

See MAINTENANCE.md at the repo root for the full step-by-step maintenance pass. In short:

python util/check_rss.py --update-activity   # update activity dates from feeds
python util/scrape_news.py                   # update activity dates from news pages
python util/check_urls.py                    # verify org URLs still resolve

check_rss.py and scrape_news.py also run automatically every week via GitHub Actions (see the bot page); check_urls.py is a manual, human-run step, done roughly quarterly.

Future ideas

  • Aggregated iCal feed from orgs that publish calendar feeds
  • Outreach to orgs to encourage RSS/iCal feed adoption

See also