Infra

The Jail Data Initiative runs across three environments. Each handles a distinct workload:

MongoDB Atlas (central database)

All application state lives in the jaildatainitiative database. Both the Lambda API and the IONOS scheduler connect to the same cluster. Collections:

• media — articles and publications
• team — members and contributors (each doc has a type field)
• rosters — roster entries keyed by facility (_id = facid)
• manual_facilities — manually added facilities (_id = facid)
• scrapers — scheduler state per scraper (status, intervals, next_due_at)
• scraper_runs — append-only run log (indexed on {scraper_name, started_at})
• bookings — longitudinal booking records built from roster snapshots (one doc per booking)
• snapshot_log — tracks which S3 stashes have been ingested into bookings (dedup index on {scraper_name, s3_prefix})
• bookings_standard — materialized standardized bookings with unified field names across all platforms (one doc per booking, keyed on booking_id)
• field_maps — per-platform field mapping definitions: raw field names → standard names + value transforms (_id = platform name)

AWS services

• S3 jaildatainitiative-facility-registry — static file hosting only. Stores facilities.json (full map data), facilities-slim.json (lightweight subset for rosters page), raw/... stashes (scraper archives), and *.html site assets. No longer used as a database.
• CloudFront E1VVUL8P0QQLJN — CDN for registry.jaildatainitiative.org. Serves S3 content. Invalidated on deploys and live facility updates.
• Lambda jdi-roster-api — API handler. All CRUD operations hit MongoDB. Still writes to facilities.json on S3 for live facility patches.
• API Gateway jdi-roster-api — HTTP API with CORS. Routes GET/POST/PUT/DELETE to Lambda.
• SES — transactional email for DUA submissions and facility issue reports.
• IAM — Lambda role scoped to facilities.json (S3), CloudFront invalidation, and SES.

IONOS server

• Scheduler daemon (jdi-scheduler.service) — systemd service that continuously picks the most overdue scraper and dispatches it in a Docker container.
• Docker — two images: jdi-scraper-base (Python 3.12-slim for HTTP scrapers) and jdi-scraper-selenium (adds Chromium for browser-based scrapers).
• ThreadPoolExecutor — up to 12 concurrent containers (4 max for Selenium). Each container is resource-limited (--memory 512m/2g, --cpus 1.0).

Chronological flow from external data sources through to the live site and scraper system. Steps are numbered in execution order.

Step 1a/1b: Fetch external datasets (independent, run in parallel)

python entities/fetch_bjs.py downloads the BJS 2019 Census of Jails from S3, geocodes each facility via the Census Bureau API (with BOP and Nominatim fallbacks), computes population coverage metrics, and writes data/bjs_facilities.json (2,936 records).

python entities/fetch_hifld.py queries the DHS HIFLD ArcGIS REST API for county/local/multi-jurisdiction jail facilities, filters out closed and juvenile facilities, and writes data/hifld_facilities.json (3,472 candidates).

Step 2: Match HIFLD to BJS

python entities/match_hifld.py classifies every HIFLD facility as MATCHED (same facility in BJS), HIFLD_NEW (not in BJS), or HIFLD_SUBFACILITY (sub-unit of a BJS facility). Uses composite scoring: 40% name similarity + 30% address + 30% geographic proximity (Haversine with 50km linear decay). Writes data/bjs_hifld_comparison.csv.

Step 3: Build the unified facility registry

python entities/build.py merges three facility sources into one file:

• BJS records from data/bjs_facilities.json
• HIFLD_NEW records from data/bjs_hifld_comparison.csv
• Manual facilities from MongoDB (manual_facilities collection)

Also downloads 2019 Census county populations for capacity_per_100k calculations, loads confirmed roster URLs and excluded facility IDs from MongoDB (rosters collection), filters out excluded facilities, and writes data/facilities.json. Also generates data/facilities-slim.json (~200 KB) containing only facid, facname, state, and county for fast loading on the rosters page.

Step 4: Deploy to production

python web/deploy.py uploads facilities.json, facilities-slim.json, and all *.html files to S3 (with Auth0/API URL substitution), then invalidates the CloudFront cache. The site goes live within seconds.

Step 5: Discover roster URLs (manual, periodic)

python entities/find_rosters.py takes the facility list, groups by jurisdiction, searches DuckDuckGo for roster pages, and scores results 0–100 based on platform detection, keyword presence, domain TLD, and locality match. Excludes 80+ aggregator domains (VineLink, JailBase, etc.). Writes data/roster_candidates.csv.

Step 6: Admin imports and reviews candidates

Candidates are bulk-imported via the admin UI (POST /roster/import) into the MongoDB rosters collection. Admins review each entry, confirming or rejecting roster URLs. Confirmed entries feed back into Step 3 on the next build.

Step 7: Scheduler dispatches scrapers (continuous, IONOS)

The scheduler daemon runs 24/7 as a systemd service. Every 10 seconds it:

• Checks the scrapers collection for overdue scrapers (next_due_at ≤ now)
• Picks the most overdue one (priority queue by next_due_at ASC)
• Launches it in an isolated Docker container with resource limits
• Up to 12 containers run concurrently (4 max for Selenium)

Inside each container, the scraper follows a 4-step pipeline: fetch → validate → spot-check IDs → stash to S3. The container outputs a JSON result line on stdout and exits.

Step 8: Scheduler records results and schedules next run

When a container finishes, the scheduler:

• Appends a run record to scraper_runs (stash status, duration, record count, content hash)
• Updates the scraper’s state in scrapers (consecutive_failures, last_success_at, etc.)
• Triggers bookings ingest asynchronously (Step 9)
• Computes next_due_at using adaptive logic (see Retry & Frequency Tuning below)

Step 9: Bookings ingest (async, after each successful stash)

After a successful stash, the scheduler submits an ingest job to a background thread pool. The ingest process:

• Loads the stashed files from S3 and parses them via the scraper class
• Matches each record against open bookings using BOOKING_ID_TIERS
• Creates new bookings for first appearances, continues existing bookings (diffing all fields for change tracking), and closes bookings for people no longer present
• Logs the snapshot to snapshot_log for deduplication

Ingest runs asynchronously in a 4-thread pool so it does not block scraper dispatch. See Bookings Pipeline below for full details.

Frequency settings

New scrapers default to a daily (24-hour) interval. Admins can change a scraper’s base interval from the Scrapers admin page using a frequency dropdown with these options:

The actual running interval (target_interval_s) starts at the admin-set value and is then adjusted by the auto-tuning system (see below). Auto-tuning can speed up to a 1-hour floor or slow down to a 1-week cap.

Retry and backoff

A scrape is successful if it stashes (fetch + validate + stash to S3). Parse failures are non-critical — raw data is archived and can be re-parsed later.

Change detection and frequency tuning

After each successful stash, the scheduler computes a SHA-256 hash of the fetched content and compares it to the previous run:

• 6 consecutive unchanged runs: doubles target_interval (capped at 1 week / 604,800s)
• Data changed: halves target_interval (floored at 1 hour / 3,600s)

This means scrapers that rarely change gradually slow down to weekly, while frequently changing rosters speed up to hourly. The unchanged counter resets after each adjustment.

Enable/disable: two layers

Either layer blocks the scraper. The scheduler checks both on every pick_next() call.

Every scrape run follows a hard-gated pipeline. If any pre-stash step fails, the run aborts and nothing is archived.

Step 1: Fetch

Downloads raw content from the roster source. For API-based platforms (e.g. Citizen RIMS), this is one or more JSON responses. For HTML-based platforms, this may include list pages and individual detail pages. Fails on HTTP errors, timeouts, or authentication failures.

Step 2: Validate files

Checks that fetched files are structurally viable before archiving:

• At least one file was returned and no files are empty (0 bytes)
• JSON files must be valid JSON, and must not be empty arrays or objects
• HTML files must not contain error markers (403 Forbidden, 503, CAPTCHA pages, etc.) and must exceed a minimum size threshold
• Platform-specific checks (e.g. Citizen RIMS verifies the fetched record count matches the API’s Count endpoint)

Step 3: Spot-check IDs

A lightweight pre-stash check confirming that the raw data contains person and booking identification fields. Every record must satisfy at least one identification tier for both person and booking. This runs on the raw field names before any normalization, ensuring we never archive data that can’t be parsed into identifiable people and bookings downstream.

Step 4: Stash

Uploads all fetched files plus a manifest.json to S3 under a timestamped prefix: raw/{scraper_name}/{YYYY-MM-DD}/{HHmmss}/. The manifest records the source URL, content type, byte size, and fetch timestamp for each file.

Step 5: Parse (post-stash)

Extracts structured records from the raw files. Parse failures don’t lose data — raw files are already in S3 and can be re-parsed later with --parse-only. The parse step also normalizes names (produces _full_name) and tags records with facility/jurisdiction metadata.

Different platforms provide different fields. The scraper uses a tiered identification system, preferring more reliable identifiers:

The bookings pipeline converts periodic roster snapshots into longitudinal booking records with start/end times, field change history, and data quality flags. It runs as Step 9 in the pipeline — automatically after each successful scrape, or manually via the backfill CLI.

Algorithm: ingest_snapshot(scraper_name, s3_prefix)

Booking lifecycle

Change tracking

Every field in latest_record is diffed between snapshots. Scalar fields store old_value / new_value. Array fields (charges, holds, etc.) use set-based comparison that is order-independent — records added and removed elements without false positives from reordering.

Coverage quality flags

Platform date extraction

Each platform declares which raw fields contain admission and release dates. When available, these provide more accurate timing than scrape timestamps alone:

Backfill CLI

Processes stashes in strict chronological order. Skips already-ingested snapshots unless --force is used (which drops all existing bookings for that scraper first).

MongoDB schema: bookings

Indexes

Files

Each scraper is a Python class inheriting from BaseScraper (or a platform subclass). The class hierarchy:

Each scraper class declares:

Docker images

Adding a new scraper

All endpoints go through api/roster_api.py (Lambda) via API Gateway. Admin endpoints require an Auth0 JWT with the Admin role.

Deploy the web app (facilities + HTML)

Runs build.py first, then uploads everything to S3 and invalidates CloudFront.

Deploy the Lambda API

Packages roster_api.py + pymongo into a Lambda zip, updates code and environment variables.

Deploy the scheduler (IONOS)

Builds both Docker images and installs/restarts the systemd service.

Backfill bookings from historical stashes

Processes S3 stashes in chronological order. Use --force to drop existing bookings and rebuild from scratch. Live ingest happens automatically via the scheduler daemon.

Run the data pipeline (Steps 1–3)

Steps 1a and 1b are independent and can run in parallel. Step 2 needs both. Step 3 needs 1a + 2.