Architecting Public Intelligence: Open Data Feeds for Accountability and Verification

Executive finding

Public intelligence is most credible when it makes institutional action easier to inspect without making readers easier to track. That requirement changes the technical design. A civil-liberties publication should not send a reader’s browser directly to every agency API, advocacy feed, analytics service, or document host. It should acquire public material on the server, preserve the source trail, normalize only what is needed, and publish a local snapshot that can be reviewed independently.

International Intelligence therefore treats a “feed” as an editorial intake channel rather than a live surveillance surface. The public site reads static JSON and report files stored on the same host. A scheduled PHP task performs bounded requests to documented public interfaces, records when the request occurred, and retains the last successful snapshot if a source fails. No database, reader account, advertising identifier, analytics beacon, or third-party browser request is required.

1. The asymmetry this system addresses

Governments, law-enforcement bodies, intelligence institutions, contractors, and data brokers often possess extensive information about the public while disclosing their own systems through fragmented notices, budget lines, procurement records, court filings, and delayed public-records releases. The accountability problem is rarely a total absence of records. More often, the relevant facts are divided among systems that use different identifiers, publication rhythms, formats, and legal vocabularies.

A public-intelligence desk can reduce that asymmetry by linking four kinds of evidence: the authority that permitted a system, the money that purchased it, the records describing how it operates, and the reporting that reveals its effects. The resulting product is not an operational intelligence service and not a target database. It is a documented map of public power, designed for journalism, research, oversight, petition, and lawful public-records work.

2. Privacy-preserving ingestion

The deployment model separates acquisition from reading. The acquisition layer runs only on the hosting server or through a trusted command-line scheduler. It contacts approved sources with a named user agent, a short timeout, a response-size ceiling, and small result sets. The publication layer reads only local files. JavaScript may filter or sort those files in the browser, but it does not make remote source requests.

The design deliberately prefers stale-but-known data over a broken page or an unreviewed partial response. Each refresh builds a temporary file and replaces the public cache only after valid JSON has been produced. A lock prevents overlapping jobs. Per-source status records distinguish “disabled,” “not yet attempted,” “last success,” and “latest attempt failed.”

3. A common record schema

Different public systems should not be flattened so aggressively that their meaning disappears. The common schema is therefore modest: source name, source key, source identifier, title, summary, canonical URL, publication date, retrieval date, original language, country, topics, evidence tier, confidence label, and a small set of optional fields such as agency, recipient, amount, author, or document type.

This schema supports discovery without pretending that a Federal Register notice, a contract award, and a civil-liberties article are equivalent. The interface visually preserves the evidence tier. A reader can search across all items while still seeing whether an item is an official record, reputable reporting, or project-authored analysis.

4. Official authority and rulemaking

Federal Register

The Federal Register is the first supported official-record connector. It provides rules, proposed rules, notices, presidential documents, and public-inspection material. Keyword monitors can surface developments involving artificial intelligence, cybersecurity, privacy, surveillance, data retention, information sharing, or national-security authorities. The connector stores the document number and canonical HTML page and displays the named agencies.

A search result establishes that a document was published through the Federal Register system; it does not by itself prove that a program was deployed, that an agency interpretation will survive litigation, or that the informational web version is the controlling legal edition. Material legal reliance should follow the source’s link to the official edition and inspect the full docket where appropriate.

GovInfo

GovInfo is a complementary source for authenticated government publications, congressional material, laws, reports, and historical packages. Its connector is included in the source registry but disabled until an operator supplies an API key and defines narrow collections. A production implementation should use modified-since retrieval, preserve package and granule identifiers, honor retry instructions, and avoid implying that a government agency endorses this publication.

5. Following the money

USAspending.gov

Procurement records often expose a system more clearly than promotional claims or source code. A contract can identify the buyer, recipient, amount, scope, dates, amendments, and funding path. USAspending.gov is used to monitor bounded combinations of terms such as surveillance, facial recognition, artificial intelligence, and data analytics across a limited date window.

An award record is strong evidence that an award was reported. It is not proof that every described capability was delivered, that the technology was used against a particular person, or that the deployment complied with law and policy. Those questions require the underlying solicitation, statement of work, invoices, privacy impact assessment, test results, acceptance records, access logs, and deletion rules.

The most useful workflow is combinatorial: a spending record identifies the award and agency; a public-records request then seeks the solicitation, bid, contract, amendments, deliverables, validation records, data-sharing terms, security reviews, and termination or deletion provisions. That sequence converts a broad concern into a precise record trail.

6. Public-records infrastructure

FOIA.gov

FOIA.gov supplies official information about federal agency components and annual-report data. The connector is disabled by default because it requires an operator-supplied key. When enabled, it should collect only bounded component metadata or selected annual-report material rather than mirroring the service. Useful measures include backlogs, processing times, disposition categories, and the structure of the office responsible for a request.

MuckRock and DocumentCloud as research services

MuckRock and DocumentCloud are valuable discovery and document-analysis ecosystems. Public request logs can reveal prior wording, agency response patterns, exemptions, and released files. DocumentCloud can make scanned documents searchable and preserve public document pages. Their terms, access controls, rate limits, workspace permissions, and copyright constraints must be checked before any automated use. International Intelligence does not ship an enabled connector for either service in this release; researchers may use their public interfaces manually and add source notes to commissioned reports.

A technically informed request can seek not only the final records but also the processing trail: search instructions, custodians, export logs, redaction logs, deduplication reports, upload summaries, and records describing the software used to process the request. Those records may help evaluate whether an agency performed an adequate search.

7. Narrative intelligence without behavioral tracking

Official records often arrive after a policy debate has begun. Narrative feeds help identify what deserves deeper records work. The initial release supports official RSS or Atom feeds from the Electronic Frontier Foundation and the Electronic Privacy Information Center. Those items are classified as reputable reporting or expert analysis rather than official records. Their value lies in issue discovery, legal context, and links to stronger underlying documents.

The server retains the title, summary, publication date, author when available, source language, and canonical link. It strips active markup and does not embed the source page. A Spanish interface may provide a project-written orientation for selected internal items, but external article titles and summaries remain in their source language unless a human-reviewed translation is added.

8. Data-broker and AI accountability sources

State data-broker registries, regulatory enforcement pages, privacy-impact assessments, algorithm inventories, audit reports, and AI incident repositories can reveal different parts of the same system. A registry may establish a company’s legal registration or self-reported practices. An enforcement record may establish an allegation, settlement, or order. An incident database can identify patterns and leads, but its narrative entries should be followed to the underlying reporting, filings, audits, or records.

Future connectors should favor static bulk downloads or narrow server-side deltas, especially where registries publish CSV or JSON snapshots. Cross-jurisdiction comparisons can be analytically powerful, but absence from one list is not automatically proof of noncompliance. Definitions, exemptions, effective dates, corporate names, and business models differ. Any “missing registration” finding should be labeled as an inference until the applicable legal duty and entity identity are confirmed.

Machine learning may assist internally with sorting, translation drafts, duplicate detection, topic suggestions, and comparisons. It must not become the citing authority. Human review owns the final source trail and must be able to explain which record supports each material statement.

9. Evidence integration

The desk uses an eight-level Evidence Ladder. Primary sources and official records occupy the strongest positions. Declassified records and court filings remain strong but carry visible limitations: redactions, date, scope, adversarial posture, and procedural status. Reputable reporting and expert analysis provide essential context. Firsthand accounts are important but require care, consent, and corroboration. Unverified claims may guide internal research but are not published as fact.

Confidence is recorded separately from evidence type. An official record can still be stale, incomplete, or disputed. Expert analysis can be corroborated by several independent records. Each public finding should answer three questions: what does the evidence establish, what does it not establish, and what new record would change the conclusion?

10. Deployment blueprint for a flat PHP host

The reference package is designed for PHP 8.1 or later. It uses no database and no package manager at runtime. The web root contains PHP pages, local CSS and JavaScript, report fragments, a seed feed, and writable cache and log directories. A command-line cron task runs php cron/refresh.php. Hosts without command-line cron may enable a secret-token web invocation in a local configuration file, but command-line execution is safer because the token never appears in a request URL or access log.

Secrets are kept in includes/config.local.php, which is excluded from the package by default and denied direct web access. The public API endpoints expose only the already-normalized local feed and non-sensitive source health fields. The Content Security Policy permits resources only from the same origin and blocks frames, plugins, external fonts, and browser connections to third parties.

The seed dataset makes the interface useful immediately. It contains project-authored orientations to the source system and links to the two foundational reports. After a successful refresh, official and narrative items are merged with those seeds, deduplicated, sorted, and capped to a predictable size.

11. Editorial operating standard

1. Use documented public interfaces and comply with published limits and terms.
2. Preserve source identity, dates, canonical links, and transformations.
3. Classify the underlying document, not merely the page that mentioned it.
4. Minimize bystander data before publication.
5. Separate fact, inference, dispute, and uncertainty.
6. Never use public-interest language to justify unauthorized access, evasion, impersonation, doxxing, or harassment.
7. Maintain a visible correction path and a right of reply for materially named parties.

Selected source interfaces

Primary interfaces and documentation should be rechecked before changing a connector: Federal Register API; USAspending API; FOIA.gov developer resources; GovInfo API; EFF feeds; EPIC feed.