Live sources: Wikinews and Wikipedia

Alongside the fictional catalogue, the exchange serves real articles from English Wikinews and Wikipedia. WikimediaAdapter (src/supply/wikimedia.ts) implements the ordinary SupplyAdapter interface, so these articles are matched, priced, licensed, retrieved, cleared and reported exactly like catalogue articles. It uses the public MediaWiki Action API. No account, key or secret is needed.

They are on in the deployed demo (fly.toml) and off by default everywhere else, so tests, local runs and the CI container make no network calls unless you ask:

WIKIMEDIA_SOURCES=all npm start
# then ask "What has the Curiosity rover found on Mars?" in the agent console
node scripts/check-live-sources.ts http://localhost:8080

Why these sources. Their content is openly licensed for any use, including commercial use, provided it is attributed (and, for Wikipedia, adaptations are shared alike). Their API is public, and Wikimedia's API etiquette and User-Agent policy permit this kind of low-volume, cached, identified access (WIKIMEDIA_CONTACT goes in the User-Agent). Nothing is scraped.

Publisher identity and demo prices#

Each wiki is one publisher with its own rate card: pub_wikinews_en (data/rate-cards/wikinews-en.json) and pub_wikipedia_en (wikipedia-en.json). The prices (USD 0.001 to 0.003 per retrieval) are demo prices for the exchange's retrieval, entitlement and telemetry service. Each card says so in extensions["com.clear-exchange"] (demo_pricing: true and a notice), and its id ends in _demo. The UI shows a "Demo price" chip on these offers and a "Demo listing" notice in the publisher view. The Wikimedia Foundation and the wikis' contributors are not parties to these rate cards, have not endorsed them, and receive nothing. The ledger entries are a demo record.

Licences and attribution#

The Wikinews dates come from Wikinews:Copyright. The adapter dates each Wikinews article from its dateline (or, failing that, its first revision).

How the exchange honours these licences:

• Offers carry the item's own content_licence (id, name, URL, share_alike, who to credit), and the terms say citation_required: true with an attribution instruction.
• Retrievals return an attribution block: title, author, source URL, a permanent link to the exact revision served, the page history (the list of authors), the licence, and a changes note. CC licences require saying what changed: the exchange converts wikitext to plain text and omits images, tables, infoboxes and references, and it notes any truncation of very long articles (over 100,000 characters). attribution.text is a ready-made credit line; for CC BY-SA it adds that adaptations must be shared under CC BY-SA 4.0. licence.content_licence on the retrieval repeats the item's licence.
• No added restrictions. CC licences forbid adding terms that restrict the rights they grant. So these rate cards set ai_training: "per_content_licence" instead of "prohibited" and have no snippet cap. Their tiers price the exchange's service by the agent's declared use; they do not narrow the licence. retention_seconds is informational for these sources, and telemetry usage warnings for them are advisory. The agent may always fetch the article from the wiki directly, for free.
• Share-alike. Clear Exchange's plain-text conversion of a Wikipedia article is itself offered under CC BY-SA 4.0 with the attribution above. Agents that show Wikipedia text should display attribution.text, or an equivalent credit with the links.
• Images and other media are never served; they can have different licences.

Resilience#

A slow or failing wiki never breaks a query:

• Each search or retrieval has one timeout (WIKIMEDIA_TIMEOUT_MS, 2.5 s) covering every request it makes.
• Search results and articles are cached (WIKIMEDIA_CACHE_TTL_SECONDS, 15 min, bounded in size), and a stale copy is served if a refresh fails. Identical concurrent requests share one call.
• After three consecutive failures the adapter opens a circuit for 30 s and fails fast.
• The fan-out budget and SUPPLY_TIMEOUT_MS bound every source. A source that misses them or errors is listed in stats.supply_errors, and offers from the other sources are returned as usual.
• A failure at retrieval returns 502 supply_unavailable with nothing recorded in the ledger, so the agent can retry with the same entitlement.
• Health (idle, ok, degraded, down) is shown in GET /v1/supply and on the operator view's supply cards.

Search quality#

The query's keywords go to CirrusSearch as an all-terms search. If that finds nothing, the adapter falls back to an any-term search and keeps only pages covering at least half the terms, so a mostly nonsense question does not match an article that happens to share one real word. The exchange then re-scores every candidate with its own BM25, as it does for all supply.

Tests and fixtures#

Tests never touch the network. test/fixtures/wikimedia/ holds recorded API responses, and test/wikimedia-fixtures.ts replays them through an injected fetch that fails loudly on any unrecorded request. To refresh them, run npm run record:wikimedia, which drives the real adapters with a recording fetch and is the only script that calls the live API. Review the diff, since live content changes.

CI also starts the production image with the sources on and probes them with scripts/check-live-sources.ts. That step is advisory: an outage warns and never blocks a deploy. The post-deploy smoke check runs the same probe against https://clear-exchange.fly.dev.