Agent-user test harness

harness/ holds synthetic AI agents that use the exchange the way real customers would, end to end: query, judge the offers, accept, receive the signed entitlement, retrieve the content and report SPUR telemetry. They then write a feedback report on ranking, pricing, entitlements and API ergonomics. CI runs it on every push.

npm run harness                                   # in-process exchange, deterministic, no model calls
npm run harness -- http://127.0.0.1:8080          # a server started with npm start
ADMIN_TOKEN=... npm run harness -- https://clear-exchange.fly.dev   # the deployed app
npm run harness -- --personas coding-agent,adversary --quiet

Each run writes reports/<run_id>.md (for people) and reports/<run_id>.json (the full result, for diffing runs). Generated reports are not committed. Exit code 0 when every contract check passes, 1 when any check fails, 2 when the harness could not run (for example, the target is unreachable).

Personas#

Buyers run first and create the market. The adversary runs next, then the publishers, who read what happened.

The report#

A summary, then suggested improvements ranked by impact (severity, then how many personas hit the issue, then how often), then sections for ranking quality, price constraint behaviour, entitlements and licensing, telemetry, error handling, API ergonomics and the publisher view. Each section has its contract checks and findings, with evidence and a suggestion. Ranking is judged against hand-labelled relevant content per task (from the seeded catalogue and the demo verticals): top-1 correctness, precision and nDCG. Price is treated as a constraint: no offer may exceed max_price, and order must follow relevance. Every task also checks the relevance contract: result matches whether there are offers (with a no_match message when there are none), every offer clears the floor and near-top band, every offer reports freshness, off-catalogue queries are answered "no good match" (below_floor), and tasks that ask for a period ("this month", "latest") get a recency block and a top offer inside that period. Session transcripts and an appendix of API calls (latency and status by route) close the report.

Checks and findings are different. A check is a contract the exchange already promises (a tampered token gets 401, the ledger balances); any failed check fails the run. A finding is product feedback (for example, a missing API description) and never fails the run.

Determinism (CI mode)#

Without HARNESS_LLM, decisions come from fixed rules (RuleBrain), so there are no model calls. Without a target URL, the harness starts its own exchange, which differs from npm start in three ways:

• It runs on a clock anchored at 2026-10-27T12:00:00Z plus real elapsed time. Tasks use date-relative constraints and rate cards have age rules, so the anchor keeps results identical on any day. Override it with --clock.
• The adversary tests expiry by moving that clock past the entitlement's exp, so CI does not wait out the TTL.
• Records stay in memory, and admin views are open to the harness.

npm test includes a full harness run and checks that two runs produce identical checks, findings and judgements.

Against a deployed URL#

Pass the URL and set ADMIN_TOKEN (or --admin-token) so the ledger, telemetry and publisher checks can read /v1/admin/*; without it those checks are reported as skipped, not failed. Note:

• The clock is the server's own, so date-relative results drift as the catalogue ages.
• The harness cannot move a remote clock. If the entitlement TTL is longer than --max-wait, which is usual, the expiry probe is skipped and reported as a finding.
• Every purchase creates real ledger entries and telemetry on that deployment. No money moves, but the records stay until the next restart. Agent ids are prefixed harness- so they can be filtered out.

Optional LLM mode#

For feedback at scale from many different customers, use the customer survey (npm run survey), which shares this provider seam. In the harness itself, an LLM can role-play the buyers' offer decisions and add a qualitative review section:

HARNESS_LLM=anthropic ANTHROPIC_API_KEY=... HARNESS_LLM_MODEL=<model id> npm run harness
HARNESS_LLM_ROLES=review ...    # only the qualitative review; decisions stay rule-based

HARNESS_LLM_EFFORT (low to max) sets the model's effort, which steers how much it thinks.

HARNESS_LLM_ROLES is a comma list of decide and review (default both). ANTHROPIC_BASE_URL overrides the API endpoint. The model is held to the persona's hard limits: it can accept only offers that exist, within the run budget and per-query cap. A failed or malformed reply falls back to the rules, and the report says so. The qualitative section is labelled non-deterministic. Providers sit behind LlmProvider in harness/brain.ts; another vendor is one entry in PROVIDERS. Never commit keys; pass them through the environment.

Code#

harness/
  cli.ts          entry point, arguments, in-process exchange on the anchored clock
  runner.ts       runs personas in order, collects checks, findings, calls and market data
  personas/       buyer.ts (shared buyer flow), buyers.ts, verticals.ts, adversary.ts, publishers.ts
  brain.ts        RuleBrain (default), LlmBrain and LlmReviewer, provider registry
  survey/         the customer survey (customer-survey.md)
  judge.ts        precision, nDCG, ranking order, money helpers (pure)
  pricing.ts      publisher repricing policy (pure)
  telemetry.ts    licence-respecting SPUR event batches
  jws.ts          offline entitlement verification against the JWKS
  findings.ts     check and finding recorder, impact ranking
  report.ts       markdown report
  client.ts, context.ts, model.ts   HTTP client, run context, shared types

To add a buyer, define it with buyerPersona({...}) in personas/buyers.ts and add it to BUYERS (vertical buyers live in personas/verticals.ts, VERTICAL_BUYERS); publishers use publisherPersona and PUBLISHERS in personas/publishers.ts. runner.ts runs ALL_PERSONAS (buyers, then the adversary, then publishers). Keep new personas deterministic under RuleBrain so npm test stays stable.