The question and the oracle
Draft 2020-12 schemas define neutral fixtures, run requests, adapter descriptions, transcripts, and evidence. Every expected transcript lives beside its scenario.
Portable testbed 0.1
Make the same promise. Prove the same behavior.
One neutral suite tests TypeScript, Rust, plugins, desktop services, and future implementations through a process boundary. Products keep their own architecture; users get comparable evidence.
The black-box boundary
The spec asks. Your implementation answers.
Each scenario starts a fresh adapter process. Only stable facts cross the boundary; generated IDs, database keys, and wall-clock timing stay inside.
Neutral scenarioProcess adapterCanonical transcript
01Contract
describerunPASS01
02Interop
describerunPASS11
03Runtime
describerunPASS03
What the testbed separates
The behavior under test
An adapter drives public collection, bridge, provider, runtime, and store boundaries. It reduces observations to stable facts; it does not reimplement the feature.
Authority
Passing proves behavior, not permission. A transcript cannot grant access, register a provider, publish an event, or authorize a side effect.
15 scenarios · 3 rings
The portable inventory
Small scenarios isolate failures. Application scenarios then repeat the critical paths with real product contracts and providers.
Contract
One contract, one implementing type, one record, two consumers.
core.shared-contract-consumersBoth consumers receive the same validated contract view.
Interop
Live sources, consumers, callers, and providers cross an authorization boundary.
interop.event-multicastOne event reaches every compatible consumer.interop.action-single-providerOne eligible provider is selected.interop.action-ambiguousTwo eligible providers stop implicit dispatch.interop.action-explicit-providerAn explicit selector resolves ambiguity.interop.provider-disappearanceAn unloaded provider is unavailable.interop.contract-digest-driftSame version with a different digest is rejected.interop.invalid-payloadSchema-invalid data never reaches a handler.interop.authority-boundaryCompatibility cannot grant authority.interop.request-replayA logical request is not applied twice.interop.application-provider-lifecycleA real app provider registers, validates, runs, and unloads.interop.application-event-multicastA real app source reaches two independent consumers.
Runtime
Durable admission, recovery, and worker fencing are observed through the store.
runtime.crash-recoveryA lost outcome reuses the same attempt and invocation.runtime.competing-workersA stale worker cannot commit after its lease expires.runtime.application-executionA real app catalog produces one exact successful run.
Fresh process per scenario
Run it from any language
The runner launches an executable directly—never through a shell—and validates both sides of every exchange before comparing observations.
Install and verify
npm install --save-dev @callumalpass/mdbase-testbed
npx mdbase-testbed validate
npx mdbase-testbed run \
--adapter ./scripts/testbed-adapter.mjs \
--evidence ./conformance/testbed-evidence.jsonImplement two commands
your-adapter describe
# writes one adapter-description JSON document
your-adapter run
# reads one scenario request from stdin
# writes one observable transcript to stdoutEvidence is precise—and deliberately limited
A passing run can emit a machine-readable evidence document containing the implementation identity, scenario IDs, profile names, and canonical transcript digests. Attach it to a conformance claim to make the tested behavior reproducible.
Supplemental evidence, not a blanket badge
A complete profile claim still runs every normative conformance test for that profile. The testbed adds cross-language behavioral evidence; it does not silently widen the claim.