warrant-conformance/1

A verifier that has only ever agreed with itself has not been verified.

This is an invitation to write the fourth independent implementation of the Warrant record format, and a contract small enough that doing so is a finite piece of work with a result at the end of it. no external implementer yet

The question the contract exists to answer is not "does your code work". It is the harder one: when your verifier says a record is valid, is it agreeing with the specification, or with the implementation it was written next to? Three implementations already agree here — and all three were written in this repository, from one reading of one document, by one author. Agreement among them is evidence about the document's clarity to us. It is not evidence about the document.

So the pack does the one thing that would be worth something: it drives your program against expectations copied from the specification, and it never runs ours.

What you implement

One process. It reads a single JSON object from stdin, writes a single JSON object to stdout, and exits 0 whenever it produced an answer — including an answer of "no". That last rule is the whole ergonomics of the thing: "this signature does not verify" and "I crashed before deciding" are different facts, and a contract that encodes both as a nonzero exit lets a broken program impersonate a strict one.

// in
{"warrant_conformance":"1", "id":"verify-sig/reject-04", "class":"verify-sig",
 "input":{"warrant_id":"…","key":"…","sig":"…"}}

// out
{"warrant_conformance":"1", "id":"verify-sig/reject-04", "output":{"valid":false}}

// or, the only way to decline a vector — scored UNRUN, never a pass
{"warrant_conformance":"1", "id":"verify-sig/reject-04",
 "unsupported":"no Ed25519 verification in this build"}

The expected answer is never sent, so there is nothing to echo back and passing requires computing. Each invocation is independent; nothing is carried between them. Your program may be a script, a compiled binary, or a shell wrapper. The full contract is one page: CONTRACT.md, and it ends with a working nine-line Python candidate that implements nothing, scores every vector UNRUN, and reports exactly that.

The nine classes

A class is one question the format can be asked. capabilities is mandatory — declining it is a contract violation, because a candidate that will not say what it claims cannot be tested against its claim. The other eight carry the 139 vectors:

62 of the 139 vectors are MUST-REJECT. That ratio is deliberate. An implementation whose validate() returns true unconditionally passes every positive vector in the pack, so the negatives get their own headline in the report rather than being averaged into a score: PERMISSIVE IMPLEMENTATION: 62 of 62 MUST-REJECT vectors were ACCEPTED. If no negative vector ran at all, the report says that, in place of a score.

Two grades, and claiming the smaller one is a complete result rather than a shortfall. base is the seven classes above the settlement rows; settlement adds ski-run and the fail-closed trust rules. You declare which you claim, the runner tests exactly that, and reports the grade achieved — which may be lower. One FAIL, ERROR or UNRUN withholds it.

Running it

curl -LO https://github.com/s0fractal/warrant/releases/latest/download/warrant-conformance-1.2.0.tar.gz
tar xzf warrant-conformance-1.2.0.tar.gz
cd warrant-conformance-1.2.0
python3 run.py --candidate "./my-verifier probe"

Python 3 standard library only. No install, no configuration, no network after the download, and no clone of this repository. Measured end to end from an empty directory with no repository present: 0.6 s to download and extract (32 KB), then ~7 s for a full settlement run against a Python candidate.

Those seconds are your program's start-up cost, paid once per vector. The runner spawns your candidate 139 times rather than holding a session open, because a stateless contract is one you can satisfy with a shell script — that trade is worth stating rather than hiding behind a faster-looking number. A compiled candidate is substantially quicker.

Before trusting a green run, break your own program on purpose:

python3 run.py --candidate "./my-verifier probe" --self-check

That wraps your candidate in a proxy that corrupts specific answers — every validity answer becomes yes; the signed message is built the superseded pre-0.6.0 way; whole classes are quietly declined while a grade is still claimed; the process exits nonzero instead of answering — and asserts that the runner catches each one. Any MISSED line means the runner is broken and its green runs mean nothing. A gate nobody has watched fail is not yet a gate; this project has shipped that mistake more than once, which is why the negative control is in the box rather than in our CI. Budget about 48 seconds for it — it runs the whole suite five times, once unmutated as a baseline and then once per mutation.

On your first afternoon most of those mutations will report INAPPLICABLE rather than DETECTED, and that is the correct answer: a mutation corrupts an answer, and a class you have not written yet gives it nothing to corrupt. The summary says how many of the four actually applied, so a partial proof is never dressed up as a complete one — and if none of them could be applied, the run is INCONCLUSIVE and exits nonzero rather than green.

The vectors themselves are checkable. Every file is listed in MANIFEST.sha256, and python3 run.py --verify-pack prints the SHA-256 of that manifest — one hex string pinning the whole pack, published in SPEC §8.6 at the revision you are implementing. Because the spec is versioned and tagged, the comparison holds wherever the tarball came from: a mirror, a fork, a colleague's copy. The runner refuses to produce a result from a pack whose manifest does not match.

What implementing it gets you

• An answer to the question at the top of this page, for your own code. Not "my tests pass" but "my reading of the format and theirs produce the same bytes, including on the 62 inputs that must be refused."
• A grade you can state, with the evidence for it re-runnable by anyone. The report names every UNRUN vector; nothing is rounded up.
• The name. TRADEMARK.md conditions calling an implementation "Warrant" on this pack, and until the pack existed that condition referred to a test nobody outside could perform.
• Standing to say the spec is wrong. A vector you disagree with is worth an issue. One drafted vector was withdrawn during development for exactly that reason: parse/top-level-array asserted that a non-object top level must be rejected by the parse layer, and the three implementations turned out to enforce it at three different layers while agreeing on the observable outcome. The layer is not the spec's business.

And plainly, what it gets us, since you should know why you are being asked: a reading of the specification produced by someone who was not in the room when it was written. That is the only kind of evidence that the document, rather than the habit, is what carries the format.