The first time I watched a kabin-nama being drafted, I was nine years old and bored half to death in a sitting room above Tipu Sultan Road in Wari. My uncle's family lawyer, Bashir saheb, sat at a low table with the bride's father, a fountain pen with a faintly bent nib, and a stack of ivory paper with cinnabar rules down the left margin. The thing took an entire afternoon. Nothing got signed until every clause had been read aloud, agreed to in plain Bangla, and witnessed by three men on a bench by the window drinking lebu cha. The deal lived in the whole stack of paper, every clause of it, and nobody in that room would have accepted a single line pulled from the bottom of the page as a summary of what had actually been agreed.
I have been thinking about Bashir saheb a lot since a team I work with moved their support flow over to an agent. The flow used to be a Python file with about ninety unit tests against it, each one with a known input and a known output. You sent in a refund request shaped a certain way, you got back a JSON shaped a certain way, and the test either matched the bytes or it didn't. CI ran in two minutes, the lead engineer signed off the release at the end of every sprint, and the worst bug anyone had filed all quarter was a slip in the date format from March.
The new flow has no known outputs.
The agent answers the same refund request differently on Tuesday than on Wednesday, because the model has a different mood and the retrieved context is slightly different and the customer phrased the message less politely. The byte string the old suite was matching against is gone; what survives is the shape of the answer underneath all that paraphrase.
Operative Clause · দফা
A unit test asks whether the answer matches a fixed string, which is a question that stops mattering the moment the model starts paraphrasing on Tuesday. The question that survives a model swap on a Saturday afternoon is whether the answer obeys the clauses Bashir saheb would have read aloud before letting anyone sign.
initialled by counsel
The discipline of asserting on shape rather than bytes is not new. Pact, the contract testing library built around what the consumer needs that came out of the microservices wave around 2015, treats the agreement between two services as a written document. The consumer declares what fields it actually reads from the producer's response, the producer agrees to keep those fields in that shape, and a third file (the pact) lives in version control as the binding artifact. The Python community grew up the same year around Hypothesis, David MacIver's testing library built around properties, in which the writer declares an invariant ("reversing a list twice returns the original list") and the library hammers it with a few hundred generated inputs. What both approaches assert against are the clauses the writer put into the file, never the bytes the producer or the property check happen to emit on any given run.
The two ideas together are roughly the shape of what an agent suite needs to look like in 2026. Bashir saheb's kabin-nama is the metaphor I keep returning to, because it is exactly what we are drafting when we write contract tests for a support agent. There are five clauses on the page, three witnesses on the bench, and a mahogany file box on the shelf for the folded document at the end of the afternoon.
Schedule B · attestations
three boxes · one seal
Clause 4 · the agreed scope
The agent shall answer in a permitted shape, calling only the tools that have been disclosed, on topics within the contracted domain, and shall reveal no particular about any customer that has not first been freely volunteered by the customer themselves.
Witness I
Attests the SHAPE
the answer must cite at least one source from the indexed corpus
Mostofa bhai
assert shape.must_cite_sourceWitness II
Attests the TOOLS
the order-lookup tool was called before any refund figure was stated
Rashida apa
assert tool_use.lookup_before_refundWitness III
Attests the SILENCE
no card number, no full address, no policy outside the contracted scope
Anwar saheb
assert privacy.no_pii_in_responseThe deal isn't done until every clause is read aloud and a witness signs. The build server, in a Wari office above the kabab place, is doing the same thing for the support agent.
The clauses look mundane until you sit with what each obliges. The first is about the permitted shape of the answer, which for a support agent usually means a JSON envelope with a refund_amount, a policy_citation, and a customer_facing_message, and not three paragraphs of free prose attempting to be helpful. After that comes a clause on tool use. The agent must have called at least one of the disclosed tools (order lookup, policy, eligibility) before quoting any monetary figure, and any number that ends up in the answer must appear verbatim in those tool outputs. The fourth clause keeps the agent inside the contracted scope, which is to say not promising a discount on a different product line out of misplaced generosity. The last forbids reciting back any private particular the customer themselves did not include in the prompt.
Each of these is a property in the Hypothesis sense, an assertion on what must hold regardless of what bytes the model produced this morning. A unit test would say "for input X, output Y." The contract test says "for any input, the answer must obey Clauses 1 through 5, witnessed by the suite." The model is allowed to phrase its Tuesday answer differently from the Wednesday one, so long as both days pass the same five witnesses.
The libraries to write these have arrived faster than I had expected. promptfoo lets you declare a set of assertions in a YAML file (contains-json, latency, cost, llm-rubric, javascript) and run them against a grid of prompts. DeepEval borrows the pytest layout and adds metrics for faithfulness, toxicity, and traces of tool use. None of them is yet what Pact was for service contracts; the field is too young. But the shape is clearly visible, and the engineering teams I know in Mohakhali are starting to ship versions of it.
Simon Willison has been writing usefully about this for a couple of years, with the heuristic that the only LLM tests worth running are the ones whose failure would change what you ship next. He calls them evals; the community of 2026 calls them contract tests; Bashir saheb in his sitting room would have called them shorto, the conditions of a deal. Whatever the name, the suite has to encode what the team meant by "a good answer," in a form the build server can keep verifying when the model gets swapped on a Tuesday morning. Without that, you are running against a fixed set of golden bytes that go stale the moment the model is updated.
The trap I keep watching teams fall into is the one Bashir saheb would have refused on principle. They write a single assertion that reads "the answer is helpful" and call the suite done. Helpful is a vibe; it cannot be witnessed by anyone other than the writer at the moment of writing. The marriage contract in Wari has no clause saying the husband shall be kind. It has clauses on mehr (the dower), on maintenance, on the wife's right to divorce by delegation, each a specific obligation written in a sentence a sharif could read aloud without blushing. Six months from now, when the support team is drafting refund letters in slightly the wrong tone, nobody in the room can agree on what "helpful" was meant to mean back in May.
What I have started doing on every agent feature whose ship date is mine to defend is the same thing Bashir saheb did with my uncle. I write the clauses in plain English flavoured with Bangla first, in a small markdown file beside the test suite, and let the agent translate them into promptfoo YAML or DeepEval pytest cases. The clauses live in contracts.md, the suite reads from there, and the file gets reviewed clause by clause the way a daag (a stamped revenue ticket) gets initialled before it goes back into the file. When the model gets swapped on a Saturday six months from now, the suite catches the regression on whichever clause the new model fails. The next engineer reads that clause in the language it was written in, which is the entire point, because the bride's father and the model and the engineer who wrote the original assertion will all have moved on by then. The kabin-nama from that afternoon in Wari is, as far as I know, still folded away unopened in Bashir saheb's office.
That is not a failure of the document, it is the document doing its job, because nobody has needed to go and check what was agreed and everybody has behaved as though the clauses said what they say. Most contract suites sit unread in the same way for months at a time, sometimes for the entire life of a feature, until the engineer who wrote them has moved teams and the model underneath has been swapped twice over. The ones that get read get read on a bad Saturday, by somebody who did not write them, well after the engineer who drafted the clauses has stopped answering pings about that particular feature. That Saturday is the only occasion on which anyone discovers whether the clauses were written well or merely written.