My grandfather, who taught Bangla at a school in old Dhaka for 31 years and is now retired in a flat off Road 15A in Dhanmondi, has a particular ritual on the Thursday evening the quarterly report card comes home. The card is placed flat on the dining table. The pencil and the chai-bhar sit on either side of it the way an old draughtsman would arrange his rule and his ink. He puts on the reading glasses that normally rest on a small Quran-rehal, and works through the card from Bangla to History, percentage by percentage, comparing each line against the corresponding line on the previous term's card.
My cousin, who is the actual subject of the report card, will at some point try to launch into the story of the school annual day, his recitation, and the certificate from the inter school competition. My grandfather, without breaking his gaze from the card, will raise a hand the size of a small dictionary and say thaak baba, card ta porte de aage. Let the card be read first, son. The speeches can wait, because the report card is the one document that tells you whether the term went anywhere worth talking about.
I have been thinking about that table for the better part of a year, because the PR descriptions on the agentic LLM repo I work near have started reading like my cousin's annual day stories. They are long, performative, and full of the names of new techniques and the diagrams of new flows. They are useful in the way the annual day is useful, and they are also not the document that tells you whether the term actually moved.
What actually serves as the release note, once the team is honest about what the customer will ever feel, is the pass rate delta on the eval suite; the rest of the merge commit is the annual day speech being read aloud over a microphone that nobody is listening to.
The change that I think has happened, somewhere between the spring of 2024 and now, is that the eval suite has taken over the role the changelog used to play in classical software. In an old web app the changelog said "added pagination to the customer list," and you could trust that pagination was now on the list because the deterministic mechanics underneath almost guaranteed it. In a product driven by an LLM, the changelog says "improved retrieval-grounded quote accuracy," and that sentence on its own is, with respect, worth roughly the paper it is printed on. The model either did improve on the eval suite that the team has agreed measures that capability, or it did not, regardless of what the team tried. The pass rate movement is the verifiable artefact. The rest is the speech the cousin wants to give.
Hamel Husain has been making this argument for about two years now, in posts that read less like blog entries and more like field notes from someone watching a small profession learn a hard lesson in public. His repeated point, with which I have come to agree, is that the only honest unit of progress in an LLM application is the eval. Without a suite that runs on every change and reports a delta, you are working on vibes. With one, you are doing engineering of a recognisable kind, even when the system itself remains a black box that occasionally tells your customer the founder of Modhumati Bank was a Mughal nobleman.
progress report · term iv
The Model, Class of v2.3
roll no. 2174 · section eval
issued
11 May 2026
prev. term: 7 Feb 2026
Retrieval-grounded quotes
fewer fabricated citations; the new chunker is working
81%
89%
↑8
Refusal calibration
stopped refusing benign questions about Bangla literature
74%
82%
↑8
Multi-turn handoff
marginal, mostly noise in the harness
68%
71%
↑3
Tool-call argument shape
regressed after the schema rename; needs follow-up
91%
86%
↓5
Tabular answer fidelity
the bigger context window is paying for itself
62%
79%
↑17
Local-context (Dhaka, Bangla)
the new sft pass on Bangla helped more than expected
54%
73%
↑19
term gpa · so far
the line the release notes used to be about
··%
↑0.0 from last term
A pass rate is not a number that floats free, which is the part of its shape that matters; it lives on the line of a particular subject, the way retrieval-grounded quote pass rate and refusal calibration on benign Bangla queries and shape conformance for tool call arguments are three separate rows on the same imagined card. Each line is one capability the team has agreed to grade. The team rewrites the model, the prompts, the retrieval pipeline, or the context window, and the next term arrives with the card reprinted. My grandfather runs the pencil from the previous term down the new card and notes the deltas. There will be twelve subjects up, seven down, and at least one that has dropped catastrophically because somebody quietly changed a tokenizer in the second week. The release note writes itself once the deltas are visible.
There is a counterargument that the academic LLM community has been quietly grinding through for a couple of years, and it is fair to admit it before I go further. Evals can be gamed, evals can drift, evals can be poorly designed in ways that flatter the model rather than measure it. Eugene Yan's writing on this is the cleanest summary I know, and he is honest about the cost of building a suite worth trusting. The suite must be representative, must be held out from training, must contain the awkward cases as well as the canonical ones, and must be graded again by humans on a cadence that catches the model gaming the grader. None of that is free, none of it is glamorous, and all of it is precisely the work that produces a card the dining table can take seriously on a Thursday evening.
But none of those objections actually unseat the argument; they sharpen the version of it that the dining table is willing to hear. A bad school's report card tells you nothing about your child; you would still rather have the bad card and know its limits than be left with only the headmaster's monthly speech. A bad eval suite tells you nothing about your model. A team running a bad eval suite is, at least, a team that has agreed there is a thing to be measured and a discipline to be answerable to. The teams that have skipped this step entirely, and there are more of them in the Dhaka tech ecosystem than I would like, are the ones who release on vibes and answer customer questions by reading their own PR descriptions back to themselves.
The reason this matters for the people writing the PR description, including the ones I work near, is that the description has stopped being the artefact carrying real weight in the merge. In an older codebase the description told you what was supposed to have changed, the diff told you what was supposed to have made the change happen, and CI told you both halves had compiled. The eval pass rate is what now tells you whether the change had its intended consequence on the system's behaviour, which is the only consequence the customer will ever actually feel. A PR that lifts no eval and breaks no eval has, in the strict accounting my grandfather would recognise, not moved the term. It can still be merged; it is just not, in any meaningful sense, a release note.
The way my grandfather closes the ritual is the part I keep returning to. He signs the bottom of the card in pencil, dates it in red ballpoint above the previous term's signature, and slides both cards back into the manila folder where the family's report cards have been kept since 1987. Nothing else in that flat has anything like that continuity. The furniture turns over every decade or so, my aunt replaces the curtains in the dining room on a schedule known only to her, and the cousins eventually emigrate to Toronto or marry into Mohammadpur or both. The folder stays, and because it stays, it is the only object in the house capable of settling an argument. Somebody will open it in 2042 to establish whether the youngest cousin was genuinely better at geography than the middle one, and it will answer.
An eval suite kept honestly across versions is that folder.
Everything else the team produces in a quarter, the docs, the slides, the demo videos, the PR descriptions in three colours of markdown, is the annual day. I don't want to be unfair to the annual day. People quote a cousin's recitation for twenty years at weddings, and a good demo has redirected more roadmaps than any scorecard I have ever written. But the annual day has never once settled a dispute about what actually happened, because it was not built to hold a comparison and the folder was. Which is why, when a customer asks what changed in this version, the true answer is a diff of eval rows, and the document we send them is a translation of that diff for people who are never going to open the repository.