A screening tool that is confidently wrong is worse than no screening tool at all. Someone spends a day applying for something they were never going to get, or is told they do not qualify for something they do. So the accuracy figure sits on every screen, and this page explains exactly what produced it.
“Accuracy” hides three different questions. Is the arithmetic right? Can the system read the facts out of how people actually write? And, given both, does it reach the right answer? They fail in different ways and are measured in different ways, so they are reported apart rather than blended into one flattering number.
31 of 31
Hand-computed cases, each with its expected figure worked out from a primary source and written down as a number: USDA’s own worked SNAP example, the IRS revenue procedure tables, the medicare.gov limits, VA’s pension rates. None of the expected figures is produced by the code being tested, so a disagreement is a disagreement with an agency.
On its first run this found a real error: at the plateau of the Earned Income Tax Credit, a filer with three children was being shown $8,230.50 against a published maximum of $8,231, because the rate times the earned income amount lands just under the rounded figure the IRS publishes. It is fixed.
96.0% of facts read correctly
Twenty descriptions written the way people actually write (“base pay is about $900 and tips add around $1,100”, “every other week, about $1,150 after taxes”, “me, my son, and our dog”), with the facts labelled by hand before anything read them, and scored once. This is the sixth such set: each earlier one moved into the development set once it had been fixed against.
Code finds every amount that could be money. The decision model says what each one is (pay, rent, a utility bill, child care) and how often it is paid, as a choice among fixed options, so it never writes a number and cannot garble one. Code copies the amount and does every sum. Counting people stays in code; where the model counts the household differently, the screener asks rather than guesses. Pattern-matching code alone scores 83.2% on the same set.
A miss that leaves a fact blank makes the screener ask for it. A miss that reads a fact wrongly is worse, because nothing prompts anyone to check it: code alone read a $140 electric bill as one household’s whole income. Every remaining miss is listed rather than quietly fixed.
99.4% mean balanced accuracy across the 20 programs the held-out set can test
The 49 households held back for this, run through the whole system: the paragraph read, every rule answered by the decision model, follow-up questions asked and answered, and each program’s verdict compared with the answer key. Accuracy is scored on the 39 of them that state every fact; the rest are missing one on purpose, to test the follow-up questions. The confidence threshold (0.3) was chosen beforehand on the development households by a rule fixed in advance.
Most households are now asked no follow-up at all. A question is chosen by how much money its answer is expected to move, weighed by how likely each answer is, and one expected to move less than $25 a year is not asked. Before that change these households were asked 81 questions in all; they are now asked 51, with no loss of accuracy. The weak spot this run showed: where a description never says whether the money comes from a job or from benefits, the model sometimes guessed confidently instead of asking, and it read some people of 62 to 64 as already on Medicare. Both are addressed since, with an explicit “not stated” answer for the first and an age check in code for the second, and on the development households the missing fact is now asked first 60% of the time rather than 40%. The figures above are from before those changes: the held-out households are not re-scored after being fixed against.
The comparison swaps only the source of answers: Claude Sonnet 5 answers the same questions about the same paragraph, through the same rules, follow-up loop and answer key, with its reply held to a schema that allows only each rule’s own options. Most of its misses are one misreading: it took “rent with utilities included” as not paying for heating, where the rule says energy paid through the rent counts. It asked more questions, partly because its stated confidence often fell below a threshold set for the other model. A second configuration, all questions in one request without a schema, is in the technical report, with the method and limits.
The first run found a mistake in the rules rather than in the model: the dependent care credit needs a child under 13, and both the answer key and the screener counted every child. It is fixed, a conformance case now pins it, and the held-out run was repeated on the corrected rules. Run to run, the model’s answers vary slightly: both runs got three verdicts wrong, not the same three.
Four out of five households in the set qualify for SNAP. A system that answered “eligible” to every household would therefore be 80% correct while being completely useless. The figure reported here is balanced accuracy: it weighs getting the ineligible households right equally with getting the eligible ones right, and that same do-nothing system would score 50%.
Telling someone they are likely eligible when they are not costs them a wasted application and their trust. Telling someone they are not eligible when they are is the exact failure this product exists to fix, and it is silent: nobody ever finds out. Both are tracked separately rather than averaged away, and every household is shown every program, including the ones judged unlikely, so a person who knows their own situation can disagree with us.
The validation households are synthetic. Writing them from known facts is what makes their correct answers checkable, and it also makes them tidier than the way people really write. A strong score here is evidence that the rules are implemented correctly and that the follow-up questions are well chosen. It is not evidence about real applicants, and we will not present it as such.
SNAP, Medicaid and state earned income credits are screened under each state’s own rules: the higher SNAP income limits and dropped savings tests most states use, Medicaid expansion or not, and the state credit rate. The other programs use federal limits, which some states and local agencies raise, so a household told it is unlikely to qualify for one of those may still qualify where it lives. Every result says so.
Every dollar figure in the system records the official page it was read from and the date it was read, and copies of those documents are kept alongside the rules so any number can be traced back. When a new table has not been published yet, as with the SNAP figures that take effect each October, the system says so on screen rather than quietly carrying on with a stale number or inventing one.