Most AI benchmarks are exams. A question goes in, an answer comes out, and somebody marks it. That tells you very little about whether an agent can be handed an actual job.
On 27 October, Surge AI published DAYJOB: Tax, a benchmark for AI agents on professional tax work. Instead of questions, it gives agents 50 complete assignments: a client request, a pile of records, and an expectation that they produce the finished thing. A return. A set of workpapers. A review of a disputed position. A letter of advice.
The best model tested, Claude Sonnet 5.5, finished 10.8% of them to standard. That number matters less than why it is so low, and the reason should change how you scope any plan to automate paperwork in your business.
The test was the whole job, not the question
According to Surge, the benchmark spans eight areas of tax practice and contains 1,789 workspace files, with a median of 30 files per assignment and as many as 123. There are 1,487 PDFs, 117 Excel workbooks, 38 Word documents, plus emails, images, CSVs and loose text.
That file list is the point. One assignment includes a photograph of a handwritten receipt for stock bought for resale, which the agent has to read, reconcile against spending records, and carry through to inventory and cost of goods sold. Another involves a company whose owner believes it only operates in one state, while its field dispatch log shows crews working across five. Nobody asked the agent to check that. The evidence was just sitting in the files.
This is what back-office work actually looks like in a small business. Incomplete instructions, records that disagree with each other, and the most expensive problem being the one nobody thought to raise.
What a 10.8% score does and does not mean
Read the scoring method before you read the score. Surge reports pass@1 at a 95% rubric threshold, meaning an attempt only counts as a pass if it satisfies at least 95% of the grading criteria for that task. That is close to flawless. A deliverable that was 85% right, useful, and fast still scores as a failure.
So this is not evidence that AI gets nine out of ten tax jobs wrong. It is evidence that AI rarely gets one completely right without supervision. Those are different claims with very different consequences for your business.
It is also worth noting what the benchmark does not report. In the Hacker News thread on the benchmark, one commenter asked the obvious question: what is the human baseline? No figure is published. Human tax professionals also miss things. Another commenter in the same thread described an accountant mis-filing their return at a cost of around $20,000, discovered only after they switched firms. The honest comparison is not "AI versus perfect". It is "AI plus review versus the process you run today".
The failures are quiet, confident and well formatted
Here is the part that should shape your budget.
Surge describes a client, Anna, asking whether to sell her flat in October or January for tax reasons. The agent correctly worked out that both dates satisfied the residence test for an exclusion worth up to $250,000. What it left out, in all five attempts, was that she would stop qualifying altogether if the sale slipped past August 2027. The answer to the question asked was right. The advice was dangerous.
In another, a separating couple asked how to split their retirement accounts. The agent opened with a flat statement that they were not legally married and built its whole recommendation on that. Their own records raised real doubt: they had lived together in Texas and held a ceremony in Colorado before moving to Florida. The agent never verified the assumption it had made.
Neither of these looks like a failure on the page. There is no error message, no blank field, no obviously botched sum. They read as competent work. A business owner reviewing that output would have no way of knowing what was missing, because knowing what is missing is the entire skill.
Software people are used to failures that announce themselves. This kind does not. That is why the useful question is never "can AI do this task", it is "when it gets this wrong, who notices, and how".
The real cost model is drafting plus review
If you are considering an agent for bookkeeping, BAS preparation, quoting, payroll checks or any compliance paperwork, the business case has three numbers in it, not one.
The first is how long the agent takes. The second is how long a competent human takes to verify its output, which is the number everyone forgets. The third is the cost of a miss that nobody catches, multiplied by how often that happens.
If verification takes nearly as long as doing the work from scratch, you have bought nothing. If verification is fast because the output is built to be checked, you have bought a lot. That difference is a design decision, and it is where we would spend the build budget.
In practice that means making the agent show its workings rather than just its conclusions: every figure traced to the source document it came from, every assumption stated as an assumption, every piece of missing evidence flagged rather than quietly assumed. It also means deciding in advance which outputs are drafts a human signs off and which, if any, are allowed to go out automatically. For anything with a regulator attached, our answer is none of them.
Where people are already getting real value
The same discussion thread is full of people using these tools well, and the pattern is consistent. One commenter who prepares their own cross-border return said feeding the draft plus supporting paperwork to Claude surfaced a genuine error, including an incorrect exchange rate, and correctly explained how the mistake had been made. Another said Gemini handled estate calculations mostly well but got withholding wrong, concluding it was more useful for preparation than for planning and advice. A third described exporting 1,700 transactions out of QuickBooks and using AI to check the categorisations in under an hour, a task that previously took a full day.
The common thread: AI as a second reader over work a human owns, or as a fast pass over high-volume, low-stakes classification. Not AI as the professional of record.
One more thing worth weighing. A commenter raised the discomfort of handing a complete financial picture to an AI vendor. For an Australian business that is not just a feeling, it is a question about where data is processed, how long it is retained, and whether it is used for training. Ask it before you pilot anything, not after.
What to do with this
If someone pitches you an AI agent that replaces a back-office role, ask three things. What does the review step look like, and who does it. How does the system make its own work checkable, rather than just confident. And what happens to our data.
Then run the smallest possible pilot on work you can verify independently, and measure the review time honestly. Budget for the reviewer. The saving is real, but it comes from making skilled people faster, not from removing them.
