Four AI assistants, same questions, same documents, different answers

Most organisations rolling out AI assistants share the same worry: the answers look right, but how do you know? We wanted to test that properly, on the kind of documents regulated firms rely on every day.

Over the past fortnight we put four widely used assistants (Microsoft 365 Copilot, Claude, ChatGPT and Gemini) through the same set of questions on the same documents, and recorded every answer.

How we tested

We used four documents: three German listed-company annual reports, and a short Word document built from pasted website screenshots, the kind of content found in internal decks and competitor packs across every enterprise.

Each document was tested in three forms:

• The original file, as published.
• The accessibility-remediated file, processed through ComplyLoft so that its tables, headings, figures and reading order are tagged to PDF/UA structure.
• A structured export generated from that remediated structure: every table cell recorded with its page, row, column, the text exactly as printed, its normalised value, its unit and its position on the page, plus the document’s own arithmetic recomputed as checks.

Each form was put to the same ten questions, chosen for the details that trip up machine reading: prior-year columns printed before the current year, totals with no label, dashes meaning “nil”, footnote markers run into the text, and dates and symbols that exist only inside images. Files were provided by direct upload and through SharePoint and Google Drive connectors. Chat history and memory were switched off. We recorded around 60 runs and scored each answer for correctness and for whether it said where in the document the answer came from.

What we found

On the original documents, every assistant made errors that looked exactly like correct answers. On the annual reports the figures were mostly right, but rarely traceable. On the screenshot document the figures themselves went wrong, and each assistant went wrong in its own way. One read a dash as a cross on every run. Another replaced a podcast name with a plausible invented one in eight of nine runs. Dates moved by a day or a year. A label reading “~5 min” was reported as 4 minutes, 45 minutes, 45 seconds and 21:05 by different assistants. One assistant cited a note in an annual report that does not exist.

The same assistant, on the same file, gave different answers minutes apart. Scores varied by four points out of twenty between identical runs. The same content as a Word file and as a PDF produced different errors. Nothing in the answer told the user which run they had received.

How the file reaches the assistant changes what it sees. Through some connectors, an assistant received only the text layer of the screenshot document: a single sentence. It honestly declined nine of the ten questions. Uploaded directly, the same assistant read the images and answered.

Remediation on its own did not change the answers. This surprised us. Accessible tagging and alt text are essential for people using screen readers, but none of the four assistants used that structure when answering, in PDF or in Word. The remediated files scored the same as the originals, within the variation we saw between runs.

The structured export changed everything. Answering from the export, every assistant scored 19 or 20 out of 20 on every run and every route. Every figure came with a cell reference that a simple script could check against the source: page, row, column and position. Across every run where an assistant quoted those references, none failed the check. The assistants declined to answer when a figure wasn’t there rather than estimating. And on the connector routes where the original document arrived as a single sentence, the export arrived complete.

Why this matters

For regulated organisations, the problem with AI on documents is rarely that it is always wrong. It is that it is usually right, and nobody can tell which answers are the exceptions. Checking means re-reading the document, which removes the time the AI was meant to save.

What this testing points to is a different approach. Rather than relying on each assistant’s own reading of a page, give it a checked, structured version of the document to answer from. In our tests that made the answers a user receives more accurate by default, consistent across all four assistants, and traceable to the exact place in the source, so a reviewer can confirm a figure in seconds rather than searching for it.

The structure that makes this possible is the same structure accessibility remediation already produces. Organisations working towards the European Accessibility Act are building it anyway.

What we haven’t tested yet

This is early evidence, and we want to be clear about its limits. It covers four clean, digitally created documents, focuses on tables, figures, labels and image content, and uses the chat versions of the assistants rather than models running inside an organisation’s own systems. Scanned documents, long narrative text and other sectors remain to be tested. As with all AI-supported work, outputs should be reviewed by a qualified person before they are relied on.

A detailed whitepaper is coming soon

We are now running a larger, independent study in collaboration with a university research centre, covering a much broader set of financial documents, multiple model families, and both standard and structure-aware ways of reading documents, with blind scoring of the results. We’ll publish the findings as a whitepaper in the coming months.

If you’d like to receive it when it’s released, or to discuss how your own documents behave with the AI tools your teams use, get in touch at contact@complyloft.com.