For UK charities · How I keep it accurate
Do private AI assistants make things up?
Any AI can. The job is making sure the ones I build don't reach your team with it when they do.
I build UK charities private AI assistants that answer questions from their own systems: their CRM, their finance system, their spreadsheets and their documents, without any of it leaving their control. Each one has a job, and some read several systems at once. This page is about the first thing worth asking of a tool like that. Can you trust what it tells you?
AI tools are known for inventing a confident, plausible answer to a question they can't really answer. The industry calls it hallucination. It's the right thing to worry about, and it's the reason most of the work on one of these assistants is not building it. It's trying to catch it being wrong before you ever see it.
You can see one of these assistants here. This page is only about the answers: how I test them, and what the testing has caught. No adjectives, just the method and the results.
What I can honestly claim
I'm not going to tell you it never makes a mistake. Nobody can say that honestly about any AI, and I'd be wary of anyone who does. Here is what I can say instead.
Every assistant I build is tested against data I control and already know the right answers for, before anyone relies on it. Every question has to be answered correctly, with working links to the source, 20 times in a row before it ships. One wrong answer out of the 20 fails the whole question. Every mistake the testing finds becomes a permanent check, so the same mistake cannot come back. And where a kind of mistake can be made impossible rather than just unlikely, I make it impossible.
There is one promise underneath all of this that matters most: a wrong answer never reaches you. When the assistant cannot check what it is about to say against your data, it does not guess, it says so plainly.
On the demonstration build the most recent test asked all twenty questions twenty times over, twice through, eight hundred answers in all, and not one wrong answer went out, and not once did it have to fall back on "I can't answer that". It was simply right every time.
That's a different promise from "trust me, it's clever". It's "here is the evidence, and here is what it got wrong on the way".
How the testing works
The idea is simple. I take a set of questions I already know the right answers to, drawn from the real data in the systems you already run: your CRM, your Xero, whichever third-party systems you use. I ask each one many times over, and mark every answer against the known result.
The answers I mark against aren't my opinion. In the demonstration here, that data is a fictional charity's, built by scripts that write out the correct figures as they go, so the member counts, the fund totals, the attendance and the species counts are all worked out separately from the assistant. On your own build it's your real figures, established the same way, independently of the assistant. The assistant never marks its own homework.
Then a few rules make the marking strict:
- Each question is asked 20 times, not once. A lucky right answer isn't a pass. The score is how many of the 20 were right.
- The facts and the source links are marked separately. A right figure with a broken link is a fail. A working link on a wrong figure is a fail.
- Every question also lists what the answer must not contain: the wrong total, a confidential detail, a broken link.
- Every failed answer is kept, word for word, so a one in twenty wobble gets diagnosed from the evidence rather than guessed at.
Here is what one full run looks like, against my Fenmere Trust demo, a fictional charity with entirely made-up data. Every question goes out 20 times, and each answer comes back marked against the known figure, green for a pass.
20 questions, 20 runs each, twice over: 800 answers, not one wrong, every figure marked against a total worked out separately from the assistant.
Why the numbers hold still
The reason this works is that the AI does less than you'd think. It writes only the sentence.
When you ask how much a fund raised, or how many members lapsed, the counting and adding are done by ordinary code reading your systems, the same way every time. The figures, the links, and the way the answer is laid out are all built by code from what your systems returned. The AI's job is only to understand the question and put the answer into words. It routes the question in and phrases the answer out, and it doesn't touch the figure, the link or the layout in between.
So it cannot drop a figure, cannot mangle a link, and cannot invent a citation, because it never types any of them. That's why a total comes out the same on the twentieth run as the first, and why every figure carries a link straight back to the record it came from. Nothing is taken on trust.
Checked on every answer, not just in testing
Testing happens before an assistant reaches you, against answers I already know. But it doesn't stop there. Once the assistant is live, every answer it gives is checked in the moment, before you ever see it.
Once it's pulled the figures from your systems, the assistant writes its sentence. Before that sentence reaches you, it's checked back against the very data the tools just returned: does it carry the figures the question turned on, does it say anything the data doesn't support, is there a real record behind every claim? If it drops a number, contradicts the data, or makes a claim with no source, the answer is thrown out and written again. And if it still doesn't hold up, the assistant tells you it can't answer, rather than show you something unchecked.
That's the difference between "we tested it and it was fine" and "it's checked every single time". The testing proves the method works; this runs on the actual answer you're given.
What the testing has caught
This is the part most people don't show you. Here are real mistakes the testing found on the demo assistant, and what I did about each one.
The six stone-curlews that were really two
Asked how many stone-curlews the reserve had, the assistant answered six. There were two. It hadn't invented anything. The birds had been counted on three separate visits, and the code had added the visits up instead of taking the highest single count. A person watching the same two birds three times does not have six birds.
This is the important kind of mistake, because it isn't the AI making things up. It's a wrong convention about what the data means, and it was invisible until it was tested against a known answer. The fix was to count the peak, never the sum, written into the code and pinned down by a new test so it can't drift back.
The nest location it was right to withhold
One reserve holds a protected breeding site whose exact grid reference is genuinely sensitive. On two runs out of 20, asked about the site, the assistant quoted the grid reference. It was polite about it, and it cited its source correctly. It was still wrong to say it.
Asking it nicely not to isn't a safeguard. So the grid reference is now removed by the code before the assistant ever sees the document text. It can't disclose what it was never given. That's the difference between unlikely and impossible, and for something like a nest location or a safeguarding detail, impossible is the only acceptable answer.
There's a fuller record of every failure, its cause and its fix. Each before and after is a real pass rate: a wrong figure on 3 runs out of 20, then 20 out of 20 once the fix and its test were in.
Each fix is also a new test, so the mistake it caught cannot come back. The full ledger records the cause and fix for every one.
What this means for your data
The stone-curlew mistake is the one to remember, because your data has its own version of it. Every organisation counts things in ways that are obvious to the people who work there and written down nowhere: what makes a member lapsed, which date counts as the gift date, when two records are really the same person. Get one of those conventions wrong and the answer looks fine and is quietly incorrect.
Finding those is most of the work, and it's the same loop every time. Take data where the right answers are known, test the assistant against it, find where it has read your world wrong, fix it in the code, and keep the test. The report at the end, this question, answered correctly, 20 times in a row, is part of what I hand over.
So the honest answer to "does it make things up" is: I assume it will try, and I build and test it so that where it matters, it can't. If you'd like to see that run against your own data, that's a large part of what the work is. Email me and tell me what your team needs to be able to trust.
The examples here come from the demonstration assistant built for a fictional charity, the Fenmere Trust. Its records are made up so I can show the testing openly. The method is the one I use on real client builds. You can see the assistant itself here.
