Your AI gave the wrong answer. Find out exactly where it broke.
When customers act on your AI’s answers, “the model got it wrong” isn’t a diagnosis. We map every path your feature can take to an answer, test each one against answers your own expert wrote first, and trace each wrong answer back to where it first went wrong.
Not just how often it’s wrong. Where it’s wrong, so you know what to fix.
One AI feature · up to 20 questions · report in two business days · €900 fixed
- 1route · look up · read · answer✕ Q9 broke at look up
- 2route · look up · retry · read · answerno question took it · one added
- 3route · search · read · rules · answer✓ 6 of 6 right
- 4route · search · read · rules · person✕ Q4 broke at rules
- 5route · person✓ 2 of 2 right
Every step’s output is compared with its right answer. A step that got the right input and gave a wrong output is where it broke: here, look up for question 9 and rules for question 4. Running the question again with that step given the right answer confirms it. When two steps fail together, the report shows both. A question sent down the wrong branch broke at the fork.
where it brokea path no question took
An example, not yours. Your map is drawn from your code, settings or logs, and every path on it gets tested.
One wrong answer can have very different causes.
- The search found the wrong passage.
- The router sent it down the wrong branch.
- A rule doesn’t cover the case.
- A reply was cut off at a length limit.
- A fallback nobody tested.
- Or the model itself.
Fixing the wrong layer costs engineering time, and can leave the real failure in place.
We ran it on ourselves firstNine problems. The biggest wasn’t the model.
Before offering this, we ran it on our own contract-reading product: one real, published distributor agreement, 13 questions answered by hand before any model saw it, five runs, 106 model calls in all.
- 1 right
- 10 declined
- 2 wrong
- on the 13 questions, our product as it was. Neither wrong answer was a false yes.
- The biggest was in our own testing. 12 of 44 model replies had been cut off at a length limit and came back empty, and nothing had noticed.
- Nine problems in total, six causes of wrong answers. Four of the six sat earlier than where we would have looked: two in reading the document, two in what the model understood.
- The app offered actions its own rules did not allow. The overall score did not show that.
- Model fees for the whole thing: $1.96.
One run, on one document, of a product we built ourselves. It shows the method finds things. What it finds in yours is what the report is for.
One AI feature. Up to 20 questions. The report in two business days.
Who it's forYou ship an answer people act on.
Engineering leads and founders whose AI feature gives a customer an answer they act on: a yes or no, a number, a field pulled from a document, where to send a request.
It matters most where a wrong yes costs money. Contracts, insurance claims, compliance checks, underwriting, refunds.
- Before a launch.
- After you change the model or the prompt.
- After a customer found a wrong answer and nobody can say where it came from.
How rigorous is this?
Why not do it yourself?You could. Most teams don’t.
The method is not a secret. What a team can’t do is check its own work from the outside.
Your tests follow the paths you know about. We draw every path your code allows, including the retry, the fallback and the rule nobody has triggered yet, and make sure each one is tested. The fixed logic along the way, your routing and your rules, we walk through every combination of inputs it reads, with no AI involved, so a case that no rule decides has nowhere to hide.
Nobody tests to break what they built. The people who built a feature decide what counts as right. Here your expert’s answers are locked first, and we write down where we expect the feature to fail before anything runs, so our prediction can be wrong in front of you.
The hard part is the middle. Knowing the right final answer is easy. Knowing what each step in between should have produced is the work, and the plan says how we establish it.
Your tests are part of what gets tested. Our biggest finding on ourselves was in our own measuring. Your checks may have the same blind spot.
It costs less than a week of an engineer’s time. One fixed price, and the report in two business days.
How it worksAnswers first. Model second.
We map every path.
From your code, settings or logs, we draw every way a question can travel to an answer: each branch, lookup, retry, fallback and hand-off to a person.
Your expert writes the answers.
Someone who knows your field answers 10 to 20 real questions about documents you can share, chosen so that every path gets tested, and notes what a harmful wrong answer would look like. We send a template. We never write your answers.
The answers are locked.
Before any model sees your documents, we both keep a fingerprint of the answers and of the documents they were written against. Any change afterwards would show.
You approve the plan.
It names the paths and the steps on them, the price of each step, which steps can be given a right answer and how we establish it, and which AI providers will see your material, on what settings. It also says where we think your feature will fail. Nothing runs until you have agreed.
We test every path.
First as it is. Then we compare every step’s output with its right answer and find each step that got the right input and gave a wrong output. Running the question again with that step given the right answer confirms it. When two steps fail together, the report shows both, and a question sent down the wrong branch broke at the fork. Your fixed logic we also check against every combination of inputs it reads, and each case it gets wrong comes with an example.
You get the report.
Every question and every run, marked right, declined or wrong. Each is counted on its own, never blended into one score, because “I don’t know” and a confident wrong “yes” need different fixes. Every reply comes with why it stopped, so a cut-off reply is never mistaken for a decline. Each wrong answer is traced to its path and step as far as the runs allow, every path is marked tested or not, each problem comes with a proposed fix, and our prediction is scored against what we found.
What you keepAll of it. It runs without us.
- Your answersAnd their fingerprint. Written by your expert, yours from the start.
- Every runThe scripts, every reply from the model, what each call cost and why it stopped.
- The re-runRun it again on your own model accounts after the next prompt change, against the same answers.
There is no account with us. Re-running needs only your own model accounts.
What it isn'tA diagnosis, not a seal of approval.
Not a benchmark, not a certificate, and not a promise that your feature is right.
The report says what your expert's answers could catch, for the questions you gave us. Where the runs cannot isolate a cause, it says how far they narrowed it and what to test next.
Price
- One AI feature
- Up to 20 questions
- Up to 25 steps, branches included
- About 50 pages of documents
- One report, in two business days
- No VAT added
Bigger than that? We quote it before starting. Until anything runs, you can withdraw for any reason and get every euro back, including when you cannot share documents or the answers cannot be written. The two business days start once your answers are locked and you have agreed the plan, time we wait on you for something the plan names as yours does not count, and if we are late you can stop and get every euro back. You can also stop once the runs have begun: steps not yet started are refunded within ten business days, and a step under way is charged at the plan's price.
What we need from you
- A person who knows your field, to write the answers.
- Documents you are allowed to share with us.
- Your feature’s code, settings or logs, so we can draw its paths, and a way to run it step by step or to see what goes into each step and what comes out.
Steps that cannot leave your systems, such as private code or live customer data, we check from the outside: what goes in and what comes out. The report says which steps we could not open.
Where your documents go
Only to the AI providers the plan names, on the settings it states. We delete our copies 30 days after the report, or sooner if you ask. The terms carry the full schedule.
Questions first? Write to sana@muoto.xyz. Read the terms and how we handle your details.