PSAK RAG: If an answer can't be proven, it doesn't get said

I wanted an answer to a PSAK standard, and I asked a language model. It gave me a confident, clean sentence. I happened to know the standard, so I knew the sentence was wrong.

That was the whole reason this project exists. Not because LLMs are useless (they’re remarkable) but because the failure mode I care about isn’t “it’s quiet.” It’s “it’s confidently wrong, in a language that sounds exactly like the truth.”

The standard

PSAK, Pernyataan Standar Akuntansi Keuangan, is the Indonesian accounting standards set. If you’re auditing or preparing financial statements, these are the rules. And like most accounting standards, a single paragraph can change what a liability is worth, or when it’s recognized.

The corpus is dense, cross-referenced, and full of register that you have to read carefully. A model that “sort of knows” PSAK is a liability, not an asset.

The failure, precisely

LLMs are next-word predictors. They don’t query a database; they generate the most plausible continuation of what you wrote. When the training data has something adjacent to the answer, the model produces a fluent paragraph that looks right. That’s the trap. For a casual question, it’s fine. For an accounting-standard question where the difference between two paragraphs is a material judgment, it’s not fine.

So the design question became: how do you make a system that can’t present an unsupported claim?

The pipeline

The answer wasn’t a better prompt. It was a hard guarantee structure:

1. Retrieve, don’t recall. Every answer has to come from a chunk of the actual standard, retrieved rather than generated from memory.
2. Hybrid retrieval. Dense embeddings (BGE-M3) + keyword search (BM25), fused with RRF so neither method’s blind spots dominate.
3. Rerank, then route. A reranker (Jina) pulls the most-relevant passages forward. Where it’s confident, the answer gets narrow; where it isn’t, the system says so.
4. Verify mechanically. The generation step has to contain exact quotes from the retrieved passages. If the answer can’t point at the source phrase-for-phrase, it’s rejected.
5. Judge it. A small NLI model reads the claim against the passage and asks: does this follow? If the claim isn’t entailed by the source, it doesn’t ship.

prompt → retrieve → rerank → generate → verify → NLI judge
                                            │
                                            └── not entailed? → drop, no answer

The output isn’t “here’s what I think.” It’s “here’s the paragraph that supports this, and here’s the claim, and they agree.”

The rule

I wrote the rule in big letters early on, because it’s the whole point:

No citation, no answer.

A model is allowed to say “I don’t have a source for that”, and honestly, that’s more useful than a confident guess. For someone who works with standards, a source you can check beats a summary you can’t.

What I learned

Building this didn’t make me trust LLMs more. It made me understand when to trust them. The entire system is a fence: it doesn’t fix the model’s ignorance, it just makes sure the ignorance can’t sound like knowledge.

And that’s the difference between “uses AI” and “uses AI responsibly.” You don’t need the model to never be wrong. You need it to be unable to pretend it’s right.