SAKbible: no citation, no answer

I asked a language model about a PSAK paragraph I know well, because I wanted to see what it would do. Fluently, confidently, in a tone identical to the truth, it cited the wrong treatment. Wrong in a way you’d only catch if you already knew the answer — which rather defeats the purpose of asking.

On an accounting standard, confidently-wrong is precisely as dangerous as it sounds. A person who says “I don’t remember” is useless for ten seconds and safe forever. A model that says the wrong thing in the right voice gets the wrong treatment written into a working paper with someone’s name at the bottom of it.

So the design constraint came first and everything else bent around it:

No citation, no answer.

You are not allowed to know accounting

The generator doesn’t get to be an expert. It’s a reading-comprehension machine over the documents, period — whatever accounting it appears to know is supposed to come out of the retrieved passages or not at all. That sounds like a strange thing to want, until you remember what an unrestricted model did in paragraph one: improvised a treatment in the voice of an authority, from memory it can’t trace to anything.

The machinery under the rule

The pipeline is a small civilization of checks, all built on the assumption that the model will lie given the chance:

1. Retrieve, don’t recall. Claims must come out of actual chunks pulled from the index — never from what the model happens to remember about Indonesian GAAP.
2. Hybrid retrieval. Dense embeddings (BGE-M3) catch the semantically-right-but-differently-worded hits; BM25 catches paragraph numbers and exact terms. RRF fuses them so neither method’s blind spots own the shortlist. This matters more than it should because the corpus is formal Indonesian and questions arrive in English too.
3. Rerank. “Was retrieved” and “is worth reading” are two different facts about a passage. A reranker separates them.
4. The quote gate. Every citation must exist, and every quote must be an exact substring of its chunk — whitespace-normalized, no paraphrase, no “close enough”. Examples get flagged as examples so implementation guidance never masquerades as mandatory.
5. The number check. A figure appearing in an answer must appear in the quote backing it. The pipeline logs it as ANGKA_TANPA_DASAR — “number without basis” — and yes, it has caught numbers that appeared from nowhere.
6. The judge. An independent NLI model (mDeBERTa-v3, multilingual entailment) reads each claim against its full cited paragraph and scores entailment. It shares nothing with the generator: different model, deterministic, no shared interest in the answer sounding good.

Confidence is a composite — the judge carries 45%, retrieval quality 35%, the generator’s own self-assessment a humble 20%. No judge verdict means the score is hard-capped below the top band. Claims that come back low or contradicted get demoted to klaim tidak didukung — claim not supported — and the user sees that, not a smooth recovery.

My favorite line in the codebase is a comment I wrote to myself: scoring them 72%/SEDANG would fabricate certainty the pipeline doesn’t have. Even the confidence score has an anti-hallucination policy.

question → decompose → retrieve (hybrid) → rerank → generate claims + quotes
                                                        │
                              quote gate + number check + NLI judge
                                                        │
                              survivors? → answer, receipts attached
                              zero survivors → CRAG: rewrite the query,
                                               try once more
                              still zero → "I don't have a source."

When nothing survives verification, the system gets exactly one retry — query reformulation, fresh retrieval, regenerate — and then it abstains. “I don’t have a source for that” is a complete, shippable answer here.

What it doesn’t know yet

One standard is fully ingested so far, end to end, cross-references between paragraphs mapped and verified. The rest are coming, one at a time, because each one deserves the same treatment. Ask SAKbible about a standard it hasn’t read and it tells you it has no source. It does not improvise. That refusal is not a missing feature; it is the entire product, working as designed.

You don’t need a machine that never gets anything wrong. You need one that is structurally incapable of pretending to be right — and then you let it be boring, safely, on your behalf.