Six LLM grounding strategies—and the trade-offs of each

You ship a chatbot. Users find a single hallucinated fact and screenshot it on Twitter. Your trust signal collapses. The features you spent six months building become irrelevant.

Hallucination rates vary sharply with the model, task, prompt, evaluation method, and what counts as an error. There is no honest universal percentage. Reducing unsupported output is therefore a stack of mitigations, each aimed at a different failure mode.

1. Temperature 0 + a clear system prompt

The cheapest win. Set temperature=0 for any task that involves factual recall (not creative writing). Add a system prompt that explicitly tells the model not to invent:

You are a precise assistant. If you don't know an answer, say "I don't know."
Never invent dates, parameter counts, paper authors, or citations.
If the user asks for a specific fact you're uncertain about, decline.

This does not make factual output trustworthy. It is a cheap baseline that makes the intended refusal behavior explicit and can make repeated runs more consistent.

2. Few-shot examples for narrow extraction tasks

For structured-output tasks (extract dates, names, prices), show the model 3-5 examples of correct extraction before asking it to do the real task. Evaluate few-shot against zero-shot on your own labeled examples; the result depends on the model and task.

Best for: information extraction, classification, formatting. Not effective for: open-ended generation, citation, summary.

3. Retrieval-augmented generation

The dominant strategy. Embed your knowledge base, retrieve top-K relevant chunks at query time, splice into the prompt:

SYSTEM: Answer using ONLY the context below. If the context doesn't
cover the question, say so. Always cite which context block you used.

CONTEXT:
[chunk 1]: Llama 3.1 was released on July 23, 2024. Three variants:
8B, 70B, 405B. Context window: 128k tokens.
[chunk 2]: ...

USER: When did Llama 3.1 come out?

Retrieval can give the model relevant evidence, but it does not force the model to use that evidence correctly. Measure retrieval recall and answer support separately on a task-specific evaluation set.

Frameworks: LangChain, LlamaIndex, Haystack. Vector DBs: FAISS, Pinecone, Weaviate, Qdrant, Chroma, pgvector.

4. Citation requirement + post-hoc check

The classic RAG failure: retriever pulls the right document, model still emits wrong number on the page. The fix: force the model to emit inline citations + post-process to check they match.

SYSTEM: For every factual claim in your response, append [^N] where N
matches the context block. Citations must be verifiable against the
context. If you cannot cite a claim, mark it [^unverified].

Post-process: scan the response for [^N] markers, verify each citation actually appears in the retrieved context. Strip or flag unverified claims before returning to user.

Costs include extra tokens, latency, and a verification pass. The benefit is inspectability: unsupported statements can be blocked or routed for review instead of silently reaching the user.

5. Signed-claim verification (catches the long tail)

Even with RAG + citations, two failure modes remain:

• Out-of-corpus assertions. The model claims something not in your retrieved context. RAG can't verify because the assertion has no source to check against.
• Fabricated citations. Model writes[^1] but the [1] doesn't exist or doesn't support the claim.

The fix: query a separate verified-claim catalog post- generation. Extract atomic assertions from the response; look each up against a source-of-truth. We built SourceScore VERITAS for this narrow AI/ML use case: 384 hand-reviewed claims, each linked to primary evidence, with stable IDs and SourceScore-issued HMAC integrity metadata. The public API is free and requires no signup. It is a fixed catalog, not a general truth oracle.

For other verticals (non-AI/ML), Wikipedia + Wolfram Alpha + curated domain-specific knowledge bases play the same role. The pattern is what matters: a verification layer after generation that catches what RAG misses.

6. Constrained decoding (last resort for high-stakes outputs)

For outputs that must conform to a specific schema (JSON, BNF grammar), use a library like Instructor, Pydantic AI, Outlines, or vendor JSON-mode APIs. Constrained decoding guarantees the output fits the schema (zero schema-violation errors) but doesn't guarantee semantic correctness — a hallucinated date still type-checks.

Use constrained decoding for: schema enforcement, downstream system contracts. Combine with strategies 3-5 for factual correctness.

The honest stack

Production-grade grounding looks like:

1. Temperature 0 + clear system prompt — free baseline
2. Few-shot examples for narrow tasks — free
3. RAG over your knowledge base — moderate cost
4. Citation requirement + post-process check — low cost
5. Catalog lookup on residual claims — adds a network request
6. Constrained decoding for schema-required outputs — moderate cost

Combining the layers can reduce different classes of failure, but the result is only as good as your evaluation set. Report task-specific supported-answer and abstention rates rather than a universal “hallucination reduction” percentage.

What doesn't work

• "Just use a smarter model." Frontier models hallucinate less than older models, but not close to zero. Capability scaling has not killed hallucination.
• "Fine-tune the model on your factual corpus." Helps for in-corpus questions; the model still hallucinates out-of-corpus. Expensive + maintenance-heavy.
• "Just ask the model if it's sure." Self-reported confidence is poorly calibrated. The model says it's confident about hallucinated answers as often as correct ones.

Practical sequencing

If you're shipping a new AI feature today:

1. Week 1: ship strategies 1 + 2 (temperature + system prompt + few-shot if applicable)
2. Week 2-3: ship strategy 3 (RAG over your corpus)
3. Week 4: ship strategy 4 (citation requirement + post-process)
4. Week 5+: add strategy 5 (verification layer) for residual claims
5. Whenever needed: strategy 6 (constrained decoding) for schema-bound outputs

Skipping straight to strategy 5 without 1-4 is a mistake — verification is most cost-effective on the long tail of claims, not the bulk. RAG + citations gets you most of the way; verification closes the last gap.

Related

• LLM grounding — full concept pillar
• Hallucination categories + root causes
• RAG vs VERITAS — when each pattern applies
• AI agent grounding use case
• RAG pipeline verification use case
• Try the verification API in browser