SourceScore

Integration guide

DSPy + VERITAS

DSPy is Stanford's compound-AI-system framework — programs instead of prompts. This guide shows two integration patterns: a custom dspy.Retrieve backed by the VERITAS catalog, and a candidate-and-review post-processor module.

Why DSPy + VERITAS

DSPy programs declare what the system should do (signatures + modules) and leave the how (exact prompts) to the optimizer. That separation makes external retrieval modules first-class — a VERITAS retriever fits naturally into the existing dspy.Retrieve interface.

The compound system gains a typed retrieval path that DSPy's optimizers can reason about — curated-record retrieval becomes a tunable step, not a brittle prompt-stuffing decision.

Install

pip install dspy-ai requests

Pattern 1 — Custom dspy.Retrieve

Subclass dspy.Retrieve and translate VERITAS search hits into DSPy passages. Each passage carries the claim id, confidence, and source URLs as metadata so downstream modules can render citations.

import dspy
import requests
from typing import List

VERITAS = "https://sourcescore.org/api/v1"

class VeritasRetriever(dspy.Retrieve):
    def __init__(self, k: int = 5, min_confidence: float = 0.8):
        super().__init__(k=k)
        self.min_confidence = min_confidence

    def forward(self, query_or_queries, k=None) -> List[dspy.Example]:
        queries = [query_or_queries] if isinstance(query_or_queries, str) else query_or_queries
        results = []
        for q in queries:
            r = requests.get(
                f"{VERITAS}/search",
                params={"q": q, "limit": k or self.k},
                timeout=8,
            )
            for hit in r.json().get("results", []):
                if hit.get("confidence", 0) < self.min_confidence:
                    continue
                results.append(
                    dspy.Example(
                        long_text=hit["statement"],
                        claim_id=hit["id"],
                        confidence=hit["confidence"],
                        canonical_url=f"https://sourcescore.org/claims/{hit['id']}/",
                        tags=hit.get("tags", []),
                    ).with_inputs("long_text")
                )
        return results

Wire into a DSPy program

import dspy

# Set up the LM + retriever
lm = dspy.OpenAI(model="gpt-4o-mini", temperature=0)
rm = VeritasRetriever(k=5, min_confidence=0.85)
dspy.settings.configure(lm=lm, rm=rm)

# Define the signature
class CitedAnswer(dspy.Signature):
    """Use only candidate records whose exact statements support the answer."""
    question: str = dspy.InputField()
    context: list[str] = dspy.InputField(desc="Candidate records with [claim_id] tags")
    answer: str = dspy.OutputField(desc="Answer with [claim_id] citations after every fact")

# Build the program
class CitedRAG(dspy.Module):
    def __init__(self):
        super().__init__()
        self.retrieve = dspy.Retrieve(k=5)
        self.generate = dspy.ChainOfThought(CitedAnswer)

    def forward(self, question: str):
        passages = self.retrieve(question).passages
        context = [
            f"{p.long_text} [{p.claim_id}] (conf={p.confidence:.2f})"
            for p in passages
        ]
        return self.generate(question=question, context=context)

program = CitedRAG()
result = program(question="When was the Transformer architecture introduced?")
print(result.answer)

The signature forces a [claim_id] citation after every assertion. DSPy's optimizer can later tune the exact prompt around this signature without changing the contract — VERITAS continues to feed reviewed catalog records regardless of which prompt-template the optimizer settles on.

Pattern 2 — Candidate-retrieval post-processor

When you want free-form generation but candidate retrieval afterwards, wrap /api/v1/verify in a DSPy module that runs after the answer generation. Review the returned primary sources before treating any assertion as factual.

class VeritasCandidateLookup(dspy.Module):
    """Retrieve a candidate catalog record for each assertion."""

    def __init__(self, min_confidence: float = 0.85):
        super().__init__()
        self.min_confidence = min_confidence

    def forward(self, answer: str) -> dict:
        lines = [l.strip() for l in answer.split("\n") if l.strip()]
        candidates, no_candidates = [], []
        for line in lines:
            r = requests.post(
                f"{VERITAS}/verify",
                json={"claim": line, "minConfidence": self.min_confidence},
                timeout=8,
            ).json()
            if r.get("bestMatch"):
                candidates.append({
                    "text": line,
                    "claim_id": r["bestMatch"]["id"],
                    "confidence": r["bestMatch"]["confidence"],
                    "url": f"https://sourcescore.org/claims/{r['bestMatch']['id']}/",
                })
            else:
                no_candidates.append(line)
        return dspy.Prediction(
            candidates=candidates,
            no_candidates=no_candidates,
            candidate_rate=len(candidates) / max(1, len(lines)),  # similarity coverage, not truth rate
        )

# Composing it into a larger program
class AnswerAndCandidateReview(dspy.Module):
    def __init__(self):
        super().__init__()
        self.generate = dspy.ChainOfThought("question -> answer")
        self.retrieve_candidates = VeritasCandidateLookup(min_confidence=0.85)

    def forward(self, question: str):
        a = self.generate(question=question)
        v = self.retrieve_candidates(a.answer)
        return dspy.Prediction(
            answer=a.answer,
            candidate_records=v.candidates,
            no_catalog_candidates=v.no_candidates,
            candidate_rate=v.candidate_rate,
        )

DSPy optimizer compatibility

DSPy's optimizers (BootstrapFewShot, MIPRO, COPRO) can tune around the VERITAS retriever — they'll adjust the prompts that consume the passages, but they can't change what the passages contain. That's by design: the catalog is the trusted layer, the optimizer improves how the model uses it.

One diagnostic is candidate_rate from the example above: the share of assertions for which the bounded catalog returned a candidate. Do not optimize this as an accuracy target; it measures catalog overlap, not truth. Evaluate support against a separately labeled evidence set.

Compose with other DSPy modules

The VERITAS retriever composes with any DSPy pattern: ReAct, MultiHopProgram, ProgramOfThought. A multi-hop pattern with candidate review:

class MultiHopCandidateReview(dspy.Module):
    def __init__(self):
        super().__init__()
        self.retrieve = VeritasRetriever(k=3)
        self.hop1 = dspy.ChainOfThought("question -> sub_question")
        self.hop2 = dspy.ChainOfThought("question, sub_answer -> final_answer")
        self.find_candidates = VeritasCandidateLookup()

    def forward(self, question: str):
        sub_q = self.hop1(question=question).sub_question
        passages = self.retrieve(sub_q).passages
        sub_a = "\n".join(p.long_text for p in passages)
        final = self.hop2(question=question, sub_answer=sub_a).final_answer
        return self.find_candidates(final)

Next steps