Loading AI::COSMOS...
“AI Application Engineering requires rigorous automated quality evaluation. The RAG Triad framework measures 3 critical metrics: 1. Context Relevance (Are retrieved chunks pertinent?), 2. Groundedness (Is the answer supported by retrieved chunks without hallucinations?), and 3. Answer Relevance (Does the answer address the user query?). Guardrails inspect inputs and outputs for Prompt Injections, Jailbreaks, and PII leakage in real time.”
Quantifying production AI systems using the RAG Triad (Context Relevance, Groundedness, Answer Relevance), PII masking, and prompt injection defenses.
// RAG Triad Automated Evaluation Metric
export interface RAGTriadScore {
contextRelevance: number; // 0.0 to 1.0
groundedness: number; // 0.0 to 1.0 (Hallucination check)
answerRelevance: number; // 0.0 to 1.0
isPassing: boolean;
}
export async function evaluateRAGRun(
query: string,
context: string,
answer: string
): Promise<RAGTriadScore> {
const evalPrompt = `Evaluate the RAG interaction across 3 metrics (0.0 to 1.0):
1. Context Relevance: Did context contain the answer?
2. Groundedness: Is the answer 100% truthful to context?
3. Answer Relevance: Does the answer directly answer query?
Query: ${query}
Context: ${context}
Answer: ${answer}`;
const scores = await callLLMJudge(evalPrompt);
const isPassing = scores.groundedness >= 0.90 && scores.answerRelevance >= 0.85;
return { ...scores, isPassing };
}Input Guardrail: Scan user prompt for jailbreak attempts, token bloat, and prompt injection signatures
Retrieval Assessment: Compute Context Relevance score (ratio of useful context tokens to total tokens)
Generation Assessment: Compute Groundedness score by extracting atomic facts and verifying against context
Answer Relevance: Measure semantic similarity between user question and synthesized response
Output Guardrail: Scrub PII (Credit cards, SSNs, API keys) before returning payload to client
Automated CI/CD RAG evaluation pipelines catch hallucination regressions before PR merges, maintaining >95% groundedness benchmarks.