Mortanas AcademyHow should LLM output quality be evaluated?
Evaluate LLM quality on a versioned set of real tasks with explicit rubrics, not on a few impressive examples. Measure factual support, task completion, instruction adherence, safety, consistency and cost or latency where relevant. Include normal, edge and adversarial cases, blind human review and clear failure labels. Compare candidate systems on the same inputs and re-run the set when the model, prompt, retrieval source or tool configuration changes.
A practical five-step workflow
- Sample representative tasks and document expected evidence or outcomes.
- Create a rubric with pass, partial and fail definitions.
- Add edge cases, adversarial inputs and protected data scenarios.
- Run blind review and calculate agreement between reviewers.
- Version results and set release, monitoring and rollback thresholds.
Worked example
A support model is tested on 200 redacted tickets; reviewers score routing, factuality, policy adherence and escalation, while latency and cost are measured separately.
Risk controls
- Testing only easy or synthetic examples
- Combining unlike quality dimensions into one opaque score
- Failing to re-test after dependency changes
Primary and authoritative sources
Use the current version of each primary source for critical, legal or regulated decisions.
- NIST AI Risk Management Frameworkwww.nist.gov
- NIST Generative AI Profile (NIST AI 600-1)nvlpubs.nist.gov
Continue with the detailed Mortanas guide
This answer brief gives the decision pattern. The linked implementation guide expands the workflow, measures, failure modes and operating notes.
Questions about this answer
How should LLM output quality be evaluated?
Evaluate LLM quality on a versioned set of real tasks with explicit rubrics, not on a few impressive examples. Measure factual support, task completion, instruction adherence, safety, consistency and cost or latency where relevant. Include normal, edge and adversarial cases, blind human review and clear failure labels. Compare candidate systems on the same inputs and re-run the set when the model, prompt, retrieval source or tool configuration changes.
What should be measured?
Risk-weighted pass rate with confidence interval and reviewer-agreement score.
What evidence should be retained?
Keep the approved purpose, input or source references, relevant system and prompt version, human reviewer, corrections and the final outcome. Retention must follow the organisation’s privacy, security and records rules.
When should a human intervene?
Human review should increase when the output can affect rights, safety, money, reputation, access or an irreversible external action, or when evidence is missing, conflicting or uncertain.
Scope: Educational guidance, not legal, medical, financial or security advice. Verify current primary rules and obtain qualified advice for regulated decisions.