Evaluation Frameworks
Measure the behavior that matters in language-model applications.
Overview
Design a small, repeatable evaluation set around real product failures. Define expected behavior and a scoring rubric before choosing a model judge. Keep holdout examples separate from tuning data.
Referenced source
OpenAI EvalsEvaluation frameworknpx reporadars add evaluation-frameworksCLI coming soon — preview only.
Draft. Independent runtime review pending.
Reference linked; draft instructions are original.
0.1.0, pinned in this draft.
Explicit permissions and limits below.
Claude Code, Codex, Cursor
Read anonymized examples. No external model calls, dataset publication, or paid runs without approval.
Capabilities
- Translate product failures into test cases
- Specify expected outputs and scoring rubrics
- Separate development and holdout examples
- Review privacy and data provenance
- Report uncertainty and failure categories
How it works
- Confirm the task, failure modes, and available examples.
- Write a rubric with observable success criteria.
- Design a bounded run and review ambiguous cases manually.
- Report results with limitations and reproducibility details.
This is an original Reporadars draft, informed by the linked source. Compatibility has not been independently runtime-tested; inspect the instructions, permissions, and placement for your agent before use.