Score answer quality, tool use, grounding, recovery, and safety as separate rubric dimensions.
The full prompt is available without an account.
Implement score_rubric(results, rubric) that groups test results by rubric category
and computes a pass rate for each category.
results is a list of dicts: {"name": str, "passed": bool}.
rubric is a dict mapping category names to lists of test names.
Return a dict {category_name: pass_rate} where pass_rate is the fraction of
tests in that category that passed (0.0 to 1.0). If a category has no matching
results, return 0.0 for that category.
Read-only Python 3.12 preview. Sign in to edit and execute it.
def score_rubric(results, rubric):
scores = {}
# Iterate rubric categories and compute pass rate for each.
# If no results match a category, score is 0.0.
return scores