Score a proposed agent architecture against a comprehensive rubric covering orchestration, tools, memory, safety, evals, and observability.
Full problem description visible. Upgrade to unlock the editor, test cases, and solution.
Implement evaluate_design(proposal, rubric) that scores an agent system design against a structured rubric.
proposal is a dict mapping category names to dicts of design decisions:
{"orchestration": {"framework": str, "loop_type": str, ...}, "tools": {...}, ...}
rubric is a dict mapping category names to lists of criteria. Each criterion is a dict:
- key (str): the design decision key to check in proposal[category].
- label (str): human-readable criterion name.
- max_points (int): maximum score if present.
- weight (float, optional): importance multiplier (default 1.0).
Requirements:
- For each category in rubric, check that the category exists in proposal and each criterion key is present.
- Score each present criterion at its max_points (weighted). Missing criteria score 0.
- Identify missing failure handling: if a category exists but lacks any "fallback", "retry", or "error" key, note it.
- Prioritize feedback: missing components first, then underdeveloped categories (< 50% of max).
- Return {"total_score": float, "max_score": float, "percentage": float, "feedback": list[str], "details": dict}.
₹999/monthLimited period launch pricing.
Get the full problem statement, test cases, interactive editor, solution explanation, and visual diagram.
def evaluate_design(proposal, rubric):
total_score = 0.0
max_score = 0.0
feedback = []
details = {}Score each rubric criterion, detect missing failure handling
priority = sorted(feedback, key=lambda x: 0 if x.startswith("Missing") else 1)
return {"total_score": total_score, "max_score": max_score,
"percentage": round(total_score / max_score * 100, 1) if max_score > 0 else 0,
"feedback": priority, "details": details}Already have Pro access? Sign in