fix: Adding detailed information on each metric evaluation

Additionally, few other small changes.
*   Updated a test fixture to support the latest eval data schema. Somehow I missed doing that previously.
*   Updated the `evaluation_generator.py` to use `run_async`, instead of `run`.
*   Also, raise an informed error when dependencies required eval are not installed.
*   Also, changed the behavior of AgentEvaluator.evaluate method to run all the evals, instead of failing at the first eval metric failure.

PiperOrigin-RevId: 775919127
This commit is contained in:
Ankur Sharma
2025-06-25 18:32:02 -07:00
committed by Copybara-Service
parent 3901fade71
commit 04de3e197d
7 changed files with 240 additions and 49 deletions
@@ -182,7 +182,7 @@ class EvaluationGenerator:
tool_uses = []
invocation_id = ""
for event in runner.run(
async for event in runner.run_async(
user_id=user_id, session_id=session_id, new_message=user_content
):
invocation_id = (