In order to make evals more resilient to temporary model failures, we add retry options to llm_requests that are made during evals.
Note that this is a fall back option, if the developer has already specified their own retry options, then those will be honored.
Co-authored-by: Ankur Sharma <ankusharma@google.com>
PiperOrigin-RevId: 832383755
ADK already has a set of metrics that don't rely expected_invocations. Also, for eval cases with conversation scenario, this would be the main line case.
PiperOrigin-RevId: 825101481
Details:
- Adds the `LlmBackedUserSimulator` which uses an LLM to generate user prompts until it decides that the conversation is complete.
- Adds unit tests for the new functionality.
PiperOrigin-RevId: 823557910
Details:
- Adds the `StaticUserSimulator` which implements the current functionality of supplying a fixed set of user prompts for an EvalCase.
- Adds the `UserSimulatorProvider` which determines the type of user simulator required for an EvalCase (StaticUserSimulator or LlmBackedUserSimulator).
- Integrates the UserSimulatorProvider and UserSimulator into the CLI and evaluation infrastructure.
- Updates and adds unit tests for the new functionality.
- Miscellaneous updates to lay groundwork for a full implementation of the LlmBackedUserSimulator in the future.
PiperOrigin-RevId: 822198401
This change removes the `evaluate`, `_evaluate_row`, `are_tools_equal`, `_remove_tool_outputs`, `_report_failures`, and `_print_results` static methods from `TrajectoryEvaluator`, along with their corresponding unit tests. These methods were previously marked as deprecated.
PiperOrigin-RevId: 817477494
We updated the one of the public methods on AgentEvaluator to take in eval metric configurations using a more formal EvalConfig data model.
We also mark "criteria" field on the method as deprecated.
Updated some integration test cases.
PiperOrigin-RevId: 814314134
The PR does two main things:
1) Introduces a new rubric based tool use metric
2) Given that we now have two rubric based metric, we refactor and create a new RubricBasedEvaluator interface.
PiperOrigin-RevId: 811983514
AppDetails require two pieces of information:
1) Instructions
2) Tools
Both these pieces of information are gathered using the llm_request that was passed to the model. This approach, slightly invasive, ensures that we capture the "exact" instructions and tools that were given to the model.
PiperOrigin-RevId: 811180648
Details:
1. Data model for storing App Details (the agentic system)
As we move towards LLM as Judge metrics, we see that some of these metrics need information about the Agentic system that was used for inferencing. We add a data model to capture that.
2. Data model for Steps
We refine the concept of intermediate data. Previously it stored data in the form of a multiple lists, thereby losing out on the chronological information. This information is needed for some of the metrics. So we refine the concept of intermediate data as series of logical steps that an Agent Take.
PiperOrigin-RevId: 811122784
Details:
- We plan on introducing Rubric based metrics in subsequent changes. This change introduces the data model needed that allows agent developer to provide rubrics.
- We also introduce a data model for the config that the eval system has been using for quite some time. It was loosely and informally described as a dictionary of metric names and expected thresholds. In this change, we actually formalize it using a pydantic data model, and extend it allow developers to specify rubrics as a part of their eval config.
What is a rubric based metric?
A rubric based metric is the assessment of a Agent's response (final or intermediate) along some rubric. This evaluation of agent's response significantly differs from the strategy where one has to provide a golden response.
PiperOrigin-RevId: 805488436
This endpoint could be used by ADK Web to dynamically know:
- What are the available eval metrics in an App
- A description of those metrics
- A value range supported by those metrics
We also update the metric registry to make it mandatory to supply these details. The goal is to improve usability and interpretability of the eval metrics.
PiperOrigin-RevId: 787277695
