The evaluate_invocations method override in Evaluator subclasses was not consistent, leading to errors during calls, especially when using kwargs. Made the overrides and calls consistent to resolve this issue.
Co-authored-by: Keyur Joshi <keyurj@google.com>
PiperOrigin-RevId: 850462752
ADK already has a set of metrics that don't rely expected_invocations. Also, for eval cases with conversation scenario, this would be the main line case.
PiperOrigin-RevId: 825101481
Details:
- Adds the `StaticUserSimulator` which implements the current functionality of supplying a fixed set of user prompts for an EvalCase.
- Adds the `UserSimulatorProvider` which determines the type of user simulator required for an EvalCase (StaticUserSimulator or LlmBackedUserSimulator).
- Integrates the UserSimulatorProvider and UserSimulator into the CLI and evaluation infrastructure.
- Updates and adds unit tests for the new functionality.
- Miscellaneous updates to lay groundwork for a full implementation of the LlmBackedUserSimulator in the future.
PiperOrigin-RevId: 822198401
Details:
1. Data model for storing App Details (the agentic system)
As we move towards LLM as Judge metrics, we see that some of these metrics need information about the Agentic system that was used for inferencing. We add a data model to capture that.
2. Data model for Steps
We refine the concept of intermediate data. Previously it stored data in the form of a multiple lists, thereby losing out on the chronological information. This information is needed for some of the metrics. So we refine the concept of intermediate data as series of logical steps that an Agent Take.
PiperOrigin-RevId: 811122784
This endpoint could be used by ADK Web to dynamically know:
- What are the available eval metrics in an App
- A description of those metrics
- A value range supported by those metrics
We also update the metric registry to make it mandatory to supply these details. The goal is to improve usability and interpretability of the eval metrics.
PiperOrigin-RevId: 787277695
This change:
- Introduces the LocalEvalService Class.
- Implements only the "perform_inference" method. Evaluate method will be implemented in the next CL.
- Adds required test coverage.
PiperOrigin-RevId: 781781954
