This endpoint could be used by ADK Web to dynamically know:
- What are the available eval metrics in an App
- A description of those metrics
- A value range supported by those metrics
We also update the metric registry to make it mandatory to supply these details. The goal is to improve usability and interpretability of the eval metrics.
PiperOrigin-RevId: 787277695
This version of the EvalSetsManager is intended to support two main behaviors
1) The agent developer wants to bring in their own eval set file, which is usually the case with `adk eval` cli. Once their eval sets are uploaded into this version of the eval sets manager, the EvalSetManager could be handed over to the Eval system for running evals.
2) As a part of AgentEvaluator testing, we expect developers to supply Eval cases in json files. The in-memory version of the EvalSetsManager will help us run those test cases using LocalEvalService.
PiperOrigin-RevId: 783198788
This change:
- Introduces the LocalEvalService Class.
- Implements only the "perform_inference" method. Evaluate method will be implemented in the next CL.
- Adds required test coverage.
PiperOrigin-RevId: 781781954
We add a new metric for evaluating safety of Agent's response to ADK Eval. We delegate the actual implementation to Vertex Gen AI Eval SDK, so using this metric will require GCP project.
As a part of this change, we created (refactored) a simple Facade for vertex gen ai eval sdk.
PiperOrigin-RevId: 778580406
