With this change we ensure that all three eval entry points, web, cli and pytest use the common LocalEvalService.
Updates to web and cli happened in a previous change.
PiperOrigin-RevId: 786445632
We update both adk web run eval endpoint and adk eval cli to use the LocalService. The old method is marked as deprecated and will be removed in later PRs.
PiperOrigin-RevId: 785612708
This version of the EvalSetsManager is intended to support two main behaviors
1) The agent developer wants to bring in their own eval set file, which is usually the case with `adk eval` cli. Once their eval sets are uploaded into this version of the eval sets manager, the EvalSetManager could be handed over to the Eval system for running evals.
2) As a part of AgentEvaluator testing, we expect developers to supply Eval cases in json files. The in-memory version of the EvalSetsManager will help us run those test cases using LocalEvalService.
PiperOrigin-RevId: 783198788
This change:
- Introduces the LocalEvalService Class.
- Implements only the "perform_inference" method. Evaluate method will be implemented in the next CL.
- Adds required test coverage.
PiperOrigin-RevId: 781781954
We add a new metric for evaluating safety of Agent's response to ADK Eval. We delegate the actual implementation to Vertex Gen AI Eval SDK, so using this metric will require GCP project.
As a part of this change, we created (refactored) a simple Facade for vertex gen ai eval sdk.
PiperOrigin-RevId: 778580406
Additionally, few other small changes.
* Updated a test fixture to support the latest eval data schema. Somehow I missed doing that previously.
* Updated the `evaluation_generator.py` to use `run_async`, instead of `run`.
* Also, raise an informed error when dependencies required eval are not installed.
* Also, changed the behavior of AgentEvaluator.evaluate method to run all the evals, instead of failing at the first eval metric failure.
PiperOrigin-RevId: 775919127
Copybara import of the project:
--
6596c477c7138df79d77e67181fd5979fd1a72e3 by Garrett Bischof <bischofgarrett@gmail.com>:
docs: change eval_dataset to eval_dataset_file_path_or_dir
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/adk-python/pull/397 from gwbischof:eval_dataset e8a0d0911c21ac76dab83e8520aaebc38d24adf5
PiperOrigin-RevId: 765638775
