In order to make evals more resilient to temporary model failures, we add retry options to llm_requests that are made during evals.
Note that this is a fall back option, if the developer has already specified their own retry options, then those will be honored.
Co-authored-by: Ankur Sharma <ankusharma@google.com>
PiperOrigin-RevId: 832383755
It is common for expected response and expected tool calls column to be empty for user simulated conversations. So, we don't render those.
Co-authored-by: Ankur Sharma <ankusharma@google.com>
PiperOrigin-RevId: 826202867
Session input file contains fields that are needed to run evals and later be able retrieve the session generated by them.
Co-authored-by: Ankur Sharma <ankusharma@google.com>
PiperOrigin-RevId: 825742522
ADK already has a set of metrics that don't rely expected_invocations. Also, for eval cases with conversation scenario, this would be the main line case.
PiperOrigin-RevId: 825101481
Right now the failure eats up the traceback information and there is no clear way for the developer to know what went wrong. Adding this traceback info could give them the needed debugging information.
PiperOrigin-RevId: 823179833
This change removes the `convert_session_to_eval_format` function and its associated unit tests. New tests for `create_gcs_eval_managers_from_uri` are also added.
PiperOrigin-RevId: 819576620
This change removes the `run_evals` function and its helper `_get_evaluator` from `cli_eval.py`, as they were marked as deprecated. Corresponding test mocks and patches in `test_fast_api.py` are also removed.
PiperOrigin-RevId: 818719422
This change removes the `evaluate`, `_evaluate_row`, `are_tools_equal`, `_remove_tool_outputs`, `_report_failures`, and `_print_results` static methods from `TrajectoryEvaluator`, along with their corresponding unit tests. These methods were previously marked as deprecated.
PiperOrigin-RevId: 817477494
Agent developers can now create an eval set and add eval cases through command line itself. Adding an eval case is limited only to specifying conversation scenarios.
Sample comamnds:
- Create an eval set:
adk eval_set create \
contributing/samples/hello_world \
set_01
- Add an eval case with scenario file
Content of scenarios.json file:
'{"scenarios": [{"starting_prompt": "hello", "conversation_plan": "world"}]}'
adk eval_set add_eval_case \
contributing/samples/hello_world \
set_01 \
--scenarios scenarios.json
PiperOrigin-RevId: 817456117
Details:
- Introduces a concept of `ConversationScenario` to represent a scenario that user simulator is supposed to follow.
- Introduces a `UserSimulator` interface, that one should implement. UserSimulator interface will be integrated with LocalEvalService in subsequent PRs.
PiperOrigin-RevId: 816883699
We updated the one of the public methods on AgentEvaluator to take in eval metric configurations using a more formal EvalConfig data model.
We also mark "criteria" field on the method as deprecated.
Updated some integration test cases.
PiperOrigin-RevId: 814314134
The PR does two main things:
1) Introduces a new rubric based tool use metric
2) Given that we now have two rubric based metric, we refactor and create a new RubricBasedEvaluator interface.
PiperOrigin-RevId: 811983514
AppDetails require two pieces of information:
1) Instructions
2) Tools
Both these pieces of information are gathered using the llm_request that was passed to the model. This approach, slightly invasive, ensures that we capture the "exact" instructions and tools that were given to the model.
PiperOrigin-RevId: 811180648
Details:
1. Data model for storing App Details (the agentic system)
As we move towards LLM as Judge metrics, we see that some of these metrics need information about the Agentic system that was used for inferencing. We add a data model to capture that.
2. Data model for Steps
We refine the concept of intermediate data. Previously it stored data in the form of a multiple lists, thereby losing out on the chronological information. This information is needed for some of the metrics. So we refine the concept of intermediate data as series of logical steps that an Agent Take.
PiperOrigin-RevId: 811122784
For advanced eval use cases, we do expect agent developers to have rubrics that are specific to an Eval Case and in some cases even specific to a single invocation/turn in the eval case conversation.
A separate PR will be created to consume this data model changes in ADK Eval.
PiperOrigin-RevId: 805588808
Details:
- We plan on introducing Rubric based metrics in subsequent changes. This change introduces the data model needed that allows agent developer to provide rubrics.
- We also introduce a data model for the config that the eval system has been using for quite some time. It was loosely and informally described as a dictionary of metric names and expected thresholds. In this change, we actually formalize it using a pydantic data model, and extend it allow developers to specify rubrics as a part of their eval config.
What is a rubric based metric?
A rubric based metric is the assessment of a Agent's response (final or intermediate) along some rubric. This evaluation of agent's response significantly differs from the strategy where one has to provide a golden response.
PiperOrigin-RevId: 805488436
Due to reasons that are being investigated, some of the recent changes got unintentionally reverted. We are adding those back in this PR.
PiperOrigin-RevId: 789384063
This endpoint could be used by ADK Web to dynamically know:
- What are the available eval metrics in an App
- A description of those metrics
- A value range supported by those metrics
We also update the metric registry to make it mandatory to supply these details. The goal is to improve usability and interpretability of the eval metrics.
PiperOrigin-RevId: 787277695
With this change we ensure that all three eval entry points, web, cli and pytest use the common LocalEvalService.
Updates to web and cli happened in a previous change.
PiperOrigin-RevId: 786445632
Please set --log_level to DEBUG, if you are interested in having those API request and responses in logs.
NOTE: Generally it is not recommended to have DEBUG log level for services that run in a production setting. It is our recommendation to only use DEBUG log level in a debug or development setting.
PiperOrigin-RevId: 785972338
We update both adk web run eval endpoint and adk eval cli to use the LocalService. The old method is marked as deprecated and will be removed in later PRs.
PiperOrigin-RevId: 785612708
This version of the EvalSetsManager is intended to support two main behaviors
1) The agent developer wants to bring in their own eval set file, which is usually the case with `adk eval` cli. Once their eval sets are uploaded into this version of the eval sets manager, the EvalSetManager could be handed over to the Eval system for running evals.
2) As a part of AgentEvaluator testing, we expect developers to supply Eval cases in json files. The in-memory version of the EvalSetsManager will help us run those test cases using LocalEvalService.
PiperOrigin-RevId: 783198788
This change:
- Introduces the LocalEvalService Class.
- Implements only the "perform_inference" method. Evaluate method will be implemented in the next CL.
- Adds required test coverage.
PiperOrigin-RevId: 781781954
We add a new metric for evaluating safety of Agent's response to ADK Eval. We delegate the actual implementation to Vertex Gen AI Eval SDK, so using this metric will require GCP project.
As a part of this change, we created (refactored) a simple Facade for vertex gen ai eval sdk.
PiperOrigin-RevId: 778580406
Additionally, few other small changes.
* Updated a test fixture to support the latest eval data schema. Somehow I missed doing that previously.
* Updated the `evaluation_generator.py` to use `run_async`, instead of `run`.
* Also, raise an informed error when dependencies required eval are not installed.
* Also, changed the behavior of AgentEvaluator.evaluate method to run all the evals, instead of failing at the first eval metric failure.
PiperOrigin-RevId: 775919127
