The `model_dump_json()` method already returns a JSON string, so wrapping it in `json.dumps()` was causing double encoding
Close#3993
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 852455534
Adds a Pydantic field validator to ToolTrajectoryCriterion to automatically convert string inputs for the match_type field into the corresponding MatchType enum member
Close#3711
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 852415560
Details:
- Allows users to provide custom instructions for the LLM-backed user simulator via the `custom_instructions` field in `LlmBackedUserSimulatorConfig`.
- The custom instructions must include placeholders for the stop signal, conversation plan, and conversation history. A pydantic validator ensures these placeholders are present.
- If no custom instructions are provided, the current default template is used.
Co-authored-by: Keyur Joshi <keyurj@google.com>
PiperOrigin-RevId: 850471448
The evaluate_invocations method override in Evaluator subclasses was not consistent, leading to errors during calls, especially when using kwargs. Made the overrides and calls consistent to resolve this issue.
Co-authored-by: Keyur Joshi <keyurj@google.com>
PiperOrigin-RevId: 850462752
In order to make evals more resilient to temporary model failures, we add retry options to llm_requests that are made during evals.
Note that this is a fall back option, if the developer has already specified their own retry options, then those will be honored.
Co-authored-by: Ankur Sharma <ankusharma@google.com>
PiperOrigin-RevId: 832383755
ADK already has a set of metrics that don't rely expected_invocations. Also, for eval cases with conversation scenario, this would be the main line case.
PiperOrigin-RevId: 825101481
Details:
- Adds the `LlmBackedUserSimulator` which uses an LLM to generate user prompts until it decides that the conversation is complete.
- Adds unit tests for the new functionality.
PiperOrigin-RevId: 823557910
Details:
- Adds the `StaticUserSimulator` which implements the current functionality of supplying a fixed set of user prompts for an EvalCase.
- Adds the `UserSimulatorProvider` which determines the type of user simulator required for an EvalCase (StaticUserSimulator or LlmBackedUserSimulator).
- Integrates the UserSimulatorProvider and UserSimulator into the CLI and evaluation infrastructure.
- Updates and adds unit tests for the new functionality.
- Miscellaneous updates to lay groundwork for a full implementation of the LlmBackedUserSimulator in the future.
PiperOrigin-RevId: 822198401
This change removes the `evaluate`, `_evaluate_row`, `are_tools_equal`, `_remove_tool_outputs`, `_report_failures`, and `_print_results` static methods from `TrajectoryEvaluator`, along with their corresponding unit tests. These methods were previously marked as deprecated.
PiperOrigin-RevId: 817477494
We updated the one of the public methods on AgentEvaluator to take in eval metric configurations using a more formal EvalConfig data model.
We also mark "criteria" field on the method as deprecated.
Updated some integration test cases.
PiperOrigin-RevId: 814314134
The PR does two main things:
1) Introduces a new rubric based tool use metric
2) Given that we now have two rubric based metric, we refactor and create a new RubricBasedEvaluator interface.
PiperOrigin-RevId: 811983514
