In order to make evals more resilient to temporary model failures, we add retry options to llm_requests that are made during evals.
Note that this is a fall back option, if the developer has already specified their own retry options, then those will be honored.
Co-authored-by: Ankur Sharma <ankusharma@google.com>
PiperOrigin-RevId: 832383755
Details:
- Adds the `StaticUserSimulator` which implements the current functionality of supplying a fixed set of user prompts for an EvalCase.
- Adds the `UserSimulatorProvider` which determines the type of user simulator required for an EvalCase (StaticUserSimulator or LlmBackedUserSimulator).
- Integrates the UserSimulatorProvider and UserSimulator into the CLI and evaluation infrastructure.
- Updates and adds unit tests for the new functionality.
- Miscellaneous updates to lay groundwork for a full implementation of the LlmBackedUserSimulator in the future.
PiperOrigin-RevId: 822198401
AppDetails require two pieces of information:
1) Instructions
2) Tools
Both these pieces of information are gathered using the llm_request that was passed to the model. This approach, slightly invasive, ensures that we capture the "exact" instructions and tools that were given to the model.
PiperOrigin-RevId: 811180648
Details:
1. Data model for storing App Details (the agentic system)
As we move towards LLM as Judge metrics, we see that some of these metrics need information about the Agentic system that was used for inferencing. We add a data model to capture that.
2. Data model for Steps
We refine the concept of intermediate data. Previously it stored data in the form of a multiple lists, thereby losing out on the chronological information. This information is needed for some of the metrics. So we refine the concept of intermediate data as series of logical steps that an Agent Take.
PiperOrigin-RevId: 811122784
This change:
- Introduces the LocalEvalService Class.
- Implements only the "perform_inference" method. Evaluate method will be implemented in the next CL.
- Adds required test coverage.
PiperOrigin-RevId: 781781954
Additionally, few other small changes.
* Updated a test fixture to support the latest eval data schema. Somehow I missed doing that previously.
* Updated the `evaluation_generator.py` to use `run_async`, instead of `run`.
* Also, raise an informed error when dependencies required eval are not installed.
* Also, changed the behavior of AgentEvaluator.evaluate method to run all the evals, instead of failing at the first eval metric failure.
PiperOrigin-RevId: 775919127
* Fix typos in docstrings of evaluation_generator.py and event.py (#101)
Corrected 'resposnes' to 'responses', 'uncertainity' to 'uncertainty', 'conversaction' to 'conversation', and 'exeuction' to 'execution' in relevant docstrings for clarity.
* Fix typos in docstrings and comments across multiple files
Corrected 'detla' to 'delta', 'buil-in' to 'built-in', 'walkaround' to 'workaround', and 'conversaction' to 'conversation' for clarity in relevant files. Updated comments for consistency.
---------
Co-authored-by: Hangfei Lin <hangfei@google.com>
