Details:
- Adds the `StaticUserSimulator` which implements the current functionality of supplying a fixed set of user prompts for an EvalCase.
- Adds the `UserSimulatorProvider` which determines the type of user simulator required for an EvalCase (StaticUserSimulator or LlmBackedUserSimulator).
- Integrates the UserSimulatorProvider and UserSimulator into the CLI and evaluation infrastructure.
- Updates and adds unit tests for the new functionality.
- Miscellaneous updates to lay groundwork for a full implementation of the LlmBackedUserSimulator in the future.
PiperOrigin-RevId: 822198401
AppDetails require two pieces of information:
1) Instructions
2) Tools
Both these pieces of information are gathered using the llm_request that was passed to the model. This approach, slightly invasive, ensures that we capture the "exact" instructions and tools that were given to the model.
PiperOrigin-RevId: 811180648
Details:
1. Data model for storing App Details (the agentic system)
As we move towards LLM as Judge metrics, we see that some of these metrics need information about the Agentic system that was used for inferencing. We add a data model to capture that.
2. Data model for Steps
We refine the concept of intermediate data. Previously it stored data in the form of a multiple lists, thereby losing out on the chronological information. This information is needed for some of the metrics. So we refine the concept of intermediate data as series of logical steps that an Agent Take.
PiperOrigin-RevId: 811122784
Google Team MemberandCopybara-Service
Ankur SharmaandCopybara-Service