This CL fixes several bugs in the BigQuery Agent Analytics plugin and refactors the internal data-passing pattern for better type safety and maintainability.
- **Stale Loop State Validation:** Use `loop.is_closed()` — a public, reliable API — to detect and clean up stale asyncio loop states in `_batch_processor_prop`, `_get_loop_state`, and `flush`. The previous approach used `asyncio.Queue._loop` which is `None` on Python 3.10+, causing the check to always treat states as stale.
- **Quota Project ID Fallback:** Remove the `or project_id` fallback when setting `quota_project_id` on `BigQueryWriteAsyncClient`. This fixes Workload Identity Federation flows where the federated identity lacks `serviceusage.services.use` on the quota project.
- **Kwargs Passthrough:** Pass `**kwargs` through to `_log_event` in all callbacks. Previously only model callbacks forwarded them, causing custom attributes (e.g. `customer_id`) to silently drop for agent, tool, run, and error events.
- **State Delta Logging:** Replace the dead `on_state_change_callback` (never invoked by the framework) with `on_event_callback`, which is already dispatched by the runner for every event. Remove duplicate `STATE_DELTA` logging from `after_tool_callback`.
- **EventData Dataclass:** Replace the `**kwargs`-as-data-bus pattern in `_log_event` with an explicit `EventData` dataclass. This makes the interface self-documenting, catches typos at construction time, and eliminates shared dict mutation across `_resolve_span_ids`, `_extract_latency`, and `_enrich_attributes`. All 12 callback call sites now construct typed `EventData` instances.
- **Multi-Subagent Tool Logging Tests:** Add `TestMultiSubagentToolLogging` (6 tests) verifying that tool events are correctly attributed to subagents in multi-turn, multi-agent scenarios. Total tests: 111 (up from 60).
Co-authored-by: Haiyuan Cao <haiyuan@google.com>
PiperOrigin-RevId: 871381533
The definition of an "invocation" for context filtering has been updated. An invocation now starts with a user message and can include multiple model turns (like the tool calls and responses) until the next user message. The filtering logic has been rewritten to identify invocation start points based on human user messages
Close#4296
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 866023290
This CL enhances asyncio event loop management and test isolation.
- **BigQuery Analytics Plugin:** Ensure the asyncio event loop is consistently closed within the BigQuery analytics plugin. This prevents potential resource leaks. Add checks to handle potential deadlocks in Python 3.13+ when creating loops during interpreter shutdown.
- **Test Thread Pool Cleanup:** Introduce a pytest fixture (`cleanup_thread_pools`) to automatically shut down and clear all tool-related thread pools after each test run in `test_functions_thread_pool.py`. This improves test isolation and prevents order-dependent test failures.
- **Streaming Test Loop Restoration:** Refactor event loop handling in `test_streaming.py`. A new `_run_with_loop` method is introduced in the custom test runners to create a temporary event loop for each test execution, run the coroutine, and crucially, restore the original event loop afterwards. This prevents tests from interfering with each other's loop state.
- **Resource Closure:** Ensure services are closed properly in tests by adding `await service.close()` in `test_service_factory.py` and using `async with session_service` in `test_session_service.py`.
PiperOrigin-RevId: 863305565
Improves test reliability by guaranteeing the BigQueryAgentAnalyticsPlugin is shut down after each test execution. This is achieved by wrapping test logic in try/finally blocks, using a yield fixture, or employing a new asynccontextmanager to ensure the plugin.shutdown() method is always called, preventing potential resource leaks between tests.
PiperOrigin-RevId: 862951816
This change introduces a `flush` method to the `BigQueryAgentAnalyticsPlugin`. This ensures that all pending log events are written to BigQuery before the agent's run completes.
Key changes:
- Added `flush()` method to `BigQueryAgentAnalyticsPlugin` to force write of pending events.
PiperOrigin-RevId: 859263853
This refactors the BigQueryAgentAnalyticsPlugin to use the standard OpenTelemetry API for trace and span ID generation and propagation, replacing the custom ContextVar implementation.
Key changes:
- Utilizes `opentelemetry.trace` for starting/ending spans.
- Correctly uses `opentelemetry.context` for context attachment and detachment.
- Span information is now derived from the OpenTelemetry context when available.
- Added a fallback mechanism to ensure span_id and parent_span_id are still populated if the OpenTelemetry SDK is not initialized.
To get standard OpenTelemetry trace information in BigQuery logs, users should install `opentelemetry-sdk` and initialize a global `TracerProvider` in their application *before* initializing ADK components.
Example minimal initialization:
```python
# Install: pip install opentelemetry-sdk
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
trace.set_tracer_provider(TracerProvider())
```
PiperOrigin-RevId: 858965562
* **Async Safety:** Improved TraceManager context variable handling to ensure correct context isolation in concurrent asynchronous operations. This was achieved by using immutable tuples for the span stack and making copies of context dictionaries before modification.
* **Enhanced Logging:** The BigQueryAgentAnalyticsPlugin now captures richer metadata, including:
* Root agent name (via a new context variable).
* LLM model name and version.
* Usage metadata from LLM requests and responses.
* **Serialization Fix:** Updated BigQueryAgentAnalyticsPlugin to prevent JSON serialization errors when logging custom objects (e.g., Dataclasses). These are now automatically converted to dictionaries or string representations to ensure successful insertion into BigQuery.
PiperOrigin-RevId: 855415320
Fix discovery engine search tool, bigquery agent analytics plugin, and application integration tool to correctly handle the ADC quota project override -- the x-goog-user-project should be set based on the ADC quota project, per gcloud auth team's requirements.
Co-authored-by: Kathy Wu <wukathy@google.com>
PiperOrigin-RevId: 853841124
When truncating conversation history, make sure function_response messages always have their corresponding function_call included
Close#4027
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 852377919
Performance: Switched to BigQuery Storage Write API with async batching, reducing agent latency.
Multimodal: Native support for GCS offloading (ObjectRef) for images, video, and large text.
Reliability: Added connection pooling, retries, and a "rescue flush" for safe shutdown on Cloud Run.
Observability: Fixed distributed tracing hierarchy with ContextVars support.
PiperOrigin-RevId: 843062561
This update enhances the BigQuery agent analytics plugin:
* **Schema Field Descriptions:** The recommended BigQuery table schema now includes descriptions for each field, improving data understandability.
* **Optimized Table Structure:** The plugin now creates the table with daily partitioning on `timestamp` and clustering on `event_type`, `agent`, and `user_id` by default.
* **Truncation Flag:** A new boolean field `is_truncated` is added to the schema to show if the `content` was truncated.
PiperOrigin-RevId: 832436799
This update improves the `BigQueryAgentAnalyticsPlugin` in several ways:
* Corrects the PyArrow schema generation to accurately reflect BigQuery field nullability based on the `mode` attribute.
* Introduces a configurable `shutdown_timeout` in `BigQueryLoggerConfig` to manage how long the plugin waits for pending logs to flush during shutdown.
* Adds more robust error handling within the `shutdown` method and background write tasks, particularly for event loop closure issues.
* Improves internal logging to provide better diagnostics.
* Ensures consistent use of safe content formatting.
PiperOrigin-RevId: 831225837
This change introduces a shutdown lifecycle hook for plugins. The `PluginManager` now has an `async def shutdown()` method that will call `await plugin.shutdown()` on any registered plugins that implement the method. This is called from `Runner.close()`, allowing plugins to perform cleanup tasks like flushing logs or closing connections when the runner instance is being closed. This improves the reliability of plugins that perform background operations.
PiperOrigin-RevId: 831037737
This change updates the `BigQueryAgentAnalyticsPlugin` (formerly in `bigquery_logging_plugin.py`) to perform BigQuery writes asynchronously in background tasks, preventing blocking of the main agent execution flow. Initialization of BigQuery clients and table creation is also made asynchronous. The content logged in various callbacks has been streamlined and simplified. The plugin file has been renamed to `bigquery_agent_analytics_plugin.py` to match the class name.
PiperOrigin-RevId: 829603260
SaveFilesAsArtifactsPlugin now resolves the canonical URI for each saved artifact. When the URI is model-accessible (`gs://`, `https://`, `http://`), we add a `Part(file_data=...)` so the LLM can fetch the filedirectly while still emitting the placeholder. If no model-accessible URI exists, we retain the original inline blob alongside the placeholder to preserve access to the uploaded bytes.
Close#2016
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 828786519
This commit refactors the `BigQueryAgentAnalyticsPlugin` to leverage the modern BigQuery Storage Write API, replacing the previous implementation that used the legacy `insert_rows_json` method (based on `tabledata.insertAll`).
**Key Changes:**
* **Switched to Storage Write API:** Event logs are now ingested using the `BigQueryWriteClient` from the `google-cloud-bigquery-storage` library.
* **Utilizes Default Stream:** We are using the `_default` stream for sending data, which is an efficient method for streaming in data without needing to manage stream lifecycles. This is ideal for continuous event logging.
* **Apache Arrow Format:** Log entries are converted to the Apache Arrow format using `pyarrow` before being sent. The BigQuery table schema is dynamically converted to an Arrow schema. This binary format is more efficient than JSON.
* **Updated Initialization:** The plugin now initializes both the standard `bigquery.Client` (for table management) and the `BigQueryWriteClient`.
* **Test Updates:** Unit tests in `test_bigquery_logging_plugin.py` have been comprehensively updated to mock the new `BigQueryWriteClient`, `bq_schema_utils`, and `pyarrow` components. Tests now verify calls to `append_rows` and the data structure passed to create the Arrow RecordBatch.
**Benefits of this change:**
* **Improved Performance:** The Storage Write API is designed for high-throughput streaming and offers better performance compared to the legacy API.
* **Reduced Cost:** Ingesting data via the Storage Write API is generally more cost-effective.
* **Enhanced Reliability:** The Storage Write API provides more robust streaming capabilities.
* **Modernization:** Aligns the plugin with the recommended best practices for BigQuery data ingestion.
This change enhances the efficiency and scalability of the BigQuery logging plugin.
PiperOrigin-RevId: 828655496
This change introduces BigQueryLoggerConfig to allow customization of the BigQueryAgentAnalyticsPlugin. Users can now enable/disable the plugin, specify event type allowlists and denylists, and provide a custom function to format or redact the content field before logging to BigQuery. The content logged for model and tool errors has also been enhanced.
PiperOrigin-RevId: 828172241
Merge #3163
END_PUBLIC
Hello,
Since global_instruction has been deprecated, I’m migrating to GlobalInstructionPlugin.
During the migration, I encountered an error and am submitting this PR to fix it.
In [df05ed6](https://github.com/google/adk-python/commit/df05ed6b3b7b218d85fddc1acd6617802cdf6f2a) ,
GlobalInstructionPlugin references invocation_context, but CallbackContext actually contains _invocation_context.
This mismatch always causes an error during execution.
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/adk-python/pull/3180 from UlookEE:fix_global_instruction_plugin e289a12d69812f0abcfe77db0114fdb2045b31bc
PiperOrigin-RevId: 827682501
