feat: Migrate BigQuery logging to Storage Write API

This commit refactors the `BigQueryAgentAnalyticsPlugin` to leverage the modern BigQuery Storage Write API, replacing the previous implementation that used the legacy `insert_rows_json` method (based on `tabledata.insertAll`).

**Key Changes:**

*   **Switched to Storage Write API:** Event logs are now ingested using the `BigQueryWriteClient` from the `google-cloud-bigquery-storage` library.
*   **Utilizes Default Stream:** We are using the `_default` stream for sending data, which is an efficient method for streaming in data without needing to manage stream lifecycles. This is ideal for continuous event logging.
*   **Apache Arrow Format:** Log entries are converted to the Apache Arrow format using `pyarrow` before being sent. The BigQuery table schema is dynamically converted to an Arrow schema. This binary format is more efficient than JSON.
*   **Updated Initialization:** The plugin now initializes both the standard `bigquery.Client` (for table management) and the `BigQueryWriteClient`.
*   **Test Updates:** Unit tests in `test_bigquery_logging_plugin.py` have been comprehensively updated to mock the new `BigQueryWriteClient`, `bq_schema_utils`, and `pyarrow` components. Tests now verify calls to `append_rows` and the data structure passed to create the Arrow RecordBatch.

**Benefits of this change:**

*   **Improved Performance:** The Storage Write API is designed for high-throughput streaming and offers better performance compared to the legacy API.
*   **Reduced Cost:** Ingesting data via the Storage Write API is generally more cost-effective.
*   **Enhanced Reliability:** The Storage Write API provides more robust streaming capabilities.
*   **Modernization:** Aligns the plugin with the recommended best practices for BigQuery data ingestion.

This change enhances the efficiency and scalability of the BigQuery logging plugin.

PiperOrigin-RevId: 828655496

pyproject.toml

@@ -32,6 +32,8 @@ dependencies = [
   "fastapi>=0.115.0, <1.119.0",                              # FastAPI framework
   "google-api-python-client>=2.157.0, <3.0.0",               # Google API client discovery
   "google-cloud-aiplatform[agent_engines]>=1.125.0, <2.0.0", # For VertexAI integrations, e.g. example store.
+  "google-cloud-bigquery-storage>=2.0.0",
+  "google-cloud-bigquery>=2.2.0",
   "google-cloud-bigtable>=2.32.0",                           # For Bigtable database
   "google-cloud-discoveryengine>=0.13.12, <0.14.0",          # For Discovery Engine Search Tool
   "google-cloud-secret-manager>=2.22.0, <3.0.0",             # Fetching secrets in RestAPI Tool
@@ -48,6 +50,7 @@ dependencies = [
   "opentelemetry-exporter-otlp-proto-http>=1.36.0",
   "opentelemetry-resourcedetector-gcp>=1.9.0a0, <2.0.0",
   "opentelemetry-sdk>=1.37.0, <=1.37.0",
+  "pyarrow>=14.0.0",
   "pydantic>=2.0, <3.0.0",                                   # For data validation/models
   "python-dateutil>=2.9.0.post0, <3.0.0",                    # For Vertext AI Session Service
   "python-dotenv>=1.0.0, <2.0.0",                            # To manage environment variables
@@ -206,7 +209,7 @@ asyncio_mode = "auto"
 python_version = "3.9"
 exclude = "tests/"
 plugins = ["pydantic.mypy"]
-# Start with non-strict mode, and switch to strict mode later.
+# Start with non-strict mode, and swtich to strict mode later.
 # strict = true
 disable_error_code = ["import-not-found", "import-untyped", "unused-ignore"]
 follow_imports = "skip"