Using one lock and checking for tables creation instead of schema version.
Closes issue #4445
Co-authored-by: Liang Wu <wuliang@google.com>
PiperOrigin-RevId: 869808097
Merge https://github.com/google/adk-python/pull/4435
### Link to Issue or Description of Change
- Closes: #4302
**Problem:**
`VertexAiSessionService.list_sessions()` only returns the first ~100 sessions. The `sessions_iterator` from `api_client.agent_engines.sessions.list()` is an `AsyncPager` — it implements `__aiter__`/`__anext__` for fetching subsequent pages, but the code uses a plain `for` loop which only calls `__iter__`/`__next__`, so it never fetches beyond the first page.
**Solution:**
Changed `for api_session in sessions_iterator` to `async for api_session in sessions_iterator` so the `AsyncPager` actually paginates. Updated the test mock to return an `AsyncIterableList` (supports both sync and async iteration) instead of a bare list, so the tests properly simulate real `AsyncPager` behaviour.
### Testing Plan
**Unit Tests:**
```
$ pytest tests/unittests/sessions/
115 passed, 1 warning in 2.25s
```
The existing `test_list_sessions`, `test_list_sessions_with_pagination`, and `test_list_sessions_all_users` all continue to pass with the updated mock.
Co-authored-by: Liang Wu <wuliang@google.com>
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/adk-python/pull/4435 from anmolg1997:fix/vertex-ai-session-service-pagination 14c71b607ecbf2215f4b9ba6eb4b0ff6b9eaf740
PiperOrigin-RevId: 868466166
Merge https://github.com/google/adk-python/pull/4365
## Summary
- Fixes `DataError` when using PostgreSQL with `asyncpg` for session storage
- PostgreSQL's default `TIMESTAMP` type is `WITHOUT TIME ZONE`, which cannot accept timezone-aware datetime objects
- The existing code handled this for SQLite but not PostgreSQL - this fix applies the same timezone stripping
## Error
When creating a session with PostgreSQL + asyncpg, the following error occurs:
```
sqlalchemy.dialects.postgresql.asyncpg.Error: <class 'asyncpg.exceptions.DataError'>:
invalid input for query argument $5: datetime.datetime(2026, 2, 3, 21, 32, 50, 353909,
tzinfo=datetime.timezone.utc) (can't subtract offset-naive and offset-aware datetimes)
```
During the INSERT:
```sql
INSERT INTO sessions (app_name, user_id, id, state, create_time, update_time)
VALUES ($1, $2, $3, $4, $5, $6)
```
Where `$5` and `$6` are timezone-aware datetimes being inserted into `TIMESTAMP WITHOUT TIME ZONE` columns.
## Root Cause
Commit 1063fa53 changed from database-generated timestamps (`func.now()`) to explicit Python datetimes (`datetime.now(timezone.utc)`). The SQLite case was handled by stripping the timezone, but PostgreSQL was overlooked.
## Test plan
- [x] Verified fix resolves the error when creating sessions with PostgreSQL + asyncpg
- [ ] Existing unit tests pass
Fixes regression from #1733
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/adk-python/pull/4365 from filipecaixeta:fix-postgresql-timestamp-timezone 9d788ba99e7167a53962d93e59a80f78af091ca9
PiperOrigin-RevId: 867800330
This change introduces an in-process `asyncio.Lock` per session to serialize `append_event` calls for the same session ID within a single process. For supported database dialects (MySQL, PostgreSQL, MariaDB), it also uses `SELECT ... FOR UPDATE` to acquire row-level locks on the session, app state, and user state records, preventing race conditions across different processes or database connections. A new test case verifies that concurrent updates to stale session objects correctly merge all state changes.
Close#1049
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 867752676
This change sets `pool_pre_ping=True` in SQLAlchemy engine kwargs for database backends other than SQLite. This helps ensure that connections from the pool are still valid before being used, preventing issues with stale or disconnected connections. Tests are added to verify the default behavior and that explicit overrides are respected
Close#4211
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 864886767
This property is a compatibility alias that returns the update timestamp as a POSIX timestamp. It infers whether the database is SQLite using sqlalchemy.inspect to call get_update_timestamp correctly
Close#4334
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 864595914
This CL enhances asyncio event loop management and test isolation.
- **BigQuery Analytics Plugin:** Ensure the asyncio event loop is consistently closed within the BigQuery analytics plugin. This prevents potential resource leaks. Add checks to handle potential deadlocks in Python 3.13+ when creating loops during interpreter shutdown.
- **Test Thread Pool Cleanup:** Introduce a pytest fixture (`cleanup_thread_pools`) to automatically shut down and clear all tool-related thread pools after each test run in `test_functions_thread_pool.py`. This improves test isolation and prevents order-dependent test failures.
- **Streaming Test Loop Restoration:** Refactor event loop handling in `test_streaming.py`. A new `_run_with_loop` method is introduced in the custom test runners to create a temporary event loop for each test execution, run the coroutine, and crucially, restore the original event loop afterwards. This prevents tests from interfering with each other's loop state.
- **Resource Closure:** Ensure services are closed properly in tests by adding `await service.close()` in `test_service_factory.py` and using `async with session_service` in `test_session_service.py`.
PiperOrigin-RevId: 863305565
This changes the behavior of how we handle stale session, instead of reject the transaction entirely, we aggresively refresh the session.
Removed synchronous inspect(self).session calls within StorageSession.update_timestamp_tz and _dialect_name. This introspection was causing deadlocks/hangs when used with sqlalchemy.ext.asyncio in Python 3.13.
Closes issue: https://github.com/google/adk-python/issues/1733
Co-authored-by: Shangjie Chen <deanchen@google.com>
PiperOrigin-RevId: 861353058
The migration tool uses synchronous SQLAlchemy engines but users often provide async driver URLs (e.g., postgresql+asyncpg://) since that's what ADK requires at runtime.
This fix:
- Makes `to_sync_url()` public in `_schema_check_utils.py` for reuse
- Updates `migrate_from_sqlalchemy_pickle.py` to convert async URLs
- Updates `migrate_from_sqlalchemy_sqlite.py` to convert async URLs
- Adds comprehensive unit tests for `to_sync_url()` function
- Adds integration test for migration with async driver URLs
Fixes#4176
Co-authored-by: Liang Wu <wuliang@google.com>
PiperOrigin-RevId: 858359061
Merge https://github.com/google/adk-python/pull/3756
move event iteration inside api_client context in get_session
Move event iteration inside the api_client context manager in VertexAiSessionService.get_session() to prevent client closure during multi-page event fetching.
**Please ensure you have read the [contribution guide](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) before creating a pull request.**
### Link to Issue or Description of Change
**1. Link to an existing issue (if applicable):**
- Closes: #3757
**2. Or, if no issue exists, describe the change:**
**Problem:**
When a session contains more than 100 events (requiring pagination), `VertexAiSessionService.get_session()` fails with:
```
RuntimeError: Cannot send a request, as the client has been closed.
```
The root cause is that the `events_iterator` is consumed **outside** the `async with self._get_api_client() as api_client:` context block. When the iterator needs to fetch page 2, 3, etc., the API client has already been closed because the `async with` block has exited.
```python
# Current buggy flow:
async with self._get_api_client() as api_client:
get_session_response, events_iterator = await asyncio.gather(...)
# ← Client closed here
async for event in events_iterator: # ← Fails on page 2+ (client closed)
session.events.append(...)
```
**Solution:**
Move the session creation, user validation, and event iteration **inside** the `async with` block so the API client remains open during the entire pagination process:
```python
async with self._get_api_client() as api_client:
get_session_response, events_iterator = await asyncio.gather(...)
# Validation and session creation...
async for event in events_iterator: # ← Now works for all pages
session.events.append(...)
# Client closed after all events are fetched
```
### Testing Plan
**Unit Tests:**
- [x] I have added or updated unit tests for my change.
- [x] All unit tests pass locally.
```bash
pytest tests/unittests/sessions/test_vertex_ai_session_service.py -v
```
**Added regression test:** `test_get_session_pagination_keeps_client_open`
- Creates a `MockAsyncClientWithPagination` that tracks whether it's inside the `async with` context
- Raises `RuntimeError` if iteration happens outside the context (matching real httpx behavior)
- Simulates 3 pages of events (100 + 100 + 50 = 250 events)
- Verifies all 250 events are successfully retrieved
**Manual End-to-End (E2E) Tests:**
1. Deploy an ADK agent to Vertex AI Agent Engine
2. Create a session and send 100+ messages to accumulate >100 events
3. Verify `get_session()` successfully retrieves all events without error
**Before fix:**
```
RuntimeError: Cannot send a request, as the client has been closed.
```
**After fix:**
- Session with 201 events (3 pages) loads successfully
- All events are retrieved and appended to the session
### Checklist
- [x] I have read the [CONTRIBUTING.md](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) document.
- [x] I have performed a self-review of my own code.
- [x] I have commented my code, particularly in hard-to-understand areas.
- [x] I have added tests that prove my fix is effective or that my feature works.
- [x] New and existing unit tests pass locally with my changes.
- [x] I have manually tested my changes end-to-end.
- [x] Any dependent changes have been merged and published in downstream modules.
### Additional context
This bug affects any production deployment where users have extended conversations. Sessions accumulating >100 events (which triggers pagination) become completely unusable as the agent cannot load the session to process new messages.
The fix is minimal and maintains backward compatibility - it only changes the scope of the `async with` block without altering any logic or return values.
**Affected versions:** Tested on google-adk 1.19.0, but the bug exists in earlier versions as well.
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/adk-python/pull/3756 from AlexisMarasigan:fix/vertex-ai-session-service-paginatio 01fbafa6524312f24f7c9feaffb07bff0ad49b77
PiperOrigin-RevId: 855451813
The fix will use quotes to escape "key", which is column name in the metadata table. Should work for different database types.
Merge https://github.com/google/adk-python/pull/4106
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/adk-python/pull/4106 from DineshThumma9:fix/mysql-reserved-keyword-issue e39d0d02f3695d6890bc3267417b5dad58f7e8ee
PiperOrigin-RevId: 854411915
The SqliteSessionService now accepts database paths in the form of SQLite URLs (e.g., "sqlite:///./sessions.db", "sqlite+aiosqlite:////absolute.db")
Close#4077
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 853922433
Final part of https://github.com/google/adk-python/discussions/3605.
This change introduces:
- A new `adk migrate session` CLI command to run database schema upgrades.
- A migration script to upgrade from the old Pickle-based session schema (v0) to the new JSON-based schema (v1).
- A migration runner that orchestrates the upgrade process, handling sequential migrations and using temporary SQLite databases for intermediate steps if needed.
- Unit tests for the v0 to v1 migration.
Co-authored-by: Liang Wu <wuliang@google.com>
PiperOrigin-RevId: 852983323
The change updates the `StorageEvent.to_event` method to use `EventActions.model_validate` when rehydrating the `actions` field. This ensures that nested models within `EventActions`, such as `EventCompaction`, are correctly reconstructed from the stored data
Close#4047
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 852408683
Part 2 of https://github.com/google/adk-python/discussions/3605.
The DatabaseSessionService now checks for the usage of a V1 schema based on the "adk_internal_metadata" table. Table creation and subsequent operations use either the V0 or V1 SQLAlchemy models accordingly. New databases will default to V1.
Migration script and CLI command will be provided in the next change.
Co-authored-by: Liang Wu <wuliang@google.com>
PiperOrigin-RevId: 845443406
Part 1 of https://github.com/google/adk-python/discussions/3605.
This change adds a new schema that uses JSON serialization to store Events data in the database. A new "adk_internal_metadata" table is also added to store information like schema version. Since we want to keep supporting existing DB, we fork from the original schema and call it "v0", while the new one is called "v1".
The change is no-op for existing users. In later change, the new schema will be used for new databases, and migration scripts will be provided for existing databases.
Co-authored-by: Liang Wu <wuliang@google.com>
PiperOrigin-RevId: 844986248
Also provide a command line tool `adk migrate session` for DB migration
Addresses https://github.com/google/adk-python/discussions/3605
Addresses https://github.com/google/adk-python/issues/3681
To verify:
```
# Start one postgres DB
docker run --name my-postgres -d -e POSTGRES_DB=agent -e POSTGRES_USER=agent -e POSTGRES_PASSWORD=agent -e PGDATA=/var/lib/postgresql/data/pgdata -v pgvolume:/var/lib/postgresql/data -p 5532:5432 postgres
# Connect to an old version of ADK and produce some query data
adk web --session_service_uri=postgresql://agent:agent@localhost:5532/agent
# Check out to the latest branch and restart ADK web
# You should see error log ask you to migrate the DB
# Start a new DB
docker run --name migration-test-db \
-d \ --rm \ -e POSTGRES_DB=agent \ -e POSTGRES_USER=agent \ -e POSTGRES_PASSWORD=agent -e PGDATA=/var/lib/postgresql/data/pgdata -v migration_test_vol:/var/lib/postgresql/data -p 5533:5432 postgres
# DB Migration
adk migrate session \
--source_db_url="postgresql://agent:agent@localhost:5532/agent" \
--dest_db_url="postgresql://agent:agent@localhost:5533/agent"
# Run ADK web with the new DB
adk web --session_service_uri=postgresql+asyncpg://agent:agent@localhost:5533/agent
# You should see the data from old DB is migrated
```
Co-authored-by: Shangjie Chen <deanchen@google.com>
PiperOrigin-RevId: 837341139
The `database_session_service` now updates the `update_time` of a session to the event's timestamp when an event is appended
Close#2721
Co-authored-by: George Weale <gweale@google.com>
PiperOrigin-RevId: 834994070
This change introduces `input_transcription` and `output_transcription` fields to session events, enabling the storage and retrieval of transcription data in both the database and Vertex AI session services.
Closes#3172
Co-authored-by: Hangfei Lin <hangfei@google.com>
PiperOrigin-RevId: 834366848
