Josh Soref
aa1233608a
Merge https://github.com/google/adk-python/pull/2447 This PR corrects misspellings identified by the [check-spelling action](https://github.com/marketplace/actions/check-spelling) The misspellings have been reported at https://github.com/jsoref/adk-python/actions/runs/16840838898/attempts/1#summary-47711379253 The action reports that the changes in this PR would make it happy: https://github.com/jsoref/adk-python/actions/runs/16840839269/attempts/1#summary-47711380479 Note: while I use tooling to identify errors, the tooling doesn't _actually_ provide the corrections, I'm picking them on my own. I'm a human, and I may make mistakes. I've included a couple of changes to make CI happy. Personally, I object to CI being in a state of "random drive by person who adds a blank line in the middle of a file must fix all the preexisting bugs in the file", but that appears to be the state for this repository. COPYBARA_INTEGRATE_REVIEW=https://github.com/google/adk-python/pull/2447 from jsoref:spelling d85398e7fd154d124d477c6af6181481a01f34e0 PiperOrigin-RevId: 827629615
3.1 KiB
Using ollama models with ADK
Model choice
If your agent is relying on tools, please make sure that you select a model with tool support from ollama website.
For reliable results, we recommend using a decent size model with tool support.
The tool support for the model can be checked with the following command:
ollama show mistral-small3.1
Model
architecture mistral3
parameters 24.0B
context length 131072
embedding length 5120
quantization Q4_K_M
Capabilities
completion
vision
tools
You are supposed to see tools listed under capabilities.
You can also look at the model's template and tweak it based on your needs.
ollama show --modelfile llama3.1 > model_file_to_modify
Then you can create a model with the following command:
ollama create llama3.1-modified -f model_file_to_modify
Using ollama_chat provider
Our LiteLlm wrapper can be used to create agents with ollama models.
root_agent = Agent(
model=LiteLlm(model="ollama_chat/mistral-small3.1"),
name="dice_agent",
description=(
"hello world agent that can roll a dice of 8 sides and check prime"
" numbers."
),
instruction="""
You roll dice and answer questions about the outcome of the dice rolls.
""",
tools=[
roll_die,
check_prime,
],
)
It is important to set the provider ollama_chat instead of ollama. Using ollama will result in unexpected behaviors such as infinite tool call loops and ignoring previous context.
While api_base can be provided inside litellm for generation, litellm library is calling other APIs relying on the env variable instead as of v1.65.5 after completion. So at this time, we recommend setting the env variable OLLAMA_API_BASE to point to the ollama server.
export OLLAMA_API_BASE="http://localhost:11434"
adk web
Using openai provider
Alternatively, openai can be used as the provider name. But this will also require setting the OPENAI_API_BASE=http://localhost:11434/v1 and OPENAI_API_KEY=anything env variables instead of OLLAMA_API_BASE. Please notice that api base now has /v1 at the end.
root_agent = Agent(
model=LiteLlm(model="openai/mistral-small3.1"),
name="dice_agent",
description=(
"hello world agent that can roll a dice of 8 sides and check prime"
" numbers."
),
instruction="""
You roll dice and answer questions about the outcome of the dice rolls.
""",
tools=[
roll_die,
check_prime,
],
)
export OPENAI_API_BASE=http://localhost:11434/v1
export OPENAI_API_KEY=anything
adk web
Debugging
You can see the request sent to the ollama server by adding the following in your agent code just after imports.
import litellm
litellm._turn_on_debug()
Look for a line like the following:
quest Sent from LiteLLM:
curl -X POST \
http://localhost:11434/api/chat \
-d '{'model': 'mistral-small3.1', 'messages': [{'role': 'system', 'content': ...