refactor(llm): add LiteLLM-backed provider abstraction#2363
Draft
refactor(llm): add LiteLLM-backed provider abstraction#2363
Conversation
Introduce an internal LLMProvider helper that wraps LiteLLM provider parsing, provider model info lookup, and provider-aware API base inference. Use the helper in LLM and Telemetry so Bedrock auth handling and cost-calculation model/provider routing stop manually splitting model strings. Extend model_features to accept an LLMProvider so provider-aware rules can operate on parsed provider/model data while keeping the raw model string available. Add focused tests for the new helper and the updated feature lookup paths. Co-authored-by: openhands <openhands@all-hands.dev>
Contributor
API breakage checks (Griffe)Result: Passed |
Contributor
Agent server REST API breakage checks (OpenAPI)Result: Passed |
12 tasks
Contributor
Coverage Report •
|
||||||||||||||||||||||||||||||||||||||||
Rework the new LLMProvider helper so it no longer keeps both raw and parsed model strings internally. Accept full model strings at the SDK boundary, then immediately normalize to LiteLLM's parsed provider/model pair for transport, telemetry, and feature checks. Unknown models still remain opaque strings, but the provider abstraction itself now avoids duplicate raw-vs-parsed state. Also route chat/responses calls through parsed provider kwargs and tighten the associated tests. Co-authored-by: openhands <openhands@all-hands.dev>
enyst
commented
Mar 9, 2026
Keep LLMProvider for LiteLLM-facing transport logic only and use the canonical model string for capability/feature lookups. Co-authored-by: openhands <openhands@all-hands.dev>
enyst
commented
Mar 9, 2026
Co-authored-by: openhands <openhands@all-hands.dev>
enyst
commented
Mar 9, 2026
Co-authored-by: openhands <openhands@all-hands.dev>
enyst
commented
Mar 9, 2026
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactor SDK LLM provider handling around an internal
LLMProviderhelper backed by LiteLLM, so provider-specific logic stops re-splitting rawprovider/modelstrings across the SDK.This PR now takes the simpler direction discussed in issue #2274 and PR review:
provider+modelview for LiteLLM-facing runtime pathsLLMinstance instead of refreshing provider state during each transport callConcretely, this PR:
openhands.sdk.llm.utils.litellm_provider.LLMProvideras a thin wrapper around LiteLLM provider parsing, provider model info lookup, API base inference, and call kwargs generationLLMProviderfocused on LiteLLM's parsed runtime shape (provider, parsedmodel, resolved API base) rather than storing duplicaterequested_*fieldsLLMchat/responses transport calls to use provider-owned LiteLLM kwargs generation (model+custom_llm_provider, plus Bedrock-awareapi_keyforwarding) instead of repeatedly passing or re-splitting the full stringapi_keyhandling intoLLMProviderso provider-sensitive transport behavior lives alongside provider parsingmodel_featureslookups on the canonical model string (model_canonical_nameormodel) instead of introducing a second provider abstractionLLMsetup, so chat/responses transport paths reuse the same resolved provider metadata for the lifetime of the instanceCloses #2274.
Checklist
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22golang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:fab6f99-pythonRun
All tags pushed for this build
About Multi-Architecture Support
fab6f99-python) is a multi-arch manifest supporting both amd64 and arm64fab6f99-python-amd64) are also available if needed