Skip to content

Litellm Model

Guides

  • Setting up most models is covered in the quickstart guide.
  • If you want to use local models, please check this guide.

minisweagent.models.litellm_model

logger module-attribute 

logger = getLogger('litellm_model')

LitellmModelConfig dataclass 

LitellmModelConfig(
    model_name: str,
    model_kwargs: dict[str, Any] = dict(),
    litellm_model_registry: Path | str | None = getenv(
        "LITELLM_MODEL_REGISTRY_PATH"
    ),
)

model_name instance-attribute 

model_name: str

model_kwargs class-attribute instance-attribute 

model_kwargs: dict[str, Any] = field(default_factory=dict)

litellm_model_registry class-attribute instance-attribute 

litellm_model_registry: Path | str | None = getenv(
    "LITELLM_MODEL_REGISTRY_PATH"
)

LitellmModel 

LitellmModel(**kwargs)
Source code in src/minisweagent/models/litellm_model.py 
30
31
32
33
34
35
def __init__(self, **kwargs):
    self.config = LitellmModelConfig(**kwargs)
    self.cost = 0.0
    self.n_calls = 0
    if self.config.litellm_model_registry and Path(self.config.litellm_model_registry).is_file():
        litellm.utils.register_model(json.loads(Path(self.config.litellm_model_registry).read_text()))

config instance-attribute 

config = LitellmModelConfig(**kwargs)

cost instance-attribute 

cost = 0.0

n_calls instance-attribute 

n_calls = 0

query 

query(messages: list[dict[str, str]], **kwargs) -> dict
Source code in src/minisweagent/models/litellm_model.py 
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
def query(self, messages: list[dict[str, str]], **kwargs) -> dict:
    response = self._query(messages, **kwargs)
    try:
        cost = litellm.cost_calculator.completion_cost(response)
    except Exception as e:
        logger.critical(
            f"Error calculating cost for model {self.config.model_name}: {e}. "
            "Please check the 'Updating the model registry' section in the documentation at "
            "https://klieret.short.gy/litellm-model-registry Still stuck? Please open a github issue for help!"
        )
        raise
    self.n_calls += 1
    self.cost += cost
    GLOBAL_MODEL_STATS.add(cost)
    return {
        "content": response.choices[0].message.content or "",  # type: ignore
        "extra": {
            "response": response.model_dump(),
        },
    }

get_template_vars 

get_template_vars() -> dict[str, Any]
Source code in src/minisweagent/models/litellm_model.py 
83
84
def get_template_vars(self) -> dict[str, Any]:
    return asdict(self.config) | {"n_model_calls": self.n_calls, "model_cost": self.cost}
bug_report Something broken/unclear?

Open an issue on GitHub!
help Open-ended discussions

Join our Slack!
Our projects
SWE-agent SWE-agent SWE-rex SWE-ReX SWE-smith SWE-smith SWE-bench SWE-bench sb-cli sb-cli