Explicitly define models

[ghostdevv]

I'd like to be able to use olla with Cloudflare Workers AI, but for one reason or another it's open ai compatibility doesn't include /v1/models. Something that I think traceloop/hub does well is that it allows you to define providers and models more explicity.

Their config would allow something like this as I understand it:

providers:
    - key: cloudflare
      type: openai
      api_key: ...
      base_url: ...

models:
    - key: llama-test
      type: @cf/meta/llama-3.2-1b-instruct
      provider: cloudflare

[thushan]

Nice idea. We did have the idea of "custom" and then manually provide models so you can (essentially) create these sorts of things. But the issue is how to be flexible enough to give good coverage across the board. There's specifying models (below) and a generic filter option (so include/exclude filter types).

Would something like this work - not supported yet obviously, just brainstorming:

      - url: "..."
        name: "flarellm"
        priority: 100
        type: "custom"
        profile: "cloudflare-ai" # <--- only available for custom types
        models:
        - meta/llama-3.2-1b-instruct
        - meta/llama-3.3-1b-instruct

The core APIs etc would be configured via the existing profile infrastructure, this way you can configure the same provider and customise the models etc. Auth would be prerequisite for this.

Auth endpoints are also coming, just having some issues to work through. I think that will be ready in a couple of weeks.

[ghostdevv]

That config looks good! The name profile may need some bike shedding, as you also have type there which I had assumed referred to the profile (unless I've misunderstood).

Also, for posterity, the model name would include that @cf as they have other prefixes such as @hf (hugging face).

Auth endpoints are also coming, just having some issues to work through. I think that will be ready in a couple of weeks.

Sweet! let me know if there is a way I can help