How to use with llama.cpp

[schlick7]

Has anybody gotten this working with llama.cpp? When i try it just says its not a valid provider.

[MODSetter]

We use litellm in backend. This should do it
BerriAI/litellm#9138
https://docs.litellm.ai/docs/providers/openai_compatible

[schlick7]

I tried that 3 times but now it works. Only change I made was to start using the new model router setting in llama.cpp . So either that was needed or I was doing something wrong. Thanks