ellmer and lm studio

General
1

Ellmer and LM Studio

Has anyone managed to use ellmer to work with a local lm studio model? I have found working on my Mac that these use memory much better than ollama - due to the MLX format apparently? - so I can run better models.

I have tried several options such as chat_google_gemini, or chat_openai_compatible:

While changing options to point to my localhost server.

base_url = "http://localhost:1234/v1"

but the former is obviously sending and receiving the wrong format.

Error: <ellmer::AssistantTurn> object properties are invalid:
- @json must be <list>, not <NULL>
In addition: Warning message:
Premature end of input; ignoring final partial chunk

The latter I believed from the LM Studio documentation could be the right transfer format.

But it asks for an Open-AI key? For a local model? Which was actually gemini?

Error in `openai_key()`:
! Can't find env var `OPENAI_API_KEY`.
Run `rlang::last_trace()` to see where the error occurred.

Does anyone know how to do this? Or is ellmer support e.g. chat_lmstudio a possibility?

1 Like
2

Sorry I have been tinkering with this all day. I understand that you now have to set credentials in the .Renviron and cannot supply it as an argument option. Security!

The chat_openai_compatible is the correct option and the base url needs to be appnded with "/v1" to tell it that.

I couldn't work out that the local model that doesn't need an API key still had to have some specific string set - in this case "lm-studio". I set it in the project .Renviron as I was working in a project Soo...

the following works for lm-studio on a macbook.

usethis::edit_r_environ("project")
# save this key in the .Renviron
# OPENAI_API_KEY="lm-studio"


client <- chat_openai_compatible(
  base_url = "http://localhost:1234/v1",
  name = "OpenAI-compatible",
  system_prompt = "You are a helpful assistant.",
  api_key = NULL,
  credentials = NULL,
  model = "google/gemma-3-12b",
  params = NULL,
  api_args = list(),
  api_headers = character(),
  echo = c("none", "output", "all")
)

# then chat away like
client$chat("an inane question?")
3

Glad you figured this out! One shortcut I can offer: instead of setting OPENAI_API_KEY in your .Renviron, you can pass a function to credentials that returns a string, e.g.

chat_openai_compatible(
  base_url = "http://localhost:1234/v1",
  credentials = function() "lm-studio",
  model = "google/gemma-3-12b"
  # remaining arguments as/if needed
)