Local LLM endpoint Roadmap?

[codetricity]

I enjoyed reading the question from @balise about blocking sensitive data:

Posit AI and blocking sensitive data

I would like to try Posit AI but I work with highly sensitive biomedical data. Can I install Posit AI but have it turned off by default for all of my RStudio projects? I would like to have the ability to enable it for projects that do not contain sensitive data but have it off until I explicitly turn it on project by project.

This led me to wonder if there is a Positron roadmap for enterprise deployments where the LLM endpoint is customer-controlled (e.g., on-prem, VPC, or local inference).

In the broader ecosystem (e.g., Visual Studio Code), there are already several approaches to this:

• Ollama has instructions to have VSCode use a local LLM
• Continue has a similar document for a local LLM
• Cline also has a doc for this

To be clear, I’m not currently an enterprise user. However, I’m interested in this capability for a few practical scenarios:

• A teacher wants to introduce AI-assisted workflows but does not have the budget to provide commercial LLM access to each student
• A junior team member wants to analyze proprietary or pre-publication data locally before sharing results more broadly
• An individual developer prefers to run models locally for cost control or data isolation

In my own case, I’ve been building a course where students use a fully local stack (LLM + embedding model + vector database) to run experiments without external API dependencies.

Given this, I’m curious whether Posit is considering support for OpenAI-compatible endpoints or customer-controlled model backends in Positron / Posit AI, as this seems increasingly relevant for both enterprise and educational use cases.

[tom_rstudio]

Howdy @codetricity thanks for the writeup and questions!

Positron has support for Custom Providers: Getting Started – Positron and an older, but long discussion post at: Positron Assistant now has preview support for Custom Providers (OpenAI compatible) · posit-dev/positron · Discussion #9988 · GitHub

Positron Assistant’s custom provider support is intended for use with any OpenAI-compatible API endpoint that uses the /v1/chat/completions endpoint for chat.

I have spoken with some Posit Workbench customers who are using Custom Providers with their own internal AI Gateways or self-hosted foundational models, using frameworks like litellm or Portkey, or even more bespoke approaches.

Personally, when I've discussed this with administrators, I have tried to clearly differentiate the model size/quality of locally-hosted models (ie on a beefy server) vs local-to-laptop models (on a laptop).

We don’t currently recommend using a local model that can fit on a laptop as a custom provider. Read more about why local models are not there (yet) on Posit’s blog. However, the models are improving rapidly, and a more recent followup talks a bit about progress in the past 4 months: LLMs running on my laptop can drive coding agents now | Simon P. Couch – Simon P. Couch

You are of course welcome to try it out with whatever models you want, but I do want to caution a bit as to the effectiveness of most consumer-grade laptops and what types of models they can support.