Self-Hosted Copilot Alternative: Private Code Assistant
A self-hosted Copilot alternative has three parts: an editor extension, a code-capable model, and a private endpoint connecting the two. The extension (Continue, Cline and Tabby are common) replaces the Copilot client, an open model such as Qwen2.5-Coder replaces the hosted model, and an OpenAI-compatible endpoint inside your network replaces the cloud call. Source code never leaves your environment.
The honest caveat comes first: chat, edits and agent-style help work well with this setup, but inline tab completion is harder, and a 7B model will not match a frontier model on difficult refactors. This post explains each part and where the trade-offs are.
Why teams do this
Code is often the most sensitive thing a company owns, and a hosted assistant sends snippets of it, plus surrounding files, to someone else's servers on every request. Some companies cannot allow that at all, whether from customer contracts, export rules or an air-gapped development network. Others allow it but want to control cost at scale. A private endpoint solves the first problem outright and can help with the second; the self-hosted LLM cost post has the break-even model.
Part 1: the editor extension
| Tool | What it is | Custom endpoint |
|---|---|---|
| Continue | Open-source extension for VS Code and JetBrains: chat, edit, autocomplete | Yes, any OpenAI-compatible URL |
| Cline | Agent-style extension for VS Code that plans and edits across files | Yes, OpenAI-compatible provider |
| Tabby | Self-hosted server with its own editor plugins, focused on completion | Runs its own model server |
| Aider | Terminal tool that edits a git repository from chat | Yes, OpenAI-compatible |
Continue is the simplest place to start. It reads a config file where you list models and say what each one is for. A minimal config.yaml pointing at a private endpoint looks like this; check Continue's docs for the current schema, since it changes:
name: Private assistant
version: 1.0.0
schema: v1
models:
- name: Internal code model
provider: openai
model: qwen2.5-coder-32b-instruct
apiBase: https://llm.internal.example.com/v1
apiKey: <your-token>
roles: [chat, edit, apply]
The model name must match what your endpoint reports. Developers then get chat, "edit this selection" and apply-to-file from a model that never leaves the network.
Part 2: the model
Code assistants need a model trained on code. Current open options include:
- Qwen2.5-Coder in sizes from 0.5B to 32B, mostly Apache 2.0 (check the card for each size). The 32B instruct model is a common choice for chat and edits.
- Qwen3-Coder variants, which are larger mixture-of-experts models aimed at agentic coding. They need more memory, so look at the Qwen VRAM requirements before planning hardware.
- StarCoder2 from the BigCode project, trained on permissively licensed code under the OpenRAIL-M license.
- Codestral from Mistral, which has its own non-production license; read it before use.
Memory is the constraint. A 32B model at 4-bit needs about 24 GB of GPU memory for one user at 8K context, and code assistants want longer context because they send open files along with your question. Size with the method in GPU memory for LLM, and plan for 16K to 32K context.
Use two models if you can: a larger one for chat and edits, where a few seconds are acceptable, and a small one (1.5B to 7B) for completion, where latency must be low.
Part 3: the endpoint and its limits
Chat, edit and apply modes use the standard chat-completions API, so any OpenAI-compatible server works. See the OpenAI-compatible API guide for how the client side behaves.
Tab autocomplete is different. It uses fill-in-the-middle (FIM) prompting, where the model sees the code before and after the cursor, and it needs a model trained for FIM, a server that supports the completions or FIM interface, and very low latency. Check these three things before promising developers inline completion. If your endpoint only offers chat completions, begin with chat and edits and treat completion as a later step.
Set expectations
- A self-hosted 32B model is good for explaining code, writing tests, generating boilerplate, and routine edits. It is weaker than the best hosted models on large, ambiguous multi-file changes.
- Latency depends on your GPUs. Developers notice a slow assistant and stop using it, so measure time to first token with the real number of simultaneous users. The LLM observability guide lists what to track.
- Set a policy on what developers may paste into prompts, and who can see the logs. A private endpoint removes the third party, not the need for rules.
Where AI Refinery fits
AI Refinery, a self-hosted LLM platform, provides the private endpoint. You download a code model from its catalog if one is available, pick a quantization level and launch an inference server inside your own infrastructure. The server exposes an OpenAI-compatible /v1/chat/completions endpoint, with per-application API tokens managed in the console, and the audit log and RBAC cover who is using which model. The customer provides the GPUs.
It does not include an editor extension, and its documented endpoint is chat completions, so Continue's chat and edit roles are the supported path and inline FIM completion would need separate checking. It also has no code index or repository search of its own. Which code models are in the catalog at a given time is shown in the console. If you also want a general private chat for non-developers, the self-hosted ChatGPT alternative post covers that side. To see the platform, visit AI Refinery.
Tool configuration formats and model licenses change. Check each project's current documentation.