Let your remote GPU power VSCode Copilot

Who is not dreaming of a virtually unlimited token budget to spend?

Unfortunately, most credit cards have rather hard limits.

So why not make use of that gaming computer sitting at home and turn it into your own AI inference server?

In this post, we will configure a Linux machine with an NVIDIA GPU to run vLLM and expose an OpenAI compatible API. We will then connect Visual Studio Code and GitHub Copilot to it, allowing Copilot Agent to use a model running entirely on our own hardware.

There are a few interesting lessons along the way, particularly around GPU memory, context windows and why a model that technically fits into VRAM might still not be a particularly good fit.

The hardware

Before we get our hands dirty, let’s establish the context.

I am using a fairly ordinary gaming computer running Ubuntu 26.04 LTS. If you do not have Ubuntu installed yet, Canonical provides an excellent installation guide. You can even keep your Windows installation and dual boot, if you really want to.

But let’s not lose direction 🙂

The machine I am using has:

• NVIDIA RTX 4090 with 24 GB VRAM
• 16 core CPU / 32 Threads
• 64 GB system RAM
• 1TB SSD storage + 16TB HDD storage

The GPU is by far the most important component here.

A smaller NVIDIA GPU will work as well, but the amount of VRAM determines which models you can realistically run and how much context they can handle.
I see there are now 48GB versions of consumer cards available, but the shortage of memory makes them /super/ expensive

vLLM also supports AMD GPUs through ROCm, but this article focuses on NVIDIA and CUDA.

Installing the NVIDIA driver and CUDA

First, we need a working NVIDIA driver.

On current Ubuntu releases, I recommend letting Ubuntu select the appropriate driver instead of hard coding a particular driver version:

sudo ubuntu-drivers install

Reboot afterwards:

sudo reboot

Then verify that the GPU is visible:

nvidia-smi

You should see your GPU, driver version, available VRAM and current GPU utilization.

Ubuntu 26.04 also provides the NVIDIA CUDA toolkit directly through its package repositories:

sudo apt install cuda-toolkit

For vLLM itself, however, we will later let its installer select the appropriate PyTorch backend based on the installed CUDA driver.

A nicer way to watch the GPU

nvidia-smi is useful, but staring at a continuously refreshing terminal table gets old quickly.

I like nvtop:

sudo apt install nvtop

Then simply run:

nvtop

It gives you a nice overview of GPU utilization, memory usage and the processes currently occupying the GPU.

This becomes surprisingly addictive once you start experimenting with local models.

Installing vLLM

Now we can install vLLM.

vLLM is an inference and serving engine designed specifically for large language models. One of its most useful features for our purposes is that it can expose models through an OpenAI compatible HTTP API.

Instead of installing everything globally, we will create a dedicated Python environment.

I use uv for this.

Create a directory:

mkdir -p ~/dev/vllm
cd ~/dev/vllm

Create a Python 3.12 environment (while newer python versions exist, make sure to use the explicitly mentioned version of vllm docs):

uv venv --python 3.12 --seed

Activate it:

source .venv/bin/activate

And install vLLM:

uv pip install vllm --torch-backend=auto

The --torch-backend=auto option lets uv inspect the installed CUDA driver and select an appropriate PyTorch build.

You can verify the installation with:

vllm --version

Running our first model

Before downloading something that consumes most of our 24 GB VRAM, let’s start small.

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The first launch takes a little longer because vLLM downloads the model from Hugging Face.

Once the server is ready, we can query it:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": "Explain what vLLM does in two sentences."
      }
    ],
    "max_tokens": 200
  }'

Congratulations. You now have an OpenAI compatible LLM API running on your gaming computer.

No cloud API involved.

Choosing the right model

Now things get more interesting.

Choosing a local model is less about asking “which model do I want?” and more about asking “which model can I actually run well?”

The main constraint is VRAM.

A 24 GB RTX 4090 does not mean that we can load a 23 GB model and call it a day, I learned through the hard way :D. GPU memory is needed for several things:

VRAM
├── model weights
├── activations
├── CUDA graphs and runtime memory
└── KV cache

The last one is particularly important.

The KV cache and context length

During inference, the model stores information about tokens it has already processed in something called the Key/Value cache, or KV cache.

For our purposes, the important relationship is simple:

more tokens = more KV cache memory

Double the context and you approximately double the KV cache requirement.

The KV cache is also shared between concurrent requests.

If vLLM has enough KV cache for 100,000 tokens, that could mean:

1 request × 100,000 tokens

or

2 requests × 50,000 tokens

or

5 requests × 20,000 tokens

This is one reason vLLM is attractive compared with simpler local model runners. It is specifically designed to schedule multiple requests efficiently and manage this memory pool.

Why this matters for GitHub Copilot

Agentic coding consumes a surprising amount of context.

Before Copilot has even sent much of your actual code, the request can already contain system instructions, tool descriptions, conversation history and other agent context.

In my experiments, roughly 20,000 tokens can disappear before substantial repository context enters the picture.

So a model with a nominal 64K context window can suddenly feel rather small.

This also explains an important distinction:

Model context limit ≠ available KV cache

A model might support 128K tokens architecturally while your GPU only has enough free VRAM to maintain 50K tokens of KV cache comfortably.

Watch the vLLM startup output carefully. It tells you how much GPU memory is being consumed by weights, activations, CUDA graphs and the KV cache.

For example, with a larger coding model on my RTX 4090 I saw approximately:

Weights:             15.74 GiB
Peak activations:     0.72 GiB
CUDA graphs:          1.22 GiB
KV cache:             3.03 GiB

Suddenly the 24 GB card does not look quite so enormous anymore.

This is the real balancing act of local inference:

larger model versus larger context versus more concurrent users.

You cannot maximize all three on a fixed amount of VRAM.

Running Qwen3 Coder

For coding, I wanted to experiment with Qwen3 Coder.

A model such as Qwen3-Coder-30B-A3B-Instruct is particularly interesting because it is a Mixture of Experts model. It contains around 30 billion parameters, but only a subset is active for each token.

Do not confuse this with memory usage, though.

The complete model weights still need to reside somewhere. MoE primarily reduces the amount of computation required per token. It does not magically turn a 30B model into a 3B model in VRAM.

On a 24 GB RTX 4090, quantization therefore becomes important.

Depending on the exact model and quantization you choose, your command will look roughly like this:

vllm serve MODEL_NAME \
  --host 0.0.0.0 \
  --port 8899 \
  --served-model-name qwen3-coder \
  --max-model-len 65535 \
  --kv-cache-dtype fp8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

--served-model-name is worth using.

It gives your clients a stable model name:

qwen3-coder

You can later replace the actual checkpoint without changing every client that talks to the server.

The tool calling options are particularly important for agentic coding. Copilot Agent does not merely ask the model to generate text. The model needs to request tools for reading files, editing code, executing commands and continuing its work based on the results.

FP8 KV cache

Notice this option:

--kv-cache-dtype fp8

Using FP8 for the KV cache substantially reduces its memory footprint compared with BF16 or FP16.

That can make the difference between a model technically loading and actually having enough context available to be useful.

There can be some loss of accuracy depending on scaling and model support, so this is another trade-off rather than free memory.

Again, local inference is mostly an exercise in deciding which compromise annoys you least.

Running vLLM as a service

Starting vLLM manually is useful for testing, but eventually we want the server to start automatically.

Create a dedicated environment file:

sudo nano /etc/vllm.env

Add your API key:

VLLM_API_KEY=replace-this-with-a-long-random-secret

Protect the file:

sudo chmod 600 /etc/vllm.env

Now create the systemd service:

sudo nano /etc/systemd/system/vllm.service

For my installation, where vLLM lives in:

/home/trueal/dev/vllm/.venv/bin/vllm

the service looks roughly like this:

[Unit]
Description=vLLM OpenAI Compatible API Server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=trueal
Group=trueal
EnvironmentFile=/etc/vllm.env

ExecStart=/home/trueal/dev/vllm/.venv/bin/vllm serve MODEL_NAME \
    --host 0.0.0.0 \
    --port 8899 \
    --api-key ${VLLM_API_KEY} \
    --served-model-name qwen3-coder \
    --max-model-len 65535 \
    --kv-cache-dtype fp8 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder

Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Replace MODEL_NAME with the actual Hugging Face model you want to serve.

Reload systemd:

sudo systemctl daemon-reload

Enable the service:

sudo systemctl enable --now vllm

Watch the startup log:

journalctl -u vllm -f

And check its status:

systemctl status vllm

Testing the authenticated API

vLLM supports an OpenAI compatible API, so clients can talk to it much like they would talk to OpenAI.

Test it with:

curl http://localhost:8899/v1/models \
  -H "Authorization: Bearer replace-this-with-a-long-random-secret"

Then try a chat request:

curl http://localhost:8899/v1/chat/completions \
  -H "Authorization: Bearer replace-this-with-a-long-random-secret" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-coder",
    "messages": [
      {
        "role": "user",
        "content": "Write a Python function that calculates Fibonacci numbers."
      }
    ]
  }'

If that works, the hard part is done.

A word about security

There is one important warning here.

Binding vLLM to:

0.0.0.0

makes it listen on all network interfaces.

Do not simply forward port 8899 from your router to the Internet.

vLLM’s API key authentication protects its main OpenAI compatible API paths, but not every server endpoint should be assumed to be safe to expose directly.

If you want to access the server remotely, put it behind a properly configured reverse proxy, firewall, VPN or another trusted access layer.

For my setup, I expose the service through HTTPS rather than exposing port 8899 directly.

The architecture then looks like:

VS Code
   |
   | HTTPS
   v
Reverse proxy
   |
   | localhost / private network
   v
vLLM :8899
   |
   v
RTX 4090

This also gives us a nice stable endpoint such as:

https://vllm.example.com/v1

Connecting GitHub Copilot in VS Code

Recent versions of Visual Studio Code allow custom language model endpoints to be configured for Copilot Chat.

Open the command palette and run:

Chat: Manage Language Models

Add a Custom Endpoint and configure it to use your vLLM server.

The important pieces are:

API type: Chat Completions
URL: https://vllm.example.com/v1/chat/completions
Model: qwen3-coder

Store the API key through VS Code’s provider configuration rather than putting the secret directly into a configuration file.

Once configured, your local model appears in the model picker.

Select it and switch Copilot Chat to Agent mode.

At this point something rather amusing happens.

GitHub Copilot is still providing the agent framework, tools and integration with Visual Studio Code, but the actual language model deciding what to do is running on the RTX 4090 under your desk.

The flow becomes roughly:

Copilot Agent
     |
     | prompt + tools + repository context
     v
Your vLLM API
     |
     v
Qwen3 Coder
     |
     | tool request
     v
VS Code
     |
     | executes tool
     v
Qwen3 Coder
     |
     | next decision
     v
...

The model can inspect files, request edits, run terminal commands, inspect test results and continue working.

That is proper agentic coding, not simply code completion.

The catch: 24 GB is both a lot and not very much

This experiment also taught me something about GPU memory.

Before running local models, I mostly thought about whether a model would fit into VRAM.

That turns out to be only half the question.

The better question is:

How much VRAM remains after the model fits?

If a quantized model consumes 16 GB of a 24 GB card, you still need memory for CUDA, activations and the KV cache.

And the KV cache determines how much context your running requests can actually retain.

For an interactive chatbot, 32K or 48K tokens can be plenty.

For an agentic coding system that may already consume around 20K tokens describing its environment and tools, things look very different.

This creates a rather interesting optimization problem:

smaller model
    =
more KV cache
    =
larger context
    =
more concurrent requests

larger model
    =
potentially better reasoning
    =
less KV cache
    =
smaller practical context

There is no universally correct answer.

For a single developer, a larger coding model with a 64K context might be preferable.

For several concurrent developers, a smaller model with a much larger KV cache could provide a considerably better experience.

This is also why vLLM is such an interesting tool. It is not merely a convenient way to load a model. It is an inference server designed around exactly these scheduling and memory management problems.

Was it worth it?

Absolutely.

Not because my RTX 4090 suddenly replaces the largest commercial AI models. It does not.

But it creates a very useful additional tier.

I can use cloud models when I genuinely need their capabilities, while routine coding, experimentation, summarization and agentic tasks can run locally without watching a token counter or wondering what the next API bill will look like.

It also gives me complete control over the model, context size, quantization, privacy and serving infrastructure.

And perhaps most importantly, it turned a gaming GPU into infrastructure.

Which is a wonderfully nerdy justification for having bought the thing in the first place.