Self-hosting open-weight models vs using hosted model APIs

Once a team has a working LLM feature, someone asks whether it would be cheaper or safer to run the model themselves. Open-weight models have become capable enough that the question is reasonable. Serving engines such as vLLM, SGLang and Hugging Face Text Generation Inference make it technically straightforward to put one behind an endpoint.

The answer depends less on the model than on the shape of the workload. This article walks through the factors that actually decide it, without pretending that a single price comparison settles anything. Prices for both GPUs and tokens change too often for that.

Two different cost structures

A hosted API charges per token. Cost is variable: it rises and falls with usage, and an idle feature costs nothing.

Self-hosting turns that into a mostly fixed cost. You pay for GPU instances by the hour, or buy hardware, whether they are serving requests or not. On top of that come costs that do not appear on a GPU invoice:

The comparison that matters is cost per useful token at your real utilization, including people. A GPU serving a few requests an hour costs as much as one serving thousands.

Utilization decides the economics

The effective cost of a self-hosted token is roughly:

cost per token = (GPU cost per hour + share of ops cost per hour)
                 ÷ tokens actually served per hour

The denominator is the whole story. Serving engines achieve high throughput by batching many concurrent requests on the same GPU. With continuous batching, a GPU that is kept busy can serve a large number of tokens per hour. The same GPU handling occasional, bursty traffic spends most of its time idle, and the cost per token is many times higher.

That leads to a few practical rules:

• Steady, high-volume workloads favor self-hosting. Batch classification, embedding large corpora, nightly summarization and other jobs you can queue keep GPUs full.
• Spiky, interactive, low-volume workloads favor APIs. You would have to provision for peak and pay for the troughs.
• Batch work can be scheduled. Queued jobs can run on spot or preemptible capacity, which changes the arithmetic further. Hosted providers often offer discounted batch endpoints too, so compare like with like.

Measure your real token volume by hour of day before modeling anything. Many teams find their traffic is far burstier than the monthly total suggests.

Model size and hardware

The model you need determines the hardware, and the hardware determines the fixed cost. A model with a few billion parameters fits on a single GPU and can be quantized to run on smaller cards. Larger models need several GPUs with tensor parallelism, and the largest open-weight models need multiple nodes.

A minimal vLLM deployment exposes an OpenAI-compatible endpoint, which keeps application code portable between self-hosted and hosted models:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --port 8000

--max-model-len matters more than it looks. The key-value cache for long contexts consumes GPU memory, so a larger maximum context leaves room for fewer concurrent requests and lowers throughput.

Before committing, check the quality gap honestly. Run your own evaluation set against the open-weight candidate and the hosted model you use now. For some tasks, such as extraction, classification and routing, a smaller open model is close enough. For complex reasoning and agentic tool use, the gap to frontier hosted models may still be large.

Latency

Self-hosting is not automatically faster. Two numbers matter: time to first token, and tokens per second for each stream.

• Network distance. A model in the same region or data center as the application removes a round trip to a provider. For short completions this can be noticeable.
• Batching trade-off. Large batches maximize throughput but slow each individual request. You tune the balance yourself, which is a benefit when you need predictable latency and a burden when you do not know your traffic.
• Queueing under load. When a self-hosted cluster is saturated, requests wait. A hosted API absorbs bursts with capacity you never see, up to your rate limit.

If latency is the main driver, measure both options with your real prompt and output lengths, at your real concurrency.

Data control

Data control is often the strongest argument for self-hosting, and also the most often overstated. The options are wider than "public API" or "our own GPUs":

• Direct provider APIs, with contractual terms on retention and training, and in some cases zero data retention agreements.
• Models through your cloud provider, such as Amazon Bedrock, Azure OpenAI or Vertex AI, which keep traffic within your cloud agreement and often offer regional processing.
• Open-weight models on managed endpoints in your own cloud account.
• Fully self-hosted on your own infrastructure, including air-gapped environments.

Start from the actual requirement. If it is "data must stay in the EU and must not be used for training", the second option often meets it without running a GPU fleet. If it is "data must never leave our network", only self-hosting will do.

Operational load

Running models in production is a real service, with the same expectations as any other:

• GPU node pools, drivers and device plugins, and capacity reservations in regions where GPUs are scarce.
• Autoscaling on queue depth or GPU utilization, with slow scale-up because model weights take minutes to load.
• Monitoring for throughput, time to first token, queue length and GPU memory.
• An upgrade path for the serving engine and the models, with an evaluation gate before each rollout.
• On-call for an endpoint that product features now depend on.

A hybrid approach

Most organizations end up with both, routed by workload:

• Hosted frontier models for complex, interactive and low-volume tasks where quality matters most.
• Self-hosted open-weight models for high-volume, well-defined tasks such as classification, extraction and embeddings, and for data that must not leave your environment.
• One internal interface in front of both. Because vLLM and most providers speak an OpenAI-compatible API, a gateway can route each request by model name, team or data classification. Applications do not need to know where a model runs.

The hybrid setup also keeps your options open. When a new open-weight model closes the gap for a task, you move that task. When a hosted model becomes cheaper, you move it back. Neither move requires changing application code.

Where to start

1. Inventory current LLM use by task, with token volume per hour, latency requirements and data classification.
2. Build an evaluation set for the highest-volume task and score one or two open-weight candidates against your current model.
3. If quality is close enough, run a time-boxed pilot on rented GPUs and measure real throughput at real concurrency.
4. Compare cost per useful token, including on-call and engineering time, at the utilization you measured.
5. Put a gateway in front of every model call first, so moving a workload later is a configuration change.