An internal LLM gateway as a platform capability

Language model use inside a company usually starts in many places at once. One team calls a provider API directly with a key from a shared password vault. Another uses a cloud provider's model service through its own account. A third runs an open-weight model on a GPU node. Each has its own credentials, its own logging, or none, and its own way of handling outages.

Within a year, the questions start. How much are we spending, and on what? Who has access to which models? Did customer data go to a provider we have not approved? What happens when that provider has an outage? Nobody can answer them, because the traffic is spread across dozens of direct integrations.

An internal LLM gateway answers them in one place. It is the same move platform teams made earlier with API gateways and service meshes: put a shared layer in the path, and make the policy live there.

What the gateway does

Applications call one internal endpoint. The gateway authenticates the caller, applies policy, routes the request to a provider and records what happened. Providers' credentials never leave the gateway.

Most providers and self-hosted serving engines offer an OpenAI-compatible chat completions API, and most gateways expose that format too. That keeps adoption cheap: teams change a base URL and a key, and existing client libraries keep working.

import os
 
from openai import OpenAI
 
client = OpenAI(
    base_url="https://llm-gateway.internal/v1",
    api_key=os.environ["LLM_GATEWAY_KEY"],
)
 
response = client.chat.completions.create(
    model="chat-standard",
    messages=[{"role": "user", "content": prompt}],
    extra_headers={"x-feature": "order-summary"},
)

The model name is an internal alias, not a provider's model ID. The gateway maps chat-standard to a concrete model, which lets the platform team change providers or versions without touching application code.

Build or adopt

Open-source options such as LiteLLM, Envoy AI Gateway and the AI plugins for Kong cover most of the routing and key management. Cloud providers also offer gateway features in front of their own model services. Building a thin proxy yourself is feasible, but the long tail of provider API differences, streaming, tool calls and token counting makes adopting an existing project the better default. Spend the engineering time on policy and integration, not on protocol translation.

Per-team keys and quotas

Issue gateway keys per team and per application, never per person and never shared across services. Each key carries metadata the gateway uses for every decision:

Quotas matter more than they first appear. Provider rate limits are usually per organization account. Without per-team limits at the gateway, one batch job that loops on retries can exhaust the shared limit and take every other team's features down with it.

Set limits in both requests per minute and tokens per minute, and a monthly token or spend budget per key. Alert the owning team at a threshold, and decide in advance whether exceeding the budget blocks requests or only notifies.

Issue keys through the same self-service path teams use for other platform resources, so getting access takes minutes and nobody is tempted to go around the gateway. Workloads should fetch the key from the secret store rather than hold it in configuration.

Logging and redaction

The gateway is the natural place to log model traffic, because every request passes through it. Decide carefully what to keep:

• Always log metadata: timestamp, key, team, feature header, model alias and resolved model, input and output token counts, latency, status and which provider served the request.
• Log content only where needed, with a defined retention and restricted access. Full prompts are valuable for debugging and evaluation and are also some of the most sensitive data the company holds.
• Redact before sending, where policy requires it. Pattern-based detection of card numbers, national identifiers and credentials can mask or block requests before they leave. Treat this as a safety net, not a guarantee; detection misses things.

Send logs to the existing observability stack rather than a separate tool, so model latency and errors sit next to the services that depend on them.

Cost attribution

Token counts per request, multiplied by the provider's prices, give a cost per request. Aggregated by key metadata, that answers the questions finance and engineering leadership ask:

• Spend per team and per feature, which needs the feature header or a key per feature.
• Spend per model, which shows when a cheaper model would do.
• Trends per environment, which catch development traffic accidentally running against the most expensive model.

Keep the price table in the gateway's configuration, versioned, and update it when providers change prices. Showback, sending each team a monthly report of its usage, changes behavior on its own without chargeback.

Fallbacks and routing

Provider outages and rate-limit responses happen. With direct integrations, each team handles them differently, or not at all. A gateway can handle them once:

• Retries with backoff for transient errors and rate-limit responses, within a total time budget.
• Fallback to an equivalent model at another provider or region when the primary fails, configured per alias.
• Routing by policy, such as sending confidential data only to a model in your own cloud account, or sending batch traffic to a discounted batch endpoint.

Fallbacks are not free. Two models give different answers to the same prompt, and prompts tuned for one may perform worse on another. Run your evaluation set against every fallback target before enabling it, and record in the logs when a fallback served the request.

Rollout steps

1. Deploy the gateway with two or three model aliases that cover current use, and connect the providers already approved.
2. Move provider credentials into the gateway and issue per-team keys through self-service.
3. Migrate existing integrations one team at a time; for most, it is a base URL and key change.
4. Turn on metadata logging and a per-team cost report from the first day.
5. Add per-key rate limits and budgets once you have a few weeks of usage data.
6. Configure fallbacks per alias after evaluating each target.
7. Revoke the old direct provider keys, and block direct egress to provider APIs from workloads where your network allows it.