LLM Gateway - Docs by LangChain

Private beta: The LLM Gateway is in private beta. APIs and features may change as we iterate. Sign up for the waitlist to get access.

The LLM Gateway is a proxy that sits between your agents (or any LLM client) and the LLM providers they call. Instead of each client storing provider API keys locally, keys are stored once in LangSmith as Provider Secrets. Clients authenticate with a LangSmith API key. When a request passes through the gateway, it:

Authenticates the caller using their LangSmith API key.
Resolves the actual provider API key from your workspace’s Provider Secrets.
Evaluates governance policies (spend limits, PII redaction, secrets redaction).
Proxies the request to the upstream provider.
Traces the call to LangSmith.

Every gateway-proxied call appears as a trace in LangSmith. When a policy fires, the event flows into LangSmith Engine for triage—you can go from a blocked request to the trace that triggered it to the fix, all in one product.

Feature availability

Capability	Description
Spend limits	Hard-block enforcement at the organization, workspace, API key, or user level. When a cap is hit, the caller receives a 402 response with an actionable error message.
Spend visibility	Real-time cost rollups by workspace, user, API key. Per-model visibility can be had via Custom Charts.
Model fallbacks	When the primary model returns a status code you’ve flagged as retryable, such as a rate limit or provider outage, the request is retried against backup model configurations in priority order.
PII redaction	Detects and redacts names, places, nationality, religion, political affiliation, and ages from requests before they reach the model.
Secrets redaction	Detects and redacts US phone numbers, US SSNs, API keys, tokens, and credentials from requests. Covers AWS, GitHub, GitLab, OpenAI, Anthropic, GCP, Azure, Slack, Datadog, PyPI, npm, private keys, and LangSmith tokens.
LangSmith Engine integration	Policy violations surface as issues in LangSmith Engine. Click through from a violation to the trace that produced it.
Audit logging	Administrative changes to gateway configuration and gateway invocations are logged.
Tracing	Every gateway-proxied call appears in the same LangSmith workspace as the rest of your agent’s traces. Routing through the gateway does not fragment your observability.

Supported providers

Provider	Gateway path	Secret name
Anthropic	`/anthropic`	`ANTHROPIC_API_KEY`
AWS Bedrock	`/bedrock`	`AWS_BEARER_TOKEN_BEDROCK`
Baseten	`/baseten`	`BASETEN_API_KEY`
Fireworks	`/fireworks`	`FIREWORKS_API_KEY`
Google Gemini	`/gemini`	`GOOGLE_API_KEY`
Google Vertex AI	`/vertex`	`VERTEX_SERVICE_ACCOUNT_JSON`
OpenAI	`/openai`	`OPENAI_API_KEY`

In addition to the providers above, LangChain offers managed models through the /langchain path. These require no provider secret of your own: you authenticate with your LangSmith API key and LangChain owns the upstream credential.

Regional gateways

If your LangSmith account is on a regional instance, use the corresponding regional gateway. Append the provider path to the base URL, for example /openai/v1 or /langchain/v1.

Region	Gateway base URL
GCP US	`https://gateway.smith.langchain.com`
GCP EU	`https://eu.gateway.smith.langchain.com`
GCP APAC	`https://apac.gateway.smith.langchain.com`
AWS US	`https://aws.gateway.smith.langchain.com`

Known limitations

Claude Desktop: no marketplace plugins supported. The Chat tab not visible, but Cowork is functional.
Codex Desktop: no marketplace plugins supported.
Cursor: IDE integration is not yet available.
ChatGPT Desktop / Gemini Desktop: not configurable.

Resources

Spend policies

Set and manage cost limits across your organization.

PII and secrets redaction

Prevent sensitive data from reaching LLM providers or trace storage.

Model fallbacks

Automatically retry against backup models when the primary model fails.

For further questions on LLM Gateway, contact support.langchain.com.

Connect these docs to Claude, VSCode, and more via MCP for real-time answers.

Edit this page on GitHub or file an issue.

​Feature availability

​Supported providers

​Regional gateways

​Known limitations

​Resources

Spend policies

PII and secrets redaction

Model fallbacks

Feature availability

Supported providers

Regional gateways

Known limitations

Resources