Skip to main content
Private beta: The LLM Gateway is in private beta. Sign up for the waitlist to get access.

Prerequisites

Before you start, confirm that:
  • Your Organization admin has enabled the LLM Gateway and added provider API keys to workspace secrets. To set this up, refer to Admin setup.
  • You have a workspace-scoped LangSmith API key attached to a role with gateway:invoke and workspaces:read permissions. Ask your org admin if you’re unsure.

1. Set environment variables

Set the following in your terminal (or add them to your ~/.zshrc to persist across sessions):
This points all provider SDKs at the LangSmith Gateway and uses your LangSmith API key for authentication. The gateway resolves actual provider keys from your workspace’s Provider Secrets—you never need local copies of provider API keys.
If your LangSmith account is on a regional instance, use the corresponding regional gateway.

Using LangChain and Deep Agents

LangChain chat models and Deep Agents (including Deep Agents Code) support the gateway through two convenience environment variables:
This routes all supported chat models through the gateway at https://gateway.smith.langchain.com. To use a different gateway (e.g. the EU instance), set its URL instead of true:
You can also configure base URLs and API keys for individual providers as described in the section above. See more details below for provider support and interactions with provider-specific environment variables.
  • Supported in Python only.
  • Supported chat models:
  • Provider-specific base URLs take precedence over the gateway, so you can still route an individual provider elsewhere. For example, with the gateway enabled, OPENAI_API_BASE sends OpenAI to that URL while every other provider continues to use the gateway:
The table below shows how the base URL and key are resolved, using OpenAI as the example (other providers use their own *_API_BASE / *_API_KEY variables). GW default is https://gateway.smith.langchain.com/openai/v1.

2. Make a call

A 200 response with a chat completion confirms the gateway, your API key, and your role permissions are all working.

3. View your trace

Open the LangSmith UI and navigate to the tracing project named gateway or gateway-<short_api_key>-<api_key_id> in the workspace associated with your API key. You should see a new trace for the call you just made.
If your application also emits its own LangSmith traces (for example, via LangChain or LangGraph tracing), the gateway-side trace and your application trace appear as separate runs. Linking gateway traces to the parent application run is not yet supported.

4. Set a spend policy (optional)

Go to Settings → Gateway → LLM Gateway in LangSmith to create a spend policy. For example, you can set a daily $10 cap on your API key. When the cap is reached, the gateway returns a 402 response with the message: "Request blocked by gateway policies: R&D Spend Cap". See Spend policies for the full guide on policy dimensions, time windows, and conflict resolution.

How the gateway handles requests

Here’s what the gateway did when you made a call:
  1. Authenticated your request using the LangSmith API key.
  2. Resolved the actual provider API key (for example, OPENAI_API_KEY) from your workspace’s Provider Secrets.
  3. Evaluated any active policies (spend limits, PII redaction, secrets redaction).
  4. Proxied the request to the upstream provider.
  5. Traced the call to LangSmith, including token counts, cost, and any policy events.
Routing through the gateway requires no application code changes.

Next steps