Large Language Models

What Is an MCP Gateway? Architecture, Examples, Rate Limits and Open Source Options

An MCP gateway is the single front door between your AI agents and every MCP server they call. This article shows how the architecture works, three real setups, rate limit patterns that hold up under agent traffic, and the open source gateways you can run today.

What Is an MCP Gateway? Architecture, Examples, Rate Limits and Open Source Options
Cristian Da Conceicao

Your first MCP server takes an afternoon. Your tenth takes a security review, three credential rotations and one angry message about a runaway agent that hammered an internal API 4,000 times in a minute. That is the moment most teams go looking for an MCP gateway.

An MCP gateway is a proxy that sits between AI clients (agents, chat apps, IDE assistants) and the MCP servers that expose tools and data. Instead of every client connecting to every server directly, they all connect to one controlled entry point. That entry point handles authentication, routing, policy, rate limits and logging, so individual servers do not have to.

This article walks through what a gateway does, how the architecture fits together, what real setups look like, how to design rate limits that survive agent traffic, and which open source options are worth your time.

What an MCP Gateway Actually Does

Aerial view of many highway lanes merging into one toll plaza with checkpoint booths

Think of a toll plaza. Traffic arrives from many directions, each vehicle gets checked once, and the flow is shaped before it enters the road beyond. A gateway does the same for tool calls.

The Problem It Solves

Model Context Protocol (MCP) standardizes how an LLM client discovers and calls tools. That is great for interoperability, but it creates a messy topology at scale. If you have 8 clients and 15 servers, you can end up with 120 separate connections, each with its own credential, its own logging format and its own failure mode.

Common pain points without a gateway:

  • Credential sprawl: every client stores a token for every server.
  • No central audit trail: nobody can say which agent called which tool, with which arguments.
  • Uneven protection: one server has rate limits, nine do not.
  • Painful onboarding: adding a server means editing config on every client.
  • Tool overload: agents see hundreds of tool definitions and burn context on descriptions they never use.

The Core Components

Most gateways, regardless of vendor, combine the same building blocks:

ComponentJobWhy it matters
Front door endpointOne MCP URL that clients connect toClients need only one config entry
Authentication layerOAuth, OIDC, API tokens, mTLSIdentity is verified once, centrally
Router / registryMaps tool names to upstream serversServers can move without client changes
Policy engineAllow or deny by user, tool, argumentLeast privilege for agents
Rate limiterCaps calls per identity, tool or tenantProtects upstream systems and budgets
ObservabilityLogs, traces, metrics per callDebugging and compliance
Transport bridgeConverts stdio, SSE and streamable HTTPOld servers work with new clients

💡 A gateway is not the same as an LLM gateway. An LLM gateway proxies calls to model providers (chat completions, embeddings). An MCP gateway proxies calls to tools and data sources. Many products now do both, but the traffic patterns are different.

How the Architecture Fits Together

Software architect sketching a system diagram on a glass whiteboard in a bright office

Request Flow Step by Step

A typical tool call moves through the gateway like this:

  1. Client connects. The agent opens a session to the gateway's MCP endpoint and presents a credential.
  2. Identity is resolved. The gateway validates the token and maps it to a user, a team or a tenant.
  3. Tool listing is filtered. When the client asks what tools exist, the gateway returns only the ones that identity may use. This also trims the context the model has to read.
  4. Call is authorized. On tools/call, the policy engine checks the tool, the arguments and any conditions such as time of day or data classification.
  5. Limits are checked. The rate limiter decrements a counter. If the budget is spent, the call is rejected with a clear error the agent can reason about.
  6. Request is routed. The gateway forwards the call to the right upstream server, injecting that server's credential so the client never sees it.
  7. Response is returned and logged. The result flows back, with latency, status and payload metadata written to the audit log.

Stateless vs Stateful Sessions

MCP sessions can carry state, and that shapes how you scale the gateway. Streamable HTTP lets you run stateless requests, which scale horizontally with no fuss. Features like server-initiated messages, subscriptions and long-running tool calls often need session affinity, meaning the same client keeps hitting the same gateway replica, or the session state lives in a shared store.

A practical rule: start stateless, add a shared store only when a tool really needs it. Sticky sessions at the load balancer are the cheaper fix for most teams.

Where It Runs

Three deployment shapes show up again and again:

  • Sidecar or local process: a lightweight gateway on the developer's machine that fronts a handful of local servers.
  • Central service: one gateway cluster in your cloud account, behind your identity provider.
  • Per-tenant instances: one gateway per customer in a multi-tenant product, which isolates blast radius at the cost of more infrastructure.

Close-up of a hand plugging a universal adapter into a laptop with several cables coiled beside it

The transport bridge deserves a mention. Plenty of useful MCP servers still speak stdio only, which means they run as a local subprocess. A gateway can wrap them, run them in a container and expose them over HTTP, so remote clients can use them without anyone installing anything locally.

Real Examples of Gateway Setups

Solo Developer Setup

You use an IDE assistant, a chat client and a terminal agent. Each needs a filesystem server, a Git server and a database server. Without a gateway you copy the same three config blocks into three apps.

With a local gateway, you register the three servers once, and every client points to http://localhost:PORT/mcp. Add a new server on Tuesday and all three apps see it immediately. Rate limits here are mostly a seatbelt: cap the database tool at something like 30 calls a minute so a looping agent cannot hammer your dev database.

Team Setup With SSO

Hands checking a visitor badge at a marble reception desk in a corporate lobby

A 40-person engineering team shares access to an issue tracker, a wiki, a CI system and a production read replica. The gateway sits behind single sign-on:

  • Engineers authenticate with their company identity. No shared tokens.
  • A group-to-tool mapping gives support staff read-only wiki tools, developers CI tools, and only the on-call group the production read replica.
  • Every call is logged with the human's identity, not a generic service account.
  • Offboarding is one switch in the identity provider.

This is the case where gateways pay for themselves fastest, because the alternative is a spreadsheet of who holds which token.

Multi-Tenant SaaS

You ship a product where customers connect their own agents to your tools. Now the gateway is part of your public surface. You need per-tenant isolation, per-tenant quotas, usage metering for billing, and strict argument validation, because the caller is no longer an employee you trust.

Here, the rate limiter does double duty as a commercial control: a free plan might allow 1,000 tool calls a day, a paid plan 100,000, and bursts get throttled rather than rejected.

💡 Whatever the setup, log the tool name, caller identity, latency, status and argument size. Be careful about logging full arguments and results, since they often contain personal or confidential data.

Rate Limits That Make Sense

Water dam spillway passing a controlled stream through a single steel gate

Agent traffic does not look like human traffic. A person clicks a button every few seconds. An agent can fire ten tool calls in parallel, retry on any error, and loop until it succeeds or someone stops it. Rate limits are how you keep a helpful automation from becoming a denial of service against your own systems.

Limit by Identity, Not IP

Behind a corporate proxy, hundreds of users share one IP address, and one agent platform may rotate IPs constantly. The useful dimensions are:

  • Per user or per agent identity: stops one runaway loop.
  • Per tool: a send_email tool deserves a far lower limit than search_docs.
  • Per tenant or team: keeps noisy neighbors from starving others.
  • Per upstream server: protects fragile backends that have their own limits.
  • Global: a last-resort ceiling for the whole gateway.

A sensible starting policy might look like this:

limits:
  default:
    per_user: 120 calls/minute
  tools:
    search_docs:   { per_user: 300 calls/minute }
    run_sql_query: { per_user: 30 calls/minute, burst: 5 }
    send_email:    { per_user: 10 calls/hour }
  tenant:
    free_plan: 1000 calls/day
    pro_plan:  100000 calls/day

This is an illustrative sketch, not the syntax of any specific product. Every gateway has its own config format, but the shape of the policy transfers.

Request Limits vs Token Budgets

Two different things can be counted:

Limit typeCountsBest for
Request-basedTool calls per time windowProtecting backends and APIs
Token-basedLLM tokens consumed by tool resultsControlling model spend and context size
ConcurrencyCalls in flight at onceSlow or expensive tools
Payload sizeBytes in or outPreventing giant results that flood context

Token-based limits matter for MCP in a way they do not for classic APIs. A tool that returns a 200,000-character document costs you nothing at the HTTP layer, but it can eat a huge slice of the model's context window and your bill. Some gateways now support token-aware limits, so check that before you assume request counts are enough.

Where Counters Live

This is the design choice that bites teams later. Counters can live in three places:

  • In memory, per replica. Fastest, adds almost no latency, but each replica counts on its own. With 4 replicas and a limit of 100, a user can effectively make 400.
  • Shared store such as Redis. Accurate across replicas, with a small latency cost per call and one more dependency to run.
  • Gateway-native distributed counters. Same accuracy, but the feature is sometimes reserved for a paid or enterprise tier in open source projects.

For a single-node gateway, in-memory is fine. For anything with several replicas and hard quotas, plan for a shared store.

💡 Return a machine-readable rejection with a retry-after hint. A well-behaved agent can back off. A bare "error" makes the model retry instantly and make things worse.

Common algorithms include token bucket (allows bursts, refills steadily), fixed window (simple, but allows double bursts at window edges) and sliding window (smoother, a bit more state). Token bucket is the usual default for tool calls because agents are bursty by nature.

Open Source Gateways Worth Trying

Developer at a standing desk with two monitors showing blurred terminal output

The landscape moves quickly, and many "best of" lists are written by vendors who rank their own product first. Treat the names below as a starting shortlist, and check each project's repository for current license, activity and feature status before committing.

Container and Kubernetes First

  • Docker MCP Gateway: described as Apache 2.0 licensed, it runs each MCP server in an isolated container, with support for signed images and injected secrets. It ships alongside Docker's desktop tooling, which makes it a natural pick for local developer setups.
  • Microsoft MCP Gateway: a reverse proxy and management layer aimed at Kubernetes, reported as MIT licensed. A good match if your platform team already lives in clusters.
  • Envoy AI Gateway: built on the Envoy proxy and now offering MCP routing. If you already run Envoy, adding MCP routing may be closer to a configuration change than a new component.

Registry and Federation Style

  • IBM ContextForge: an Apache 2.0 project that acts as gateway, registry and proxy, federating tools, agents and APIs behind one MCP endpoint. It is feature rich and popular, but the repository states it has no official support from IBM, so budget your own maintenance.
  • Obot: an open source platform that bundles a gateway with catalog and access management.
  • Lasso: security-focused, with an emphasis on inspecting traffic between agents and tools.

Performance and API Platform Style

  • agentgateway: a Linux Foundation project with token-based rate limiting in the open source tier. Reports indicate that global, cross-replica limits and budget controls sit in an enterprise control plane, so verify this against your needs.
  • Bifrost: Apache 2.0, with in-memory rate limiting for very low overhead. Remember the caveat above: counters are local to each instance.
  • Kong and Apache APISIX: mature API gateways that handle MCP through plugins. If you already run one of them, you get distributed rate limiting and a familiar operating model, though MCP-specific features (tool-level filtering, session handling) may be thinner than in purpose-built gateways.

Quick Comparison

GatewayBest fitRate limit angleWatch out for
Docker MCP GatewayLocal and container workflowsBasicLess suited to big multi-tenant setups
Microsoft MCP GatewayKubernetes platformsPlatform dependentRequires cluster skills
Envoy AI GatewayTeams already on EnvoyEnvoy filtersSteeper configuration
IBM ContextForgeFederation and registryBuilt inCommunity support only
agentgatewayToken-aware limitsToken-basedCross-replica limits may be paid
BifrostLow overheadIn-memoryPer-replica counters
Kong / APISIXExisting API estatesPlugin basedThinner MCP features

Security and Observability

Woman reviewing a wall of blurred screens in an operations room while holding a coffee cup

A gateway is also your best place to catch the weird stuff. Agents read tool descriptions written by third parties, and those descriptions can contain hidden instructions. Tool results can carry injected text. A central point lets you inspect, filter and record all of it.

Controls Worth Turning On

  • Allowlists: expose only the tools a role needs, rather than everything a server offers.
  • Argument validation: reject malformed or suspiciously broad inputs, such as a SQL tool receiving a DROP statement.
  • Human approval for risky tools: require confirmation for anything that sends, pays, deletes or deploys.
  • Pinned tool definitions: alert when a server quietly changes a tool's description after you approved it.
  • Secret injection: keep upstream credentials in the gateway, never in client configs or model context.

Top-down desk with a network diagram, magnifying glass and a small padlock

What to Measure

You want dashboards for calls per tool, error rate per upstream, p95 latency, rejections from rate limits, and top callers. A spike in rejections for one identity is usually a looping agent. A spike across everyone usually means an upstream slowed down and retries piled up.

How to Pick One

A Short Checklist

Before choosing, answer these questions:

  1. Where will it run? Laptop, cloud VM, Kubernetes, or edge?
  2. Who authenticates? Do you need your existing OIDC provider, or are tokens enough?
  3. How many replicas? If more than one, how will rate limit counters be shared?
  4. Which transports? Do you need to wrap stdio servers?
  5. Who maintains it? A project with a foundation or active company behind it beats a weekend repo.
  6. Can you export logs? Audit data should flow into the tools you already use.

Three engineers around a laptop in a loft office with exposed brick walls

Three Common Mistakes

  • Putting everything behind one flat permission. If every user can call every tool, you have built an expensive proxy, not a gateway.
  • Ignoring the replica problem. In-memory limits look perfect in a demo and leak badly in production.
  • Skipping a pilot. Roll out to one team, watch the logs for a week, tune limits against real usage, then widen access.

A sensible rollout order: single gateway in front of two or three low-risk servers, then SSO, then tool-level policy, then rate limits tuned from real numbers, then high-risk tools with approval flows.

Test Your Agents With Better Models

A gateway is only as useful as the agents behind it, and tool calling quality varies a lot between models. If you are wiring up an MCP setup and want to compare how different LLMs handle multi-step tool use, you can try several on Picasso IA without setting up separate accounts:

Draft your tool descriptions, paste in a sample policy, and ask each model to find gaps or ambiguities. Short, precise descriptions make agents choose the right tool more often, and that saves you rate limit budget too.

When you are ready to go beyond text, browse everything available at picassoia.com/en/all-models and start experimenting with your own images, video and audio projects on Picasso IA today.

Share this article

Choose your language

FreeCreate AI Images and VideosFree, No Credit CardFree Generations to StartSeedance 2.5 Lite IncludedStart Free Now