What Is an MCP Gateway? Architecture, Examples, Rate Limits and Open Source Options
An MCP gateway is the single front door between your AI agents and every MCP server they call. This article shows how the architecture works, three real setups, rate limit patterns that hold up under agent traffic, and the open source gateways you can run today.
Your first MCP server takes an afternoon. Your tenth takes a security review, three credential rotations and one angry message about a runaway agent that hammered an internal API 4,000 times in a minute. That is the moment most teams go looking for an MCP gateway.
An MCP gateway is a proxy that sits between AI clients (agents, chat apps, IDE assistants) and the MCP servers that expose tools and data. Instead of every client connecting to every server directly, they all connect to one controlled entry point. That entry point handles authentication, routing, policy, rate limits and logging, so individual servers do not have to.
This article walks through what a gateway does, how the architecture fits together, what real setups look like, how to design rate limits that survive agent traffic, and which open source options are worth your time.
What an MCP Gateway Actually Does
Think of a toll plaza. Traffic arrives from many directions, each vehicle gets checked once, and the flow is shaped before it enters the road beyond. A gateway does the same for tool calls.
The Problem It Solves
Model Context Protocol (MCP) standardizes how an LLM client discovers and calls tools. That is great for interoperability, but it creates a messy topology at scale. If you have 8 clients and 15 servers, you can end up with 120 separate connections, each with its own credential, its own logging format and its own failure mode.
Common pain points without a gateway:
Credential sprawl: every client stores a token for every server.
No central audit trail: nobody can say which agent called which tool, with which arguments.
Uneven protection: one server has rate limits, nine do not.
Painful onboarding: adding a server means editing config on every client.
Tool overload: agents see hundreds of tool definitions and burn context on descriptions they never use.
The Core Components
Most gateways, regardless of vendor, combine the same building blocks:
Component
Job
Why it matters
Front door endpoint
One MCP URL that clients connect to
Clients need only one config entry
Authentication layer
OAuth, OIDC, API tokens, mTLS
Identity is verified once, centrally
Router / registry
Maps tool names to upstream servers
Servers can move without client changes
Policy engine
Allow or deny by user, tool, argument
Least privilege for agents
Rate limiter
Caps calls per identity, tool or tenant
Protects upstream systems and budgets
Observability
Logs, traces, metrics per call
Debugging and compliance
Transport bridge
Converts stdio, SSE and streamable HTTP
Old servers work with new clients
💡 A gateway is not the same as an LLM gateway. An LLM gateway proxies calls to model providers (chat completions, embeddings). An MCP gateway proxies calls to tools and data sources. Many products now do both, but the traffic patterns are different.
How the Architecture Fits Together
Request Flow Step by Step
A typical tool call moves through the gateway like this:
Client connects. The agent opens a session to the gateway's MCP endpoint and presents a credential.
Identity is resolved. The gateway validates the token and maps it to a user, a team or a tenant.
Tool listing is filtered. When the client asks what tools exist, the gateway returns only the ones that identity may use. This also trims the context the model has to read.
Call is authorized. On tools/call, the policy engine checks the tool, the arguments and any conditions such as time of day or data classification.
Limits are checked. The rate limiter decrements a counter. If the budget is spent, the call is rejected with a clear error the agent can reason about.
Request is routed. The gateway forwards the call to the right upstream server, injecting that server's credential so the client never sees it.
Response is returned and logged. The result flows back, with latency, status and payload metadata written to the audit log.
Stateless vs Stateful Sessions
MCP sessions can carry state, and that shapes how you scale the gateway. Streamable HTTP lets you run stateless requests, which scale horizontally with no fuss. Features like server-initiated messages, subscriptions and long-running tool calls often need session affinity, meaning the same client keeps hitting the same gateway replica, or the session state lives in a shared store.
A practical rule: start stateless, add a shared store only when a tool really needs it. Sticky sessions at the load balancer are the cheaper fix for most teams.
Where It Runs
Three deployment shapes show up again and again:
Sidecar or local process: a lightweight gateway on the developer's machine that fronts a handful of local servers.
Central service: one gateway cluster in your cloud account, behind your identity provider.
Per-tenant instances: one gateway per customer in a multi-tenant product, which isolates blast radius at the cost of more infrastructure.
The transport bridge deserves a mention. Plenty of useful MCP servers still speak stdio only, which means they run as a local subprocess. A gateway can wrap them, run them in a container and expose them over HTTP, so remote clients can use them without anyone installing anything locally.
Real Examples of Gateway Setups
Solo Developer Setup
You use an IDE assistant, a chat client and a terminal agent. Each needs a filesystem server, a Git server and a database server. Without a gateway you copy the same three config blocks into three apps.
With a local gateway, you register the three servers once, and every client points to http://localhost:PORT/mcp. Add a new server on Tuesday and all three apps see it immediately. Rate limits here are mostly a seatbelt: cap the database tool at something like 30 calls a minute so a looping agent cannot hammer your dev database.
Team Setup With SSO
A 40-person engineering team shares access to an issue tracker, a wiki, a CI system and a production read replica. The gateway sits behind single sign-on:
Engineers authenticate with their company identity. No shared tokens.
A group-to-tool mapping gives support staff read-only wiki tools, developers CI tools, and only the on-call group the production read replica.
Every call is logged with the human's identity, not a generic service account.
Offboarding is one switch in the identity provider.
This is the case where gateways pay for themselves fastest, because the alternative is a spreadsheet of who holds which token.
Multi-Tenant SaaS
You ship a product where customers connect their own agents to your tools. Now the gateway is part of your public surface. You need per-tenant isolation, per-tenant quotas, usage metering for billing, and strict argument validation, because the caller is no longer an employee you trust.
Here, the rate limiter does double duty as a commercial control: a free plan might allow 1,000 tool calls a day, a paid plan 100,000, and bursts get throttled rather than rejected.
💡 Whatever the setup, log the tool name, caller identity, latency, status and argument size. Be careful about logging full arguments and results, since they often contain personal or confidential data.
Rate Limits That Make Sense
Agent traffic does not look like human traffic. A person clicks a button every few seconds. An agent can fire ten tool calls in parallel, retry on any error, and loop until it succeeds or someone stops it. Rate limits are how you keep a helpful automation from becoming a denial of service against your own systems.
Limit by Identity, Not IP
Behind a corporate proxy, hundreds of users share one IP address, and one agent platform may rotate IPs constantly. The useful dimensions are:
Per user or per agent identity: stops one runaway loop.
Per tool: a send_email tool deserves a far lower limit than search_docs.
Per tenant or team: keeps noisy neighbors from starving others.
Per upstream server: protects fragile backends that have their own limits.
Global: a last-resort ceiling for the whole gateway.
This is an illustrative sketch, not the syntax of any specific product. Every gateway has its own config format, but the shape of the policy transfers.
Request Limits vs Token Budgets
Two different things can be counted:
Limit type
Counts
Best for
Request-based
Tool calls per time window
Protecting backends and APIs
Token-based
LLM tokens consumed by tool results
Controlling model spend and context size
Concurrency
Calls in flight at once
Slow or expensive tools
Payload size
Bytes in or out
Preventing giant results that flood context
Token-based limits matter for MCP in a way they do not for classic APIs. A tool that returns a 200,000-character document costs you nothing at the HTTP layer, but it can eat a huge slice of the model's context window and your bill. Some gateways now support token-aware limits, so check that before you assume request counts are enough.
Where Counters Live
This is the design choice that bites teams later. Counters can live in three places:
In memory, per replica. Fastest, adds almost no latency, but each replica counts on its own. With 4 replicas and a limit of 100, a user can effectively make 400.
Shared store such as Redis. Accurate across replicas, with a small latency cost per call and one more dependency to run.
Gateway-native distributed counters. Same accuracy, but the feature is sometimes reserved for a paid or enterprise tier in open source projects.
For a single-node gateway, in-memory is fine. For anything with several replicas and hard quotas, plan for a shared store.
💡 Return a machine-readable rejection with a retry-after hint. A well-behaved agent can back off. A bare "error" makes the model retry instantly and make things worse.
Common algorithms include token bucket (allows bursts, refills steadily), fixed window (simple, but allows double bursts at window edges) and sliding window (smoother, a bit more state). Token bucket is the usual default for tool calls because agents are bursty by nature.
Open Source Gateways Worth Trying
The landscape moves quickly, and many "best of" lists are written by vendors who rank their own product first. Treat the names below as a starting shortlist, and check each project's repository for current license, activity and feature status before committing.
Container and Kubernetes First
Docker MCP Gateway: described as Apache 2.0 licensed, it runs each MCP server in an isolated container, with support for signed images and injected secrets. It ships alongside Docker's desktop tooling, which makes it a natural pick for local developer setups.
Microsoft MCP Gateway: a reverse proxy and management layer aimed at Kubernetes, reported as MIT licensed. A good match if your platform team already lives in clusters.
Envoy AI Gateway: built on the Envoy proxy and now offering MCP routing. If you already run Envoy, adding MCP routing may be closer to a configuration change than a new component.
Registry and Federation Style
IBM ContextForge: an Apache 2.0 project that acts as gateway, registry and proxy, federating tools, agents and APIs behind one MCP endpoint. It is feature rich and popular, but the repository states it has no official support from IBM, so budget your own maintenance.
Obot: an open source platform that bundles a gateway with catalog and access management.
Lasso: security-focused, with an emphasis on inspecting traffic between agents and tools.
Performance and API Platform Style
agentgateway: a Linux Foundation project with token-based rate limiting in the open source tier. Reports indicate that global, cross-replica limits and budget controls sit in an enterprise control plane, so verify this against your needs.
Bifrost: Apache 2.0, with in-memory rate limiting for very low overhead. Remember the caveat above: counters are local to each instance.
Kong and Apache APISIX: mature API gateways that handle MCP through plugins. If you already run one of them, you get distributed rate limiting and a familiar operating model, though MCP-specific features (tool-level filtering, session handling) may be thinner than in purpose-built gateways.
Quick Comparison
Gateway
Best fit
Rate limit angle
Watch out for
Docker MCP Gateway
Local and container workflows
Basic
Less suited to big multi-tenant setups
Microsoft MCP Gateway
Kubernetes platforms
Platform dependent
Requires cluster skills
Envoy AI Gateway
Teams already on Envoy
Envoy filters
Steeper configuration
IBM ContextForge
Federation and registry
Built in
Community support only
agentgateway
Token-aware limits
Token-based
Cross-replica limits may be paid
Bifrost
Low overhead
In-memory
Per-replica counters
Kong / APISIX
Existing API estates
Plugin based
Thinner MCP features
Security and Observability
A gateway is also your best place to catch the weird stuff. Agents read tool descriptions written by third parties, and those descriptions can contain hidden instructions. Tool results can carry injected text. A central point lets you inspect, filter and record all of it.
Controls Worth Turning On
Allowlists: expose only the tools a role needs, rather than everything a server offers.
Argument validation: reject malformed or suspiciously broad inputs, such as a SQL tool receiving a DROP statement.
Human approval for risky tools: require confirmation for anything that sends, pays, deletes or deploys.
Pinned tool definitions: alert when a server quietly changes a tool's description after you approved it.
Secret injection: keep upstream credentials in the gateway, never in client configs or model context.
What to Measure
You want dashboards for calls per tool, error rate per upstream, p95 latency, rejections from rate limits, and top callers. A spike in rejections for one identity is usually a looping agent. A spike across everyone usually means an upstream slowed down and retries piled up.
How to Pick One
A Short Checklist
Before choosing, answer these questions:
Where will it run? Laptop, cloud VM, Kubernetes, or edge?
Who authenticates? Do you need your existing OIDC provider, or are tokens enough?
How many replicas? If more than one, how will rate limit counters be shared?
Which transports? Do you need to wrap stdio servers?
Who maintains it? A project with a foundation or active company behind it beats a weekend repo.
Can you export logs? Audit data should flow into the tools you already use.
Three Common Mistakes
Putting everything behind one flat permission. If every user can call every tool, you have built an expensive proxy, not a gateway.
Ignoring the replica problem. In-memory limits look perfect in a demo and leak badly in production.
Skipping a pilot. Roll out to one team, watch the logs for a week, tune limits against real usage, then widen access.
A sensible rollout order: single gateway in front of two or three low-risk servers, then SSO, then tool-level policy, then rate limits tuned from real numbers, then high-risk tools with approval flows.
Test Your Agents With Better Models
A gateway is only as useful as the agents behind it, and tool calling quality varies a lot between models. If you are wiring up an MCP setup and want to compare how different LLMs handle multi-step tool use, you can try several on Picasso IA without setting up separate accounts:
Draft your tool descriptions, paste in a sample policy, and ask each model to find gaps or ambiguities. Short, precise descriptions make agents choose the right tool more often, and that saves you rate limit budget too.
When you are ready to go beyond text, browse everything available at picassoia.com/en/all-models and start experimenting with your own images, video and audio projects on Picasso IA today.