Enterprises have deployed AI into production: agents are running, workflows are live, and demos are a success. But beneath the momentum, a security problem is quietly forming. Who is accessing which models? What data flows through these systems? Are these effective guardrails in place? For most organizations, the honest answer is not entirely.
This is the gap that makes Palo Alto Networks’ intent to acquire Portkey a logical move. Portkey sits between enterprise applications and LLM providers like OpenAI, Anthropic, and Mistral, handling routing, failover, caching, cost tracking, and observability. Under Palo Alto, that functionality would be filtered through a security and governance lens, access controls, guardrails and visibility into what models are being accessed and by whom. For a cybersecurity company, that’s a natural and strategic extension. But what it signals about where enterprise AI infrastructure is headed is the more important story.
The infrastructure gap no one warned about
Over the past two years, enterprises have moved quickly on AI. Copilots, AI agents, and LLM-powered workflows are now staples of enterprise production. Yet, rapid progress at the application layer comes at the cost of building robust supporting infrastructure.
The result? Teams operate AI in production with limited visibility into agent behavior. Data from TrueFoundry’s 2026 Enterprise AI research revealed that over 75% of teams lack fully unified logging across their models and agent workflows. Over 40% only see AI costs after usage – no real-time alerts or budget controls. And 83% report token amplification, where a task expected to consume 1,000 tokens ends up consuming 8,000 or more through agent call chains. Latency spikes are difficult to diagnose, and tracking root causes of failures often requires engineering effort that compounds the original problem.
What an AI Gateway actually does
An AI Gateway is a control layer between enterprise applications and the large language models powering them. Rather than apps calling providers like OpenAI or Anthropic directly, requests flow through the gateway, centralizing routing, observability, reliability, and cost management in one place.
If a provider lags or fails, the gateway redirects traffic. When costs spike, teams can pinpoint causes in real time. Unexpected prompt behavior reveals full trace logs for investigation. Testing a new model doesn’t require redeploying apps.
For teams managing multiple AI features, this infrastructure quickly becomes essential. Without it, there is no single inference point for policy, no unified view of what the AI stack is doing, and no clean way to attribute costs or investigate incidents.
AI Gateways go well beyond security
Security and governance are one critical dimension of what a Gateway provides, and a legitimate reason Palo Alto’s move makes strategic sense. But enterprises also need cost controls, performance monitoring, provider flexibility, data governance, and model routing, including support for private and open-source options. Gateways scoped primarily to security force IT and engineering teams to cobble together multiple tools to fill the gaps.
The broader industry is recognizing this. According to Gartner’s 2025 Market Guide for AI Gateways, by 2028, 70% of software engineering teams building multimodal applications will use AI Gateways to improve reliability and optimize costs, not security alone. Gartner defines an AI Gateway as a technology that “acts as an intermediary between applications and various AI services or models,” providing a central control pane to secure, govern, and observe AI workloads. That scope is deliberately broad.
Organizations that get this right treat the AI Gateway as foundational infrastructure rather than a security add-on or an afterthought. They integrate it into the stack from the outset, just as they would an API gateway or service mesh, and expand their tool surface on top of it, not the reverse.
Key questions before building or buying
For enterprise teams evaluating AI Gateway options, or deciding whether they need one at all, the questions worth asking are:
- Do we have visibility into every LLM call running in production, including cost and latency?
- Can we reroute traffic if a provider goes down or becomes too expensive?
- Do we have the flexibility to use both commercial APIs and self-hosted models?
- Where does sensitive data flow, and who controls it?
If the honest answer to any of these is “not really,” there’s an infrastructure gap worth addressing. As AI shifts from experimental projects to core product features, the cost of leaving this gap unfilled only increases.
Palo Alto’s move to acquire Portkey marks a pivotal moment: AI Gateways are being recognized as essential enterprise infrastructure. The question now is how broadly enterprises choose to build on them.
Learn more about TrueFoundry here.
Related News:
