Envoy AI Gateway Capabilities
Welcome to the Envoy AI Gateway capabilities documentation! This section provides detailed information about the various features and capabilities that Envoy AI Gateway offers to help you manage and optimize your AI/LLM traffic.
LLM Providers Integrations
Support for various Large Language Model providers:
- Connecting to AI Providers: Learn how to establish connectivity with any supported AI provider
- Supported Providers: Compatible AI/LLM service providers
- Supported Endpoints: Available API endpoints and operations
- Vendor-Specific Fields: Use backend-specific parameters and access provider-unique capabilities in your OpenAI-compatible requests
Inference Optimization
Advanced inference optimization capabilities for AI/LLM workloads:
- InferencePool Support: Intelligent routing and load balancing for inference endpoints
- HTTPRoute + InferencePool: Basic inference routing with standard Gateway API
- AIGatewayRoute + InferencePool: Advanced AI-specific routing with enhanced features
Gateway Configuration
- GatewayConfig: Gateway-scoped configuration for the external processor (env vars, resources, shared settings)
Traffic Management
Comprehensive traffic handling and routing capabilities:
- Model Virtualization: Abstract and virtualize AI models
- Provider Fallback: Automatic failover between AI providers
- Usage-based Rate Limiting: Token-aware rate limiting for AI workloads
Security
Robust security features for AI gateway deployments:
- Upstream Authentication: Secure authentication to upstream AI services
Model Context Protocol (MCP)
Connect AI agents to external tools and data sources:
- MCP Gateway: Server multiplexing, tool routing, OAuth authentication, and observability for MCP workloads
Observability
Monitoring and observability tools for AI workloads:
- Metrics: Comprehensive metrics collection and monitoring