Skip to content
AIBRAINWORX / THE INDEPENDENT AI LABRESEARCH · HARDWARE · OPEN INTELLIGENCE
Inference & Hardware / Enterprise & Legal / AI News

Amazon SageMaker AI Adds KV Cache-Aware Routing for Real-Time LLM Endpoints

AWS introduces KV cache-aware routing for Amazon SageMaker AI real-time endpoints, pairing live container cache telemetry with load-aware routing for multi-turn chat, RAG, and agent workloads.

Listen to this articleGoogle Voice
0:00
Official Amazon Web Services data center server racks photograph
AI-generated conceptual illustration by AiBrainWorX; not a photograph or product screenshot.
Event / context: 2026-11-17. Source check: October 28, 2026.

Amazon Web Services has launched KV cache-aware routing for real-time endpoints on Amazon SageMaker AI, giving teams serving large language models a way to steer incoming requests toward compute instances that already hold relevant prompt computations in GPU memory. Detailed in an AWS Machine Learning Blog announcement published November 17, 2026, the feature aims to reduce redundant prompt prefill work across multi-turn conversations, Retrieval Augmented Generation (RAG) pipelines, and agentic applications with long shared system prompts.

Moving Beyond Blind Prefix Hashing

In LLM inference, reusing the key-value (KV) cache lets a model skip recomputing attention states for prompt prefixes it has already processed. SageMaker AI previously offered prefix-aware routing, which hashed a fixed byte or character window (PrefixLength) of an incoming request to send matching traffic to the same instance. According to AWS, that stateless approach had two practical blind spots: it could not detect when an instance evicted older KV entries under GPU memory pressure, and it struggled with short multi-turn chats where each added message altered the hashed text until the conversation grew longer than PrefixLength.

KV cache-aware routing replaces those inferences with actual state reported by the serving container. Instead of guessing where a prefix might still reside, the endpoint tracks which instances or inference component copies currently hold cached blocks and factors in real-time instance load before dispatching traffic.

How Live Cache Telemetry and Token Matching Work

AWS breaks the architecture into three components: container-level cache event reporting, a SageMaker-managed cache index, and a load-aware routing layer. When enabled, the model container emits an event whenever it stores or evicts KV cache entries. SageMaker AI ingests these updates into an endpoint-level index that maps one-way, truncated hashes of cached content to specific instances or inference component copies. AWS notes that the index stores only pointer hashes and never retains raw prompts, token IDs, request payloads, or KV tensors.

When a new request reaches the endpoint, SageMaker AI tokenizes the prompt and queries the index at the token level rather than across a rigid character boundary. Because matching happens on tokens, growing multi-turn histories can retain affinity with the instance holding earlier turns, and requests sharing only partial prefixes can be directed to the instance with the longest cached overlap. Crucially, the router balances cache affinity against instance utilization: if the instance with the longest cached match is too busy, SageMaker AI skips it and routes the request to the next-best available candidate.

Configuration, Scope, and Practical Considerations

Adopting the strategy requires two configuration updates and no changes to client API calls, according to AWS. Operators add the SM_VLLM_KV_EVENTS_CONFIG environment variable to their model container so it emits cache events, and set RoutingStrategy to KV_AWARE in the endpoint configuration. The capability applies to both single-model endpoints and inference component endpoints.

While AWS has not published independent third-party latency or throughput benchmarks in the announcement excerpt, the design targets a clear operational bottleneck in modern LLM serving: balancing memory locality against tail-latency spikes on overloaded GPUs. Teams evaluating the feature should note that the container configuration variable explicitly targets vLLM-based serving containers, meaning workloads built on other inference engines will need compatible event reporting or fall back to earlier routing modes.

Automated Vertex AI (gemini-4-hydrogen) primary-source report; reviewed 2026-11-17.

Follow your curiosity.

Pass the signal.