ContextPilot: Fast Long-Context Inference via Context Reuse

4–12× cache hits | 1.5–3× faster prefill | ~36% token savings across vLLM, SGLang, RAG, AI Agents, and more.

---

| Documentation | Examples | Benchmarks | Docker | Paper |

News

• [2026/03] ContextPilot now can run on macOS / Apple Silicon via llama.cpp.
• [2026/02] ContextPilot v0.3.2 released, supporting PageIndex and Mem0.
• [2026/01] ContextPilot has been accepted to MLSys 2026 🎉! See you in Bellevue, WA, USA.

About

Long-context workloads (RAG, memory chat, tool-augmented agents) prepend many context blocks. Across requests, these blocks often overlap but get reordered or duplicated, changing token prefixes and triggering cache misses and redundant KV recomputation. Common examples include (1) Trending Topic QA, (2) Closed-Domain Long-Context QA, (3) Batched Long-Context Inference, (4) multi-turn conversations with long-term memory and many more.

ContextPilot sits between context assembly and inference to maximize prefix reuse and remove duplicates:

1. Higher throughput & cache hits — boosts prefill throughput and prefix cache hit ratio via context reuse.
2. Drop-in solutions — works with PageIndex, Mem0, LMCache, and backends like vLLM / SGLang / llama.cpp.
3. No compromise in reasoning quality — can even improve with extremely long contexts.
4. Widely tested — validated across diverse RAG and agentic workloads.

It maintains a Context Index of cached content, then per request applies Reorder (align shared blocks into a common prefix) and/or Deduplicate (replace repeats with reference hints), plus cache-aware scheduling to maximize prefix sharing. The optimized prompt is sent via the OpenAI-compatible API; POST /evict keeps the index synced when KV cache is reclaimed. See its design overview below.

For more design details, see Paper and Documentation.

Performance at a Glance

ContextPilot is validated across three representative settings: single-node academic RAG, multi-node production MoE inference, and multi-turn memory-augmented chat. In every case it delivers significant speedups with comparable answer quality.

Qwen3-32B on 4×A6000 — single-node academic RAG with a 32B model on consumer GPUs.

Benchmark	Method	Prefill TP (tok/s)	Cache Hit	F1 (%)
MultihopRAG	SGLang	7,290	4.64%	60.42
	SGLang + ContextPilot	14,214	33.97%	64.39
NarrativeQA	SGLang	7,921	5.91%	28.41
	SGLang + ContextPilot	12,117	20.82%	29.64

DeepSeek-R1-671B on 16×H20 — production-scale 671B MoE inference on a multi-node GPU cluster.

Benchmark	Method	Prefill TP (tok/s)	Cache Hit	F1 (%)
MultihopRAG	SGLang	9,636	5.12%	64.15
	SGLang + ContextPilot	17,498	60.37%	64.68
NarrativeQA	SGLang	8,687	6.08%	40.20
	SGLang + ContextPilot	13,201	38.24%	41.08

Qwen3-4B on 1×A6000 — multi-turn memory chat with Mem0 on the LoCoMo benchmark.

Context Size	Method	TTFT (s)	LLM Judge
100 memories	SGLang	0.1012	0.437
	SGLang + ContextPilot	0.0554	0.420

ContextPilot results in mem0 table are without context annotation — an optional feature that adds original importance ranking to reordered context blocks, which can further improve answer quality (see Paper).

Llama-3.2-1B on Apple M3 (MacBook Air, 16 GB) — MultihopRAG on Apple Silicon with llama.cpp, no GPU server required.

Method	Avg Latency (ms)
llama.cpp	3,315
llama.cpp + ContextPilot	1,378

Settings: Llama-3.2-1B-Instruct-Q4_K_M.gguf, Metal offload (-ngl 99), --cache-reuse 256, --parallel 4, context 32768 tokens. See the Mac + llama.cpp guide.

Installation

Requirements: Python >= 3.10

---

vLLM / SGLang

ContextPilot works with both CPU and GPU backends for building the context index. The [gpu] extra enables GPU-accelerated distance computation (via cupy-cuda12x) and is faster for large batches; without it, ContextPilot falls back to the CPU backend automatically.

From PyPI — the vLLM and SGLang hooks are installed automatically:

pip install contextpilot          # CPU index computation
pip install "contextpilot[gpu]"   # GPU index computation (CUDA 12.x)

From source — run install_hook manually after install, since editable installs do not copy the .pth file to site-packages:

git clone https://github.com/EfficientContext/ContextPilot.git
cd ContextPilot
pip install -e .                  # CPU
pip install -e ".[gpu]"           # GPU (CUDA 12.x)
python -m contextpilot.install_hook   # one-time: enables automatic vLLM / SGLang integration

The install_hook step writes a .pth file into your site-packages so the vLLM and SGLang hooks load automatically at Python startup — no code changes required. To uninstall: python -m contextpilot.install_hook --remove.

---

Mac / Apple Silicon — llama.cpp

From PyPI:

pip install contextpilot
xcode-select --install    # one-time: provides clang++ to compile the native hook

From source:

git clone https://github.com/EfficientContext/ContextPilot.git
cd ContextPilot
pip install -e .
xcode-select --install    # one-time: provides clang++ to compile the native hook

Why xcode-select? The llama.cpp integration uses a small C++ shared library injected into llama-server via DYLD_INSERT_LIBRARIES. It is compiled automatically on first use and requires clang++ from Xcode Command Line Tools.

---

More detailed installation instructions are available in the docs.

Docker images are also available for both all-in-one and standalone deployment. See the Docker guide.

Getting Started

Quick Start with Context Ordering

Add one call (cp_instance.optimize()) before inference to rearrange context blocks so that shared content aligns into a common prefix, enabling cache reuse. An importance ranking in the prompt preserves accuracy.

Mode	When to Use	How It Works
Online	Multi-turn (e.g., chatbot + Mem0)	Tracks previously cached blocks; moves overlapping ones to the prefix each turn
Offline	Batch / single-shot	Globally reorders and schedules all requests for maximum prefix sharing

Both modes work with any OpenAI-compatible endpoint (vLLM, SGLang, etc.) — no changes to your inference deployment. They support both direct API calls (shown below) and HTTP server deployment (see the online usage guide).

---

Accelerating Online Inference

Multi-turn chatbot with Mem0 or RAG where each turn's context blocks partially overlap. cp_instance.optimize() moves shared blocks to the prefix so the engine reuses cached KV states.

from openai import OpenAI
# Step 1: Import ContextPilot
import contextpilot as cp

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
# Step 2: Create a ContextPilot instance
cp_instance = cp.ContextPilot(use_gpu=False)

for query in queries:
    contexts = get_contexts(query)                         # Mem0, Retriever, ...
    # Step 3: Optimize context ordering and build ready-to-use messages
    messages = cp_instance.optimize(contexts, query)

    response = client.chat.completions.create(
        model="Qwen/Qwen3-4B",
        messages=messages,
    )
    print(f"Q: {query}\nA: {response.choices[0].message.content}\n")

Note: When the engine evicts KV-cache entries under memory pressure, ContextPilot's index can go stale. Set CONTEXTPILOT_INDEX_URL when launching SGLang or vLLM to enable automatic eviction sync. For distributed setups, see Distributed Setup.

---

Accelerating Offline Inference

Batch of requests with overlapping context blocks. cp_instance.optimize_batch() globally reorders blocks and schedules execution order so queries with similar contexts run consecutively, maximizing cache reuse. See the offline usage guide for details. Offline mode can also be deployed as an HTTP server without eviction sync — see Stateless Mode.

import asyncio
import openai
# Step 1: Import ContextPilot
import contextpilot as cp

BASE_URL = "http://localhost:30000/v1"
# Step 2: Create a ContextPilot instance
cp_instance = cp.ContextPilot(use_gpu=False)

all_contexts = [get_contexts(q) for q in queries]          # Mem0, Retriever, ...
# Step 3: Optimize — reorder, schedule, and build prompts in one call
messages_batch, order = cp_instance.optimize_batch(all_contexts, queries)

# Send all requests concurrently
async def generate_all():
    ac = openai.AsyncOpenAI(base_url=BASE_URL, api_key="EMPTY")
    return await asyncio.gather(*[ac.chat.completions.create(
        model="Qwen/Qwen3-4B", messages=m
    ) for m in messages_batch])

for resp, idx in zip(asyncio.run(generate_all()), order):
    print(f"Q: {queries[idx]}\nA: {resp.choices[0].message.content}\n")

For a detailed walkthrough with concrete examples, see the Quick Start Guide. For more fine-grained control, you can also use cp_instance.reorder() and cp_instance.deduplicate() directly — see the API reference and multi-turn deduplication guide.

Adoption Examples

See many useful adoption examples: Mem0 integration, PageIndex RAG, offline batch scheduling, and multi-turn deduplication.

Citation

@inproceedings{contextpilot2026,
  title     = {ContextPilot: Fast Long-Context Inference via Context Reuse},
  author    = {Jiang, Yinsicheng and Huang, Yeqi and Cheng, Liang and Deng, Cheng and Sun, Xuan and Mai, Luo},
  booktitle = {Proceedings of the 9th Conference on Machine Learning and Systems (MLSys 2026)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2511.03475}
}

Contributing

We welcome and value all contributions! Please feel free to submit issues and pull requests.