Unified KV Cache Compression Methods for Auto-Regressive Models
-
Updated
Jan 4, 2025 - Python
#
kv-cache-compression
Here are 34 public repositories matching this topic...
LLM KV cache compression made easy
python transformers inference pytorch kv-cache large-language-models llm long-context kv-cache-compression
-
Updated
May 26, 2026 - Python
Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.
-
Updated
Mar 3, 2025
llama.cpp fork with TurboQuant WHT-rotated KV cache & weight compression + Gemma 4 MTP and Qwen 3.6 NextN speculative decoding (+30-50% throughput).
metal cpp vulkan cuda quantization mtp multimodal apple-silicon low-bit-quantization llama-cpp local-llm llm-inference gguf speculative-decoding kv-cache-compression qwen3 multi-token-prediction turboquant gemma4 nextn
-
Updated
May 14, 2026 - C++
[NeurIPS'25 Oral] Query-agnostic KV cache eviction: 3–4× reduction in memory and 2× decrease in latency (Qwen3/2.5, Gemma3, LLaMA3)
-
Updated
Feb 11, 2026 - Python
Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)
-
Updated
Apr 13, 2025 - Python
[ICLR 2025] Palu: Compressing KV-Cache with Low-Rank Projection
-
Updated
Feb 20, 2025 - Python
First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression with near-zero quality loss.
machine-learning compression deep-learning pytorch transformer attention quantization iclr vector-quantization memory-optimization kv-cache google-research llm vllm llm-inference kv-cache-compression
-
Updated
May 25, 2026 - Python
Pytorch implementation for "Compressed Context Memory For Online Language Model Interaction" (ICLR'24)
-
Updated
Apr 18, 2024 - Python
This is the official repo of "QuickLLaMA: Query-aware Inference Acceleration for Large Language Models"
-
Updated
Jul 16, 2024 - Python
xKV: Cross-Layer SVD for KV-Cache Compression
-
Updated
May 27, 2026 - Python
(ACL2025 oral) SCOPE: Optimizing KV Cache Compression in Long-context Generation
-
Updated
May 28, 2025 - Jupyter Notebook
Accurate and fast KV cache compression with a gating mechanism
-
Updated
Apr 5, 2026 - Python
Native Windows build of vLLM 0.21.0 — no WSL, no Docker. Now for RTX 50-series (Blackwell, sm_120): Python 3.13 + CUDA 12.8 + PyTorch 2.11. Pre-built wheel + Windows patch, 10 KV-cache compression dtypes, and the OpenAI API server fixed to run on Windows.
windows gpu cuda pytorch nvidia triton msvc quantization kv-cache awq llm llm-serving vllm llm-inference flash-attention qwen kv-cache-compression turboquant vllm-windows multi-turboquant
-
Updated
May 26, 2026 - Python
First open-source KVTC implementation (NVIDIA, ICLR 2026) -- 8-32x KV cache compression via PCA + adaptive quantization + entropy coding
compression pytorch nvidia transformer pca attention dynamic-programming quantization deflate entropy-coding memory-optimization kv-cache llm llm-inference kv-cache-compression iclr-2026
-
Updated
Apr 17, 2026 - Python
A list of awesome papers on compression and acceleration of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs).
acceleration compression survey pruning quantization knowledge-distillation awesome-papers large-language-models multimodal-large-language-models speculative-decoding kv-cache-compression
-
Updated
May 12, 2026
Official repo for AMD hybrid models training and inference workflow
amd attention-mechanism mamba mla hybrid-models kv-cache large-language-models llm kv-cache-compression zebra-llama
-
Updated
May 14, 2026 - Python
Discrete Kakeya cover for LLM KV cache: D4/E8 nested-lattice quantisation realising a Kakeya-style tube-cover over the direction sphere. 2.4x-2.8x compression at <1% perplexity loss on Qwen3, Llama-3, DeepSeek, GLM-4, Gemma. Drop-in transformers.DynamicCache. pip install kakeyalattice.
transformers quantization discrete-geometry kv-cache long-context vllm llm-inference kv-cache-compression qwen3 lattice-quantization e8-lattice d4-lattice kakeya kakeya-set
-
Updated
Apr 30, 2026 - Python
LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
-
Updated
Sep 17, 2025 - Python
Research and training stack for AVA — a tool-using, memory-aware virtual assistant targeting 4 GB VRAM. Spans custom transformers, verifier-RL, external memory, multi-domain benchmarks, and Gemma 4 inference optimization.
ai reproducible-research benchmarks personal-assistant quantization external-memory gemma virtual-assistant distillation inference-optimization tool-use edge-inference int4 llm long-context small-language-models kv-cache-compression recurrent-depth 4gb-vram verifier-rl
-
Updated
May 20, 2026 - Python
Improve this page
Add a description, image, and links to the kv-cache-compression topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the kv-cache-compression topic, visit your repo's landing page and select "manage topics."