Rui Li, Shuang Cao
Hill Research
ACM SoCC 2026 · 2026-11-17
MosaicKV treats KV-cache pages as a shared resource that can move between GPU HBM and host DRAM as workloads compete for memory. It uses the decoder's predictable layer order to prefetch pages and schedule microbatches around PCIe transfers. This gives a cloud operator a way to manage tail latency for time to first token when tenants collectively need more cache capacity than the GPU provides.
MosaicKV将KV缓存页视为多个租户共享的资源,根据显存竞争情况在GPU的HBM与主机DRAM之间迁移。系统利用解码器逐层访问顺序可预测的特点进行预取,并按PCIe传输进度安排微批次。在租户的缓存总需求超过GPU显存容量时,这种调度可用于控制首词元延迟的长尾。
Feng Yu, Hongshi Tan, Yao Chen, Gustavo Alonso, Weng-Fai Wong, Bingsheng He
National University of Singapore, Huazhong University of Science and Technology, ETH Zürich
ACM ATC 2026 · 2026-11-15
JetFire scans a quantized KV-cache copy in GPU HBM to identify the tokens that contribute most to attention, then uses full-precision data in host DRAM to correct their contributions. At the longest context lengths shared by both systems, JetFire-K3V2 ran end to end 1.3–3.0× as fast as TurboQuant in vLLM. The split lets the system store a compact copy of the full context on the GPU and use high precision selectively.
JetFire先扫描GPU HBM中的量化KV缓存副本,找出对注意力贡献最大的词元,再利用主机DRAM中的全精度数据修正这些词元的贡献。在两种方法均支持的最长上下文条件下,JetFire-K3V2在vLLM中的端到端速度为TurboQuant的1.3至3.0倍。这种分工让GPU只需保留完整上下文的低精度副本,并将高精度计算集中在关键位置。
Tuowei Wang, Liyun Chu, Ruwen Fan, Ju Ren
Tsinghua University
ACM ATC 2026 · 2026-11-15
Octopus uses correlations in sparse attention to identify groups of KV-cache entries that are frequently fetched together. It distributes those groups across SSDs in advance and selectively replicates them so reads can be spread across drives. The design addresses the I/O bottleneck of using flash as a lower-cost capacity tier for persistent inference caches.
Octopus利用稀疏注意力中的访问相关性,找出经常被一同读取的KV缓存条目。系统预先将这些条目组分布到多块SSD,并有选择地复制部分条目组,使读取负载可以在多块盘之间分摊。该设计针对把闪存用作持久化推理缓存的低成本容量层时遇到的I/O瓶颈。