AI Accelerators & Compute-in-Memory
AI加速器与存算一体
Shixin Zhao, Lian Liu, Tian Han, et al.
arXiv:2608.06989 · 2026-08-07
This paper argues that a single device-biased memory layout breaks down when an LLM tensor can move between an NPU and PIM across prefill, decode or MoE routing. Its PIM-as-Flexible-Memory design keeps a joint physical layout while exposing device-specific logical views, avoiding duplication and relayout. The open question is the translation and scheduling overhead on real NPU-PIM hardware.
论文认为，当同一大模型张量会在预填充、解码或MoE路由阶段于NPU和PIM之间移动时，单一的设备偏置内存布局会失效。其PIM-as-Flexible-Memory设计保留联合优化的物理布局，同时向不同设备提供各自的逻辑视图，从而避免复制和重排。未解问题是实际NPU-PIM硬件上的地址转换和调度开销。
Przemyslaw Forys, Haoran Wu, Can Xiao, et al.
arXiv:2608.03741 · 2026-08-04
HeteroPanacea is a simulation framework for agentic inference that separates prefill, decode, attention and FFN onto heterogeneous hardware choices. It combines disaggregated quantization, automated parallelization scheduling and device specialization to explore architecture trade-offs. Its conclusions necessarily depend on modeled hardware and workload assumptions.
HeteroPanacea是面向智能体推理的仿真框架，将预填充、解码、注意力和FFN映射到不同的异构硬件选择。它结合了解耦量化、自动并行调度和设备专用化来考察架构取舍。其结论必然依赖所建模的硬件和负载假设。
AI Inference Systems & Algorithms
AI推理系统与算法
Xu Gao
arXiv:2608.06763 · 2026-08-07
CubicQuant proposes a monotonic cubic mapping that turns dense integer codes into non-uniform weight reconstruction levels from 1 to 8 bits. It aims to retain the flexibility of learned codebooks without irregular decoding or extensive metadata, and describes packed-weight GPU execution. The central trade-off is whether per-group curve fitting improves enough over conventional quantization to justify extra format and kernel complexity.
CubicQuant提出单调三次映射，将密集整数编码转换为1至8比特的非均匀权重重建等级。它试图保留学习型码本的灵活性，同时避免不规则解码和大量元数据，并给出了打包权重的GPU执行方式。关键取舍在于，按组拟合曲线带来的提升是否足以抵消额外的格式与内核复杂度。
Muhammad Adnan, R. Mahapatra, Prashant J. Nair, et al.
arXiv:2608.06557 · 2026-08-06
Cascade assigns each request a continuously updated latency budget derived from its SLO, predicted remaining work, KV-cache state and current load. It uses that budget to coordinate both request ordering and KV-cache management, rather than treating deadlines as a queueing rule alone. Accurate remaining-time prediction remains the critical dependency.
Cascade为每个请求分配持续更新的延迟预算，依据其SLO、剩余工作量预测、KV缓存状态和当前负载计算。该预算同时协调请求排序与KV缓存管理，而不是只把截止时间当作排队规则。关键仍在于剩余时间预测是否准确。
Guotao Yang, M. Zhao, Haopeng Li, et al.
arXiv:2608.04991 · 2026-08-05
RAC reduces the communication bottleneck in local-cloud-local split inference by compressing boundary activations with references from repeated context. It combines exact-token retrieval, causal predictors, grouped affine alignment and calibrated residual quantization. The authors report raw-to-RAC ratios of 1.24–2.72× for time to first token and 1.01–2.79× for time per output token; the benefit should be strongest where repeated context exists and network transfer dominates.
RAC利用重复上下文提供的参考信息压缩边界激活值，以缓解本地—云端—本地分割推理中的通信瓶颈。它结合精确token检索、因果预测器、分组仿射对齐和校准残差量化。作者报告首token时间的Raw-to-RAC比值为1.24–2.72倍、每输出token时间为1.01–2.79倍；收益在重复上下文存在且网络传输占主导时最明显。
Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, et al.
arXiv:2608.02947 · 2026-08-03
ATFlash uses the frequency-pair structure of rotary position embeddings to give each RoPE wavelength its own distance window. Unlike a conventional sliding window, every key remains reachable through low-frequency pairs while shorter-wavelength terms are pruned. On Qwen2.5-0.5B and Llama-3.2-3B, it removes 37–48% of query-key inner-product terms while retaining a 96–98% top-1 match rate, though the evidence is limited to the evaluated models and benchmarks.
ATFlash利用旋转位置编码的频率对结构，为每个RoPE波长设置独立的距离窗口。不同于传统滑动窗口，每个key仍可通过低频对访问，而短波长项被裁剪。在Qwen2.5-0.5B和Llama-3.2-3B上，它去除了37–48%的query-key内积项，同时保持96–98%的top-1匹配率；不过证据仍限于所评测的模型与基准。