AI Accelerators & Compute-in-Memory
AI加速器与存算一体
Rocky Wang, Joyce Zhou, Keith Jiang, et al.
arXiv:2611.07535 · 2026-11-05T06:44:09Z
Measurements on an eight-B300 node show that CUDA 13.4 Locality Domains can improve performance through die-aware placement and execution, with reported speedups up to 43.5% for NCCL AllGather and 37% for DeepGEMM EP. The study also identifies allocation overhead above 32 MB associated with barriers and TLB invalidation. These preprint results make cross-die locality a concrete tuning consideration, with gains dependent on access patterns and workload.
在配备8颗B300的节点上,研究通过感知裸片位置的数据放置与执行方式,评估CUDA 13.4局部性域,报告NCCL AllGather和DeepGEMM EP的最高性能提升分别为43.5%和37%。研究还发现,超过32 MB的内存分配存在与屏障和TLB失效相关的额外开销。这些预印本实测结果说明,跨裸片局部性已成为具体的调优因素,但收益取决于访问模式和工作负载。
Nick Brown, Daniyal Arshad
arXiv:2611.04740 · 2026-11-03T11:57:36Z
Wolfpine simulates Tenstorrent's Wormhole and Blackhole Tensix architectures in Python and integrates with TT-Metal. Built from public documentation, open software and hardware measurements, it provides component tracing, typed events and timeline export for investigating execution bottlenecks. The preprint lowers the barrier to architecture exploration, while timing fidelity is added incrementally through an optional cost model.
Wolfpine以Python模拟Tenstorrent的Wormhole与Blackhole两代Tensix架构,并与TT-Metal集成。它基于公开文档、开源软件和硬件测量构建,提供组件追踪、类型化事件和时间线导出,便于定位执行瓶颈。该预印本工具降低了架构探索门槛,时序精度则通过可选的成本模型逐步补充。
AI Models & Inference Systems
AI模型与推理系统
Jake Hyun, Ziyu Zhao, Jae-sun Seo, et al.
arXiv:2611.08363 · 2026-11-05T17:42:06Z
StrataFlux streams selected precision refinements from host RAM and SSDs to improve low-bit weights and KV caches while overlapping transfers with computation. For 2-bit Llama-3.1-8B-Instruct weights on RTX 5090, it reduces WikiText-2 perplexity from 9.46 to 8.40 while retaining 95.4% of the quantized baseline's throughput. The preprint shows how the memory hierarchy can recover quantization quality, with the throughput result tied to the reported model and platform.
StrataFlux从主机内存与SSD中选择性传输用于精度修正的数据,并将传输与计算重叠,以改善低比特权重和KV缓存的精度。在RTX 5090上,针对采用2比特权重的Llama-3.1-8B-Instruct,它将WikiText-2困惑度从9.46降至8.40,同时保留量化基线95.4%的吞吐量。该预印本展示了利用存储层次结构弥补量化精度损失的路径,上述吞吐表现对应其报告的特定模型与平台。
Quentin Anthony, Ganesh Bikshandi, Praneeth Medepalli, et al.
arXiv:2611.07496 · 2026-11-05T05:58:36Z
ZYFA develops MI355X prefill kernels that combine direct FP8 probability computation, ping-pong scheduling of softmax and matrix multiplication, and power-of-two probability scaling. The motivation is a measured utilization gap: at 32K tokens, the cited AITER kernels reach 39% of peak FP8 matrix throughput versus 49% in BF16. The preprint addresses vector-unit work and instruction ordering that limit attention performance, so the headline FP8 arithmetic advantage alone is insufficient to predict speed.
ZYFA面向MI355X开发prefill内核,将直接计算FP8概率、交替执行softmax与矩阵乘法的乒乓调度,以及按2的幂缩放概率结合起来。其出发点是实测利用率差距:在32K token下,文中引用的AITER内核达到FP8矩阵峰值吞吐的39%,BF16则为49%。该预印本针对限制注意力性能的向量单元计算与指令排序开销展开优化,表明仅凭FP8算术峰值优势不足以预测实际速度。
Alexander Magro, Aristotle Martin, Ye Wang, et al.
arXiv:2611.07477 · 2026-11-05T05:36:08Z
KOSMOS lets GPU workgroups switch between communication and computation as data becomes ready, using consumer-demand-ordered queues inside persistent kernels. On MI350X and MI355X, the preprint reports 1.10–1.28× end-to-end BF16 TP+SP training speedups in Megatron-LM and 1.29–1.83× speedups over DeepEP for DeepSeek-V3 and Qwen3-235B proxy workloads. The contribution targets idle gaps from fixed schedules; the MoE results apply to the evaluated proxy training configurations.
KOSMOS在常驻内核中按消费者需求组织队列,使GPU工作组随数据就绪情况,在通信与计算任务之间切换。该预印本在MI350X和MI355X上报告,Megatron-LM的BF16张量并行加序列并行训练端到端加速为1.10至1.28倍;针对DeepSeek-V3与Qwen3-235B的代理工作负载,相对DeepEP加速1.29至1.83倍。其贡献在于减少固定调度造成的空闲间隙,MoE性能数字对应所评估的代理训练配置。
Yichao Fu, Yuxuan Zhang, Minshen Zhang, et al.
arXiv:2611.07149 · 2026-11-04T21:21:57Z
FastAFD separates attention and MoE expert execution on GB200 NVL72 and overlaps the work of two microbatches. Against the best tested vLLM wide-expert-parallel configuration, it reaches up to 1.17× throughput on Qwen3 and 1.16× on Qwen3.5, while matching the baseline on MiniMax-M2.5. An analytical model predicts throughput within 3% in the reported evaluation, offering a pre-deployment check of whether disaggregation pays for a particular workload.
FastAFD在GB200 NVL72上分离注意力与MoE专家执行,并将两个微批次的工作重叠。相对测试中最优的vLLM大规模专家并行配置,Qwen3和Qwen3.5的最高吞吐提升分别为1.17倍和1.16倍,MiniMax-M2.5则与基线持平。文中分析模型在所报告评估中的吞吐预测误差不超过3%,可用于在部署前判断特定工作负载是否适合解耦。
Thisara Rathnayaka Weerakoon, Ishan Dewpura, Niman Ransindu, et al.
University of Moratuwa / Independent Researcher
arXiv:2611.07105 · 2026-11-04T20:17:40Z
NUMA-Torch makes TorchInductor partition weights and compute by CPU socket, keeping thread pools and memory allocations local through CPU affinity. The approach targets cross-socket weight traffic that reduces effective DRAM bandwidth during prefill. The preprint evaluates GPT-2 variants from 117M to 762M parameters and BERT Small and Large, so extrapolating its findings to much larger models requires further measurement.
NUMA-Torch改造TorchInductor,按CPU插槽划分权重与计算,并通过CPU亲和性将线程池和内存分配限制在本地。该方法针对prefill阶段跨插槽读取权重、导致有效DRAM带宽下降的问题。预印本评估了参数量从1.17亿到7.62亿的GPT-2变体,以及BERT Small和Large,将结论推广到更大模型仍需要进一步测量。
Junhong Wu, Zedong Liu, Xinyang Ma, et al.
arXiv:2611.04382 · 2026-11-03T06:13:07Z
FastKVComp fuses distribution transformation and low-bit quantization in a TileLang kernel and replaces KVServe's lossless backend with a searchable compression pipeline. The preprint reports up to 5.9× speedup for the fused kernel and lossless-compression throughput rising from below 90 GB/s to over 500 GB/s. Faster compression can preserve the benefit of reduced KV transfers on high-bandwidth links, although these component metrics do not establish the same gain in end-to-end serving throughput.
FastKVComp以TileLang内核融合分布变换与低比特量化,并为KVServe换用可搜索的无损压缩流水线。该预印本报告融合内核最高加速5.9倍,无损压缩吞吐量从不足90 GB/s提升至超过500 GB/s。更快的压缩有助于在高带宽互连下保留减少KV传输的收益,但这些组件指标并不代表端到端推理吞吐量也有同等提升。