AI Research & Inference Systems
AI研究与推理系统
Yu Tian
arXiv:2611.14399 · 2026-11-10T17:45:12Z
REAL derives a draft-model training objective that directly estimates expected acceptance length in speculative decoding, using path weights and Rao-Blackwellization. In a controlled comparison with a Qwen3.6-35B-A3B target and a Domino drafter, it improves acceptance length against WTV, VSD, DBLast, and a static LK baseline across nine task subsets and five seeds. Longer accepted blocks could reduce target-model passes per generated token.
REAL利用路径权重与Rao-Blackwell化,推导出直接估计推测解码期望接受长度的草稿模型训练目标。在采用Qwen3.6-35B-A3B目标模型和Domino草稿模型的受控比较中,它在9个任务子集、5个随机种子下取得了优于WTV、VSD、DBLast及静态LK基线的接受长度。每轮接受更多token,有望减少生成每个token所需的目标模型前向计算次数。
Amir Mann, Jonathan Fhima, Shaked Brody, et al.
arXiv:2611.14387 · 2026-11-10T17:40:24Z
B3 refines two adjacent token blocks within a shared forward pass and opens the next block adaptively in self-editing diffusion language models. Across 10 datasets, the authors report an 11.5–14.5% speedup with comparable generation quality. The method targets latency lost when a block waits for its slowest token and requires no additional training or hyperparameter tuning.
B3在自编辑扩散语言模型中,让两个相邻token块共享一次前向计算进行迭代细化,并自适应地启动下一个块。作者在10个数据集上报告,生成质量相当时可获得11.5%至14.5%的加速。该方法减少了一个块等待最慢token所造成的延迟,无需额外训练或超参数调优。
Susav Shrestha, Brad Settlemyer, Narasimha Reddy
arXiv:2611.13293 · 2026-11-09T22:01:46Z
Decode Token Parallelism separates model-weight placement from attention ownership, distributing one model copy in layer blocks across nodes. Each active node runs projections and MLP work for the full batch within its NVLink domain, and every node computes attention for its own requests using local KV caches. This arrangement targets the memory-capacity and cross-node communication costs that constrain high-concurrency, long-context decoding.
Decode Token Parallelism将模型权重的放置与注意力计算的归属解耦,把单份模型按层块分布到多个节点。持有当前层块的节点在自身NVLink域内处理整个批次的投影与MLP计算,各节点则利用本地KV缓存为各自请求计算注意力。这种安排针对高并发、长上下文解码中的内存容量与跨节点通信开销。
Zhengxiang Huang, Yujie Sun, Shengheng Chen, et al.
arXiv:2611.14356 · 2026-11-10T17:16:47Z
EngineAgent automates support for new mobile LLMs by mapping dependencies across model conversion, graph construction, operators, and hardware backends. It prepares engine knowledge and backend-specific build and debugging workflows, then uses a builder and an auditor with device-side validation to coordinate implementation. The evaluation spans 12 tasks, three mobile LLMs, and four engine–backend pairs across CPUs, GPUs, and NPUs, focusing on the engineering work required for engine portability.
EngineAgent通过梳理模型转换、计算图构建、算子与硬件后端之间的依赖,自动完成移动端LLM的引擎适配。系统先整理引擎知识,预先验证后端的构建与调试流程,再通过构建者、审查者和设备端验证协调实现。评估覆盖12项任务、3个移动端LLM及涉及CPU、GPU和NPU的4组引擎—后端组合,聚焦引擎可移植性所需的工程工作。
Evan Li, Krishna Deshpande, Wen Cheng, et al.
arXiv:2611.13509 · 2026-11-10T03:29:29Z
Mobile AI Factory searches model sizes, fine-tuning settings, and quantization choices using measurements from the target phone. Across 10 tasks and four Android phones, the system combines an LLM orchestrator with rollback rules that reject accuracy regressions. On an entity-extraction task, its search uses about one nineteenth of the GPU time required to exhaustively train 27 variants, illustrating how hardware measurements can guide a smaller model-selection budget.
Mobile AI Factory利用目标手机的实测数据,搜索模型规模、微调设置与量化方案。该系统在10项任务和4款Android手机上接受评估,结合LLM编排器与回滚规则,撤销导致准确率下降的调整。在一项实体抽取任务中,其搜索所用GPU时间约为穷举训练27种模型变体的1/19,展示了硬件实测如何帮助压缩模型选择的计算预算。
Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri
arXiv:2611.14413 · 2026-11-10T17:53:44Z
Simulated round-to-nearest weight quantization on Qwen3-0.6B and Qwen3-1.7B finds no accuracy loss from six-bit weights in tests spanning four multiple-choice benchmarks. For the 0.6B model at four bits, averaging four cyclic answer-option orderings recovers 12.6 percentage points. The study exposes a tradeoff between weight storage and repeated inference passes, with 300 items per benchmark defining the evaluation scope.
在Qwen3-0.6B和Qwen3-1.7B上进行的最近舍入权重量化模拟显示,覆盖4个多项选择基准的测试中,6 bit权重未带来准确率损失。对于采用4 bit权重的0.6B模型,对4种循环排列的选项顺序进行平均可恢复12.6个百分点。该研究揭示了权重存储与多次推理之间的预算权衡,评估范围为每个基准300道题。
Senhao Liu, Yuheng Zhang, Chunyu Wei, et al.
arXiv:2611.13675 · 2026-11-10T07:24:50Z
OpticalRAG renders retrieved documents as page images, uses a frozen vision encoder, and compresses their visual representations into as few as 16 tokens injected into an LLM. Its shared projection connects text-query retrieval with the visual-token compressor. It targets the context-window pressure of long-document RAG by transferring document processing into a visual encoding and compression stage.
OpticalRAG将检索文档渲染成页面图像,经冻结的视觉编码器处理后,将视觉表示压缩为最少16个注入LLM的token。它通过共享投影连接文本查询检索与视觉token压缩器。该方法将文档处理转移到视觉编码和压缩阶段,以缓解长文档RAG的上下文窗口压力。
Liwei Zhang, ZhiMing Tong
arXiv:2611.13607 · 2026-11-10T05:33:41Z
One Byte, One Token converts Qwen3.8-27B from subword tokens to a pure byte-level model, shrinking its vocabulary from 248,320 entries to 384. The two-stage conversion uses 4.77 GB of supervised bytes and takes 178.6 hours on eight H200 GPUs. Changing the vocabulary and sequence representation shifts the costs of embedding tables, output projections, and sequence processing, giving hardware-aware deployment studies a concrete conversion recipe.
One Byte, One Token将Qwen3.8-27B从子词token模型转换为纯字节级模型,把词表从248320项缩减至384项。两阶段转换使用4.77 GB的受监督字节数据,在8张H200 GPU上耗时178.6小时。词表与序列表示的改变会重新分配嵌入表、输出投影及序列处理的开销,为面向硬件的部署研究提供具体转换方案。