ToshLLM: local LLM inference on Intel Macs with AMD GPUs

Engineering notes

Engineering notes will be added by the aipentium editorial team.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为大语言模型、生成式AI、ChatGPT相关技术、计算机视觉、深度学习等高价值论文补充中文说明。

Original abstract

A native SwiftUI application that runs large language models locally on Intel Macs with AMD GPUs, a configuration mainstream inference stacks leave unsupported or incorrect. Beyond packaging, it contributes original work to the Metal backend of llama.cpp: ToshGEMM, a manually tiled matrix multiply that replaces the simdgroup-matrix path AMD GPUs do not provide. FA-AMD, flash-attention decode, tile and prefill kernels written for AMD, where the upstream vectorised kernel miscompiles. A wave64 port for GCN and Vega: reductions, quantized decode, batched mat-vec and prefill on 64-wide simdgroups. Multi-GPU tensor parallelism with a butterfly all-reduce and a per-batch choice between peer copies over Infinity Fabric and event hand-off. A reimplementation of TurboQuant KV cache compression and a speculative decoding planner. Patches apply on top of llama.cpp and stable-diffusion.cpp, which remain under the copyright and MIT licence of their own authors.

Links and sources

Comments