Kimi Linear 48B A3B Instruct

Kimi Linear, published on 30 October 2025, is not a flagship chat model but a test of an attention architecture that Moonshot AI wanted to prove before scaling it. Standard transformers compare every token with every other token, which makes long inputs expensive in both memory and time. Kimi Linear replaces three out of every four attention layers with Kimi Delta Attention (KDA), a refined gated delta rule that keeps a fixed-size recurrent memory instead of a growing cache, and leaves full multi-head latent attention in only seven of its twenty-seven layers. The remaining twenty run on KDA. On top of that sits a mixture of experts with 256 experts, eight of which fire per token plus one shared, so 48 billion total parameters cost about 3 billion per token. The context window is 1,048,576 tokens and the published weights hold 49.1 billion parameters. The point of the exercise is cost, and the vendor's own figures are about speed rather than intelligence: up to 75 per cent less key-value cache, up to six times faster decoding at a one-million-token context, and 3.98x faster generation at 128K while scoring 84.3 on RULER. At a short 4K context it reports 51.0 on MMLU-Pro, which the card presents as parity with full attention rather than an advantage. One caveat belongs next to the headline claim. Moonshot writes that the architecture outperforms full attention on the strength of controlled 1.4-trillion-token training runs, but the checkpoints actually released were trained on 5.7 trillion tokens. The comparison and the shipped model are therefore two different experiments, and no like-for-like full-attention twin of the released weights exists to check the claim against. Weights for both the base and the instruction-tuned checkpoint are on Hugging Face under an MIT licence, with the KDA kernel open-sourced in the FLA library; there is no paid Moonshot endpoint for this model, so using it means running it yourself. The architecture paper is arXiv:2510.26692.

▸Videos

No videos yet.

▸Reviews