Mistral 7B

Mistral 7B was published on 27 September 2023, four months after the company was founded, and it is the model the rest of Mistral's catalogue descends from. The claim in the announcement was narrow and testable: 7.3 billion parameters beating Llama 2 13B on every benchmark Mistral ran, and beating the original 34B Llama on many of them. The weights went out under Apache 2.0 — the permissive licence, not a research licence — so anyone could download the model, fine-tune it and sell what they built on top of it without asking. Two architectural choices carried the result. Grouped-query attention cut the memory traffic of generation, which is what makes a small model fast rather than merely small. Sliding window attention limited each layer to a 4,096-token window while letting information propagate across layers, so longer inputs cost less than a full attention matrix would imply; the published configuration declares 32,768 positions with that 4,096-token window. Neither idea was invented here, but the combination in a freely licensed 7B model was new to the open ecosystem. What followed mattered more than the benchmarks. Mistral 7B became the default starting point for open fine-tuning work through late 2023 and 2024 — the Hugging Face repository still records hundreds of thousands of downloads a month years later — and it established the pattern the company repeated: release open weights, keep the frontier models commercial. Mistral's current API line-up no longer offers it, and the company has never announced a shutdown date; because the licence is Apache 2.0, the weights cannot be withdrawn from anyone who already has them.

▸Videos

No videos yet.

▸Reviews