Mixtral 8x7B

Mixtral 8x7B was announced on 11 December 2023 and it changed what an open model was expected to cost to run. Instead of one dense network, each feed-forward block holds eight separate groups of parameters; for every token, at every layer, a small router picks two of them. The model therefore stores 46.7 billion parameters but spends only 12.9 billion per token — it answers at the speed and the cost of a 13B model while scoring like something far larger. Mistral's own comparison put it at or above Llama 2 70B on most benchmarks with roughly six times faster inference, and at or above the GPT-3.5 base model — the model behind the free tier of ChatGPT at the time. It handled a 32,000-token context, English, French, Italian, German and Spanish, and generated code competently. The licence was again Apache 2.0, which is the part that mattered commercially: a company could match a paid frontier product on its own hardware without a negotiation. The release was also a piece of theatre that the industry copied afterwards. Mistral put the weights out first as a bare magnet link on X, with no paper, no benchmark table and no blog post, and let the open-source community measure the model for three days before publishing its own numbers. Sparse mixture-of-experts designs went on to become the default for large models across the field. Mixtral 8x7B no longer appears in Mistral's API line-up and no shutdown date was ever announced; under Apache 2.0 the weights remain downloadable.

▸Videos

No videos yet.

▸Reviews