DeepSeek-V3

DeepSeek-V3 is the model on which everything DeepSeek released later stands. The weights appeared on Hugging Face on 25 December 2024: a sparse mixture-of-experts transformer with 671 billion parameters in total and 37 billion activated per token, spread over 61 layers with a hidden size of 7,168. The first three layers are dense; the remaining 58 route each token to 8 of 256 experts plus one shared expert. The vocabulary holds 129,280 tokens and the context window is 163,840 tokens, usually quoted as 128K. The technical claim that mattered at the time was not the size but the bill. DeepSeek states the full training run took 2.788 million H800 GPU-hours, of which 2.664 million went to pre-training on 14.8 trillion tokens — an order of magnitude below what comparable Western models were understood to cost. Three design decisions carry that result: attention compressed through Multi-head Latent Attention, an auxiliary-loss-free strategy for balancing expert load (earlier mixtures paid for balance with a penalty term that degraded quality), and a mixed-precision FP8 training framework, validated here at very large scale for the first time. The lab also notes that the run finished without a single unrecoverable loss spike or rollback, which for a model of this size is a claim about engineering discipline rather than about architecture. One detail is routinely reported wrongly: DeepSeek-V3 is not an MIT-licensed model. The repository ships two separate files. The code is MIT; the weights are covered by the DeepSeek License Agreement version 1.0, a use-based licence with its own restrictions. Only from the March 2025 refresh, DeepSeek-V3-0324, did the company move the weights themselves to MIT. Read on 25 August 2026 — twenty months after publication and long after two newer generations shipped — the original repository still records 934,114 downloads in thirty days.

▸Videos

No videos yet.

▸Reviews