gte-modernbert-base

gte-modernbert-base, published by Alibaba's Tongyi Lab on 20 January 2025, turns a passage of English text into 768 numbers. A search engine stores those numbers for every document it indexes and finds related material by measuring distance between them. It answers no questions and holds no conversation; it is infrastructure that sits under a search box or a retrieval-augmented chatbot. The model is 149 million parameters over 22 layers, hidden dimension 768, vocabulary 50,368, and takes up to 8,192 tokens of input. That window is the point of the release. The BERT-family encoders that dominated this job for years stopped at 512 tokens, which forced anyone indexing a long document to chop it into fragments and lose the connections between them. Sixteen times the window means a full article, a contract or a manual can be read in one piece. What makes this release worth stating plainly: Alibaba did not build the underlying encoder. The model is a fine-tune of ModernBERT-base, published in December 2024 by Answer.AI and LightOn, and the configuration file says so — every third layer attends globally, the rest work on a 128-token local window, which is ModernBERT's design, not Alibaba's. The Chinese company supplied the contrastive training that turns a general encoder into a retrieval model, and nothing below it. On the vendor's own numbers it scores 64.38 on MTEB English, 55.33 on BEIR retrieval, 87.57 on the long-context LoCo suite and 79.31 on the code-retrieval CoIR suite. The long-context result is the one that follows from the architecture, and it is where the model beats its 512-token predecessors by the widest margin. Weights are Apache 2.0, with no added restrictions. In the 30 days to 23 September 2026 the model was downloaded 158,273 times and holds 201 likes on Hugging Face - respectable, but far behind the reranking model released alongside it the same day, which sees sixteen times the traffic with half the likes.

▸Videos

No videos yet.

▸Reviews