IBM printed a rival's better score in its own model card — and it is worth reading why

IBM's reranking model for English search, published under Apache 2.0, comes with a comparison table in which a competing model of exactly the same size wins two of the three benchmarks. The gap on long documents is 5.4 points, and IBM printed it next to its own result rather than leaving it out. The model, granite-embedding-reranker-english-r2, does not search for anything. It is the second stage of a search system: a fast retrieval model fetches twenty plausible documents, and the reranker re-reads each one together with the query and decides the final order. That extra pass is expensive — it runs once per candidate — but it is also where most of the accuracy in a modern search stack comes from. On IBM's own measurements the effect is large. Reranking the top twenty results from the company's own retriever lifts BEIR retrieval accuracy from 53.1 to 55.8, long-document search from 41.6 to 45.8, and the English split of MIRACL from 43.6 to 55.2 — nearly twelve points, obtained without retrieving a single new document. The same table shows gte-reranker-modernbert-base, an open competitor with an identical 149 million parameters and the same 8,192-token window, scoring 56.1 on BEIR against Granite's 55.8, and 51.2 on long documents against 45.8. Granite wins only the multilingual benchmark, 55.2 against 54.8. A fourth row is easy to miss and more useful than the winner. The seven-year-old ms-marco-MiniLM-L12-v2, at 33 million parameters and a 512-token limit, scores 53.2 on BEIR and 55.4 on MIRACL — beating both modern models on the multilingual test at a fifth of the size. What it cannot do is long documents, where its short context caps it at 34.5. That is the practical lesson for anyone choosing a reranker: context length, not parameter count, decides whether a model is usable on real documents, and the benchmark that matters is the one closest to your own corpus. The averages hide it. There is also a reason IBM can afford the candour. Across the whole R2 family the company excluded MS MARCO — the retrieval dataset most open models train on — because its licence forbids commercial use. IBM loses accuracy for it and says so, the same trade-off it documented in August when it published a speech model twice and forbade commercial use of the better version. A card that admits a competitor wins is easier to trust on everything else it claims. The reranker and its retrieval sibling have been added to the catalogue.