Blog LLMs & Texto Robótica & RL

Baidu Releases Unlimited OCR, a 3B Model That Keeps the KV Cache Flat for Long-Document Parsing

Baidu open-sourced Unlimited OCR, a 3B-parameter MoE model that parses dozens of document pages in a single forward pass. Its Reference Sliding Window Attention (R-SWA) holds the KV cache constant, so memory and latency stay flat as output grows. It scores 93.23 on OmniDocBench v1.5, beating the DeepSeek OCR baseline by 6.22 points, under an MIT license. The post Baidu Releases Unlimited OCR, a 3B Model That Keeps the KV Cache Flat for Long-Document Parsing appeared first on MarkTechPost .

MarkTechPost ·Asif Razzaq · 25 de janeiro de 2026

Most end-to-end OCR models slow down as output grows. Each generated token adds to the KV cache. Memory rises and generation drags. Parsing dozens of pages becomes impractical. Baidu’s Unlimited OCR addresses this directly. It swaps the decoder’s attention for a design that keeps memory constant.

• Unlimited OCR is a 3B-parameter Mixture-of-Experts model, with only 500M parameters active.

• It replaces decoder attention with Reference Sliding Window Attention (R-SWA), keeping the KV cache constant.

• The model parses dozens of pages in one forward pass under a 32K maximum length.

• It scores 93.23 on OmniDocBench v1.5, beating the DeepSeek OCR baseline by 6.22 points.

• It builds on DeepSeek OCR via continue-training, not a from-scratch run.

What is Unlimited OCR?

Unlimited OCR takes DeepSeek OCR as its baseline. It keeps the DeepEncoder and the Mixture-of-Experts decoder. The MoE design holds 3B total parameters but activates only 500M at inference.

The DeepEncoder is the compression engine. It cascades a SAM-ViT under window attention with a CLIP-ViT under global attention. At the bridge, it applies 16× token compression. A 1024×1024 PDF image becomes just 256 visual tokens. Fewer input tokens mean a smaller prefill.

DeepEncoder natively supports five resolution modes, and Unlimited OCR keeps two. ‘Base’ mode runs at 1024×1024 for multi-page work. ‘Gundam’ mode uses dynamic resolution for single pages.

https://arxiv.org/pdf/2606.23050

How R-SWA Keeps the Cache Constant

The contribution is Reference Sliding Window Attention. Standard Multi-Head Attention stores a key and value for every token. As output length T grows, the cache grows with it. The size is CMHA(T) = Lm + T. Memory and latency climb without bound.

R-SWA breaks that link. Each generated token attends to all reference tokens, meaning the visual tokens and the prompt. It also attends to the preceding n output tokens, where n defaults to 128. Everything older is evicted. The cache becomes a fixed queue of size m + n.

The size is CR-SWA(T) = Lm + min(n, T) ≤ Lm + n. It is bounded by a constant. As T grows far beyond n, the cache ratio trends toward zero. So memory stays flat and per-step latency stays flat.

The research team compare this to soft forgetting. A person copying a book glances at the source and the last few words. They do not re-read everything transcribed so far. Visual tokens never undergo state updates. That avoids the progressive blurring seen in linear attention. The interactive simulator below lets you vary T and watch both caches respond.

Animate decoding

MHA KV cache

10,048

key/value entries

R-SWA KV cache

2,176

key/value entries

Cache ratio ρ

0.217

R-SWA ÷ MHA

Memory saved

78%

vs standard MHA

Standard MHA — grows with every token10,048

R-SWA — bounded by L_m + n2,176

C_MHA(T) = L_m + T = 2,048 + 8,000 = 10,048

C_R-SWA(T) = L_m + min(n, T) = 2,048 + 128 = 2,176

Attention view: each new token sees all reference tokens plus only the last n output tokens. Earlier output is evicted from the cache.

Reference tokens (visual + prompt, always visible)

Active window (last n tokens)

Evicted output (soft-forgotten)

Grounding: Unlimited OCR keeps the full reference cache of size L_m but holds only the most recent n output tokens (n defaults to 128). As output length T grows far beyond n, the cache ratio ρ(T) trends toward zero, so MHA's linear growth is replaced by a constant footprint. The page-to-token estimate uses the DeepEncoder figure of 256 tokens per 1024×1024 page. Numbers illustrate the cache formulas in the report, not a benchmark run.

R-SWA cache formulas from the Unlimited OCR technical report (arXiv:2606.23050)

Built by Marktechpost

(function(){

var pages=document.getElementById('pages'),

tokens=document.getElementById('tokens'),

win=document.getElementById('window');

var TOK_PER_PAGE=256;

var anim=null;

function fmt(n){return Math.round(n).toLocaleString('en-US');}

function render(){

var P=+pages.value, T=+tokens.value, n=+win.value;

var Lm=P*TOK_PER_PAGE;

var mha=Lm+T;

var swa=Lm+Math.min(n,T);

var ratio=swa/mha;

var saved=Math.round((1-ratio)*100);

document.getElementById('pagesVal').textContent=P+(P===1?' page':' pages');

document.getElementById('tVal').textContent=fmt(T);

document.getElementById('nVal').textContent=n;

document.getElementById('mhaVal').textContent=fmt(mha);

document.getElementById('swaVal').textContent=fmt(swa);

document.getElementById('ratioVal').textContent=ratio.toFixed(3);

document.getElementById('saveVal').textContent=saved+'%';

var maxC=mha;

document.getElementById('mhaBar').style.width='100%';

document.getElementById('swaBar').style.width=Math.max(2,(swa/maxC)*100)+'%';

document.getElementById('mhaBarTxt').textContent=fmt(mha);

document.getElementById('swaBarTxt').textContent=fmt(swa);

document.getElementById('fMhaN').textContent=fmt(Lm)+' + '+fmt(T)+' = '+fmt(mha);

document.getElementById('fSwaN').textContent=fmt(Lm)+' + '+fmt(Math.min(n,T))+' = '+fmt(swa);

drawStream(T,n);

postHeight();

function drawStream(T,n){

var s=document.getElementById('stream');

s.innerHTML='';

var REF=10, OUT=34;

for(var i=0;i

var r=document.createElement('div'); r.className='tok ref'; s.appendChild(r);

var prog=Math.min(1, T/120000);

var generated=Math.round(prog*OUT);

var winCells=Math.max(1,Math.round((n/512)*8));

for(var j=0;j

var t=document.createElement('div'); t.className='tok';

if(j

if(j>generated-winCells){ t.className='tok win'; }

else { t.className='tok evicted'; }

s.appendChild(t);

function postHeight(){

try{ parent.postMessage({type:'uocr-resize',height:document.body.offsetHeight+40},'*'); }catch(e){}

function play(){

if(anim){stopAnim();return;}

document.getElementById('play').textContent=' Pause';

tokens.value=256;

anim=setInterval(function(){

var v=+tokens.value+3000;

if(v>=120000){v=120000; render(); stopAnim(); return;}

tokens.value=v; render();

function stopAnim(){clearInterval(anim);anim=null;document.getElementById('play').textContent=' Animate decoding';}

pages.addEventListener('input',render);

tokens.addEventListener('input',function(){ if(anim) stopAnim(); render();});

win.addEventListener('input',render);

document.getElementById('play').addEventListener('click',play);

document.getElementById('reset').addEventListener('click',function(){

stopAnim(); pages.value=8; tokens.value=8000; win.value=128; render();

window.addEventListener('load',render);

window.addEventListener('resize',postHeight);

window.addEventListener("message",function(e){

if(e.data && e.data.type==="uocr-resize"){

var f=document.getElementById("uocr-frame");

if(f){ f.style.height=e.data.height+"px"; }

How It Was Trained

Unlimited OCR was not trained from scratch. The research team continue-trained from the DeepSeek OCR checkpoint for 4,000 steps. They froze the DeepEncoder and trained only the decoder. Training used about 2M document samples on 8×16 A800 GPUs. The 9:1 split favored single-page data, with multi-page samples built by concatenation.

The research team evaluates on OmniDocBench v1.5 and v1.6. The main finding/stat is 93.23 overall on v1.5. That beats the DeepSeek OCR baseline by 6.22 points. The table below compares the three related models. All three share the same 3B-A0.5B size.

Metric (v1.5)DeepSeek-OCRDeepSeek-OCR 2Unlimited-OCROverall ↑87.0189.1793.23Text Edit ↓0.0730.0490.038Formula CDM ↑83.3786.8592.61Table TEDS ↑84.9785.6090.93Read-order Edit ↓0.0860.0600.045

On OmniDocBench v1.6, Unlimited OCR reaches 93.92 overall. That is the top score in the research paper’s v1.6 comparison. Gains hold across text, formula, and table recognition.

Speed improves too. On OmniDocBench in Base mode, Unlimited OCR hits 5,580 TPS against DeepSeek OCR’s 4,951 TPS. That is a 12.7% increase. The gap widens with longer output. At a 6,000-token output ceiling, DeepSeek OCR lags Unlimited OCR by 35%.

Where It Fits: Use Cases

The constant cache suits workloads that page-by-page systems handle poorly.

• Whole-book transcription: Feed 40+ pages and parse them in one continuous pass. The reported edit distance stays below 0.11 at 40+ pages, with 96.90% Distinct-35.

• Document parsing pipelines: Extract text, tables, formulas, and reading order in a single forward pass.

• High-throughput batch parsing: The included infer.py launches an SGLang server and sends concurrent requests over a folder or PDF.

• Beyond OCR: The research team call R-SWA a general parsing attention, applicable to ASR and translation.

Running It: Minimal Code

The Transformers path needs trust_remote_code=True and a CUDA GPU. Single-image parsing uses Gundam mode.

Copy CodeCopiedUse a different Browser

import torch

from transformers import AutoModel, AutoTokenizer

name = "baidu/Unlimited-OCR"

tokenizer = AutoTokenizer.from_pretrained(name, trust_remote_code=True)

model = AutoModel.from_pretrained(

name, trust_remote_code=True, use_safetensors=True,

torch_dtype=torch.bfloat16,

).eval().cuda()

model.infer(

prompt="document parsing.",

image_file="your_image.jpg",

output_path="your/output/dir",

base_size=1024, image_size=640, crop_mode=True, # gundam mode

max_length=32768,

no_repeat_ngram_size=35, ngram_window=128,

save_results=True,

Multi-page and PDF parsing call model.infer_multi in Base mode at image_size=1024. For production throughput, SGLang serves an OpenAI-compatible API using the fa3 attention backend.

Strengths and Weaknesses

• Constant KV cache holds memory and latency flat across long outputs.

• End-to-end SOTA scores on OmniDocBench v1.5 and v1.6.

• Only 500M active parameters keep inference cheap.

• MIT license, open weights, and dual Transformers plus SGLang support.

• R-SWA gains arrive without a measured accuracy cost on single pages.

Weaknesses:

• Parsing is not truly unlimited; a 32K context still bounds the prefill.

• Long prefills grow as page count accumulates, despite heavy compression.

• Multi-page runs use Base mode only, so very small text can be missed.

• ASR and translation transfer remains future work, not a shipped result.

Check out the Paper, Repo and Model Weights. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post Baidu Releases Unlimited OCR, a 3B Model That Keeps the KV Cache Flat for Long-Document Parsing appeared first on MarkTechPost.

Ver no Hugging Face

// relacionados

Baidu Releases Unlimited OCR, a 3B Model That Keeps the KV Cache Flat for Long-Document Parsing

Leia também

Amazon ups India bet with fresh $13B AI infrastructure investment

Jalapeño: a OpenAI projeta seu primeiro chip de inferência — e usou IA para fazer isso em 9 meses

SkillOpt: como ensinar agentes de IA a melhorar suas próprias habilidades — +23 pontos em GPT-5.5

Authors Guild test finds some AI detectors perfectly identify human writing while others fail on every single text