ReToken: One Token to Improve Vision-Language Models for Visual Retrieval
Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infe…
Hugging Face · Daily Papers
·Yao Xiao, Reuben Tan
·
·▲ 1 upvotes
Este artigo está em destaque na seleção diária de papers do Hugging Face, curada pela comunidade de pesquisa em IA.
Autores: Yao Xiao, Reuben Tan, Zhen Zhu, Yuqun Wu, Jianfeng Gao, Derek Hoiem
- 1 upvotes da comunidade
Resumo
Resumo original (em inglês), extraído do paper:
Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints. We present ReToken, a single learnable embedding trained as an explicit retrieval target that selects a sparse set of query-relevant visual tokens from a pre-filled visual KV cache. Trained on only a small image-QA dataset, ReToken yields consistent gains across image and video benchmarks: on Visual Haystacks it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative), and on LVBench it transfers zero-shot to long video for an 8.0-point gain with Qwen3VL-8B. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReTokenOnde ler
// relacionados
Leia também
Blog
Claude published malicious code to the Internet and attacked 3 real companies
Blog
LingBot-Map Tutorial: GPU-Aware Inference and Point Cloud Export
Blog
Thinking Machines bets on efficiency over size with its second model, Inkling Small
Blog