Recall Before You Rank: Similarity-Guided Top-$K$ Reuse for Efficient Long-Context Attention

arXiv:2607.27692v1 Announce Type: new Abstract: Top-$K$ sparse attention reduces the cost of Softmax and value aggregation by attending to only a small subset of key--value (KV) entries. However, identifying this subset still requires scoring the current query against the full KV cache and performing global Top-$K$ selection, leaving selector cost linear in context length and limiting the practical efficiency of sparse attention for long-context decoding. In this paper, we introduce ReTopK, a tr...