RecSA enables million-token attention for Qwen

haoailab.com13 min read

Researchers present RecSA (Recursive Sparse Attention), a method that builds a tree over the KV cache to let each attention query search it level by level at logarithmic cost. By combining candidate-local RoPE and a distilled pooler, RecSA enables models trained only on short 32K sequences to extrapolate to millions of tokens while outperforming existing dense and sparse baselines on RULER. Open article →