ASAP: Attention Sink Anchored Pruning

Abstract

Vision Transformers (ViTs) face severe computational bottlenecks due to the quadratic complexity of self-attention at high resolutions. Existing token reduction methods rely on local metrics, such as single-layer attention scores, that are inherently vulnerable to the attention sink phenomenon, where uninformative tokens are paradoxically preserved over salient foreground objects. We propose ASAP (Attention Sink Anchored Pruning), a training-free framework that recasts this sink as a feature. Modeling ViT information flow as a Lazy Random Walk, ASAP identifies the sink as a dominant accumulator of probability mass. By computing the diffusion distance to the sink within the cumulative transition matrix, ASAP partitions tokens via Radial Diffusion Clustering and compresses background redundancy through Transition Weight Pooling in a single shot. Extensive experiments across image, video, and vision-language tasks demonstrate ASAP outperforms state-of-the-art methods, accelerating throughput by up to 48% while maintaining or even exceeding baseline accuracy.

Publication
NeurIPS 2026

The project repository currently contains a README announcing code release in December 2026; an implementation is not yet available there.

Jaehyuk Lee
Jaehyuk Lee
Ph.D. Student

I am professional ggureogi

Hanyoung Kim
Hanyoung Kim
Ph.D. Student

I’m the last M.S. in Financial Engineering graduate managed from Econ department! You can easily find my name on most Korean keyboards :)

Yanggee Kim
Yanggee Kim
M.S. in Mathematics

The very first Kim of 640.

Donghun Lee
Donghun Lee
Associate Professor

Connecting artificial intelligence and mathematics, in both directions.