Independent study · sparse attention

How much of its context does an LLM actually need to read?

Ezekiah Livingston A five-baseline study of sparse attention on frozen open models, with a downloadable evaluation harness. All numbers from logged runs on free Kaggle GPUs. Find me on LinkedIn.

Published September 1, 2026

Why I ran this

I spent several months building a sparse-attention prototype of my own. To find out whether it worked, I had to build increasingly paranoid instrumentation, because every quick metric I tried turned out to have a way of lying to me. The instrumentation eventually killed the prototype — cleanly, with numbers — and outlived it. This page documents what the harness measures, the three ways this kind of evaluation silently fails, and the results across three open models. The harness itself is downloadable below.

Method

Take a frozen open-weight model (never trained or modified). Feed it a long real document. Intercept every attention layer and inject an additive mask that restricts which blocks of earlier tokens each query may read — nothing else about the computation changes, and this is verified: with an all-open mask, the patched model reproduces the original bit-for-bit (max difference 0.0). Score each selection policy by real next-token negative log likelihood on the final 1,024 tokens.

Why five arms? Each one closes an escape hatch.

A score means nothing on its own; each baseline exists because leaving it out lets a broken result look like a good one.

• dense — the target. Without it, "my method scored 0.85" is unanswerable: 0.85 of what? Every arm is read as distance from dense.
• oracle — the ceiling. When a method scores badly there are two possible stories: the method is bad, or the task is impossible at this reading budget. The oracle, which cheats with the model's real attention, separates them; here it showed the budget costs almost nothing, so every point lost is a selection failure.
• meanpool — the incumbent. The most common flattery in this field is comparing a new method against a weak strawman. Putting the actual state-of-the-art mechanism on the same scoreboard, every run, makes that impossible.
• random — the null, and the arm that earned its place. It sounds too dumb to include. During this work, a method we had reported as "second best" across three separate evaluations turned out, once the random arm was added, to score below random selection: it was not a weaker selector, it was not selecting at all. Without this arm, "second best out of two" reads as respectable; with it, the truth was one line of the table. Anything at or below random is guessing with extra steps.
• local — the floor, and the harness's self-destruct. It measures whether the document even needs long-range reading. If local nearly matches dense, no selection method was actually tested, and the harness declares the run invalid instead of publishing numbers; that guard has fired in practice and stopped a meaningless-but-plausible table from being reported.

Most sparse-attention results we have read omit at least two of these five, and every omission is a specific, demonstrated way to fool yourself.

The floor-to-dense gap is the recoverable signal; each method is scored as the share of it recovered. If the floor nearly matches dense, the harness declares the whole run invalid rather than reporting numbers, because the document never tested long-range reading.

Three ways this evaluation silently lies

These are the heart of the page: the failure modes that made every quick metric untrustworthy until it was hardened against them.

1. Checks that pass on NaN. if error >= tolerance: fail never fires when the error is NaN, because every comparison with NaN is false. An earlier version of this harness printed a complete, confident results table where every number was NaN, above a verification line that had silently "passed." Guards must test the positive condition: if not (error < tolerance): fail.
2. Masks that let the model peek at the future. Supplying any explicit attention mask disables the framework's built-in causality, so the mask must enforce token-level causality itself. A block-granularity mask let every query see up to 31 tokens ahead — and the tell was that every sparse method beat full attention, which is impossible. If a restricted model outperforms an unrestricted one, the evaluation is broken, full stop.
3. Proxy metrics that invert rankings. We first scored methods by "how much attention mass lands inside the selected blocks." By that proxy, method A beat method B by 32 points. By real next-token loss, B beat A decisively — the proxy didn't just add noise, it reversed the answer. Every number on this page is real NLL for that reason.

Results

Same harness, same corpus recipe, ~13% attention density, three base models:

(percentages = share of recoverable long-range signal; oracle at ~100% means ideal selection fully matches dense attention at one-eighth the reading)

A fourth run is worth reporting precisely because it failed. TinyLlama-1.1B-Chat scored 9.43 NLL under dense attention — near random-guessing for its vocabulary — while every one of the harness's integrity checks stayed green. The instrument correctly measured an unsuitable substrate (a chat-tuned model collapses on raw text) instead of producing a plausible-looking wrong table. Lesson folded into the docs: base models only, and an implausibly high dense NLL means the substrate, not the selectors, is what you're measuring.

What it means

Selection is the entire quality problem: the oracle shows the reading budget itself costs nothing.

The incumbent is strong: a two-line, training-free baseline captures >90%. New selection methods must beat that, not just "random" or a strawman.

The open ground is the last 7-9 points between meanpool and oracle — block summaries richer than a mean — and the deployment frontier is kernels, not selection: masking proves quality but doesn't skip memory reads.

Limitations

• Context capped at ~4k tokens on free-tier GPUs (explicit masks force a dense mask in memory; longer needs a block-sparse kernel).
• No end-to-end speed claims anywhere: selection time is measured, memory traffic is arithmetic, and the harness says so on its own output.
• Selection is per query-block and shared across heads; per-head selection may differ.
• One corpus type (source code / docs); three model families; small document samples.

Get the harness

Download the notebook (runs on a free Kaggle T4 in ~25 minutes) and the README. Adding your own method is one function:

def sel_mine(layer_idx, q, k, T, device):
    scores = ...   # (query_block, key_block); higher = more worth reading
    return build_mask(scores, T // BLOCK, T, device)

Study and harness by Ezekiah Livingston, built in pair with an AI research assistant (Claude). Every number above comes from logged Kaggle runs; the harness re-produces the table from scratch, downloads included. Contact via the main site.