The Art of Selective Precision

K and V are Not Equal

The attention formula has two caches. They look symmetric — both are matrices of shape (seq_len × d). But they play fundamentally different roles in producing the output:

K determines which tokens get attention — its values flow through softmax before contributing to output. V determines what information flows — its values are simply weighted and summed after the softmax weights are fixed.

This asymmetry has a dramatic consequence for quantization errors. Add the same amount of noise to K and to V — and see which causes more damage:

Why Softmax Amplifies K Errors

The difference comes down to one equation. Softmax is exponential:

A small perturbation ε in logit space doesn't add a small amount to the attention weight — it multiplies it by eᵉ. For ε = 0.5, that's a 65% weight increase. For ε = 1.0, it's 172%.

V errors bypass softmax entirely. An error ε in a V value contributes at most wᵢ × ε to the output — where wᵢ is an attention weight always ≤ 1. The error gets scaled down, not amplified.

Drag the slider to add noise to the top attention score. Watch how the weight distribution shifts:

Same Bits, Opposite Results

The most dramatic demonstration: take a fixed bit budget of ~5.8 bits per value average. Split it two ways:

Identical average bits. 500× quality difference — purely from deciding which cache gets compressed.

Click each config to see its attention output quality:

The Safe Decision Tree

Model weights?
│
├── Q8_0 or higher
│   ├── Best:        -ctk turbo4 -ctv turbo4
│   └── More:        -ctk turbo3 -ctv turbo3
│
└── Q4_K_M
    ├── Qwen2.5     → MUST asymmetric: -ctk q8_0 -ctv turbo3
    ├── Unknown      → Safe default:  -ctk q8_0 -ctv turbo4
    └── Llama/Mistral 24B+ → Try symmetric, fallback to asymmetric

Boundary V: Not All Layers Are Equal

You know V can be compressed aggressively. But not every layer tolerates the same level of compression. The first 2 and last 2 layers are uniquely exposed:

Boundary V protects exactly those 4 layers with high precision V, compresses everything else aggressively:

Adjust the model depth to see how the compression cost and quality recovery change:

The 64-layer model recovers 91% of the quality gap because 4 boundary layers represent a smaller fraction of a deeper model — same protection, lower proportional cost.

Sparse V: Skip What Doesn't Matter

After computing softmax(QKᵀ/√d), you get attention weights over all tokens. At long context, most of those weights are negligibly small — the model is paying essentially zero attention to most past tokens.

If a token has weight 0.000001, its V vector contributes 0.000001 × V to the output — essentially nothing. Yet standard attention still dequantizes every V vector, loading all those indices and norm values from memory.

Sparse V skips dequantization for tokens below a threshold. The weights are computed from K before V is touched — so we already know which tokens to skip.

Block Size 128: Free Compression

PolarQuant stores each vector as: indices (2/3/4 bits each) + one norm (2 bytes). The original implementation organised storage in 32-element blocks — each block getting its own norm.

For a 128-dimensional vector:

Block size 32:  128 dims / 32 = 4 blocks → 4 norms × 2 bytes = 8 bytes
But all 4 norms store the SAME VALUE — 3 are pure redundancy!

Block size 128: 128 dims / 128 = 1 block → 1 norm × 2 bytes = 2 bytes
Save 6 bytes per vector. Zero quality change.

The Complete Optimization Stack

All six layers are orthogonal — they don't interfere with each other. Enable all six and you get all six benefits simultaneously.