TANGRAM Icon Use Cases

Running Tangram end to end, one compression method at a time.

Project Page Code

Compression Methods

Tangram's non-uniform KV cache is orthogonal to which importance scorer decides what to evict. Every scorer returns the same [num_kv_heads, chunk_len] score, so the compression level, the head grouping and the budget allocation stay unchanged when you swap one for another. Each method below has its own page in the repository, carrying the commands to reproduce what is shown here.

KeyDiff

Scores post-RoPE keys by how far they point from the chunk's mean direction, so the least distinctive keys are evicted first. Reads no queries and loads no checkpoint.

gate-free query-independent

Full guide →  ·  Paper

SnapKV · Expected Attention · FastKVzip

Also supported by the engine and covered by the paper's evaluation. Their use-case pages are not written yet; until then, see the benchmark scripts under benchmarks/tangram/ in the repository.

KeyDiff, End to End

Throughput

Tangram end-to-end speedup, KeyDiff scorer
Request throughput on scbench-variable-tracking, A100 80GB, over KV retention ratios. The number on each Tangram bar is the speed-up against the uncompressed r=1.0 reference drawn beside it.

Accuracy

SCBench accuracy, KeyDiff scorer, Tangram vs PyTorch across five models
Mean score over all 13 SCBench datasets, across the benchmark's short / mid / long / multi task groups, with Tangram against the stock PyTorch path running the same method. The dashed line is the full-KV reference and the violet series is the gap between the two bars, as a percentage of the without-Tangram score, on the right axis.

Reproducing this

The commands, the SCORER / LEVEL knobs and the cluster maps each level needs are documented in the repository rather than duplicated here, so there is one copy to keep current:

docs/usecases/keydiff.md →