Checked automatically
Whole-file structured reads become candidates without asking the agent to pick a mode or change its prompt.
for local agents reading structured files
Make exact data reads cost less.
Context Compression sits behind your AI tool. When an agent reads JSON, JSONL, CSV, or TSV, it can swap the file for a lower-token file only after proving the smaller file decodes to the same parsed data.
measured savings
Cut the noisy tokens before they reach the model.
On 28 real files, Context Compression cut 17.5M tokens down to 15.2M. That is 2,324,959 tokens gone, and every selected read still decoded back to the exact same parsed data.
raw tokens
17,496,442
original structured files in the corpus
optimized tokens
15,171,483
selected after round-trip verification
tokens removed
2,324,959
advanced benchmark candidates, May 22 2026
corpus savings
13.3%
with raw fallback included
SQuAD QA rows
72.7%
Titanic rows
50.5%
LogHub logs
41.5%
GitHub metadata
29.1%
read path
Every supported read checks for a cheaper exact path.
Context Compression belongs in the file-read layer. For JSON, JSONL, CSV, and TSV, it tries a lower-token sidecar, proves parsed equality, and otherwise hands back the original file.
Only exact views win
Rows, keys, fields, and values must round-trip to the parsed source. Source files stay unchanged.
Agents keep their workflow
Codex, Claude Code, Pi, Hermes, MCP, OpenClaw, and generic agents keep their normal file-read path while the adapter chooses the verified file.
Raw is the safety path
Unsupported files, failed proofs, unsafe candidates, and partial reads return the original file instead of an approximate substitute.
Shell intent stays raw
jq, grep, head, pipes, flags, and raw-shell intent stay untouched.
No semantic thinning
It does not summarize rows, drop fields, or turn equality checks into a model-answer parity claim.
how it works
A selector, a verifier, and a fallback.
Adapters stay thin. They ask the selector for a decision, then read only the verified path. If the proof fails, the agent receives the original file.
Catch a whole-file read
The adapter handles supported local JSON, JSONL, CSV, and TSV reads.
Try reversible candidates
The selector counts model tokens for safe views and benchmark-only advanced views.
Verify parsed equality
A sidecar wins only when it decodes back to the same parsed source data.
Return sidecar or raw
The report records hashes, token counts, selection policy, and the trusted read_path.
install
Free. MIT open source. Keep the fallback on.
Clone it, point it at your worst file, and keep the raw fallback on. Start with a local checkout, run the selector, then wire the matching adapter into your runtime.
Local checkout
No hosted service is required. The selector runs on local files and writes sidecars under the project cache.
git clone https://github.com/saminkhan1/context-compression
cd context-compression
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
chmod +x run-hook.shCodex hook
Use the Bash PreToolUse hook for simple whole-file cat reads. Runtime hooks use the safer candidate tier.
[features]
hooks = true
[[hooks.PreToolUse]]
matcher = "Bash"
[[hooks.PreToolUse.hooks]]
type = "command"
command = "/absolute/path/to/context-compression/run-hook.sh"Manual proof
Run the selector directly when you want the full JSON report before enabling transparent rewrites.
.venv/bin/python selector.py \
--cwd "$PWD" \
--model gpt-5.4-mini \
--adapter manual \
--include-candidates \
--verify-report \
sample-repetitive.jsonEvidence gate
Use the repo checks before repeating benchmark or product claims on your own data.
.venv/bin/python -m unittest discover -s tests
.venv/bin/python scripts/verify_evidence.py --full-tests
python3 scripts/verify_clean_install.py