Prompt Compression, Explained: Lossy vs Lossless, and When Each Is Safe

Prompt compression means sending a language model fewer tokens for the same job, so each request costs less, returns faster and leaves more room in the context window. It comes in two kinds. Lossy compression (LLMLingua-style token dropping, or summarising old conversation turns) throws information away; it is safe on long documents and old history where some detail can go. Lossless compression removes only repeats and noise, so the model receives the same content; it is safe everywhere, and it saves most on coding agents, which resend the same files every turn. Everything below was checked on the papers and vendors' own pages on 23 September 2026.

Searches for "prompt compression" run at around 590 a month, and most of the pages that rank are research summaries. This one is for people paying the bill: what each method throws away, what the claimed numbers are, and which one you can switch on without checking every answer afterwards.

What is prompt compression, and why it exists

Every request to a model is billed by the token, and the input side is usually the larger share. A chat question is a few hundred tokens. A coding agent turn is tens of thousands: the system prompt, every tool definition, the conversation so far, every file it read and the output of every command it ran, sent again on every turn.

Compression attacks that input. It is one lever of eleven in our guide to reducing LLM API costs, and it is not the same thing as caching. Prompt caching keeps your text identical and bills the repeated start of it at a discount; compression sends less text. The two stack, with one condition we come back to below.

Lossy vs lossless prompt compression

The question that matters is not how many tokens a method removes. It is whether the model can still see everything it needed.

Lossy compression changes the content. It deletes tokens a small model judges unimportant, drops whole sentences, or replaces twenty turns of conversation with a paragraph of summary. The ratios are large. The cost is that you have made a decision on the model's behalf about what it will not need, and if that decision was wrong, the model cannot tell you: it answers from what is left.

Lossless compression changes only the packaging. A file the model already read is replaced with a pointer to the first copy. Two reads of the same file become one read plus a diff. A log line that repeats four hundred times becomes one line with a count. Terminal colour codes, which the model could never read, disappear. The ratios are smaller, but the information the model receives is the same, so the answer should be too.

An analogy from ordinary files: a ZIP archive is lossless, a JPEG is lossy. You would not JPEG a contract.

LLMLingua and the research methods

Most of the published work is lossy, and most of it comes from one place.

LLMLingua (Microsoft Research, EMNLP 2023) uses a small language model to score each token of your prompt and remove the ones it finds predictable, with a budget controller that decides how hard to squeeze each part. The paper reports "up to 20x compression with little performance loss" across four datasets, including GSM8K maths problems and ShareGPT conversations.

LongLLMLingua (ACL 2024) is tuned for long documents and retrieval, where the question tells you which passages matter. On NaturalQuestions it reports up to 21.4% better accuracy with around 4x fewer tokens on GPT-3.5-Turbo, and a 94% cost reduction on the LooGLE benchmark. The accuracy gain is real and worth understanding: long contexts bury the answer, and removing the noise around it can help the model find it.

LLMLingua-2 (Findings of ACL 2024) replaces the small language model with a trained classifier that decides, token by token, keep or drop. It reports 2x to 5x compression, 3x to 6x faster than earlier compression methods, and 1.6x to 2.9x lower end-to-end latency. All three live in one MIT-licensed Python library, microsoft/LLMLingua.

Two older approaches show the range. Selective Context (EMNLP 2023) prunes low-information phrases and reports a 50% reduction in context cost with a small drop in BERTScore and faithfulness. Gist tokens (NeurIPS 2023) teach a model to squeeze a prompt into a few special tokens, up to 26x, but they require fine-tuning a model you control, so they are no use against Claude or GPT through an API.

Context compression for agents: what the providers ship

The tools you can use today without a research library are all about agents that run out of room.

• Claude Code /compact and auto-compact. /compact replaces the conversation with a structured summary. Claude Code then re-injects your project CLAUDE.md and memory and re-reads the files it most recently touched. Auto-compact does the same when the window fills; /autocompact sets how full it gets first, and /compact focus on the auth bug tells the summary what to keep. Our Claude Code rate limit guide explains why a huge context also eats your plan.
• Anthropic compaction (API). Claude writes a summary of older turns on the server, either when you ask or when input reaches a threshold you set. Both modes are in beta. The docs say plainly that you should write your own summarisation prompt when the default summary drops something a later turn needs.
• Anthropic context editing (API, beta). Instead of summarising, it clears the oldest tool results once the prompt passes a threshold (100,000 input tokens by default), leaving a placeholder so Claude knows something was removed. The docs note that clearing invalidates cached prompt prefixes.
• OpenAI compaction (Responses API). A compact_threshold setting, or a separate /responses/compact endpoint, returns a compaction item that "carries forward key prior state and reasoning" in fewer tokens. It is opaque: you cannot read what was kept.

All four are lossy by design. They are the right answer when the alternative is hitting the context limit, and the wrong answer if you use them to save money on a session that still fits.

Does prompt compression hurt quality?

Lossy compression can, and the published numbers understate how.

The papers measure averages on benchmarks. An independent comparison of methods (Jha et al., "Characterizing Prompt Compression Methods for Long Context Inference", 2024) found that simple extractive compression, keeping or dropping whole passages, reached up to 10x with minimal accuracy loss, while token pruning "often lag[s] behind" it. Averages also hide where the losses land. Token-level methods are most likely to drop what looks predictable: a digit in a price, a negation, a variable name that appears once, a rule you stated at the start. Those are exactly the tokens a marketer's brief or a developer's codebase depends on.

For agents there is a second risk. A summary is written once and then trusted. If turn 12 established that the staging database is not to be touched and the summary lost it, turn 40 does not know.

Lossless compression does not carry this risk in the same way, because the model still receives every piece of information. What it can do wrong is mechanical, for instance a pointer the model fails to follow, so the vendor's claim is worth testing on your own work before you rely on it. That is what our field test does.

The methods side by side

Method	Lossy?	Typical saving (as claimed)	Quality risk	Where it runs	Examples
Token pruning	Yes	Up to 20x (LLMLingua paper); 2x to 5x (LLMLingua-2)	Drops rare but critical tokens: numbers, names, negations	Your code, a Python library	LLMLingua, LLMLingua-2, Selective Context
Query-aware document compression	Yes	Around 4x with higher accuracy on NaturalQuestions (LongLLMLingua)	Low for question answering over long documents; higher for open tasks	Your RAG pipeline	LongLLMLingua, extractive rerankers
Summarising history	Yes	Depends on session length	Summary loses a detail a later turn needs	Your agent or the provider	Claude Code /compact, Anthropic compaction, OpenAI compaction
Clearing old tool results	Yes	Depends on tool use	Model cannot re-check an old result; breaks the cached prefix	Provider (beta)	Anthropic context editing
Learned compression	Yes	Up to 26x (gist tokens paper)	Needs a fine-tuned model	Your own model	Gist tokens
Removing repeats and noise	No	20%+ on agent workloads (Gate, vendor figure)	None to the content; test the mechanics	A gateway or proxy	Constellation Gate AI, tamp's lossless stages

What lossless compression looks like in practice

Constellation Gate AI calls itself "the accountability layer for AI" and sits "between your agent and the model". Its compression is six specific, lossless fixes, and its own summary of the principle is short: "The first copy of anything stays intact, so what reaches the model is the same content."

1. Repeated content becomes a pointer. A file already read in the conversation is not resent in full: Gate's example goes from 5,214 tokens to 214.
2. Tool schemas are minified, without changing which tools the model can call.
3. Terminal control codes are removed: colours, cursor moves, progress bars.
4. Duplicate log lines collapse into one line with a count.
5. Harness scaffolding is trimmed, such as Claude Code's system reminders and Cursor's workspace state.
6. Overlapping reads become diffs. Gate's three-turn example goes from 15,000 tokens to 5,000.

Gate also keeps the prefix it forwards byte-identical, which is what your provider's prompt caching needs, so compression and caching stack instead of cancelling out. Its published figures: 20% or more fewer tokens on agent workloads, 23% lower token cost in the Rocket Resume case study, and a homepage request that goes from 6,500 tokens to 5,162. All vendor-published; our review has the detail.

Priced on Claude Sonnet 5 at $2 per million uncached input tokens (our checked price list), that homepage request costs $0.0130 before and $0.0103 after. Across 10,000 such requests a month, $130 becomes $103. Cached input changes the maths, which is why the calculator has a cached-input slider.

Try it on your own agent: start free with Gate. Affiliate link: we earn a commission on Pro seats at no cost to you, and it pays for the testing, not the verdict.

For contrast, tamp is an MIT-licensed local proxy for coding agents with nine compression levels. Its lossless stages (minify, dedup, diff and others) sit alongside lossy ones, and its default level includes an LLMLingua stage that runs when its Python sidecar is available. It claims 52.6% fewer input tokens. That is more than Gate claims, and the difference is largely the lossy part. Our LLM gateway comparison covers both.

Two honest limits on lossless compression. On a single chat, little repeats, so there is little to remove; the saving is an agent story. And on a flat Claude or ChatGPT subscription it does not lower the bill. Fewer tokens per turn may stretch your allowance further, but that is unmeasured; our field test reports on 30 September 2026. Routing is one setting: see Claude Code through a proxy and Codex CLI through a proxy.

When to use which

• Coding agents (Claude Code, Codex, Cursor). Lossless first, always: it is where the repeats are. Add /compact with a focus when a session genuinely outgrows the window, and /clear between unrelated tasks, which costs nothing.
• Long documents and retrieval. Query-aware lossy compression (LongLLMLingua, or plain extractive selection of the relevant passages) is a reasonable trade, provided you spot-check answers against the full text.
• Long chats and customer-facing assistants. Summarise old turns if you must, but keep rules, prices and names in the system prompt, never only in history.
• Contracts, code, numbers, legal or medical text. No lossy compression. Trim what you send instead.
• Batch pipelines with a fixed prefix. Prompt caching beats compression here. Do not rewrite the prefix.

Your first fifteen minutes

Following the rule we apply to every tool: real input, one output, hard stop.

1. Minutes 0 to 4. Open your last long agent session, or yesterday's usage page, and note input tokens per request. If it is under a few thousand, compression will not move your bill; stop here.
2. Minutes 4 to 8. Look for the repeats: the same file read twice, a long test log, a big tool list. That is what lossless compression removes.
3. Minutes 8 to 12. Create a free Gate account and point one agent at it; the Gate page has the setup for each tool. Run one ordinary task.
4. Minutes 12 to 15. Compare input tokens per request before and after, and put both into the cost calculator. Write the number down. Stop.

The short version

Lossy compression buys large ratios by guessing what the model will not need; use it on long documents and old history, and check the answers. Lossless compression removes only what the model has already seen or cannot read; switch it on for agents and forget about it.

Free records, Pro blocks. Gate's free plan ($0, 20,000 recorded requests a month) includes basic compression on every request. Advanced compression ("20%+ saved per message, by sending repeated content once"), prompt-injection blocking, redaction and spend limits are Pro at $20 per user per month. Start free with Gate (affiliate link; we earn a commission on Pro seats).

Checked on 23 September 2026 on the LLMLingua (arXiv 2310.05736), LongLLMLingua (2310.06839), LLMLingua-2 (2403.12968), Selective Context (2310.06201), gist tokens (2304.08467) and Jha et al. (2407.08892) papers, the microsoft/LLMLingua and sliday/tamp repositories, Anthropic's compaction and context-editing docs, Claude Code's context-window docs, OpenAI's compaction guide, and Constellation Gate AI's homepage and pricing page. Gate's figures are vendor-published; our own field-test numbers follow on 30 September. If something above is out of date, tell us and we will fix it with a dated note.