How Compaction Works in Pi

Great thread, I was just thinking about compaction. My current line of thought is that compaction/pruning/ctx management in general should be something ongoing and maybe recursive. For example: User:'How is auth implemented?' -> [thinking] [codebase exploration with [thinking] in between, 10 file reads, 3 of which were "wrong"] [thinking] -> agent_response This little exchange contains a WHAT (how auth actually is implemented) and a HOW (where that info is and how to retrieve it). Maybe this question was part of a larger task. I think that whole exchange could be summarised before it enters context, kind of like what happens with subagents. The main thread would then consist mostly of [summaries]. Eventually the context will fill up anyway and we would summarise those summaries again. Alternatively one could maintain a [master_summary], kind of like an internal state. So new [summaries] get integrated directly and the [master_summary] gets updated.

Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages. For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration. Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations lead to more frustrating future chats because the LLM misses intent and or context. (Or, the presence of paragraphs and paragraphs of LLM output makes the next token predictor dumber? Unsure.)

Compaction is painful if you run just one local LLM, the best way to avoid it is to keep context as small as possible. One trick I find useful is to have one model with two KV caches running and while first cache has produced tokens, second cache immediately summarizes them during input tokens are being generated (tools time), then harness switches to the second KV cache which takes newly produced input tokens while KV in first cache is getting replaced with compacted summary tokens. This is a kind of ping pong, so we trade more space for less time. Still experimenting but it looks it works, and nice bonus it improves GPU utilization. Btw I have my own harness and model serving code, but it can be easily implemented in any other harness and model server.

In my experience, the best approach to compaction is to never get to the point where you need compaction and to generally stay below about 30% context window utilization. Even for long agentic workflows this can be accomplished for quite a while, much longer than most people might think. Here's what I do for each of my sessions: 1. For asides, off-topic work, or repetitive work that has already been done in the session, branch backwards (with /tree) and summarize. 2. If I've exceeded 30% or the 'price-doubling' multi-tier pricing, prune (my custom extension). 3. If I've already pruned and I'm still close to 30%, 'prune all' (more extensive prune). Definition: '/prune': Removes ~50% context on a fresh session (not previously pruned) - Keeps: User messages, normal assistant prose, commands/status markers, extension receipts, model settings, and a plain-text receipt for each tool call. - Removes: Thinking, signatures, actual tool calls/results, tool output, images, compaction summaries, and other extensions’ state. '/prune-extended': Removes ~80% context on a fresh session - Keeps: User messages, normal assistant prose and conclusions, commands/status markers, extension receipts, and model settings. - Removes: Thinking, signatures, all tool calls/results and output, images, compaction summaries, other extensions’ state, and any tool-activity receipts created by /prune. Both create a new session and delete the old one after a successful switch. Using these I can keep a session going for weeks (or longer), even with extensive use and almost all the important context is preserved while dumping the less important context. Neither command requires an LLM summarization so they execute quickly.

I think the way prompt caching works really discourages more creative compaction techniques. Like perhaps some kind of heuristic progressive compaction that replaces tool results and thinking traces after use with pointers could potentially keep the model smart for much longer, but that'd mean breaking cache every turn, and possibly even within a turn, seriously driving up cost.