Raghib Hasan

Does Your Coding Agent Need a Token Diet?

Claim
Do simple principles, caveman, rtk, or Token Optimizer make a coding agent use fewer tokens?
Result
One task, five runs per setup. Median token use ranged from −14.6% (rtk) to +39.2% (All three), versus the plain agent (baseline).
Why
86.3% of tokens were conversation history re-read on each call.

Before and after, call by call

Baseline

  1. git --no-pager log --oneline | head -8 && git ls-files | head -50 && cat packag… 21k
  2. python -m pytest -q 2>&1 | tail -40; cat CONTRIBUTING.md; wc -l invoicer/*.py 21k
  3. cat invoicer/money.py invoicer/invoice.py; sed -n 1,60p invoicer/rates.py; pyth… 24k
  4. sed -i 's/ROUND_HALF_EVEN/ROUND_HALF_UP/g' invoicer/money.py && python - <<'E' … 29k
  5. edit invoicer/money.py · edit invoicer/money.py · edit invoicer/rates.py · edit invoicer/invoice.py 31k
  6. git status --short; python -m pytest -q 2>&1 | tail -12 31k
  7. T="Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>" git ad… 32k
  8. final reply 32k

Principles +7.9%

  1. git --no-pager log --oneline | head -8 && git ls-files | head -40 && python -m … 21k
  2. python -m pytest -q --tb=short 2>&1 | grep -v "test_sample_totals\[" | head -40… 23k
  3. cat invoicer/money.py invoicer/discounts.py invoicer/invoice.py invoicer/rates.… 23k
  4. wc -c invoicer/*.py CONTRIBUTING.md && grep -n -i -B2 -A15 "commit" CONTRIBUTIN… 24k
  5. sed -n 66,98p CONTRIBUTING.md; sed -n 70,90p invoicer/money.py; cat invoicer/di… 25k
  6. sed -i 's/ROUND_HALF_EVEN/ROUND_HALF_UP/g' invoicer/money.py && python -m pytes… 28k
  7. python - <<'E' import re p='invoicer/money.py';s=open(p).read().replace('ROUND_… 31k
  8. git status --short && git add invoicer/money.py && git commit -q -m "fix(money)… 32k
  9. final reply 32k

caveman +11.4%

  1. git --no-pager log --oneline | head -8 && git status --short && ls && cat packa… 24k
  2. cat CONTRIBUTING.md; ls invoicer tests; python -m pytest -q 2>&1 | tail -30 25k
  3. cat invoicer/money.py invoicer/discounts.py invoicer/invoice.py; python -m pyte… 28k
  4. sed -i 's/ROUND_HALF_EVEN/ROUND_HALF_UP/g' invoicer/money.py && grep -n "rate_f… · edit invoicer/invoice.py 32k
  5. edit invoicer/money.py · edit invoicer/money.py · grep -n "def rate_for" -B3 -A12 invoicer/rates.py; python -m pytest -q 2>&1 | t… 33k
  6. edit invoicer/rates.py · python -m pytest -q 2>&1 | tail -8 34k
  7. T="Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>" && git… 35k
  8. final reply 35k

rtk −14.6%

  1. rtk git log --oneline -8 && ls && cat CONTRIBUTING* 2>/dev/null | head -30; ls … 21k
  2. sed -n 30,200p CONTRIBUTING.md; python -m pytest -q 2>&1 | tail -40; wc -l invo… 22k
  3. cat invoicer/money.py invoicer/discounts.py; sed -n 1,60p invoicer/rates.py · cat invoicer/invoice.py 24k
  4. sed -i 's/ROUND_HALF_EVEN/ROUND_HALF_UP/g' invoicer/money.py && python - <<'E' … 29k
  5. edit invoicer/money.py · edit invoicer/money.py · edit invoicer/rates.py · edit invoicer/invoice.py · rtk pytest -q 2>&1 | tail -30 30k
  6. rtk git status --short T="Co-authored-by: Copilot <223556219+Copilot@users.nore… 31k
  7. final reply 31k

All three +39.2%

  1. rtk git log --oneline -8 && ls && (python -m pytest -q --tb=short 2>&1 | tail -… 25k
  2. cat CONTRIBUTING.md | head -50; wc -l invoicer/*.py; cat invoicer/money.py invo… 27k
  3. sed -i 's/ROUND_HALF_EVEN/ROUND_HALF_UP/g' invoicer/money.py && sed -i 's/ retu… 33k
  4. edit invoicer/money.py · edit invoicer/money.py · edit invoicer/rates.py 34k
  5. python -m pytest -q --tb=short 2>&1 | tail -n 25 34k
  6. rtk read invoicer/discounts.py invoicer/invoice.py; python -m pytest -q --tb=sh… 35k
  7. edit invoicer/invoice.py · python -m pytest -q --tb=short 2>&1 | tail -n 15 39k
  8. T='Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>' && git… 39k
  9. final reply 40k

Token Optimizer −1.8%

  1. git --no-pager log --oneline | head -8 && git status --short && ls && cat packa… 21k
  2. cat CONTRIBUTING.md; ls invoicer tests; wc -l invoicer/*.py; python -m pytest -… 21k
  3. cat invoicer/money.py invoicer/discounts.py invoicer/invoice.py; python -m pyte… 24k
  4. sed -i 's/ROUND_HALF_EVEN/ROUND_HALF_UP/g' invoicer/money.py && python - <<'E' … 28k
  5. edit invoicer/money.py · edit invoicer/money.py · edit invoicer/invoice.py · edit invoicer/rates.py 30k
  6. python -m pytest -q 2>&1 | tail -15 30k
  7. T="Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>" git ad… 31k
  8. final reply 31k
One row per model call; gridlines every 10k tokens. Grey is the conversation so far, re-read on every call.

What the agent reads first, in tokens

The task prompt every run starts with, and the six instructions the principles setup adds, split into tokens:

Baseline 41 tokens · 213 characters

Task prompt · 41 tokens

CI is red on main and finance reports wrong totals on some invoices. Find and fix the root causes, make sure the whole test suite passes, and commit the fixes following this repository's commit message convention.

[14325, 382, 3592, 402, 2758, 326, 17496, 10988, 8201, 92279, 402, 1236, 87586, 13, 9764, 326, 9295, 290, 6577, 16431, 11, 1520, 3239, 290, 6062, 1746, 13800, 29169, 11, 326, 8737, 290, 60095, 3992, 495, 23229, 885, 8737, 3176, 39395, 13]

Principles 149 tokens · 646 characters

Instructions file · 108 tokens

# Working principles - Be brief. No preamble, no recap of what you did. - Search before you read: find the relevant code with grep or glob, then view only the lines you need. - Never re-read a file you have not changed. - Keep command output small: run `python -m pytest -q --tb=short`, and pipe long output through `tail -n 40` or `grep`. - Make independent tool calls in parallel. - Stop as soon as the task is done and verified.

[2, 29330, 23477, 279, 12, 2439, 14567, 13, 3004, 876, 47712, 11, 860, 97266, 328, 1412, 481, 2242, 558, 12, 10497, 2254, 481, 1729, 25, 1646, 290, 12331, 3490, 483, 51548, 503, 19753, 11, 1815, 2325, 1606, 290, 8698, 481, 1309, 558, 12, 28741, 322, 53808, 261, 1974, 481, 679, 625, 9180, 558, 12, 16835, 6348, 4733, 3291, 25, 2461, 2700, 29010, 533, 76, 63194, 533, 80, 2230, 32411, 28, 16090, 15007, 326, 22906, 1701, 4733, 1819, 2700, 20472, 533, 77, 220, 1723, 63, 503, 2700, 81938, 36060, 12, 9393, 13313, 4584, 11666, 306, 26697, 558, 12, 22535, 472, 6780, 472, 290, 5296, 382, 4167, 326, 29313, 558]

+ Task prompt · 41 tokens, same as baseline

Split with OpenAI’s o200k_base, an illustration: Claude’s tokenizer isn’t public. Measured, the principles added a median 169 first-call tokens (the file plus the CLI’s wrapper). The runs didn’t record the system prompt, 7,877 tokens of tool definitions, the skill list, or the setup hooks and tools.

Every run

Total tokens per run. Each dot is a run; the line marks the median.
Medians over five runs. First-call prompt: all the agent sends before acting — system prompt, 7,877 tokens of tool definitions, instructions, skills, task.
Setup Succeeded Median tokens Range Calls AIU First-call prompt
Baseline 5/5 220,306 183k–265k 8 10.90 20,531
Principles 5/5 237,809 198k–374k 9 11.50 20,700 (+169)
caveman 5/5 245,468 208k–316k 8 12.58 24,330 (+3,799)
rtk 5/5 188,084 152k–272k 7 10.79 20,840 (+309)
All three 5/5 306,595 194k–418k 9 13.48 24,791 (+4,260)
Token Optimizer 5/5 216,448 196k–301k 8 12.59 20,535 (+4)

What the runs show

The baseline’s median run used 220,306 tokens. The medians for principles, caveman, rtk, all three, token optimizer were 237,809, 245,468, 188,084, 306,595, 216,448. All 30 runs succeeded.

The narrowest within-setup range was 82,177 tokens, while the widest gap between two setups’ medians was 118,511. With one task and five runs per setup, treat these results as exploratory rather than conclusive. The single largest run, e3 (All three), used 418,085 tokens over 12 calls.

Across all 30 runs, total tokens tracked the number of model calls (Pearson r = 0.90). Every call re-reads the whole conversation so far, so calls grow as a run goes on: the last call of a run processed a median 33,534 tokens.

Cache reads were 86.3% of all tokens but 17.4% of the AIU. Cache writes were 12.5% of tokens and 62.9% of AIU; output was 0.9% of tokens and 18.5% of AIU. So the two can disagree: Token Optimizer’s median was below the baseline’s in tokens (216,448 against 220,306) but above it in AIU (12.59 against 10.90).

The six rules added 169 tokens to the prompt. Their runs took a median of 9 calls against the baseline’s 8.

Installing it added 3,799 tokens to the first prompt — a 4,466-character session-start instruction and 20 more skills in the skill list. Writing them to the cache made its first call cost 2.92 AIU against the baseline’s 1.97, and every later call re-reads them. Median prose per run went from 912 to 649 characters. Prose is the small part anyway: 75% of what the baseline agent wrote was tool input, such as commands and edits.

A plain run of the failing suite prints 749 lines. The agent ran the suite 130 times across the 30 runs; 130 test invocations piped output through a filter such as tail or grep. The largest single tool result was smaller with rtk (3,048 tokens, against 3,892 at baseline), and rtk reported a median 6,722 tokens saved per run — its own estimate, and less than a single late call processes. Its instruction and hook added 309 tokens to the prompt.

All three together added 4,260 tokens to the prompt, made the first call cost 3.06 AIU and had median total cost 13.48 AIU.

Token Optimizer v5.13.48 installed session-start continuity restore, capability-gated bash compression, post-tool nudges and stop-time rollup in an isolated Copilot home. Its median used 216,448 tokens (−1.8% versus baseline) and 12.59 AIU, with 5 of 5 runs succeeding. The adapter’s hooks added a median 0 characters of context across a run. These results cover this one task and do not establish broader savings.

What I’d take from it

Opinion, from one task and one model.

  • Count calls before words. Each call re-reads everything so far, so the number of calls set the token count here far more than how much the agent wrote.
  • Price what a tool adds to every prompt. Instructions, hooks and skill lists ride along on every call. A tool has to save more than that before it saves anything.
  • Check what your agent already does. This one trimmed its own command output, so an output filter had little left to take. Output filters should matter more where an agent reads long, raw logs.
  • Measure with enough runs. Single sessions vary too much to judge a setup. Run your own tasks many times and compare the spread, not one before and one after.

How it was run

A small Python invoicing library with 164 tests, 40 of them failing with 749 lines of test output. Three seeded bugs: money rounded half to even instead of half up; a zero tax-rate override fell back to the standard rate; and in tax-exclusive regions, tax was charged before a line’s share of the coupon. The prompt, the same for every run:

CI is red on main and finance reports wrong totals on some invoices. Find and fix the root causes, make sure the whole test suite passes, and commit the fixes following this repository's commit message convention.

GitHub Copilot CLI 1.0.99-3, non-interactive (copilot -p), model claude-sonnet-5.5 at medium reasoning effort, memory off.
  • Baseline: the CLI as installed.
  • Principles: six rules in the user instructions file:
    1. Be brief. No preamble, no recap of what you did.
    2. Search before you read: find the relevant code with grep or glob, then view only the lines you need.
    3. Never re-read a file you have not changed.
    4. Keep command output small: run python -m pytest -q --tb=short, and pipe long output through tail -n 40 or grep.
    5. Make independent tool calls in parallel.
    6. Stop as soon as the task is done and verified.
  • caveman (source, commit d262ab3): installed with its own installer for Copilot, which adds a session-start hook that switches the agent to terse replies, and 20 skills.
  • rtk (source, 1.0.1): rtk init -g --copilot --auto-patch --no-trust-filters, which adds an instruction to run commands through rtk and a hook that rewrites them, so rtk filters their output.
  • All three: everything above in one setup.
  • Token Optimizer (source, noncommercial license, v5.13.48): installed into an isolated Copilot home. Its Copilot adapter adds session-start continuity restore, capability-gated bash compression, post-tool tally/nudges, and stop-time rollup.
Every run got a fresh copy of the repository and a fresh home directory holding only its setup, with no other environment from my machine. Five runs per setup, each capped at 100 AIU. All setups used the same Copilot CLI version.
The fixes are committed, every commit subject follows the repository’s contributing guide, and the original tests plus a hidden acceptance test pass on the committed code. No run edited the tests.
Tokens per model call from the CLI’s session log. AIU is Copilot’s usage unit: the log records a rate for each kind of token, the same in every run (AIU per 1k tokens: cache read 0.01, cache write 0.25, fresh input 0.2, output 1), and tokens times those rates reproduce each run’s recorded total exactly. The first run’s opening 13,640 prompt tokens were already cached. Python 3.12.10, pytest 8.4.2.
One task, one agent, one model, five runs per setup. rtk’s savings figure is its own estimate. Other token-saving tools weren’t tested.

The kit rebuilds the repository, re-runs every setup and regenerates this page’s data.

Round 2: Regression hunt

Round 1 gave the tools little to compress. This second task buries the same three invoice bugs in a longer history, produces verbose test output, and asks for a written incident note. It is designed to give command filtering and terse replies more opportunity to help; compare it separately from Round 1.

Round-two waterfall

Baseline

  1. git --no-pager log --oneline --decorate | head -30 && ls && cat CONTRIBUTING* 2… 21k
  2. sed -n 30,200p CONTRIBUTING.md; python -m pytest -q 2>&1 | tail -30; git tag 22k
  3. python -m pytest -q 2>&1 | grep FAILED | grep -v test_sample_totals; python -m … 24k
  4. cat invoicer/money.py; git --no-pager log --oneline -- invoicer/money.py invoic… 29k
  5. sed -i 's/ROUND_HALF_EVEN/ROUND_HALF_UP/g' invoicer/money.py && sed -i 's/retur… 37k
  6. edit invoicer/money.py · edit invoicer/money.py · edit invoicer/rates.py 38k
  7. python -m pytest -q 2>&1 | tail -15; git --no-pager log --oneline -S"or self.st… 39k
  8. python -m pytest -q 2>&1 | grep FAILED | grep -v samples; python -m pytest -q -… 39k
  9. git --no-pager log --oneline -S"tax = line_tax(line_gross, rate, inclusive=Fals… 42k
  10. git --no-pager show d658235 -- invoicer/invoice.py | grep -n "^[-+].*line_tax\|… 43k
  11. python - <<'E' p='invoicer/invoice.py' s=open(p).read() a=""" if region.inclusi… 43k
  12. git add invoicer/money.py && git commit -q -m "fix(money): round half up again … 44k
  13. final reply 44k

Principles

  1. git --no-pager log --oneline --decorate | head -30 && git status --short && ls … 21k
  2. cat CONTRIBUTING.md | head -40; python -m pytest -q --tb=short 2>&1 | grep FAIL… 23k
  3. sed -i 's/ROUND_HALF_EVEN/ROUND_HALF_UP/g' invoicer/money.py && python -m pytes… 26k
  4. git --no-pager show 80a6029 -- invoicer | head -150; sed -n 70,100p CONTRIBUTIN… 27k
  5. python - <<'E' p='invoicer/money.py' s=open(p).read().replace('ROUND_HALF_EVEN'… 29k
  6. python -m pytest -q --tb=short tests/test_rates.py tests/test_cli.py 2>&1 | gre… 29k
  7. python - <<'E' p='invoicer/rates.py' s=open(p).read().replace("return self.over… 31k
  8. cat invoicer/discounts.py; git --no-pager log --oneline -- invoicer/discounts.p… 32k
  9. cat invoicer/invoice.py | sed -n 1,200p | grep -n "gross\|discount\|tax" | head… 34k
  10. sed -n 160,175p invoicer/invoice.py; git --no-pager log -L166,172:invoicer/invo… 34k
  11. git --no-pager show d658235 -- invoicer/invoice.py | grep -n "^[-+].*line_tax" … 35k
  12. python - <<'E' p='invoicer/invoice.py' s=open(p).read() s=s.replace(""" if regi… 36k
  13. T=' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>' git c… 36k
  14. final reply 37k

caveman

  1. git --no-pager log --oneline --decorate | head -30 && git status --short && ls … 25k
  2. sed -n 30,200p CONTRIBUTING.md | grep -n -i -B2 -A15 commit | head -50; git tag… 26k
  3. python -m pytest -q 2>&1 | grep FAILED | sed 's/\[.*//' | sort | uniq -c; ls in… 28k
  4. cat invoicer/money.py; git --no-pager diff v0.9.0 HEAD -- invoicer/rates.py | h… 29k
  5. sed -i 's/return self.overrides.get(category) or self.standard/return self.over… 37k
  6. edit invoicer/rates.py · git --no-pager show 80a6029 -- invoicer/money.py | head -60 38k
  7. git --no-pager diff --stat; python -m pytest -q 2>&1 | tail -8 39k
  8. sed -i '' 's/ROUND_HALF_EVEN/ROUND_HALF_UP/g' invoicer/money.py && python -m py… 39k
  9. python -m pytest -q 2>&1 | grep FAILED | sed 's/\[.*//' | sort | uniq -c; pytho… 40k
  10. git --no-pager show 55006b2 -- invoicer/discounts.py; git --no-pager show d6582… 41k
  11. edit invoicer/invoice.py · python -m pytest -q 2>&1 | tail -5; ruff check . 2>&1 | tail -3 43k
  12. T='Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>' git ad… 44k
  13. final reply 45k

rtk

  1. rtk git log --oneline -30 && ls && git tag && (cat Makefile package.json 2>/dev… 21k
  2. rtk python -m pytest -q -x 2>&1 | tail -30; cat CONTRIBUTING.md | head -40; git… 22k
  3. sed -n 40,100p CONTRIBUTING.md; python -m pytest -q 2>&1 | tail -15; ls invoice… 23k
  4. python -m pytest -q 2>&1 | grep -E "^(FAILED|[0-9]+ )" | sed 's/ - .*//' | awk … 25k
  5. rtk read invoicer/money.py; git log --oneline -- invoicer/money.py; git log -S"… 28k
  6. git show 80a6029 -- invoicer/money.py | grep -E "^[+-]" | head -40; git show 35… 30k
  7. python - <<'E' import re p='invoicer/money.py'; s=open(p).read() s=s.replace("R… 31k
  8. python -m pytest -q 2>&1 | grep -E "^FAILED" | sed 's/\[.*//' | sort | uniq -c;… 31k
  9. git log --oneline -S"line_tax(line_gross, rate, inclusive=False)" -- invoicer/i… 34k
  10. git show d658235 -- invoicer/invoice.py | grep -E "^[+-]" | grep -n "line_tax\|… 34k
  11. python - <<'E' p='invoicer/invoice.py'; s=open(p).read() old=""" if region.incl… 35k
  12. T="Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>" git ad… 35k
  13. final reply 36k

All three

  1. rtk git log --oneline -20 --stat | head -60; ls; python -m pytest -q --tb=short… 25k
  2. git --no-pager log --oneline | cat | wc -l; git --no-pager tag; python -m pytes… 27k
  3. sed -n 60,120p CONTRIBUTING.md; cat invoicer/money.py invoicer/discounts.py; gi… 29k
  4. git --no-pager show 80a6029 -- invoicer | head -60; git --no-pager log --onelin… 33k
  5. sed -i 's/ROUND_HALF_EVEN/ROUND_HALF_UP/g' invoicer/money.py && python -m pytes… 35k
  6. edit invoicer/money.py · edit invoicer/money.py 36k
  7. git --no-pager diff --stat | cat; python -m pytest -q --tb=line 2>&1 | grep -E … 36k
  8. cat invoicer/invoice.py | sed -n 1,200p; python -m pytest -q --tb=short tests/t… 36k
  9. sed -n 1,90p invoicer/rates.py; rtk git --no-pager log --oneline --format='%h %… 39k
  10. git --no-pager show 35e7746 -- invoicer/rates.py | grep -n "rate_for\|overrides… 41k
  11. python - <<'E' import re p='invoicer/rates.py';s=open(p).read() s=s.replace("""… 41k
  12. C="Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>" git ad… 42k
  13. final reply 43k
One row per call in each setup’s median-token run; gridlines every 10k tokens. Grey is the conversation re-read on every call.

Every round-two run

Total tokens per run. Each dot is a run; hollow dots are failed runs; the line marks the median.
Round 2 medians over 5 runs per setup. A first-call prompt includes the system prompt, 7,877 tool-definition tokens, instructions, skills, and task.
Setup Succeeded Median tokens Range Calls AIU vs baseline
Baseline 4/5 465,251 359k–514k 15 17.66 —
Principles 5/5 431,215 259k–495k 14 15.63 −7.3%
caveman 5/5 473,176 333k–682k 13 18.64 +1.7%
rtk 5/5 385,254 290k–564k 13 15.19 −17.2%
All three 5/5 463,485 448k–540k 13 17.74 −0.4%

What Round 2 found

  • Tokens: rtk had the lowest median at 385,254 tokens (−17.2% versus the baseline’s 465,251).
  • Usage: rtk had the lowest median charge at 15.19 AIU, against 17.66 for baseline. Token count and usage charge are different measures.
  • Outcomes: 24 of 25 runs succeeded. Failed runs: a4 (Baseline).
  • Variation: within-setup ranges spanned 92,040 to 348,819 tokens. Five runs on one task are exploratory, not evidence of general savings.

Round-two task and method

The fixture starts from the known-good 0.9.0 release, then adds 38 commits; three later commits each reintroduce one of the bugs. The full test suite has 1,119 tests, including 89 failures before the fix and 1,724 lines of pytest output.

Finance says invoice totals were right in 0.9.0 but some are wrong now, and CI is red on main. Find the commit that introduced each problem, fix the root causes, make sure the whole test suite passes, and commit the fixes following this repository's commit message convention. Then reply with an incident note for finance: for each problem, the commit that introduced it, what went wrong, and how you verified the fix.

Copilot CLI 1.0.99-3, claude-sonnet-5.5 at medium reasoning effort; Python 3.12.10, pytest 8.4.2. Each run used a fresh repository copy and an isolated setup home. The same five-run median, token, AIU, range, and success measures are used here, but this dataset has its own baseline and is not pooled with Round 1.

  • Baseline: The CLI as installed.
  • Principles: Six rules in the user instructions file.
  • caveman: A session-start hook that switches the agent to terse replies, plus its skills.
  • rtk: An instruction and hook that route commands through rtk to filter their output.
  • All three: Principles, caveman, and rtk together.

Round-two token rates per 1k: cache read 0.01, cache write 0.25, fresh input 0.2, output 1. The recorded token counts and charges are reported separately. Token Optimizer was added after these runs and was not measured in Round 2.