Teaching an agent new tools without forgetting the old ones
Every time you fine-tune an agent on a new set of tools, it picks up the new capabilities but often at the cost of the old ones. We ran a three stage benchmark to measure exactly how much earlier knowledge gets lost while new knowledge accumulates, then tried four training recipes to see if that tradeoff is inevitable.
Fine tune a tool search model on three batches of app integrations (Airtable → Gmail → Slack) and it forgets the first batch. Standard SFT drops stage-0 accuracy from 62% to 6%. Soft distillation (SDFT) forgets less but peaks lower. Three changes: warm-start teacher, a hybrid hard+soft loss, and frozen multi-teacher ensembles bring stage-0 accuracy back to 62% with no extra trainable parameters.
The problem we wanted to solve — Catastrophic Forgetting
Agents in production keep getting new integrations. This week you ship Airtable. Next month, Gmail. Then Slack. Each time, you fine tune the model on the new tools and deploy.
After you train on Gmail, does the model still know how to pick an Airtable tool? Or did learning the new stuff overwrite the old stuff?
Here is a concrete example of what the model has to do. A user says:
“Pull up a specific airtable record by id”
The model should answer with the right API slug, wrapped in tags:
That part works fine after stage 0 training. The model hits about 62% accuracy on held-out Airtable queries. Then we train it on Gmail tools for another 100 steps. Accuracy on those same Airtable queries drops to 38%. Train on Slack next, and Airtable accuracy falls to 6%.
The model mostly forgot how to do Airtable queries. That is catastrophic forgetting, and it is the default outcome when you keep fine-tuning the same weights on new data.
Catastrophic forgetting with standard fine-tuning
Accuracy on stage-0 queries (Airtable / GitHub / Notion) after each training stage · SFT baseline
61.6% → 5.7% on the same held-out Airtable queries — while the model learns each new toolkit.
How we set up the experiment
We built a benchmark called composio-bench. The task is tool search: read a natural language query, output the correct Composio API slug. We used Qwen3.5-4B with a small LoRA adapter (rank 8). Same base model throughout, only the adapter weights change between stages.
Training happens in three stages. Each stage introduces a new set of app integrations. The model trains for 100 steps on that stage's data, we save a checkpoint, then move on. We never retrain on old data unless a recipe explicitly says to.
| Stage | New toolkits added | Example query |
|---|---|---|
| Stage 0 | Airtable, GitHub, Notion | “Make a fresh table in an airtable base”→ AIRTABLE_CREATE_TABLE |
| Stage 1 | Gmail, HubSpot, Instagram, Linear, Supabase, Twitter | “Send an email with gmail”→ GMAIL_SEND_EMAIL |
| Stage 2 | Google Calendar, Drive, Sheets, Outlook, Slack, Telegram | “Post a message to a slack channel”→ SLACK_SEND_MESSAGE |
What is validation accuracy?
After each stage, we test the model on queries it did notsee during training. Each stage has its own held-out validation set (about 10% of that stage's queries, split by toolkit).
When we write val_stage0 = 61.6%, we mean:
- Take the model as it exists right now (after however many stages of training have run).
- Run it on the stage-0 validation queries (Airtable / GitHub / Notion only).
- Count how often the output slug exactly matches the correct answer. That fraction is validation accuracy.
So if you see val_stage0 drop from 62% after stage 0 to 6% after stage 2, that means: the model finished learning Slack tools, and when you ask it an Airtable question it used to handle well, it gets it wrong 94% of the time.
We also report full_stage0, full_stage1, and full_stage2— larger held-out splits (~200 queries each) drawn from the full Composio eval suite, stratified by toolkit. Same pass@1 metric, harder distribution, more stable than the small val sets. When we say a result “held up on the full benchmark,” we mean these numbers.
Four training recipes
We ran the same 3 stage benchmark four ways. Every time we change the way the model learns, note that the model and the dataset remains the same.
Recipe 1: Standard fine-tuning (SFT)
The baseline. Show the model a query and the correct answer slug. Train it to predict that slug directly. This is what most people do. It learns the current stage fast and forgets previous stages just as fast, that is the 62% → 6% drop we opened with.
Recipe 2: Self-distillation (SDFT)
Instead of hard labels, the student model first writes its own answer. A teacher model then scores that answer token-by-token (with the correct answer shown as an in-context example). The student learns to match the teacher's softer probability distribution rather than memorizing one exact slug. This way we preserve hinton's dark knowledge.
SDFT did help with forgetting. After all three stages, stage-0 accuracy was 15.7% vs. SFT's 5.7% — nearly 3× better on the same metric. Relative to each method's own peak, SDFT kept about 42% of stage-0 knowledge (37.7% → 15.7%) while SFT kept only 9% (61.6% → 5.7%). The catch: SDFT never peaked as high as SFT (37.7% vs. 61.6% after stage 0), because training used a few-shot format scaffold. Zero-shot eval read as 0% — an eval mismatch, not proof the model learned nothing. Re-evaluating with the same few-shot prompt used in training showed real capability and milder forgetting. Still not good enough to ship zero-shot, which is why we kept iterating.
Recipe 3: Hybrid SDFT + SFT
Mix two losses: 40% standard cross-entropy on the correct answer, 60% distillation from the teacher. The hard labels keep the output format stable. The soft distillation keeps the weight updates gentle. Same trainable parameters as before.
Recipe 4: Multi-teacher hybrid
Same hybrid loss, plus: after each stage, freeze that stage's adapter as a read-only teacher. On new data, average predictions from all past teachers plus the current one. On old data (25% of each batch), replay queries from prior stages and distill from the specialist teacher for that stage only.
Results
The number we care about most is val_stage0 after all three stages are done: after learning everything, how good is the model at the first toolkit set? SFT, hybrid, and multi-teacher use zero-shot eval (same prompt as production). Plain SDFT trained with a few-shot scaffold, so we report it with a matching few-shot eval.
Stage-0 accuracy after learning stages 1 and 2
val_stage0 after stage 2 · zero-shot, except SDFT* (few-shot eval matching training)
SFT ends at 5.7%. Plain SDFT at 15.7% (few-shot eval), lower peak, but roughly 3× the retained stage-0 accuracy and a much gentler slide from peak to final. Hybrid gets to 31.4%. Multi-teacher gets to 61.6%.
Does stage-0 knowledge survive new toolkits?
val_stage0 after each stage · SDFT* uses few-shot eval; others zero-shot
The dashed SDFT line shows the same pattern: accuracy drops as new stages land, but the slope is flatter than SFT. SDFT was pointed in the right direction on forgetting; hybrid and multi-teacher built on that and fixed the zero-shot deploy gap.
Harder held-out split (full_stage0) after all 3 stages
~1,800 stage-0 queries · zero-shot, except SDFT* (few-shot)
Full benchmark after stage 2
Complete pass@1 on every val and full split, after all three training stages. This is the cleanest side-by-side: rows are recipes, columns are which toolkit era each eval probes.
| Recipe | val_s0 | val_s1 | val_s2 | full_s0 | full_s1 | full_s2 |
|---|---|---|---|---|---|---|
| SFT | 5.7% | 21.2% | 40.2% | 0.5% | 18.5% | 43.5% |
| Plain SDFT* | 15.7% | 23.0% | 23.2% | 7.5% | 27.5% | 25.0% |
| Hybrid (α=0.4) | 31.4% | 28.5% | 39.6% | 28.5% | 36.0% | 48.0% |
| Multi-teacher hybrid | 61.6% | 35.2% | 36.6% | 49.5% | 43.5% | 49.0% |
* Plain SDFT: few-shot eval (matches training prompt). Zero-shot eval was 0% across all stages — a format mismatch, not zero capability. SFT, hybrid, and multi-teacher: zero-shot throughout. Per-stage numbers for all recipes are in the appendix.
Why these changes worked
Multi-teacher did not add parameters. It is still one LoRA adapter being trained. The gains came from three fixes we made while iterating on SDFT. Each one addresses a specific failure mode we hit.
Fix 1: The teacher has to know what the student already knows
Our first SDFT runs used the raw base model as the teacher. That sounds reasonable — the base model is the starting point, so distill from it. But at stage 1 the student already learned Airtable. The base model did not. Every distillation step was pulling the student back toward generic pretraining behavior.
The fix: at the start of each stage, snapshot the student's current weights and use thatas the teacher. Stage 0 teacher = base model. Stage 1 teacher = stage-0 checkpoint. The teacher always represents what the model knew going in, plus an in-context demo of the right answer. Distillation now means “stay close to what you already know, adjusted by this example” instead of “forget everything and start over.”
Fix 2: Hard labels for format, soft targets for retention
Pure distillation is good at not overwriting weights. It is bad at locking in output format when training includes a few-shot example the model never sees at deploy time. Our first SDFT runs looked like total failure in zero-shot eval (0% everywhere). Re-running eval with the training-matched few-shot prompt showed 15.7% stage-0 accuracy after stage 2, the model had learned, we were just measuring it wrong.
SFT has the opposite problem: hard labels lock format, but the updates are aggressive enough to wipe old knowledge. So we split the loss:
The 40% SFT anchor trains on zero-shot prompts (no few-shot scaffold), so training matches deployment. The 60% distillation keeps the update small enough that prior stages do not get bulldozed.
Fix 3: Keep the specialists around
Hybrid fixed a lot, but it still had one teacher that gets replaced each stage. By stage 2 that teacher is a generalist, decent at everything, sharp on nothing. Stage-0 knowledge is in there somewhere, but diluted.
Multi-teacher keeps a frozen copy of each stage's adapter (~30 MB each, not a full 4B model). Two rules for how to use them:
- New queries (75% of batch): ask every frozen teacher plus the current one what they think. Average their token-level predictions. The student learns from the consensus of all its past selves.
- Old queries (25% of batch): replay stage-0 or stage-1 examples and distill from the specialist for that stage only. Airtable rows go to the stage-0 teacher. Gmail rows go to the stage-1 teacher.
You are not storing three full models. You are storing three small snapshots, each frozen at the point where the model was best at that stage's toolkits. Replay keeps those snapshots relevant. The ensemble keeps new learning from drifting too far from all of them at once.
What this means in practice
If you are shipping new agent capabilities on a schedule — new integrations, new tool sets, new domains — the default fine-tuning loop will eat its own tail. You do not necessarily need more data from old stages or a bigger model. You need to think about who teaches the model, on what data, and whether old teachers stick around.
Multi-teacher adds about 10% wall-clock time (extra teacher queries per batch) and ~60 MB of frozen adapters at stage 2. No extra trainable parameters.
We are running follow-ups — multi-teacher at 9B, a second seed, and wiring these knobs into our autoresearch loop so a coding agent can search over teacher configs automatically. More in our continual learning system post.
Appendix: complete benchmark log
Pass@1 on every val and full split, after each training stage. Read across a row to see what the model knew at that checkpoint; read down a column to see forgetting on one toolkit era as later stages land.
| Checkpoint | val_s0 | val_s1 | val_s2 | full_s0 | full_s1 | full_s2 |
|---|---|---|---|---|---|---|
| After stage 0 | 61.6% | 17.6% | 7.9% | 42.0% | 30.5% | 17.0% |
| After stage 1 | 38.4% | 36.4% | 4.3% | 32.0% | 48.5% | 15.0% |
| After stage 2 | 5.7% | 21.2% | 40.2% | 0.5% | 18.5% | 43.5% |
| Checkpoint | val_s0 | val_s1 | val_s2 | full_s0 | full_s1 | full_s2 |
|---|---|---|---|---|---|---|
| After stage 0 | 37.7% | 14.5% | 8.5% | 24.5% | 23.0% | 13.0% |
| After stage 1 | 20.1% | 32.7% | 4.9% | 15.0% | 38.5% | 15.5% |
| After stage 2 | 15.7% | 23.0% | 23.2% | 7.5% | 27.5% | 25.0% |
| Checkpoint | val_s0 | val_s1 | val_s2 | full_s0 | full_s1 | full_s2 |
|---|---|---|---|---|---|---|
| After stage 0 | 63.5% | 17.6% | 8.5% | 49.5% | 25.5% | 18.5% |
| After stage 1 | 44.7% | 41.8% | 11.0% | 40.5% | 45.5% | 19.0% |
| After stage 2 | 31.4% | 28.5% | 39.6% | 28.5% | 36.0% | 48.0% |
| Checkpoint | val_s0 | val_s1 | val_s2 | full_s0 | full_s1 | full_s2 |
|---|---|---|---|---|---|---|
| After stage 0 | 58.5% | 19.4% | 9.1% | 49.0% | 30.5% | 18.5% |
| After stage 1 | 58.5% | 41.8% | 12.8% | 46.5% | 49.0% | 21.5% |
| After stage 2 | 61.6% | 35.2% | 36.6% | 49.5% | 43.5% | 49.0% |
* Plain SDFT evaluated with few-shot prompt (matches training). All others zero-shot. Qwen3.5-4B · LoRA rank 8 · 100 steps/stage.