Probe-and-Refine Tuning of Repository Guidance for Coding Agents

TL;DR

We propose probe-and-refine tuning, which iteratively diagnoses and patches a repository's guidance file (AGENTS.md) using synthetic bug-fix probes, improving coding agent performance.

Problem

LLM-based coding agents require repository-level operational knowledge (file structure, test procedures, past failure patterns) not present in code. Engineers write AGENTS.md files to provide this context, but studies disagree on whether LLM-generated guidance helps or harms agent performance.

Approach

The method uses synthetic bug-fix probes to diagnose the guidance file and patches it via single-shot LLM calls, with no agent loop or tool use during tuning.

Results & Contribution

On SWE-bench Verified with Qwen3.5-35B-A3B at 200 steps, probe-and-refine achieves 33.0% mean resolve rate vs. 28.3% for static knowledge base and 25.5% for unguided baseline (p<0.001). Improvement stems from coverage (14.5pp more instances patched) rather than precision (~59%, p=0.119). Guidance enables productive use of larger step budgets.