Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade

TL;DR

This paper proposes a cascade mechanism that predicts LLM agent failures early from internal representations to abort computation and save costs.

Problem

LLM agents often commit to failing trajectories in multi-step tasks, consuming substantial inference compute before failure becomes observable. Predicting failure early from observable behavior alone is challenging.

Approach

The authors use lightweight per-round probes on the agent's hidden activations to anticipate episode failure. They design an abort cascade with a distribution-free calibrated gate per round, where per-round recall budgets are jointly searched to guarantee that successful episodes survive all gates at a user-specified global rate.

Results & Contribution

Experiments on TextCraft with two agent models (Qwen-2.5-7B, Llama-3.2-3B) show the cascade meets recall targets from 90% to 97%. At a 90% target, it saves 47.1% and 37.2% of inference compute, respectively, which is 1.6–1.7x the best single-gate policy. A cascade reading only behavior saves roughly half as much, and adding behavioral features to the probe yields no further gain, indicating hidden states capture what behavior reveals. The work also characterizes the sample complexity of certifying high recall targets for practitioners.