Autodata: An agentic data scientist to create high quality synthetic data

TL;DR

Autodata proposes a method where an AI agent acts as a data scientist to create high-quality synthetic data, and meta-optimizes the agent itself to further improve data quality.

Problem

Existing synthetic data creation methods often have limitations in data quality or diversity, making it challenging to efficiently obtain large-scale, high-quality data for AI model training.

Approach

Autodata sets an AI agent as a data scientist to perform data creation tasks, and meta-optimizes the agent itself to learn to generate better data. A specific implementation called 'Agentic Self-Instruct' is presented.

Results & Contribution

The method achieves superior results over classical synthetic dataset creation methods on computer science research tasks, legal reasoning tasks, and reasoning with mathematical objects. Meta-optimizing the agent itself delivers an even larger performance uplift, suggesting a way to convert increased inference compute into higher quality model training.