Abhinav Singh, Harsha Vardhan Khurdula, Yoeven D. Khemlani, Vineet Agarwal
SOB is the first multi-source benchmark evaluating LLMs' ability to extract structured data from diverse sources (text, image, audio), measuring both schema compliance and value accuracy.
Existing benchmarks evaluate either schema compliance or value accuracy within a single source domain, or conflate structured output capability with vision/speech processing. Thus, a fair benchmark for comparing structured output quality itself is lacking.
SOB comprises three source modalities: text (5,000 records), image (209 records), and audio (115 records). All models receive a text-normalized representation regardless of source modality, isolating structured-output capability. Each record includes a natural-language question, a JSON schema, and a ground-truth answer, evaluated with seven metrics (schema compliance, value accuracy, etc.).
Evaluation of 21 models shows near-perfect schema compliance, but value accuracy reaches only 83.0% on text, 67.2% on images, and 23.7% on audio, where longer context substantially hinders extraction. The dataset and evaluation pipeline are released to provide a benchmark for LLM structured output research.