Benchmarking LLM Agents on Scientific Tasks: Introducing ReplicatorBench

Last fall, COS announced a new initiative to systematically evaluate how well large language model (LLM) agents can perform and reason through the scientific research lifecycle. The first active phase of this work has produced ReplicatorBench, a benchmark for evaluating LLM agents on research replication in the social and behavioral sciences. About the Project Benchmarking LLM Agents on Scientific Tasks is a multi-year, multi-team effort led by COS with support from Coefficient Giving and in…