ORC IAP Seminar 2026 Talk 5 Kehang Zhu
Kehang Zhu
PhD Student
Harvard University
Title
Automated Social Science: Language Models as Scientist and Subjects
Abstract
We present an approach for automatically generating and testing, in silico, social scientific hypotheses. This automation is made possible by recent advances in large language models (LLM), but the key feature of the approach is the use of structural causal models. Structural causal models provide a language to state hypotheses, a blueprint for constructing LLM-based agents, an experimental design, and a plan for data analysis. The fitted structural causal model becomes an object available for prediction or the planning of follow-on experiments. We demonstrate the approach with several scenarios: a negotiation, a bail hearing, a job interview, and an auction. In each case, causal relationships are both proposed and tested by the system, finding evidence for some and not others. We provide evidence that the insights from these simulations of social interactions are not available to the LLM purely through direct elicitation. When given its proposed structural causal model for each scenario, the LLM is good at predicting the signs of estimated effects, but it cannot reliably predict the magnitudes of those estimates. In the auction experiment, the in silico simulation results closely match the predictions of auction theory, but elicited predictions of the clearing prices from the LLM are inaccurate. Combined with the exact replication with human subject experiments, the system can explore a large design space and discover novelty patterns about human society.
Bio
Kehang is a 5th year PhD student co-advised by Prof. John Horton in MIT Sloan and Prof. David Parkes, Harvard.
Kehang is interested in studying AI agents as proxies for human decision-making, as well as how individuals collaborate with these agents in economic contexts. His research focuses on using LLM simulations to find novel patterns about human society and also trade-offs of deploying large language models (LLMs) as autonomous agents.