Are Emergent Abilities of Large Language Models a Mirage?
Recent work claims that LLMs display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. This paper presents an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, emergent abilities appear due the researcher’s choice of metric rather than due to fundamental changes in model behavior with scale. Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous, predictable changes in model performance. This alternative explanation is presented in a simple mathematical model, then tested in three complementary ways: (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities, (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on BIG-Bench; and (3) show how to choose metrics to produce never-before-seen seemingly emergent abilities in multiple vision tasks across diverse deep networks. Via all three analyses, they show evidence that alleged emergent abilities evaporate with different metrics or with better statistics, and may not be a fundamental property of scaling AI models.
In this video, I will talk about the following: What is emergent abilities of LLMs? Emergent abilities are a mirage? Analyzing GPT-3’s Emergent Arithmetic Abilities. Inducing Emergent Abilities in Networks on Vision Tasks.
For more details, please look at https://arxiv.org/pdf/2304.15004.pdf
Schaeffer, Rylan, Brando Miranda, and Sanmi Koyejo. "Are emergent abilities of Large Language Models a mirage?." NeuRIPS 2023.