Source
ACL
DATE OF PUBLICATION
05/02/2022
Authors
Oleg Serikov Ryan Teehan Miruna Clinciu Eliza Szczechla Natasha Seelam Shachar Mirkin Aaron Gokaslan
Share

Emergent Structures and Training Dynamics in Large Language Models

Abstract

Large language models have achieved success on a number of downstream tasks, particularly in a few and zero-shot manner. As a consequence, researchers have been investigating both the kind of information these networks learn and how such information can be encoded in the parameters of the model. We survey the literature on changes in the network during training, drawing fr om work outside of NLP when necessary, and on learned representations of linguistic features in large language models. We note in particular the lack of sufficient research on the emergence of functional units, subsections of the network wh ere related functions are grouped or organised, within large language models and motivate future work that grounds the study of language models in an analysis of their changing internal structure during training time.

Join AIRI on social media