Do Language Models Learn Grammar Like Humans? New Research Says No

The quest to build artificial intelligence that truly understands language has taken another surprising turn. A new study posted on arXiv.org throws cold water on the idea that current language models learn grammatical rules in a way that mirrors human language acquisition. This could have significant implications for how we approach training the next generation of AI.

Tolerance Principle Challenged

The research, led by a team of computational linguists, specifically investigated whether the "Tolerance Principle" – a concept in linguistics that describes how children learn grammatical rules despite exposure to exceptions – applies to transformer-based language models. The Tolerance Principle, proposed by Yang in 2016, suggests that children can learn a grammatical rule even if they encounter a certain number of exceptions to that rule. The study authors sought to determine if language models exhibit similar behavior.

The researchers used BabyBERTa, a transformer model designed for efficient learning with small datasets developed by Huebner et al. in 2021, and trained it on artificial grammars. They carefully controlled the size of the training data, the diversity of sentences, and the ratio of rule-following examples to exceptions. The goal was to see if BabyBERTa could generalize grammatical rules in accordance with the Tolerance Principle, just as human infants do.

BabyBERTa's Unexpected Behavior

The results were not what the researchers expected. "We found that, unlike human infants, BabyBERTa's learning dynamics do not align with the Tolerance Principle," the paper states. This suggests that the mechanisms by which current language models learn grammar are fundamentally different from how humans acquire language. While models like GPT-3, BERT, and LLaMA are incredibly powerful at generating text, they may be doing so based on pattern recognition rather than true grammatical understanding. It seems that even with optimizations like those in BabyBERTa, machines learn in ways distinct from humans.

This has major implications for the future of AI development. If we want to create AI that truly understands and uses language in a human-like way, we may need to rethink our approach to training and architecture. Simply scaling up models and feeding them massive amounts of data may not be enough. We might need to incorporate principles of human language acquisition into the design of these systems. The Verge reports that several labs are already exploring alternative architectures that mimic the brain's language processing centers more closely. The journey toward artificial general intelligence is long and complex, and this latest research highlights just how far we still have to go. It also suggests the current benchmarks might be missing crucial elements for truly assessing language understanding.