Miloš Stanojević, DeepMind

The role of syntax in the world of large language models

Tuesday 21st November 2023, 16:00. Room L3.36 at LAB42, Amsterdam Science Park, plus live streaming on Zoom.

Large Language Models (LLMs) have shown impressive results in a recent period to the extent that some cognitive scientists are claiming that syntactic theories should be abandoned as an explanation of human language in favour of LLMs. I will provide evidence that syntax is still beneficial both in scientific and engineering pursuits with human language. First, LLMs do not provide a prediction nor an explanation of what are the universal properties of all human languages, unlike the syntactic theory considered here. Second, neural activity of some brain regions during language processing can be accounted for better by an incremental syntactic parser than by a LLM surprisal. Finally, LLMs can work even better if augmented with a syntactic compositional structure. If that is so, you might ask, why is syntax not more popular in NLP then? I believe it is because the modern hardware accelerators (GPUs and TPUs) are not optimal for tree-like computation which makes it difficult to scale syntactic models, i.e. syntax is losing a hardware lottery. To address this problem we have created a JAX library, called SynJAX, that makes training of large scale syntactic models on GPUs and TPUs possible.