Large Language Models (LLMs) are AI systems trained on huge amounts of data. They are capable of understanding and generating human language and they represent natural language, intended as any type of spoken language. What these models work on is language, which should not be considered as a mere sequence of words, since it also conveys cultural and linguistic aspects, to which humans are extremely attached to. Nowadays they are almost universally known thanks to one of their famous interfaces, ChatGPT. Their success is linked to the wide range of applications they enable, starting with their use as computational tools to support writing. 

Malvina Nissim, Professor of Computational Linguistics & Society at the University of Groningen, has been studying and developing these models for years. During her attendance at this year’s AIxIA, the international conference of the Italian Association for Artificial Intelligence at our university, we had the opportunity to discuss with her about AI and LLMs. 

“One of the most important concepts that needs to be understood regarding LLMs is that they are language generation models, which means they generate language: they are not knowledge-based, and they should not be treated as such” explains prof. Nissim.  

To understand LLMs output, it is important to understand how they work and how they are trained. To train these models, different texts are used, such as those on Wikipedia. These texts contain facts, but this is not what these models learn: they actually learn how to produce language, word by word.  

“These systems acquire knowledge only as a by-product of being exposed to a lot of language. They work as a statistical model: if they have seen a piece of text, for example a piece of news, many times, they consider this fact to be probable. And they do this not because they know the fact, but because they have learned that after that word describing that fact comes this word, and then that word and then that word. Because of this, LLMs should not be treated as a retrieval model and used instead of a Google search, because they make things up”, warns Prof. Nissim. 

Another important aspect is context, both cultural and linguistic: do LLMs understand it and can we teach it to them? 

Language models are trained on massive amount of text data coming from different sources, which may sometimes convey the same content expressed in different ways. Let’s take Wikipedia and X, for example: they can deal with the same fact but express it in very different styles. If asked to produce a text in the same style as a particular source, as X for example, LLMs might even be able to reproduce it, but that doesn't mean that they understand the differences in context. Interestingly, thanks to the fact that they have access to many textual contexts, current models look back up to 20000 words to build their probabilities and predict the next word: this explains why they seem to be able to contextualise words, recognise different meanings and write in different styles, even though they are not aware of any pragmatics of communication. 

What kind of development can we expect in the next five to ten years when it comes to LLMs for the general public?  

I think we will see even more integration of LLMs in everyday tasks, even in applications that we will not be aware of. That is why it is important to invest in literacy. From a research perspective, I think there will be some branching in the research fields, with a focus on those working on deployment, while in academia I expect their study as a product of natural language.  

How can we balance the rapid advances in artificial intelligence with societal and ethical responsibilities? 

Developing AI, and especially LLMs, with an eye on the ethical consequences for society is a huge challenge. I don't have a clear answer to this, but I think that we, as researchers, should be aware of this without being frozen in our research by ethical issues. By reflecting societal habits, LLMs prove to be a window to study society. Furthermore, we should focus on using these models into applications and tools that people use daily and that are not dangerous. Still, defining what is dangerous and what is ethical can be very subjective and a lot of research is needed from the community in this sense. As researchers, we need to make as many people as possible aware of how these systems work, because only if they know what these objects are, they can be better prepared to not put themselves in danger by using them.