On Oct. 9, the launch of the Erasmian Language Model (ELM) took place. Academic staff, staff and students, as well as external parties such as government agencies and startups, joined the ELM development team to learn more about the generative AI model. How was it developed, what can it do and how can we use it?
The idea behind the Erasmian Language Model (ELM) originated about six months ago, says Evert Stamhuis, Professor of Law and Innovation and Academic Lead for AI Convergence at Erasmus University Rotterdam (EUR). Within the minor AI and Societal Impact, students learn what AI models are, how they work and what possible problems and improvements could be. For the students involved in the minor, issues such as data privacy of closed-source models linked to a central, remote server became clear. As did the environmental impact of the data used. "Another issue at play are the biases that generative AI models have, such as racist and sexist biases that can be seen in programs like ChatGPT," explained Michele Murgia, ELM project leader and coordinator of the minor.
ELM is a generative AI model developed and based on EUR. Unlike other available AI models, ELM is software that is downloaded onto the hard drive of the computer you use to access it. This solves several privacy issues, unlike an AI model on a remote server. Also, the model is kept small to reduce environmental impact. The model is specifically suited for academic research and teaching. It is a truly open-source model (you have insight into both the model and the data) and it is trained in both English and Dutch, to counter certain English-language biases.
The process behind the development of ELM consisted of three steps. First, LLM (Large Language Model) pre-training was required for the model. This was done using the University Library, through the publicly accessible repository of all master's theses and publications published at EUR and Erasmus MC. Thus, at its core, ELM was trained on the academic research conducted at EUR. The second step of the process is fine-tuning under supervision. During this step, the program is given specific examples and instructions for completing a particular request. The current version of ELM does not have a chat interface like other AI (e.g. ChatGPT), but it may do so in the future. As a final step, reinforcement learning with feedback from humans was used to train the program. That way it can distinguish between good and bad generated results.
Currently, there are two versions of ELM: ELM Small and the full ELM. Of these versions, ELM Small will continue to be developed because the full ELM is trained on Llama-2, Google's generative AI. That is not fully open-source and thus would interfere with the desire to make ELM a community-based model. ELM Small is the version intended to be downloaded on laptops for personal use because it can run on most laptop hard drives and uses only 1.8 GB of storage space. This version uses 160 million parameters and gives the user full control over editing the program. This includes adding training material to improve the model, as well as controlled fine-tuning and reinforcement learning options. This is the version of ELM that students in the AI and Societal Impact minor tested, extended and improved.
"We want this model to be efficient and serve the goals we have in mind at EUR. For that reason, we deliberately chose to keep it a smaller model," explained João Goncalves, academic lead of ELM and lecturer in the AI and Societal Impact minor. Both Michele and João emphasized that ELM is not a traditional model, but a community-based model. The end users for whom it was designed, everyone at EUR, are also the co-creators of ELM. Thus, everyone who uses ELM can directly influence the design of the model for their specific academic needs. "The success of the model depends on you, the users, becoming co-creators," says João.
By the end of the event, it was clear that many students and staff were excited about the project and asked relevant questions. These included departmental biases within research, the ability of the program to identify the source of information (which is not feasible due to the nature of the LLM), and whether the program can be aware of its limitations in answering questions based on the information it is trained on.
The next step for ELM is further co-creation. The call for anyone interested in using and developing ELM is open!
To assist in the development of ELM, the team is looking for people interested in the project. Contributions can range from providing information to train the model with, to providing examples and directions for fine-tuning under supervision. You can also provide general feedback on the model. If you are interested, please contact Michele Murgia or João Goncalves.
