About the Slovenian Chat Arena
The Slovenian Chat Arena is an interactive platform that enables comparative evaluation of conversations between different large language models. Users chat with two language models simultaneously and then evaluate which response was better. This allows us to collect valuable feedback on the quality of responses from different models.
The collected evaluations help improve the quality of generated responses and support the development of Slovenian open-access large language models.
Methodology
Every vote is a blind comparison: models A and B stay anonymous until you vote. A vote for the better answer counts as a win; a tie and “both are bad” count as half a win for each model. Votes in conversations where one of the models failed to answer are not counted.
Ratings and intervals
We compute ratings from all counted votes at once with a Bradley–Terry model, which estimates each model's strength from the outcomes of the comparisons. To measure the uncertainty, we repeat the calculation on 1,000 random samples of the votes (bootstrap): the ± value marks the 95% confidence interval; the wider it is, the less certain the rating. A model's rank is one more than the number of models whose whole interval lies above its own, so several models can share a rank. The average rating of all models is 1,000.
Statistics
- Sent messagesNumber of messages between users and language models.
- 13,774
- Evaluated conversationsNumber of conversation evaluations that help us rank model performance.
- 2,954
- Active modelsNumber of language models currently active in the Arena.
- 12
Acknowledgements and sponsors
The development of the platform and projects is carried out under the auspices of the Centre for Language Resources and Technologies (CJVT) and the Faculty of Computer and Information Science (FRI), University of Ljubljana.
Funding provided by
The Ministry of Higher Education, Science and Innovation of the Republic of Slovenia and the European Union – NextGeneration EU, under the PoVeJMo project.
We sincerely thank all institutions and individuals who contribute in any way to the advancement of Slovenian language technologies.