Povedz loď: do language models know our playground jokes?
Let's start with a game. Do what it says, and do it out loud. Not in your head, actually out loud:
Povedz biele 5 krát za sebou.
Done? Now answer quickly, without thinking:
Čo pije krava?
If you said mlieko, you're in good company. Krava pije vodu.
Generating Slovak with large language models often gives you text that's overly formal and unnatural. Understanding informal Slovak is just as hard, if not harder. Casual Slovak has a lot of slang, English words that sometimes get conjugated (lajknúť, šerovať), inconsistent diacritics and plain spelling mistakes. On top of that, Slovaks and Czechs online can be very snarky and sarcastic.
I wanted a way to pick the best model for processing this kind of text from online data. Playground tricks seemed like a good test: everybody here knows them, they're as informal as it gets, and they only work if you get the joke. So I put together Povedz Bench and ran it against 23 hosted model configurations.