Povedz loď: do language models know our playground jokes?
Let's start with a game. Do what it says, and do it out loud. Not in your head, actually out loud:
Povedz biele 5 krát za sebou.
Done? Now answer quickly, without thinking:
Čo pije krava?
If you said mlieko, you're in good company. Krava pije vodu.
Generating Slovak with large language models often gives you text that's overly formal and unnatural. Understanding informal Slovak is just as hard, if not harder. Casual Slovak has a lot of slang, English words that sometimes get conjugated (lajknúť, šerovať), inconsistent diacritics and plain spelling mistakes. On top of that, Slovaks and Czechs online can be very snarky and sarcastic.
I wanted a way to pick the best model for processing this kind of text from online data. Playground tricks seemed like a good test: everybody here knows them, they're as informal as it gets, and they only work if you get the joke. So I put together Povedz Bench and ran it against 23 hosted model configurations.
Methodology
Dataset
I collected playground tricks from blogs, forums and from memory. Then I expanded the data and the expected answers with GPT models, mostly gpt-6.1-terra, with some items prepared by gpt-6.2-sol and gpt-6.2-astra. That could give GPT models a small home advantage, so keep it in mind when you look at the results. 109 of the 124 punchlines have some evidence on the web, the other 15 are from memory.
The testing consists of two task classes.
Punchlines
The model gets the setup and has to answer with the missing line only. Half of the items are dialogues. This is exactly what Gemini 3.8 Flash saw, in the variant without diacritics:
Dopln poslednu repliku A v slovenskom slovnom chytaku. Napis iba pointu.
A: Povedz lod.
B: Lod.
A:
And this is what it answered, verbatim:
Do riti choď!
Close enough for me, and it's in the answer key, so it counts as a match. That was with reasoning effort set to high. At other settings Gemini went with Trhni si nohou a choď!, Kúp si lístok a choď! or Zjedol si hovno, poď!, so the same model can give quite different answers to the same prompt.
The other half are verses, where the model continues a familiar rhyme. That's 124 items, 74 Slovak and 50 Czech. Each one runs twice, with and without diacritics, so 248 conversations per model.
The scoring is strict. After folding case, punctuation and diacritics, the whole reply has to equal one of the recorded punchlines. If the model adds an explanation, or gives a variant that isn't in the key, it's a miss. So all the numbers below are lower bounds.
Priming
The model first has to repeat a word five times and then gets a trick question. This is exactly what DeepSeek V4.1 Flash saw with reasoning off, including its own reply in the middle:
User: Řekni pětkrát za sebou: první.
DeepSeek: první. první. první. první. první.
User: V závodě právě předběhneš běžce na druhém místě. Na kterém místě jsi teď?
And this is what it answered, verbatim:
Jsi na **prvním místě** – předběhl jsi běžce, který byl druhý, takže jsi se posunul na jeho pozici a on klesl na třetí.
The explanation describes second place and the answer says first. Asked the same question in a fresh conversation, it just said Na druhém místě. 😊
That fresh conversation is the control. There are 28 tricks like this, each in Slovak and Czech and in both spellings, and I count how often the model gives the bait answer with and without the cue.
Grading here isn't just answer-key matching. Replies that match the key exactly are graded right away, and everything else goes to a calibrated Claude Sonnet 5.5 judge that decides whether it's the correct answer, the bait or something else. Only pairs where both replies could be graded are counted, which is about 112 per model.
All hosted runs used the provider's default temperature, a 4,096-token limit per reply, and ran once. Four models also got a full sweep over reasoning effort, 29 runs in total.
Results
Punchlines
Gemini 3.8 Flash matched 27 of 248, GPT-6 Sol 24 and GPT-6 Luna 10. Everyone else got three or fewer, and twelve configurations didn't get a single one.
Some misses are a different but reasonable ending. For Kdo jí tvaroh, the two leaders answered:
- Gemini 3.8 Flash: vidí za roh. (match)
- GPT-6 Sol: ten má sílu jako hroch. (miss)
GPT's line rhymes and works fine, it's just not the one everyone knows.
What I think is interesting is that Gemini doesn't only understand the punchline, it also delivers it and doesn't ruin it. Sonnet just ruins the joke:
Čo robí chobotnica, keď jej režeš chápadlá?
- Gemini 3.8 Flash: Nechápe. (match)
- Claude Sonnet 5.5: Nechápe, veď jej režeš chápadlá. (miss)
Sonnet clearly knows the joke, it just can't help explaining it. Exact matching counts that as a miss, and for what I want I think that's fair. If a model is going to read snarky comments, it should get the tone too.
Reasoning effort
- Gemini 3.8 Flash
- GPT-6 Sol
- Claude Sonnet 5.5
- Qwen3.8 Flash
- second run at the default level
Gemini also accepted “disable”, which scored 21, the same as “minimal”. Not every model offers every level.
The numbers
| Model | default | none | minimal | low | medium | high | xhigh | max |
|---|---|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | 27 (medium) | 20 | 21 | 22 | 25 | 23 | — | — |
| GPT-6 Sol | 24 (medium) | 12 | — | 25 | 20 | 18 | 22 | — |
| Claude Sonnet 5.5 | 2 (high) | 1 | 0 | 0 | 1 | 1 | 1 | 3 |
| Qwen3.8 Flash | 2 (xhigh) | 0 | 0 | 1 | 2 | 0 | 1 | 1 |
More reasoning didn't help. Turning it off cut GPT-6 Sol from 24 to 12, but above low it didn't add anything. Gemini stayed between 20 and 25 at every level I asked for, and Claude and Qwen stayed between 0 and 3. That makes sense if this is mostly recall: the model has either seen the rhyme or it hasn't, and thinking longer won't produce it.
The rings are a second run of each model at its default level, which is what you get when you don't ask for any effort (medium for Gemini and GPT-6 Sol). Same model, same setting, and the scores still differ by one to four matches. That's roughly how much a single run moves, so I wouldn't read much into small gaps.
At the high end it got worse. Some reasoning-heavy routes hit the 4,096-token limit, usually before writing any answer: Muse Ember on 38% of calls, Step-5 Preview on 29%, GLM-5.3 and Qwen3.8 Flash on 26%. Sonnet 5.5 at max returned 150 empty replies. For these models the score says as much about the token budget as about what they know.
Cost
Prices differ a lot between providers, so here are Google, OpenAI and Anthropic at every reasoning effort I ran, against what each whole run actually cost. For the rest of the field I picked Alibaba's Qwen3.8 Flash, the one other model I ran at every effort level, with a reported cost for every run. Each point is one full run, and the line connects the runs that beat everything cheaper.
- Gemini 3.8 Flash
- GPT-6 Sol
- Claude Sonnet 5.5
- Qwen3.8 Flash
- best so far for the money
The numbers
| Run | Cost | Punchlines | Clean pairs | Score |
|---|---|---|---|---|
| Qwen3.8 Flash · none | $0.02 | 0 | 93 | 41.5 |
| Qwen3.8 Flash · minimal | $0.12 | 0 | 95 | 42.4 |
| Qwen3.8 Flash · low | $0.13 | 1 | 99 | 44.4 |
| GPT-6 Luna · medium (default) | $0.13 | 10 | 110 | 51.1 |
| GPT-6 Sol · none | $0.13 | 12 | 106 | 49.7 |
| Qwen3.8 Flash · medium | $0.16 | 2 | 98 | 44.2 |
| Gemini 3.8 Flash · disable | $0.41 | 21 | 112 | 54.2 |
| Gemini 3.8 Flash · low | $0.43 | 22 | 111 | 54 |
| Qwen3.8 Flash · high | $0.43 | 0 | 97 | 43.3 |
| Qwen3.8 Flash · xhigh (default) | $0.43 | 2 | 103 | 46.4 |
| Qwen3.8 Flash · max | $0.44 | 1 | 99 | 44.4 |
| Gemini 3.8 Flash · minimal | $0.44 | 21 | 112 | 54.2 |
| Qwen3.8 Flash · xhigh | $0.45 | 1 | 95 | 42.6 |
| GPT-6 Sol · low | $0.48 | 25 | 107 | 52.8 |
| Gemini 3.8 Flash · none | $0.50 | 20 | 112 | 54 |
| GPT-6 Sol · medium | $1.10 | 20 | 109 | 52.7 |
| GPT-6 Sol · medium (default) | $1.11 | 24 | 110 | 53.9 |
| Claude Sonnet 5.5 · low | $1.13 | 0 | 107 | 47.8 |
| Claude Sonnet 5.5 · minimal | $1.23 | 0 | 109 | 48.7 |
| Claude Sonnet 5.5 · medium | $1.26 | 1 | 110 | 49.3 |
| GPT-6 Sol · high | $1.89 | 18 | 110 | 52.7 |
| Claude Sonnet 5.5 · high (default) | $2.04 | 2 | 110 | 49.5 |
| Claude Sonnet 5.5 · none | $2.05 | 1 | 109 | 48.9 |
| Claude Sonnet 5.5 · high | $2.06 | 1 | 108 | 48.4 |
| Gemini 3.8 Flash · medium | $2.15 | 25 | 112 | 55 |
| Gemini 3.8 Flash · medium (default) | $2.15 | 27 | 111 | 55 |
| Claude Sonnet 5.5 · xhigh | $3.06 | 1 | 108 | 48.4 |
| GPT-6 Sol · xhigh | $3.12 | 22 | 110 | 53.5 |
| Gemini 3.8 Flash · high | $3.43 | 23 | 112 | 54.6 |
| Claude Sonnet 5.5 · max | $11.57 | 3 | 109 | 49.3 |
The cheap end is where it gets interesting. GPT-6 Luna and GPT-6 Sol with reasoning off cost about $0.13 for the whole run and matched 10 and 12. Gemini at its cheap settings gets 20–22 for $0.41–0.50. Past that you pay about five times more for a few extra matches. Claude at max cost $11.57 and matched 3.
Punchlines alone don't say anything about falling for tricks, so I also made a combined score:
score = 50 × punchlines / 248
+ 50 × clean priming pairs / 112
A priming pair is clean if the model got the question right both without and with the cue.
- Gemini 3.8 Flash
- GPT-6 Sol
- Claude Sonnet 5.5
- Qwen3.8 Flash
- best so far for the money
The numbers
| Run | Cost | Punchlines | Clean pairs | Score |
|---|---|---|---|---|
| Qwen3.8 Flash · none | $0.02 | 0 | 93 | 41.5 |
| Qwen3.8 Flash · minimal | $0.12 | 0 | 95 | 42.4 |
| Qwen3.8 Flash · low | $0.13 | 1 | 99 | 44.4 |
| GPT-6 Luna · medium (default) | $0.13 | 10 | 110 | 51.1 |
| GPT-6 Sol · none | $0.13 | 12 | 106 | 49.7 |
| Qwen3.8 Flash · medium | $0.16 | 2 | 98 | 44.2 |
| Gemini 3.8 Flash · disable | $0.41 | 21 | 112 | 54.2 |
| Gemini 3.8 Flash · low | $0.43 | 22 | 111 | 54 |
| Qwen3.8 Flash · high | $0.43 | 0 | 97 | 43.3 |
| Qwen3.8 Flash · xhigh (default) | $0.43 | 2 | 103 | 46.4 |
| Qwen3.8 Flash · max | $0.44 | 1 | 99 | 44.4 |
| Gemini 3.8 Flash · minimal | $0.44 | 21 | 112 | 54.2 |
| Qwen3.8 Flash · xhigh | $0.45 | 1 | 95 | 42.6 |
| GPT-6 Sol · low | $0.48 | 25 | 107 | 52.8 |
| Gemini 3.8 Flash · none | $0.50 | 20 | 112 | 54 |
| GPT-6 Sol · medium | $1.10 | 20 | 109 | 52.7 |
| GPT-6 Sol · medium (default) | $1.11 | 24 | 110 | 53.9 |
| Claude Sonnet 5.5 · low | $1.13 | 0 | 107 | 47.8 |
| Claude Sonnet 5.5 · minimal | $1.23 | 0 | 109 | 48.7 |
| Claude Sonnet 5.5 · medium | $1.26 | 1 | 110 | 49.3 |
| GPT-6 Sol · high | $1.89 | 18 | 110 | 52.7 |
| Claude Sonnet 5.5 · high (default) | $2.04 | 2 | 110 | 49.5 |
| Claude Sonnet 5.5 · none | $2.05 | 1 | 109 | 48.9 |
| Claude Sonnet 5.5 · high | $2.06 | 1 | 108 | 48.4 |
| Gemini 3.8 Flash · medium | $2.15 | 25 | 112 | 55 |
| Gemini 3.8 Flash · medium (default) | $2.15 | 27 | 111 | 55 |
| Claude Sonnet 5.5 · xhigh | $3.06 | 1 | 108 | 48.4 |
| GPT-6 Sol · xhigh | $3.12 | 22 | 110 | 53.5 |
| Gemini 3.8 Flash · high | $3.43 | 23 | 112 | 54.6 |
| Claude Sonnet 5.5 · max | $11.57 | 3 | 109 | 49.3 |
Priming is easy for the big models. Gemini got all 112 pairs right in five of its seven settings, so the score mostly separates on punchlines again. GPT-6 Luna scores 51.1 for $0.13, more than any Claude or Qwen run at any price. Gemini with reasoning disabled gets 54.2 for $0.41, and the best score overall, 55.0 at medium, costs about five times that.
Priming
- asked fresh
- after the cue
- more bait after the cue
- less
The numbers
| Configuration | Fresh | After cue | Graded pairs |
|---|---|---|---|
| Claude Sonnet 5.5 · high | 2 | 1 | 112 |
| GPT-6 Sol · medium | 2 | 0 | 112 |
| Gemini 3.8 Flash · medium | 0 | 0 | 111 |
| Qwen3.8 Flash · xhigh | 2 | 2 | 106 |
| Gemma 4 26B A4B · off | 9 | 10 | 110 |
| Gemma 4 31B · off | 11 | 8 | 109 |
| Ling 3.0 Flash · on | 8 | 7 | 111 |
| LFM-3 1.7B · on | 7 | 12 | 97 |
| DeepSeek V4.1 Flash · minimal | 2 | 3 | 112 |
| DeepSeek V4.1 Flash · off | 8 | 8 | 111 |
| GLM-5.3 · on | 4 | 4 | 109 |
| GPT-6 Luna · medium | 0 | 1 | 111 |
| Kimi K3 · on | 4 | 4 | 111 |
| Muse Ember · max | 4 | 4 | 109 |
| Mistral Large 4 · high | 3 | 5 | 110 |
| Mistral Medium 3.5 · off | 13 | 8 | 112 |
| Mistral Small 4 · off | 16 | 19 | 111 |
| Nemotron 3 Ultra · high | 5 | 6 | 111 |
| Nemotron 3.5 Lightning · on | 10 | 12 | 106 |
| Nemotron 3.5 Super VL · on | 5 | 4 | 110 |
| Step-5 Preview · medium | 3 | 6 | 108 |
| HY4 Flash · on | 4 | 3 | 108 |
| GLM-5.3 Flash · max | 3 | 5 | 110 |
I expected this to work better than it did. After the cue, ten configurations gave more bait answers, seven gave fewer and six stayed the same. The biggest changes were five pairs either way: LFM-3 1.7B went from 7 to 12 and Mistral Medium 3.5 from 13 to 8.
The bigger difference is between models. GPT-6 Sol, GPT-6 Luna and Gemini almost never took the bait. Mistral Small 4, Mistral Medium 3.5, both Gemmas and Nemotron 3.5 Lightning fell for 8–14% of the questions even without the cue. Turning off DeepSeek V4.1 Flash's reasoning raised its bait answers from 2–3 to 8.
The race answer from the methodology section came from that reasoning-off run. Over all 111 pairs, though, it gave eight bait answers with the cue and eight without, so that one is just a good anecdote.
Diacritics
I expected missing diacritics to hurt, since that's how a lot of people type. They didn't. Gemini matched 13 punchlines with diacritics and 14 without, GPT-6 Sol 11 and 13, GPT-6 Luna 3 and 7. Everyone else was too close to zero to compare.
So which model?
For generating natural Slovak or Czech, this leads me to believe that Gemini and OpenAI models are the best bet. They know the most punchlines and they deliver them without explaining the joke. GPT-6 Luna does a decent job for very little money.
For understanding, I think any frontier model from Google, OpenAI or Anthropic will get it. All of them almost never took the bait, and Sonnet seems to know more jokes than its score shows, it just can't leave one alone. But I think Google has that cultural edge.
Keep in mind that my sample sizes are low, every setting ran only once, and this benchmark is mostly for fun.
Both hosted studies together cost about $63. I also ran three open models locally on two Radeon RX 9070 XTs with a smaller panel, and none of them matched a single punchline.
If you'd like the data, the code or the full model replies, get in touch. And if you know a povedz joke that's missing, send it my way too.