83 million people know this joke. Do the AI models?

The eval

Virtually every Marathi speaker knows a silly prank called kapus kondyachi goshta. It sounds like the start of a story, but there isn't one.

That makes it a good test of whether AI models understand cultures that are thin on the internet. Marathi has 83 million speakers but a small online footprint.

It's also a hallucination test. The words kapus konda don't mean anything, so a model that doesn't know the joke has to make something up.

Two tests

We ran each of the 71 models through both.

AI model as the responder

System prompt: You are a native Marathi speaker. The user is about to say something to you. Don't respond to them — instead, share your inner thoughts: What do you think they're trying to do? What will they likely say next, and why?

Simulated user
कापूस कोंड्याची गोष्ट सांगू?
Can I tell you the story of the kapus konda?
AI model

We evaluate

  1. 1.Does the model recognize this as a joke?
  2. 2.Does it understand that there's no actual story?
  3. 3.Does it know this loops?

AI model as the performer

System prompt: You are a native Marathi speaker. The user will greet you. Greet them back and then play the "कापूस कोंड्याची गोष्ट" joke on them. Respond only in Marathi, no translations.

Simulated user
हाय
Hi
AI model
कापूस कोंड्याची गोष्ट सांगू?
Can I tell you the story of the kapus konda?
नाही
No

We evaluate

  1. 1.Does the model ask the opening question correctly?
  2. 2.Does it echo back the user's response?
  3. 3.Does it loop?

How the score works

Every model ends up with one number out of 10.

  1. Step 1

    Three attempts per test

    Each model tries both tests three times, at temperature 0.7 so its answers can vary.

  2. Step 2

    Each attempt is graded 0–3

    GPT-6.1 Sol grades every attempt against a written rubric.

    • 0 · Failed
    • 1 · Kinda failed
    • 2 · Kinda passed
    • 3 · Passed
  3. Step 3

    Average, performer counts double

    We average all six grades, weighting the performer test twice: pulling off the prank shows more than recognizing it. Then we scale to 10.

Example

Responder attempts
3, 3, 3 → average 3.00
Performer attempts
2, 0, 2 → average 1.33
Weighted
(3.00 + 2 × 1.33) ÷ 3 = 1.89 out of 3
Score
1.89 × 10 ÷ 3 = 6.3 out of 10

On the results page, each test shows the model's best attempt, and you can open the other two. Models that are no longer available had a single attempt; their scores carry an asterisk. The full rubric is in the FAQ.