Frequently asked questions

What is an eval? #

An eval (short for evaluation) is a test for AI language models. People run evals to measure what models can and can't do.

Most evals test math, coding or reading comprehension: skills with clear right and wrong answers. This one tests cultural knowledge.

What is the Kapus Konda Eval? #

It tests whether AI models know a prank that virtually every Marathi speaker grows up with. One person offers to tell a story, and whatever you answer gets thrown back at you with the same offer. There is no story.

Each model is told it's a native Marathi speaker, then either starts the prank or has it played on it. What it says shows whether it knows this piece of oral culture or is guessing.

What does kapus konda mean? #

It doesn't matter.

Why does this eval matter? #

AI models learn mostly from English-language internet content, so they absorb Western culture far more deeply than others. Marathi has 83 million speakers but is badly underrepresented online.

This joke is a small piece of shared culture. If models miss it, they're probably missing whole layers of human experience that just aren't well documented on the internet.

Do 83 million people really know this joke? #

83 million is the estimated number of Marathi speakers. We believe nearly all of them know the joke, because it's such a common part of growing up. There's no survey behind that. It's vibes, not science.

Why is this also a hallucination test? #

Kapus means cotton and konda could mean chaff or dandruff. Together, kapus kondyachi goshta is nonsense dressed up as a story title.

That makes a perfect trap. Models that don't know the joke try to make sense of the words, and confidently invent stories about cotton farmers, kings buying cotton pods, or schoolchildren playing football with cotton.

The more elaborate the story, the surer you can be that the model has never seen the real joke.

Aren't you worried about explaining the joke? #

A little. Models train on internet data, possibly including this site, so future models might learn the answer here rather than from the culture. This is called benchmark contamination, and benchmarks like GSM8K, HumanEval and MMLU have all suffered from it.

But the results mean nothing if you don't know the joke. And honestly, we'd be thrilled if models learned it from here.

Why does the simulated user always say nahi? #

The joke works the same whatever you answer. We use nahi (no) because it's the most confusing reply: saying no to a story should end the conversation. Instead, it loops.

How does scoring work? #

Each model gets three attempts at each test, at temperature 0.7 so the answers can vary. Another model, GPT-6.1 Sol, grades every attempt against the rubric below.

A model's score out of 10 is the average of all its attempts, with the performer test counting double: pulling off the prank shows more than recognizing it. The conversations we show are each model's best attempt, and you can switch to the others.

Models that are no longer available had only one attempt, and their scores are marked with an asterisk. The grader is an OpenAI model, so it also grades OpenAI's own models.

As responder

Someone else starts the prank. Does the model know what's coming?

Passed
Gets that its own answer will be thrown back with the same question, and there's no story.
Kinda passed
Knows it loops forever, but misses that its answer gets echoed back.
Kinda failed
Knows it's a joke, but not how it works.
Failed
Expects a real story.

As performer

The model starts the prank. Can it pull it off?

Passed
Asks the question, then echoes the reply and asks again.
Kinda passed
Knows how the joke works, but fumbles the delivery.
Kinda failed
Knows it's a joke, but not how it works.
Failed
Misses the joke and tells a story.