The Eval
There's a silly joke virtually every Marathi speaker knows: kapus kondyachi goshta.
Don't know the joke? Read about it →
Its ubiquity in Marathi culture, combined with the fact that Marathi is underrepresented on the internet, makes it a good test for whether AI models truly understand non-western cultures. And since "kapus konda" is pure absurdity (the words don't mean anything coherent), it's also a good test for hallucination.
We challenged 31 leading AI models in two ways:
AI Model as the responder
System prompt for the AI Model: You are a native Marathi speaker. The user is about to say something to you. Don't respond. Just share your inner thoughts about what they're doing.
Simulated User
कापूस कोंड्याची गोष्ट सांगू?
Can I tell you the story of the kapus konda?
AI Model
AI
We evaluate:
- Does the model recognize this as a joke?
- Does it understand that there's no actual story?
- Does it know this loops?
AI Model as the performer
System prompt for the AI Model: You are a native Marathi speaker. The user will greet you. Greet them back and then play the "kapus kondyachi goshta" joke on them. Respond only in Marathi.
Simulated User
हाय
Hi
AI Model
AI
कापूस कोंड्याची गोष्ट सांगू?
Can I tell you the story of the kapus konda?
Simulated User
नाही
No
AI Model
AI
We evaluate:
- Does the model ask the opening question correctly?
- Does it echo back the user's response?
- Does it loop?