Knights and knaves with no consistent answer
What it tests: Detecting a contradiction instead of forcing an assignment.
Prompt length: 314 characters, roughly 80 tokens (estimate). · Evaluation results for many models ↗
You arrive on an island inhabited solely by two types of people - Knights who always tell the truth, and Knaves who always lie. Standing at a fork in the road, you meet two inhabitants named A and B. A says "B is a Knave." B says "A is telling the truth." You need to determine who is who to find the correct path.
How to grade it
No consistent assignment exists. If A is a knight, B is a knave, but then B's statement 'A is telling the truth' would be true, which a knave can't say. If A is a knave, B is a knight, and B's true statement would mean A tells the truth, another contradiction. So the puzzle as stated has no solution.
Answer check: Brute-forced all four knight/knave assignments: none is consistent.
Scorecard tick what the model got right: 0/4
Runs in your browser; nothing is saved or sent.
Where models usually fail
Answering 'A is a knight, B is a knave' and glossing over B's statement.
Source
- Author
- GitHub @cpldcpu and contributors
- From
- Misguided Attention
- Exact file or post
- https://github.com/cpldcpu/MisguidedAttention/blob/bbae011e1fb5ce239373cb2f381787baec0dd236/README.md
- Licence
- CC0 1.0
- How we use it
- Full text (licence allows reuse)
Testing a local model? Check first whether it fits your hardware with Can I run this LLM?