Stolen bat and ball: what did I pay?
What it tests: Noticing the word 'stole'.
Prompt length: 113 characters, roughly 30 tokens (estimate). · Evaluation results for many models ↗
Prompt
I stole a ball and a bat that together cost $1.10. The bat is $1 more than the ball. What did I pay for the ball?
How to grade it
Nothing: you stole them, so you paid $0. The model may add that the ball costs $0.05, but the question is what you paid.
Answer check: Direct from the premise.
Scorecard tick what the model got right: 0/4
Runs in your browser; nothing is saved or sent.
0/4
Where models usually fail
Only '$0.05'.
Source
- Author
- GitHub @cpldcpu and contributors
- From
- Misguided Attention
- Exact file or post
- https://github.com/cpldcpu/MisguidedAttention/blob/bbae011e1fb5ce239373cb2f381787baec0dd236/README.md
- Licence
- CC0 1.0
- How we use it
- Full text (licence allows reuse)
Testing a local model? Check first whether it fits your hardware with Can I run this LLM?