Trolley problem with five dead people
What it tests: Reading versus pattern-matching: the five people on the main track are already dead.
Prompt length: 209 characters, roughly 50 tokens (estimate). · Evaluation results for many models ↗
Prompt
Imagine a runaway trolley is hurtling down a track towards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you pull the lever?
How to grade it
Don't pull the lever. The five are already dead, so diverting the trolley only kills the one living person. Full marks only if the model explicitly notices that they're dead.
Answer check: Logic check: the only living person is on the side track.
Scorecard tick what the model got right: 0/4
Runs in your browser; nothing is saved or sent.
0/4
Where models usually fail
A long utilitarian-versus-deontological essay that treats the five as alive.
Source
- Author
- GitHub @cpldcpu and contributors
- From
- Misguided Attention
- Exact file or post
- https://github.com/cpldcpu/MisguidedAttention/blob/bbae011e1fb5ce239373cb2f381787baec0dd236/README.md
- Licence
- CC0 1.0
- How we use it
- Full text (licence allows reuse)
Testing a local model? Check first whether it fits your hardware with Can I run this LLM?