Prompt Engineering Unearths Hidden AI Skills

A benchmark study found that the celebrated GeoGuessr-style prompt for OpenAI’s o3 image model didn’t improve geolocation accuracy and in fact performed slightly worse than a simple prompt. The author evaluated 200 images from Wikimedia Commons, Geograph, and iNaturalist, comparing a detailed, iteratively refined “GeoGuessr” prompt against a basic instruction to infer location. Median and mean error distances were higher with the elaborate prompt, and it increased inference time modestly. The piece argues that iterative prompt engineering can create illusions of improved capability because models will endorse prompt additions, and stresses the importance of controlled benchmarks to validate claimed gains.

A benchmark test shows OpenAI’s o3 model’s celebrated “GeoGuessr” prompt didn’t improve geolocation performance: across 200 real-world images the author ran o3 with a basic prompt and with Kelsey Piper’s elaborate GeoGuessr prompt, and the default prompt yielded slightly better median and mean distance-to-true-location scores. Both prompts performed well overall, and the long prompt only modestly increased inference time. The experiment suggests prompt engineering can create persuasive narratives about capability gains without real improvement, especially when researchers iterate prompts with the model itself. The author notes such checks are straightforward and cheap but weren’t performed when the prompt became widely shared.