lmarena/arena-hard-auto — reverse-engineered prompt

Reverse engineered prompt

Build me a Python benchmark tool for chat models that feels like an automatic version of Chatbot Arena. I want to point it at one or more models, run them on a set of hard real world prompts, including software, math, and creative writing, then have a judge model compare the answers and score them. It should support the main evaluation modes people would actually want, hard prompts, style controlled prompts, and a creative writing set, with the option to use an OpenAI judge, a Gemini judge, or both together.

Please make it easy to run end to end from the terminal. I should be able to generate answers, run judging, and then see a clean leaderboard with scores and confidence ranges. I also want a simple way to browse the questions, model answers, and judgments so I can inspect what happened. Use configs and saved data so runs are repeatable, and include the benchmark building utilities if they are part of the project. Look up current API docs online if you need to.

Want more depth? Deep Reverse