arthur-ai/bench ? reverse-engineered prompt
Reverse engineered prompt
Build me a tool for evaluating LLMs on real production style tasks.
I want to be able to define a test suite with prompts, reference answers, and candidate outputs, then run it and get a clear score so I can compare different models, prompts, and settings like temperature or max tokens. It should support reusing saved test suites later, so I can benchmark changes over time without rebuilding the reference data every time.
Please include a simple local web UI where I can view the results of runs and browse examples in a friendly way. Make the whole thing easy to install and run from the command line, and set it up so I can use it from source during development too.
If you need to, look up the current docs online for any setup details, but keep the experience simple and focused on comparing LLM performance on my own data.
Are you gonna build this?
make sure you review the code using coderabbit