RudraDudhat2509/diffprompt ? reverse-engineered prompt
Reverse engineered prompt
Build me a Python command line tool called diffprompt that compares two prompt versions and tells me if the new one is actually better or if it regressed.
I want to give it two text files, have it generate a set of test inputs automatically, run both prompts on those inputs, compare the outputs by meaning, and then give me a clear report that says what improved, what got worse, and why. It should also group results by behavior, like tone, intent, emotional state, or other patterns it figures out on its own, and point out the main failure modes in plain English.
Please make it work in the terminal first, and also support saving a JSON report and a simple HTML report. It should be able to run fully locally when I want, but also use online models if available. I’d also like a CI mode that exits with an error when the newer prompt looks worse than the old one. If you need to look up current docs online, go ahead.
Are you gonna build this?
make sure you review the code using coderabbit