theschoolofai/S18Code ? reverse-engineered prompt

Build me a small Python project that can run a fair side by side evaluation of coding agents. I want one simple loop that runs the same task set twice, once as a baseline and once with a protected path guard plus a repeated failure limit, then saves the raw run before anything gets scored.

It should support running locally with Ollama and also a hosted model mode if API keys are present. Include a handful of tasks, a scorer that can be rerun later from saved results, and a way to compare the two setups honestly. Make sure it writes out proof files and run logs so I can inspect what happened after the fact, and keep any bad scoring results around instead of hiding them.

I’d like a command to run the full benchmark, a command to run one task, and a separate script to rescore everything from disk without calling the model again. If you need current docs for the local or hosted model setup, look them up online.