uiuc-kang-lab/InjecAgent ? reverse-engineered prompt
Reverse engineered prompt
Build me a Python project that can benchmark how well tool using AI agents resist hidden prompt injection attacks. I want it to load the provided user cases and attacker cases, combine them into test cases, and then run evaluations for both direct harm and data stealing scenarios. The app should let me test different model providers, like OpenAI, Claude, Together AI, and local Llama style models, and it should also support both a basic prompt and a stronger prompt version.
Please make it easy to run from the command line, with one script for prompted agents and another for fine tuned agents. It should print clear results at the end, including valid rate and attack success rates for each attack type. If the run gets interrupted, I want to be able to resume from cache. Use the existing data files in the repo, and if you need to check current model docs online, go ahead.
Are you gonna build this?
make sure you review the code using coderabbit