CoderRdm/AI_obeserve ? reverse-engineered prompt

Build me a small Python tool that can watch an AI agent’s answer and check it against a few real source files, so it can catch wrong claims, contradictions, skipped files, and made up details.

I want to be able to run one script that sends a task plus a small set of test documents to an LLM, saves the raw answer, then run another script that audits that answer against the source files and gives a clear pass or fail. Please also include a simple way to run the same task many times, collect results, and print basic stats like fail rate and common mistake types.

If it’s not too much extra, add a second experimental mode that uses a local model and looks at hidden state signals during generation, then compares that detector with the LLM judge in a dashboard or report. Use current docs online if you need to, and make it easy to run on a normal machine.