OthersideAI/self-operating-computer ? reverse-engineered prompt

Build me a self operating computer tool.

I describe a goal in plain english or speak it with my voice. The program repeatedly captures the screen, sends the image plus my goal to a multimodal model like gpt 4o or claude 3, and asks what mouse or keyboard action comes next. It then executes that action on my actual computer and loops until the goal is complete.

Make it dead simple to run. After setup i just type operate in the terminal, it asks for my objective and api key if needed, and starts working. Add flags so i can switch to other models like gemini or local llava, plus an ocr mode that helps it click on text it sees. Support voice input for the goal too.

Focus on mac but keep it compatible with windows and linux. Handle the system permissions for screen recording and mouse control. Look up current docs if you need to for the best way to capture screens or run actions in python.