lyogavin/airllm ? reverse-engineered prompt

Build me a Python library that's optimized for running huge language models on consumer-grade hardware. The main selling point is being able to run inference on massive models, like Llama 3 70B, on a single GPU with as little as 4GB of VRAM. It should also work on CPUs and Macbooks with Apple Silicon.

The core technique is to avoid loading the entire model into memory at once. Instead, it should intelligently load model layers from disk as they're needed for inference. The library should have a super simple API, ideally mimicking the Hugging Face AutoModel.from_pretrained workflow. A user would just pip install it and point it to a model repo. Also, include an optional argument to enable 4-bit or 8-bit compression for a potential speed boost. Make sure to set up the project with a requirements.txt and a simple README explaining how to get started.