TheTom/llama-cpp-turboquant ? reverse-engineered prompt

Build me a local AI inference app based on llama.cpp, but with the TurboQuant plus improvements turned on.

I want it to load LLMs like a normal llama.cpp setup, then use the new KV cache and weight compression options to save memory and run faster, especially on Apple Silicon, NVIDIA CUDA, AMD ROCm, and Vulkan. Keep the existing llama.cpp behavior working too, so I can still use standard models and quantization formats, but make the TurboQuant formats available as an opt in choice.

It would be great if you set it up so I can run inference from the command line and also use the server or example app if that already exists here. If anything is unclear from the repo, look up the current docs online and follow the project’s README style. Please make sure it builds cleanly and include a simple way to test that the new quantization and cache options are working.