Thirumurugan240/Model_Quantization ? reverse-engineered prompt

Reverse engineered prompt

Build me a small Python project that teaches model quantisation and lets me run a quantised Qwen2.5 0.5B instruct model locally on a CPU only Windows machine.

I want a simple notebook that explains the idea step by step, with a few runnable examples, plus two scripts, one that downloads the model and quantises it into a local folder, and another that loads that saved quantised model and generates text from a question I can edit in the file. It should also show how many tokens per second it is getting when it runs.

Please keep it practical and easy to follow, so someone new to this can understand what quantisation is and why it helps with memory and speed. Make sure it works without an NVIDIA GPU, and if you need to check current install or model loading details, look up the latest docs online.

Are you gonna build this?

make sure you review the code using coderabbit

Try freeSponsored — opens CodeRabbit in a new tab