NVIDIA/TensorRT-LLM ? reverse-engineered prompt

Reverse engineered prompt

Build me a project like TensorRT LLM that makes it easy to run large language models and visual generation models really fast on NVIDIA GPUs.

I want a simple Python friendly way to define or load models, optimize them for inference, and serve them efficiently with a runtime that can handle batching and orchestration under the hood. It should include support for the usual performance tricks like specialized kernels, quantization, and other GPU optimizations so the same model runs faster and uses hardware well.

Please make it usable from Python first, but also include the pieces needed for a C++ runtime and server side execution. Add a few practical examples so I can see how to get started with a chat model and a visual generation model, and make sure the docs explain how to build, run, and test it locally. If you need to look up current NVIDIA and CUDA docs online, go ahead and do that.

Are you gonna build this?

make sure you review the code using coderabbit

Try freeSponsored — opens CodeRabbit in a new tab