Lucas Wilkinson

Explore inference performance improvements that help vLLM serve DeepSeek AI models more efficiently in this technical deep dive.

Machete, Neural Magic’s optimized kernel for NVIDIA Hopper GPUs, achieves 4x memory savings and faster LLM inference with mixed-input quantization in vLLM.

Learn about Marlin, a mixed-precision matrix multiplication kernel that delivers 4x speedup with FP16xINT4 computations for batch sizes up to 32.