Compare Q4, Q5, Q8 and FP16 trade-offs.
How to use Quantization Planner
Enter the model identifier. Type the model repo into Model identifier, for example org/model-name. If you leave it empty the command uses org/model-name as a placeholder.
Choose a target format. Pick AWQ, GPTQ or GGUF under Target format.
Set the format options. For AWQ or GPTQ choose Bits (4-bit or 8-bit) and a Group size. For GGUF choose a GGUF quant level: Q4_K_M, Q5_K_M, Q6_K or Q8_0.
Choose target hardware. Select Consumer GPU, Datacenter GPU or CPU only under Target hardware.
Copy the command. Review the Conversion command and press Copy. Read the quality, speed and VRAM trade-off text below it.
Limits and privacy
- The tool only generates a command from a template. It does not download, convert or benchmark the model.
- The generated commands use AutoAWQ, AutoGPTQ, and llama.cpp's convert_hf_to_gguf.py followed by llama-quantize. You need those tools installed.
- Target hardware only changes the AWQ command, switching the device between cuda:0 and cpu. It does not change the GPTQ or GGUF output.
- The GPTQ command contains a calibration_examples placeholder that you must replace with your own tokenized samples.
- The trade-off text gives general ranges from public benchmarks, not measured results for your model.
- The command is built in your browser. Nothing is uploaded.
Frequently asked questions
Which should I choose, AWQ, GPTQ or GGUF?
The tool describes GGUF as the option that runs well on CPU-only hardware through llama.cpp, while AWQ and GPTQ target GPU inference. Its trade-off text compares them using general published ranges.
Does this tool quantize my model?
No. It writes the conversion command for the format you pick, and you run that command yourself on a machine with the model and the required libraries.
What GGUF quant levels can I generate?
Q4_K_M, Q5_K_M, Q6_K and Q8_0. The command first converts to an f16 GGUF, then runs llama-quantize with the level you chose.
How accurate is the quality loss summary?
It is a general summary of ranges reported in public benchmarks. Actual quality and VRAM savings vary by model, calibration data and settings.