vLLM, llama.cpp and Ollama flags, generated.
How to use Serving Command Builder
Choose an inference engine. Select vLLM, llama.cpp, Ollama or TGI (Text Generation Inference) under Inference engine.
Enter the model identifier. Type the model into Model identifier, for example org/model-name. An empty field falls back to that placeholder.
Pick a quantization format. Choose from Quantization format. The list changes with the engine, so only formats that engine's command can use are shown.
Set GPU count, context length and port. Fill in GPU count, Context length (tokens) and Server port. Each engine uses only the values that apply to its command.
Copy the command. Press Copy on the generated Command and read the flag explanations underneath.
Limits and privacy
- The tool only builds a command string. It does not check that the model exists or that your hardware can run it.
- Quantization choices per engine: vLLM offers None, AWQ, GPTQ and FP8. llama.cpp offers GGUF Q4_K_M, Q5_K_M and Q8_0. Ollama offers None and the same three GGUF levels. TGI offers None, AWQ, GPTQ, bitsandbytes NF4 and EETQ.
- The vLLM command uses a fixed --gpu-memory-utilization of 0.90.
- The llama.cpp command assumes a local file named after the model and quantization (for example model-name.Q4_K_M.gguf) and uses -ngl 999. Change the file path to match your download.
- For llama.cpp, a GPU count above 1 only adds a note about --split-mode and --tensor-split. The command itself is not changed.
- The TGI command is a docker run using the latest text-generation-inference image, with the port mapped to container port 80.
- Ollama uses OLLAMA_NUM_GPU and a quantization tag such as q4_K_M, which only works if that tag is published for the model.
- The command is built in your browser. Nothing is uploaded.
Frequently asked questions
How do I serve a model with vLLM?
Pick vLLM, enter the model ID, GPU count and context length, and copy the result. It produces a vllm serve command with --tensor-parallel-size, --gpu-memory-utilization and --max-model-len, plus --quantization if you chose one.
Why is my quantization list shorter for some engines?
Each engine only supports certain formats. For example llama.cpp runs GGUF files, while vLLM takes AWQ, GPTQ or FP8 checkpoints.
Does the command work as is?
It uses each engine's real flag names, but you may need to adjust paths, image tags or the model name for your setup. The tool does not validate any of them.
Can I run llama.cpp on multiple GPUs with this?
The generated command targets a single llama-server process. For more than one GPU the tool adds a note telling you to add --split-mode or --tensor-split yourself.