Hyperparameter & llama-server Reference

This document explains key runtime flags used across llm-runner and the llm-tui form.

1. Engine & Acceleration Flags

Flag	Recommended Value	Purpose
`-ngl, --n-gpu-layers`	`99`	Offload all layers to GPU VRAM.
`-fa, --flash-attn`	`on`	Enables Flash Attention kernel for faster generation and lower memory pressure.
`--load-mode`	`none` or `mmap`	Use `none` for large models under high VRAM pressure on ROCm to prevent memory stalls; use `mmap` for lighter models.
`-ctk` / `-ctv`	`q8_0`	Quantize KV cache vectors to 8-bit.
`-b, --batch-size`	`2048`	Logical batch size for prompt processing.
`-ub, --ubatch-size`	`512`	Micro-batch size for physical memory chunking.
`--reasoning-preserve`	`true`	Preserves `<think>...</think>` tags without stripping them during chat template processing.

Optimized for logical step planning, architecture design, and math:

Temperature (--temp): 0.6 (prevents hallucination while allowing creative problem exploration)
Top-P (--top-p): 0.95
Top-K (--top-k): 40
Min-P (--min-p): 0.05 (truncates low-probability noise tokens)

Optimized for precise code edits, syntax consistency, and fast tool execution:

When --mmproj <file> is passed:

Enables visual image understanding (screenshots, architecture diagrams, charts).
--image-min-tokens 1024: Ensures sufficient patch token density for grounding and OCR clarity.