43 lines
1.8 KiB
Markdown
43 lines
1.8 KiB
Markdown
# Hyperparameter & llama-server Reference
|
|
|
|
This document explains key runtime flags used across `llm-runner` and the `llm-tui` form.
|
|
|
|
---
|
|
|
|
## 1. Engine & Acceleration Flags
|
|
|
|
| Flag | Recommended Value | Purpose |
|
|
| :--- | :--- | :--- |
|
|
| `-ngl, --n-gpu-layers` | `99` | Offload all layers to GPU VRAM. |
|
|
| `-fa, --flash-attn` | `on` | Enables Flash Attention kernel for faster generation and lower memory pressure. |
|
|
| `--load-mode` | `none` or `mmap` | Use `none` for large models under high VRAM pressure on ROCm to prevent memory stalls; use `mmap` for lighter models. |
|
|
| `-ctk` / `-ctv` | `q8_0` | Quantize KV cache vectors to 8-bit. |
|
|
| `-b, --batch-size` | `2048` | Logical batch size for prompt processing. |
|
|
| `-ub, --ubatch-size` | `512` | Micro-batch size for physical memory chunking. |
|
|
| `--reasoning-preserve` | `true` | Preserves `<think>...</think>` tags without stripping them during chat template processing. |
|
|
|
|
---
|
|
|
|
## 2. Sampling Hyperparameters
|
|
|
|
### Reasoning & Planning Profile (SuperQwen / Muse-Glimmer)
|
|
Optimized for logical step planning, architecture design, and math:
|
|
- **Temperature (`--temp`)**: `0.6` (prevents hallucination while allowing creative problem exploration)
|
|
- **Top-P (`--top-p`)**: `0.95`
|
|
- **Top-K (`--top-k`)**: `40`
|
|
- **Min-P (`--min-p`)**: `0.05` (truncates low-probability noise tokens)
|
|
|
|
### Instruct / Implementation Profile (Qwen 3.5 4B)
|
|
Optimized for precise code edits, syntax consistency, and fast tool execution:
|
|
- **Temperature (`--temp`)**: `0.7`
|
|
- **Top-P (`--top-p`)**: `0.80`
|
|
- **Top-K (`--top-k`)**: `20`
|
|
- **Min-P (`--min-p`)**: `0.05`
|
|
|
|
---
|
|
|
|
## 3. Multimodal / Vision Projector
|
|
|
|
When `--mmproj <file>` is passed:
|
|
- Enables visual image understanding (screenshots, architecture diagrams, charts).
|
|
- `--image-min-tokens 1024`: Ensures sufficient patch token density for grounding and OCR clarity.
|