49 lines
2.1 KiB
Markdown
49 lines
2.1 KiB
Markdown
# VRAM Budgeting & Hardware Tuning (AMD ROCm)
|
||
|
||
## Target Platform: AMD ROCm (~28 – 30GB Usable VRAM)
|
||
|
||
Serving large reasoning models with long context windows requires careful VRAM accounting to prevent ROCm memory allocation faults or context truncation.
|
||
|
||
---
|
||
|
||
## 1. Multi-Agent Cluster Footprint
|
||
|
||
Running all 3 models concurrently on a single ~30GB GPU:
|
||
|
||
| Component | Model / Vision Weights | Weights in VRAM | Context Size | KV Cache (`q8_0`) | Total VRAM |
|
||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||
| **Planner (SuperQwen 27B)** | `Qwen3.8-27B-UD-Q4_K_XL` + `mmproj-Q8_0` | ~18.1 GB | 65,536 (64K) | ~2.6 GB | ~20.7 GB |
|
||
| **Worker #1 (Qwen 4B)** | `Qwen3.5-4B-Q5_K_M` + `mmproj-BF16` | ~3.9 GB | 16,384 (16K) | ~0.4 GB | ~4.3 GB |
|
||
| **Worker #2 (Qwen 4B)** | `Qwen3.5-4B-Q5_K_M` + `mmproj-BF16` | ~3.9 GB | 16,384 (16K) | ~0.4 GB | ~4.3 GB |
|
||
| **Total** | | **~25.9 GB** | | **~3.4 GB** | **~29.3 GB** |
|
||
|
||
> [!NOTE]
|
||
> Headroom is approximately ~0.7 – 1.0 GB. Keep system desktop composition (Niri / Wayland) overhead low.
|
||
|
||
---
|
||
|
||
## 2. Solo Model Memory Footprints
|
||
|
||
### Muse-Glimmer 30B Solo
|
||
- **Weights Footprint**: ~17.5 GB (`UD-Q4_K_XL` + `mmproj-Q8_0`).
|
||
- **Context Ceiling**: 131,072 tokens (128K).
|
||
- **KV Cache Footprint (`q8_0`)**: ~8.5 GB at full 128K context.
|
||
- **Total Footprint**: ~26.0 GB. Comfortably fits inside 30GB VRAM with ~4GB headroom.
|
||
|
||
### SuperQwen 3.8 27B Solo
|
||
- **Weights Footprint**: ~17.5 GB (`SuperQwen3.8-27b-abliterated-Q4_K_M` + `mmproj-Q8_0`).
|
||
- **Context Ceiling**: 262,144 tokens (256K).
|
||
- **KV Cache Footprint (`q8_0`)**: ~10.5 GB at full 256K context.
|
||
- **Total Footprint**: ~28.0 GB. Fits inside 30GB VRAM with ~2GB headroom.
|
||
|
||
---
|
||
|
||
## 3. KV Cache Quantization (`-ctk` and `-ctv`)
|
||
|
||
By default, llama.cpp stores key and value cache vectors in 16-bit floating point (`f16`).
|
||
For 64K–256K contexts, full `f16` KV caches require 20–40+ GB alone!
|
||
|
||
Using **`-ctk q8_0 -ctv q8_0`**:
|
||
- **50% Memory Reduction**: Halves KV cache memory requirements.
|
||
- **Near-Lossless Perplexity**: Q8_0 maintains >99.8% precision with no degradation in reasoning benchmarks.
|
||
- **Native ROCm Acceleration**: Supported with Flash Attention (`-fa on`).
|