VRAM Budgeting & Hardware Tuning (AMD ROCm)

Target Platform: AMD ROCm (~28 – 30GB Usable VRAM)

Serving large reasoning models with long context windows requires careful VRAM accounting to prevent ROCm memory allocation faults or context truncation.

1. Multi-Agent Cluster Footprint

Running all 3 models concurrently on a single ~30GB GPU:

Component	Model / Vision Weights	Weights in VRAM	Context Size	KV Cache (`q8_0`)	Total VRAM
Planner (SuperQwen 27B)	`Qwen3.8-27B-UD-Q4_K_XL` + `mmproj-Q8_0`	~18.1 GB	65,536 (64K)	~2.6 GB	~20.7 GB
Worker #1 (Qwen 4B)	`Qwen3.5-4B-Q5_K_M` + `mmproj-BF16`	~3.9 GB	16,384 (16K)	~0.4 GB	~4.3 GB
Worker #2 (Qwen 4B)	`Qwen3.5-4B-Q5_K_M` + `mmproj-BF16`	~3.9 GB	16,384 (16K)	~0.4 GB	~4.3 GB
Total		~25.9 GB		~3.4 GB	~29.3 GB

Note

Headroom is approximately ~0.7 – 1.0 GB. Keep system desktop composition (Niri / Wayland) overhead low.

2. Solo Model Memory Footprints

Muse-Glimmer 30B Solo

Weights Footprint: ~17.5 GB (UD-Q4_K_XL + mmproj-Q8_0).
Context Ceiling: 131,072 tokens (128K).
KV Cache Footprint (q8_0): ~8.5 GB at full 128K context.
Total Footprint: ~26.0 GB. Comfortably fits inside 30GB VRAM with ~4GB headroom.

SuperQwen 3.8 27B Solo

Weights Footprint: ~17.5 GB (SuperQwen3.8-27b-abliterated-Q4_K_M + mmproj-Q8_0).
Context Ceiling: 262,144 tokens (256K).
KV Cache Footprint (q8_0): ~10.5 GB at full 256K context.
Total Footprint: ~28.0 GB. Fits inside 30GB VRAM with ~2GB headroom.

3. KV Cache Quantization (`-ctk` and `-ctv`)

By default, llama.cpp stores key and value cache vectors in 16-bit floating point (f16). For 64K–256K contexts, full f16 KV caches require 20–40+ GB alone!

Using -ctk q8_0 -ctv q8_0:

50% Memory Reduction: Halves KV cache memory requirements.
Near-Lossless Perplexity: Q8_0 maintains >99.8% precision with no degradation in reasoning benchmarks.
Native ROCm Acceleration: Supported with Flash Attention (-fa on).

4. Sleep & Hibernation Protection

The Problem

On AMD ROCm architectures, entering system suspend (systemd-suspend), hybrid sleep, or hibernation with active ~28-30GB VRAM allocations and open /dev/kfd contexts causes amdgpu driver deadlocks, resulting in full system freezes during or immediately after sleep state transitions.

Declarative Systemd Sleep Hook

In dotfiles (modules/nixos/gpu.nix), a declarative systemd service (llama-server-sleep-stop.service) is bound to sleep.target:

Executes synchronously before sleep.target, systemd-suspend, and systemd-hibernate.
Sends SIGTERM to allow clean buffer detachment, followed by a hard SIGKILL if processes linger.
Flushes GPU VRAM and closes /dev/kfd handles before the kernel initiates D3 power state transition, preventing machine freezes.
Once resumed, servers can be relaunched instantly with F5 in llm-tui or via Rofi.

3.1 KiB Raw Blame History Unescape Escape