# VRAM Budgeting & Hardware Tuning (AMD ROCm)

## Target Platform: AMD ROCm (~28 – 30GB Usable VRAM)

Serving large reasoning models with long context windows requires careful VRAM accounting to prevent ROCm memory allocation faults or context truncation.

---

## 1. Multi-Agent Cluster Footprint

Running all 3 models concurrently on a single ~30GB GPU:

| Component | Model / Vision Weights | Weights in VRAM | Context Size | KV Cache (`q8_0`) | Total VRAM |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Planner (SuperQwen 27B)** | `Qwen3.8-27B-UD-Q4_K_XL` + `mmproj-Q8_0` | ~18.1 GB | 65,536 (64K) | ~2.6 GB | ~20.7 GB |
| **Worker #1 (Qwen 4B)** | `Qwen3.5-4B-Q5_K_M` + `mmproj-BF16` | ~3.9 GB | 16,384 (16K) | ~0.4 GB | ~4.3 GB |
| **Worker #2 (Qwen 4B)** | `Qwen3.5-4B-Q5_K_M` + `mmproj-BF16` | ~3.9 GB | 16,384 (16K) | ~0.4 GB | ~4.3 GB |
| **Total** | | **~25.9 GB** | | **~3.4 GB** | **~29.3 GB** |

> [!NOTE]
> Headroom is approximately ~0.7 – 1.0 GB. Keep system desktop composition (Niri / Wayland) overhead low.

---

## 2. Solo Model Memory Footprints

### Muse-Glimmer 30B Solo
- **Weights Footprint**: ~17.5 GB (`UD-Q4_K_XL` + `mmproj-Q8_0`).
- **Context Ceiling**: 131,072 tokens (128K).
- **KV Cache Footprint (`q8_0`)**: ~8.5 GB at full 128K context.
- **Total Footprint**: ~26.0 GB. Comfortably fits inside 30GB VRAM with ~4GB headroom.

### SuperQwen 3.8 27B Solo
- **Weights Footprint**: ~17.5 GB (`SuperQwen3.8-27b-abliterated-Q4_K_M` + `mmproj-Q8_0`).
- **Context Ceiling**: 262,144 tokens (256K).
- **KV Cache Footprint (`q8_0`)**: ~10.5 GB at full 256K context.
- **Total Footprint**: ~28.0 GB. Fits inside 30GB VRAM with ~2GB headroom.

---

## 3. KV Cache Quantization (`-ctk` and `-ctv`)

By default, llama.cpp stores key and value cache vectors in 16-bit floating point (`f16`).
For 64K–256K contexts, full `f16` KV caches require 20–40+ GB alone!

Using **`-ctk q8_0 -ctv q8_0`**:
- **50% Memory Reduction**: Halves KV cache memory requirements.
- **Near-Lossless Perplexity**: Q8_0 maintains >99.8% precision with no degradation in reasoning benchmarks.
- **Native ROCm Acceleration**: Supported with Flash Attention (`-fa on`).

---

## 4. Sleep & Hibernation Protection

### The Problem
On AMD ROCm architectures, entering system suspend (`systemd-suspend`), hybrid sleep, or hibernation with active ~28-30GB VRAM allocations and open `/dev/kfd` contexts causes `amdgpu` driver deadlocks, resulting in full system freezes during or immediately after sleep state transitions.

### Declarative Systemd Sleep Hook
In `dotfiles` ([`modules/nixos/gpu.nix`](file:///home/noelbautista91/Documents/Projects/dotfiles/modules/nixos/gpu.nix)), a declarative systemd service (`llama-server-sleep-stop.service`) is bound to `sleep.target`:
- Executes synchronously **before** `sleep.target`, `systemd-suspend`, and `systemd-hibernate`.
- Sends `SIGTERM` to allow clean buffer detachment, followed by a hard `SIGKILL` if processes linger.
- Flushes GPU VRAM and closes `/dev/kfd` handles before the kernel initiates D3 power state transition, preventing machine freezes.
- Once resumed, servers can be relaunched instantly with `F5` in `llm-tui` or via Rofi.