Local LLM Serving & Multi-Agent Inference

High-performance inference orchestration using llama-server on an AMD ROCm GPU (~28–30GB usable VRAM).

llms/
├── bin/
│   ├── llm-runner       # Unified CLI orchestrator (cluster, muse, superqwen, qwen4b, status, stop)
│   └── llm-tui          # Interactive Textual TUI configurator form (run via uv)
├── templates/           # Jinja chat templates for Qwen and Muse-Glimmer
├── models/              # Quantized GGUF weights and vision encoders (ignored by git)
├── docs/                # Comprehensive documentation
│   ├── architecture.md
│   ├── vram-budgeting.md
│   └── parameter-reference.md
├── run.sh               # Backwards-compatible wrapper -> bin/llm-runner cluster
├── run_muse.sh          # Backwards-compatible wrapper -> bin/llm-runner muse
├── run_superqwen.sh     # Backwards-compatible wrapper -> bin/llm-runner superqwen
└── shell.nix            # Development environment shell

---

Quickstart

1. Launching via Interactive TUI Form

Launch the full interactive TUI form to visually pick presets or customize parameters with real-time VRAM estimates:

uv run bin/llm-tui
# Or directly if executable:
./bin/llm-tui

Inside the TUI:

• Tied Configurations: GGUF model weights, vision multimodal projectors (mmproj), chat templates, and inference presets are automatically bound bi-directionally. Selecting an inference preset auto-selects its matching GGUF model and mmproj; selecting a GGUF model auto-selects its paired mmproj and appropriate preset tuning.
• F5 / Launch: Starts the server(s) in the background with selected weights, projector, and parameters.
• F8 / Stop: Stops server on the selected port.
• F9 / Stop All: Kills all running llama-server instances.
• F2 / Refresh: Refreshes health status.
• Real-time VRAM budget calculation alerts you if selected context sizes risk VRAM overflow.

2. Launching via CLI Runner

# Multi-Agent Cluster (SuperQwen 27B + 2x Qwen 4B on ports 11434-11436)
./bin/llm-runner cluster

# Muse-Glimmer 30B Solo (128K Context on port 11434)
./bin/llm-runner muse

# SuperQwen 3.8 27B Solo (256K Context on port 11434)
./bin/llm-runner superqwen

# Qwen 3.5 4B Solo (port 11434)
./bin/llm-runner qwen4b

# Custom Model Instance (with custom GGUF and vision projector)
MODEL=models/muse-glimmer/Muse-Glimmer-30B-UD-Q4_K_XL.gguf MMPROJ=models/muse-glimmer/mmproj-Muse-Glimmer-30B-Q8_0.gguf ./bin/llm-runner custom

# Check status of running servers
./bin/llm-runner status

# Stop all running instances
./bin/llm-runner stop all

---

Server Endpoints & Ports

Port	Role	Default Model	Context Window
11434	Planner / Solo	SuperQwen 3.8 27B or Muse-Glimmer 30B	64K (cluster) / 128K (muse) / 256K (superqwen)
11435	Worker #1	Qwen 3.5 4B	16K
11436	Worker #2	Qwen 3.5 4B	16K

Each instance exposes standard OpenAI-compatible API routes:

• Chat Completions: http://localhost:<port>/v1/chat/completions
• Models: http://localhost:<port>/v1/models
• Health Check: http://localhost:<port>/health

---

Detailed Documentation

• Architecture & Multi-Agent Orchestration
• VRAM Budgeting & ROCm Memory Limits
• Parameter Reference & Sampling Profiles
• Model Weights & Storage Guide