# Local LLM Serving & Multi-Agent Inference

High-performance inference orchestration using `llama-server` on an AMD ROCm GPU (~28–30GB usable VRAM).

```
llms/
├── bin/
│   ├── llm-runner       # Unified CLI orchestrator (cluster, muse, superqwen, qwen4b, status, stop)
│   └── llm-tui          # Interactive Textual TUI configurator form (run via uv)
├── templates/           # Jinja chat templates for Qwen and Muse-Glimmer
├── models/              # Quantized GGUF weights and vision encoders (ignored by git)
├── docs/                # Comprehensive documentation
│   ├── architecture.md
│   ├── vram-budgeting.md
│   └── parameter-reference.md
├── run.sh               # Backwards-compatible wrapper -> bin/llm-runner cluster
├── run_muse.sh          # Backwards-compatible wrapper -> bin/llm-runner muse
├── run_superqwen.sh     # Backwards-compatible wrapper -> bin/llm-runner superqwen
└── shell.nix            # Development environment shell
```

---

## Quickstart

### 1. Launching via Interactive TUI Form
Launch the full interactive TUI form to visually pick presets or customize parameters with real-time VRAM estimates:

```bash
uv run bin/llm-tui
# Or directly if executable:
./bin/llm-tui
```

Inside the TUI:
- **Tied Configurations**: GGUF model weights, vision multimodal projectors (`mmproj`), chat templates, and inference presets are automatically bound bi-directionally. Selecting an inference preset auto-selects its matching GGUF model and mmproj; selecting a GGUF model auto-selects its paired mmproj and appropriate preset tuning.
- `F5` / **Launch**: Starts the server(s) in the background with selected weights, projector, and parameters.
- `F8` / **Stop**: Stops server on the selected port.
- `F9` / **Stop All**: Kills all running `llama-server` instances.
- `F2` / **Refresh**: Refreshes health status.
- Real-time VRAM budget calculation alerts you if selected context sizes risk VRAM overflow.

### 2. Launching via CLI Runner
```bash
# Multi-Agent Cluster (SuperQwen 27B + 2x Qwen 4B on ports 11434-11436)
./bin/llm-runner cluster

# Muse-Glimmer 30B Solo (128K Context on port 11434)
./bin/llm-runner muse

# SuperQwen 3.8 27B Solo (256K Context on port 11434)
./bin/llm-runner superqwen

# Qwen 3.5 4B Solo (port 11434)
./bin/llm-runner qwen4b

# Custom Model Instance (with custom GGUF and vision projector)
MODEL=models/muse-glimmer/Muse-Glimmer-30B-UD-Q4_K_XL.gguf MMPROJ=models/muse-glimmer/mmproj-Muse-Glimmer-30B-Q8_0.gguf ./bin/llm-runner custom

# Check status of running servers
./bin/llm-runner status

# Stop all running instances
./bin/llm-runner stop all
```

---

## Server Endpoints & Ports

| Port | Role | Default Model | Context Window |
| :--- | :--- | :--- | :--- |
| **`11434`** | Planner / Solo | SuperQwen 3.8 27B or Muse-Glimmer 30B | 64K (cluster) / 128K (muse) / 256K (superqwen) |
| **`11435`** | Worker #1 | Qwen 3.5 4B | 16K |
| **`11436`** | Worker #2 | Qwen 3.5 4B | 16K |

Each instance exposes standard OpenAI-compatible API routes:
- **Chat Completions**: `http://localhost:<port>/v1/chat/completions`
- **Models**: `http://localhost:<port>/v1/models`
- **Health Check**: `http://localhost:<port>/health`

---

## Detailed Documentation
- [Architecture & Multi-Agent Orchestration](docs/architecture.md)
- [VRAM Budgeting & ROCm Memory Limits](docs/vram-budgeting.md)
- [Parameter Reference & Sampling Profiles](docs/parameter-reference.md)
- [Model Weights & Storage Guide](models/README.md)