88 lines
3.5 KiB
Markdown
88 lines
3.5 KiB
Markdown
# Local LLM Serving & Multi-Agent Inference
|
||
|
||
High-performance inference orchestration using `llama-server` on an AMD ROCm GPU (~28–30GB usable VRAM).
|
||
|
||
```
|
||
llms/
|
||
├── bin/
|
||
│ ├── llm-runner # Unified CLI orchestrator (cluster, muse, superqwen, qwen4b, status, stop)
|
||
│ └── llm-tui # Interactive Textual TUI configurator form (run via uv)
|
||
├── templates/ # Jinja chat templates for Qwen and Muse-Glimmer
|
||
├── models/ # Quantized GGUF weights and vision encoders (ignored by git)
|
||
├── docs/ # Comprehensive documentation
|
||
│ ├── architecture.md
|
||
│ ├── vram-budgeting.md
|
||
│ └── parameter-reference.md
|
||
├── run.sh # Backwards-compatible wrapper -> bin/llm-runner cluster
|
||
├── run_muse.sh # Backwards-compatible wrapper -> bin/llm-runner muse
|
||
├── run_superqwen.sh # Backwards-compatible wrapper -> bin/llm-runner superqwen
|
||
└── shell.nix # Development environment shell
|
||
```
|
||
|
||
---
|
||
|
||
## Quickstart
|
||
|
||
### 1. Launching via Interactive TUI Form
|
||
Launch the full interactive TUI form to visually pick presets or customize parameters with real-time VRAM estimates:
|
||
|
||
```bash
|
||
uv run bin/llm-tui
|
||
# Or directly if executable:
|
||
./bin/llm-tui
|
||
```
|
||
|
||
Inside the TUI:
|
||
- **Tied Configurations**: GGUF model weights, vision multimodal projectors (`mmproj`), chat templates, and inference presets are automatically bound bi-directionally. Selecting an inference preset auto-selects its matching GGUF model and mmproj; selecting a GGUF model auto-selects its paired mmproj and appropriate preset tuning.
|
||
- `F5` / **Launch**: Starts the server(s) in the background with selected weights, projector, and parameters.
|
||
- `F8` / **Stop**: Stops server on the selected port.
|
||
- `F9` / **Stop All**: Kills all running `llama-server` instances.
|
||
- `F2` / **Refresh**: Refreshes health status.
|
||
- Real-time VRAM budget calculation alerts you if selected context sizes risk VRAM overflow.
|
||
|
||
### 2. Launching via CLI Runner
|
||
```bash
|
||
# Multi-Agent Cluster (SuperQwen 27B + 2x Qwen 4B on ports 11434-11436)
|
||
./bin/llm-runner cluster
|
||
|
||
# Muse-Glimmer 30B Solo (128K Context on port 11434)
|
||
./bin/llm-runner muse
|
||
|
||
# SuperQwen 3.8 27B Solo (256K Context on port 11434)
|
||
./bin/llm-runner superqwen
|
||
|
||
# Qwen 3.5 4B Solo (port 11434)
|
||
./bin/llm-runner qwen4b
|
||
|
||
# Custom Model Instance (with custom GGUF and vision projector)
|
||
MODEL=models/muse-glimmer/Muse-Glimmer-30B-UD-Q4_K_XL.gguf MMPROJ=models/muse-glimmer/mmproj-Muse-Glimmer-30B-Q8_0.gguf ./bin/llm-runner custom
|
||
|
||
# Check status of running servers
|
||
./bin/llm-runner status
|
||
|
||
# Stop all running instances
|
||
./bin/llm-runner stop all
|
||
```
|
||
|
||
---
|
||
|
||
## Server Endpoints & Ports
|
||
|
||
| Port | Role | Default Model | Context Window |
|
||
| :--- | :--- | :--- | :--- |
|
||
| **`11434`** | Planner / Solo | SuperQwen 3.8 27B or Muse-Glimmer 30B | 64K (cluster) / 128K (muse) / 256K (superqwen) |
|
||
| **`11435`** | Worker #1 | Qwen 3.5 4B | 16K |
|
||
| **`11436`** | Worker #2 | Qwen 3.5 4B | 16K |
|
||
|
||
Each instance exposes standard OpenAI-compatible API routes:
|
||
- **Chat Completions**: `http://localhost:<port>/v1/chat/completions`
|
||
- **Models**: `http://localhost:<port>/v1/models`
|
||
- **Health Check**: `http://localhost:<port>/health`
|
||
|
||
---
|
||
|
||
## Detailed Documentation
|
||
- [Architecture & Multi-Agent Orchestration](docs/architecture.md)
|
||
- [VRAM Budgeting & ROCm Memory Limits](docs/vram-budgeting.md)
|
||
- [Parameter Reference & Sampling Profiles](docs/parameter-reference.md)
|
||
- [Model Weights & Storage Guide](models/README.md)
|