75 lines
2.7 KiB
Markdown
75 lines
2.7 KiB
Markdown
# LLM Inference Architecture
|
|
|
|
This repository orchestrates high-throughput local inference using `llama-server` on an AMD ROCm GPU (~28-30GB usable VRAM).
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
subgraph Clients["Agents & Applications"]
|
|
PiAgent["Pi Coding Agent"]
|
|
HermesAgent["Hermes Agent"]
|
|
RofiUI["Rofi / Ghostty TUI"]
|
|
Curl["HTTP / OpenAI API Clients"]
|
|
end
|
|
|
|
subgraph Router["Ports & Endpoints"]
|
|
P11434["Port 11434 (Planner / Reasoning / Solo)"]
|
|
P11435["Port 11435 (Worker #1 / Instruct)"]
|
|
P11436["Port 11436 (Worker #2 / Instruct)"]
|
|
end
|
|
|
|
subgraph Models["Running Models"]
|
|
SuperQwen["SuperQwen 3.8 27B / Muse-Glimmer 30B<br/>(Vision mmproj + Reasoning Preservation)"]
|
|
Worker1["Qwen 3.5 4B #1<br/>(Vision mmproj + Instruct)"]
|
|
Worker2["Qwen 3.5 4B #2<br/>(Vision mmproj + Instruct)"]
|
|
end
|
|
|
|
PiAgent -->|"Planner (models.json)"| P11434
|
|
PiAgent -->|"Worker-1 (models.json)"| P11435
|
|
PiAgent -->|"Worker-2 (models.json)"| P11436
|
|
HermesAgent --> P11434
|
|
Curl --> P11434
|
|
RofiUI -->|"Spawns & Monitors"| Router
|
|
|
|
P11434 --> SuperQwen
|
|
P11435 --> Worker1
|
|
P11436 --> Worker2
|
|
```
|
|
|
|
## Multi-Agent Cluster Mode
|
|
|
|
When running in cluster mode (`bin/llm-runner cluster` or via `llm-tui`):
|
|
1. **Planner / Reasoning Agent (`11434`)**:
|
|
- Model: `SuperQwen3.8-27B` (or `Qwen3.8-27B-UD-Q4_K_XL.gguf`)
|
|
- Purpose: High-level architectural reasoning, step planning, code review.
|
|
- Context Window: 65,536 tokens.
|
|
- Temperature: `0.6`, Top-P: `0.95`, Top-K: `40`, Min-P: `0.05`.
|
|
- Multimodal projector enabled for screenshot/diagram analysis.
|
|
2. **Worker Agents #1 & #2 (`11435` and `11436`)**:
|
|
- Model: `Qwen 3.5 4B` (`Qwen3.5-4B-Q5_K_M.gguf`)
|
|
- Purpose: Fast parallel file edits, tool calls, unit test generation.
|
|
- Context Window: 16,384 tokens each.
|
|
- Temperature: `0.7`, Top-P: `0.8`, Top-K: `20`, Min-P: `0.05`.
|
|
- Multimodal projector enabled for visual grounding.
|
|
|
|
## Solo Mode
|
|
|
|
For single-model deep thinking tasks requiring maximal context windows:
|
|
- **Muse-Glimmer 30B Solo (`bin/llm-runner muse`)**:
|
|
- Bound to port `11434`.
|
|
- Max native training context: **131,072 tokens (128K)**.
|
|
- Multimodal vision encoder enabled.
|
|
- **SuperQwen 3.8 27B Solo (`bin/llm-runner superqwen`)**:
|
|
- Bound to port `11434`.
|
|
- Max native context: **262,144 tokens (256K)**.
|
|
|
|
## Integration with Coding Agents
|
|
|
|
### Pi Agent Configuration
|
|
`dotfiles/config/pi/models.json` maps directly to these endpoints:
|
|
- `planner`: `http://localhost:11434/v1`
|
|
- `worker-1`: `http://localhost:11435/v1`
|
|
- `worker-2`: `http://localhost:11436/v1`
|
|
All models expose standard OpenAI-compatible endpoints:
|
|
- Completions: `/v1/chat/completions`
|
|
- Models: `/v1/models`
|
|
- Health: `/health`
|