# LLM Inference Architecture

This repository orchestrates high-throughput local inference using `llama-server` on an AMD ROCm GPU (~28-30GB usable VRAM).

```mermaid
flowchart TD
    subgraph Clients["Agents & Applications"]
        PiAgent["Pi Coding Agent"]
        HermesAgent["Hermes Agent"]
        RofiUI["Rofi / Ghostty TUI"]
        Curl["HTTP / OpenAI API Clients"]
    end

    subgraph Router["Ports & Endpoints"]
        P11434["Port 11434 (Planner / Reasoning / Solo)"]
        P11435["Port 11435 (Worker #1 / Instruct)"]
        P11436["Port 11436 (Worker #2 / Instruct)"]
    end

    subgraph Models["Running Models"]
        SuperQwen["SuperQwen 3.8 27B / Muse-Glimmer 30B<br/>(Vision mmproj + Reasoning Preservation)"]
        Worker1["Qwen 3.5 4B #1<br/>(Vision mmproj + Instruct)"]
        Worker2["Qwen 3.5 4B #2<br/>(Vision mmproj + Instruct)"]
    end

    PiAgent -->|"Planner (models.json)"| P11434
    PiAgent -->|"Worker-1 (models.json)"| P11435
    PiAgent -->|"Worker-2 (models.json)"| P11436
    HermesAgent --> P11434
    Curl --> P11434
    RofiUI -->|"Spawns & Monitors"| Router

    P11434 --> SuperQwen
    P11435 --> Worker1
    P11436 --> Worker2
```

## Multi-Agent Cluster Mode

When running in cluster mode (`bin/llm-runner cluster` or via `llm-tui`):
1. **Planner / Reasoning Agent (`11434`)**:
   - Model: `SuperQwen3.8-27B` (or `Qwen3.8-27B-UD-Q4_K_XL.gguf`)
   - Purpose: High-level architectural reasoning, step planning, code review.
   - Context Window: 65,536 tokens.
   - Temperature: `0.6`, Top-P: `0.95`, Top-K: `40`, Min-P: `0.05`.
   - Multimodal projector enabled for screenshot/diagram analysis.
2. **Worker Agents #1 & #2 (`11435` and `11436`)**:
   - Model: `Qwen 3.5 4B` (`Qwen3.5-4B-Q5_K_M.gguf`)
   - Purpose: Fast parallel file edits, tool calls, unit test generation.
   - Context Window: 16,384 tokens each.
   - Temperature: `0.7`, Top-P: `0.8`, Top-K: `20`, Min-P: `0.05`.
   - Multimodal projector enabled for visual grounding.

## Solo Mode

For single-model deep thinking tasks requiring maximal context windows:
- **Muse-Glimmer 30B Solo (`bin/llm-runner muse`)**:
  - Bound to port `11434`.
  - Max native training context: **131,072 tokens (128K)**.
  - Multimodal vision encoder enabled.
- **SuperQwen 3.8 27B Solo (`bin/llm-runner superqwen`)**:
  - Bound to port `11434`.
  - Max native context: **262,144 tokens (256K)**.

## Integration with Coding Agents

### Pi Agent Configuration
`dotfiles/config/pi/models.json` maps directly to these endpoints:
- `planner`: `http://localhost:11434/v1`
- `worker-1`: `http://localhost:11435/v1`
- `worker-2`: `http://localhost:11436/v1`
All models expose standard OpenAI-compatible endpoints:
- Completions: `/v1/chat/completions`
- Models: `/v1/models`
- Health: `/health`