llms/docs/architecture.md

LLM Inference Architecture

This repository orchestrates high-throughput local inference using llama-server on an AMD ROCm GPU (~28-30GB usable VRAM).

flowchart TD
    subgraph Clients["Agents & Applications"]
        PiAgent["Pi Coding Agent"]
        HermesAgent["Hermes Agent"]
        RofiUI["Rofi / Ghostty TUI"]
        Curl["HTTP / OpenAI API Clients"]
    end

    subgraph Router["Ports & Endpoints"]
        P11434["Port 11434 (Planner / Reasoning / Solo)"]
        P11435["Port 11435 (Worker #1 / Instruct)"]
        P11436["Port 11436 (Worker #2 / Instruct)"]
    end

    subgraph Models["Running Models"]
        SuperQwen["SuperQwen 3.8 27B / Muse-Glimmer 30B<br/>(Vision mmproj + Reasoning Preservation)"]
        Worker1["Qwen 3.5 4B #1<br/>(Vision mmproj + Instruct)"]
        Worker2["Qwen 3.5 4B #2<br/>(Vision mmproj + Instruct)"]
    end

    PiAgent -->|"Planner (models.json)"| P11434
    PiAgent -->|"Worker-1 (models.json)"| P11435
    PiAgent -->|"Worker-2 (models.json)"| P11436
    HermesAgent --> P11434
    Curl --> P11434
    RofiUI -->|"Spawns & Monitors"| Router

    P11434 --> SuperQwen
    P11435 --> Worker1
    P11436 --> Worker2

Multi-Agent Cluster Mode

When running in cluster mode (bin/llm-runner cluster or via llm-tui):

1. Planner / Reasoning Agent (11434):
   • Model: SuperQwen3.8-27B (or Qwen3.8-27B-UD-Q4_K_XL.gguf)
   • Purpose: High-level architectural reasoning, step planning, code review.
   • Context Window: 65,536 tokens.
   • Temperature: 0.6, Top-P: 0.95, Top-K: 40, Min-P: 0.05.
   • Multimodal projector enabled for screenshot/diagram analysis.
2. Worker Agents #1 & #2 (11435 and 11436):
   • Model: Qwen 3.5 4B (Qwen3.5-4B-Q5_K_M.gguf)
   • Purpose: Fast parallel file edits, tool calls, unit test generation.
   • Context Window: 16,384 tokens each.
   • Temperature: 0.7, Top-P: 0.8, Top-K: 20, Min-P: 0.05.
   • Multimodal projector enabled for visual grounding.

Solo Mode

For single-model deep thinking tasks requiring maximal context windows:

• Muse-Glimmer 30B Solo (bin/llm-runner muse):
  • Bound to port 11434.
  • Max native training context: 131,072 tokens (128K).
  • Multimodal vision encoder enabled.
• SuperQwen 3.8 27B Solo (bin/llm-runner superqwen):
  • Bound to port 11434.
  • Max native context: 262,144 tokens (256K).

Integration with Coding Agents

Pi Agent Configuration

dotfiles/config/pi/models.json maps directly to these endpoints:

• planner: http://localhost:11434/v1
• worker-1: http://localhost:11435/v1
• worker-2: http://localhost:11436/v1 All models expose standard OpenAI-compatible endpoints:
• Completions: /v1/chat/completions
• Models: /v1/models
• Health: /health