noelbautista91/llms
1
0
Fork 0
Code Activity Actions
llms/README.md

2.8 KiB
Raw Blame History

Local LLM Serving & Multi-Agent Inference

High-performance inference orchestration using llama-server on an AMD ROCm GPU (~28–30GB usable VRAM).

llms/
├── bin/
│   ├── llm-runner       # Unified CLI orchestrator (cluster, muse, superqwen, qwen4b, status, stop)
│   └── llm-tui          # Interactive Textual TUI configurator form (run via uv)
├── templates/           # Jinja chat templates for Qwen and Muse-Glimmer
├── models/              # Quantized GGUF weights and vision encoders (ignored by git)
├── docs/                # Comprehensive documentation
│   ├── architecture.md
│   ├── vram-budgeting.md
│   └── parameter-reference.md
├── run.sh               # Backwards-compatible wrapper -> bin/llm-runner cluster
├── run_muse.sh          # Backwards-compatible wrapper -> bin/llm-runner muse
├── run_superqwen.sh     # Backwards-compatible wrapper -> bin/llm-runner superqwen
└── shell.nix            # Development environment shell

Quickstart

1. Launching via Interactive TUI Form

Launch the full interactive TUI form to visually pick presets or customize parameters with real-time VRAM estimates:

uv run bin/llm-tui
# Or directly if executable:
./bin/llm-tui

Inside the TUI:

  • F5 / Launch: Starts the server(s) in the background.
  • F8 / Stop: Stops server on the selected port.
  • F9 / Stop All: Kills all running llama-server instances.
  • F2 / Refresh: Refreshes health status.
  • Real-time VRAM budget calculation alerts you if selected context sizes risk VRAM overflow.

2. Launching via CLI Runner

# Multi-Agent Cluster (SuperQwen 27B + 2x Qwen 4B on ports 11434-11436)
./bin/llm-runner cluster

# Muse-Glimmer 30B Solo (128K Context on port 11434)
./bin/llm-runner muse

# SuperQwen 3.8 27B Solo (256K Context on port 11434)
./bin/llm-runner superqwen

# Check status of running servers
./bin/llm-runner status

# Stop all running instances
./bin/llm-runner stop all

Server Endpoints & Ports

Port Role Default Model Context Window
11434 Planner / Solo SuperQwen 3.8 27B or Muse-Glimmer 30B 64K (cluster) / 128K (muse) / 256K (superqwen)
11435 Worker #1 Qwen 3.5 4B 16K
11436 Worker #2 Qwen 3.5 4B 16K

Each instance exposes standard OpenAI-compatible API routes:

  • Chat Completions: http://localhost:<port>/v1/chat/completions
  • Models: http://localhost:<port>/v1/models
  • Health Check: http://localhost:<port>/health

Detailed Documentation