noelbautista91/llms
1
0
Fork 0
Code Activity Actions
llms/README.md

3.5 KiB
Raw Blame History

Local LLM Serving & Multi-Agent Inference

High-performance inference orchestration using llama-server on an AMD ROCm GPU (~28–30GB usable VRAM).

llms/
├── bin/
│   ├── llm-runner       # Unified CLI orchestrator (cluster, muse, superqwen, qwen4b, status, stop)
│   └── llm-tui          # Interactive Textual TUI configurator form (run via uv)
├── templates/           # Jinja chat templates for Qwen and Muse-Glimmer
├── models/              # Quantized GGUF weights and vision encoders (ignored by git)
├── docs/                # Comprehensive documentation
│   ├── architecture.md
│   ├── vram-budgeting.md
│   └── parameter-reference.md
├── run.sh               # Backwards-compatible wrapper -> bin/llm-runner cluster
├── run_muse.sh          # Backwards-compatible wrapper -> bin/llm-runner muse
├── run_superqwen.sh     # Backwards-compatible wrapper -> bin/llm-runner superqwen
└── shell.nix            # Development environment shell

Quickstart

1. Launching via Interactive TUI Form

Launch the full interactive TUI form to visually pick presets or customize parameters with real-time VRAM estimates:

uv run bin/llm-tui
# Or directly if executable:
./bin/llm-tui

Inside the TUI:

  • Tied Configurations: GGUF model weights, vision multimodal projectors (mmproj), chat templates, and inference presets are automatically bound bi-directionally. Selecting an inference preset auto-selects its matching GGUF model and mmproj; selecting a GGUF model auto-selects its paired mmproj and appropriate preset tuning.
  • F5 / Launch: Starts the server(s) in the background with selected weights, projector, and parameters.
  • F8 / Stop: Stops server on the selected port.
  • F9 / Stop All: Kills all running llama-server instances.
  • F2 / Refresh: Refreshes health status.
  • Real-time VRAM budget calculation alerts you if selected context sizes risk VRAM overflow.

2. Launching via CLI Runner

# Multi-Agent Cluster (SuperQwen 27B + 2x Qwen 4B on ports 11434-11436)
./bin/llm-runner cluster

# Muse-Glimmer 30B Solo (128K Context on port 11434)
./bin/llm-runner muse

# SuperQwen 3.8 27B Solo (256K Context on port 11434)
./bin/llm-runner superqwen

# Qwen 3.5 4B Solo (port 11434)
./bin/llm-runner qwen4b

# Custom Model Instance (with custom GGUF and vision projector)
MODEL=models/muse-glimmer/Muse-Glimmer-30B-UD-Q4_K_XL.gguf MMPROJ=models/muse-glimmer/mmproj-Muse-Glimmer-30B-Q8_0.gguf ./bin/llm-runner custom

# Check status of running servers
./bin/llm-runner status

# Stop all running instances
./bin/llm-runner stop all

Server Endpoints & Ports

Port Role Default Model Context Window
11434 Planner / Solo SuperQwen 3.8 27B or Muse-Glimmer 30B 64K (cluster) / 128K (muse) / 256K (superqwen)
11435 Worker #1 Qwen 3.5 4B 16K
11436 Worker #2 Qwen 3.5 4B 16K

Each instance exposes standard OpenAI-compatible API routes:

  • Chat Completions: http://localhost:<port>/v1/chat/completions
  • Models: http://localhost:<port>/v1/models
  • Health Check: http://localhost:<port>/health

Detailed Documentation