History

Noel Bautista f78402d8bf Initial commit: LLM inference orchestrator with TUI form and ROCm configs		2026-10-21 12:17:34 -07:00
..
muse-glimmer	Initial commit: LLM inference orchestrator with TUI form and ROCm configs	2026-10-21 12:17:34 -07:00
Qwen3.5	Initial commit: LLM inference orchestrator with TUI form and ROCm configs	2026-10-21 12:17:34 -07:00
Qwen3.8	Initial commit: LLM inference orchestrator with TUI form and ROCm configs	2026-10-21 12:17:34 -07:00
README.md	Initial commit: LLM inference orchestrator with TUI form and ROCm configs	2026-10-21 12:17:34 -07:00

README.md

Model Storage Directory

This directory stores GGUF quantized model weights and vision encoders (mmproj). Model binary weights (*.gguf) are excluded from Git version control via .gitignore.

Expected Directory Layout

models/
├── Qwen3.5/
│   ├── Qwen3.5-4B-Q5_K_M.gguf                     # Instruct / Worker Agent
│   ├── mmproj-BF16.gguf                          # Multimodal Vision Projector
│   └── chat_template.jinja -> ../../templates/Qwen3.5/chat_template.jinja
├── Qwen3.8/
│   ├── Qwen3.8-27B-UD-Q4_K_XL.gguf               # Reasoning / Planning Agent
│   ├── SuperQwen3.8-27b-abliterated-Q4_K_M.gguf  # Alternative 4-bit quant
│   ├── mmproj-SuperQwen3.8-27b-abliterated-Q8_0.gguf
│   └── chat_template.jinja -> ../../templates/Qwen3.8/chat_template.jinja
└── muse-glimmer/
    ├── Muse-Glimmer-30B-UD-Q4_K_XL.gguf          # Deep Reasoning Solo Model
    ├── mmproj-Muse-Glimmer-30B-Q8_0.gguf         # Vision Projector
    └── chat_template.jinja -> ../../templates/muse-glimmer/chat_template.jinja

Recommended Quantizations & Sourcing

Model Name	Quantization	Size	Purpose	Recommended Port
SuperQwen 3.8 27B	`UD-Q4_K_XL` or `Q4_K_M`	~17-18 GB	Planner / Architect agent	`11434`
Qwen 3.5 4B	`Q5_K_M`	~3.9 GB	Instruct / Implementation agent	`11435`, `11436`
Muse-Glimmer 30B	`UD-Q4_K_XL`	~17 GB	Deep reasoning solo agent	`11434`

All chat templates are tracked in templates/ and referenced automatically by bin/llm-runner.