33 lines
1.6 KiB
Markdown
33 lines
1.6 KiB
Markdown
# Model Storage Directory
|
|
|
|
This directory stores GGUF quantized model weights and vision encoders (`mmproj`).
|
|
Model binary weights (`*.gguf`) are excluded from Git version control via `.gitignore`.
|
|
|
|
## Expected Directory Layout
|
|
|
|
```
|
|
models/
|
|
├── Qwen3.5/
|
|
│ ├── Qwen3.5-4B-Q5_K_M.gguf # Instruct / Worker Agent
|
|
│ ├── mmproj-BF16.gguf # Multimodal Vision Projector
|
|
│ └── chat_template.jinja -> ../../templates/Qwen3.5/chat_template.jinja
|
|
├── Qwen3.8/
|
|
│ ├── Qwen3.8-27B-UD-Q4_K_XL.gguf # Reasoning / Planning Agent
|
|
│ ├── SuperQwen3.8-27b-abliterated-Q4_K_M.gguf # Alternative 4-bit quant
|
|
│ ├── mmproj-SuperQwen3.8-27b-abliterated-Q8_0.gguf
|
|
│ └── chat_template.jinja -> ../../templates/Qwen3.8/chat_template.jinja
|
|
└── muse-glimmer/
|
|
├── Muse-Glimmer-30B-UD-Q4_K_XL.gguf # Deep Reasoning Solo Model
|
|
├── mmproj-Muse-Glimmer-30B-Q8_0.gguf # Vision Projector
|
|
└── chat_template.jinja -> ../../templates/muse-glimmer/chat_template.jinja
|
|
```
|
|
|
|
## Recommended Quantizations & Sourcing
|
|
|
|
| Model Name | Quantization | Size | Purpose | Recommended Port |
|
|
| :--- | :--- | :--- | :--- | :--- |
|
|
| **SuperQwen 3.8 27B** | `UD-Q4_K_XL` or `Q4_K_M` | ~17-18 GB | Planner / Architect agent | `11434` |
|
|
| **Qwen 3.5 4B** | `Q5_K_M` | ~3.9 GB | Instruct / Implementation agent | `11435`, `11436` |
|
|
| **Muse-Glimmer 30B** | `UD-Q4_K_XL` | ~17 GB | Deep reasoning solo agent | `11434` |
|
|
|
|
All chat templates are tracked in `templates/` and referenced automatically by `bin/llm-runner`.
|