--- base_model: bottlecapai/ThinkingCap-Qwen3.6-27B base_model_relation: quantized library_name: gguf tags: - qwen3_6 - gguf - llama.cpp - token-efficient - efficient-thinking pipeline_tag: image-text-to-text --- ThinkingCap — BottleCap AI # bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF GGUF / [llama.cpp](https://github.com/ggml-org/llama.cpp) quantizations of [bottlecapai/ThinkingCap-Qwen3.6-27B](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B) — capability of Qwen3.6-27B with **50% less** thinking tokens on average, achieved by finetuning [Qwen3.6-27B (Qwen Team, 2026)](https://huggingface.co/Qwen/Qwen3.6-27B) with online reinforcement learning while preserving the original answer quality and style. ➡️ Full model description, evaluation results (multi-seed, statistically tested), recommended sampling params, and citation: see the main model card at [bottlecapai/ThinkingCap-Qwen3.6-27B](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B). ## About GGUF and quantization [GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) is a single-file model format for running LLMs locally with llama.cpp and compatible runtimes (Ollama, LM Studio, …). The quantized variants below store weights at reduced precision — e.g. ≈4.7 bits per weight for `Q4_K_M` instead of the 16-bit `f16` source — cutting download size and memory severalfold at a small, measured quality cost. ## Files | File | Quant | Size | |---|---|---| | `ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf` | Q4_K_M | 15.7 GB | | `ThinkingCap-Qwen3.6-27B-Q6_K.gguf` | Q6_K | 20.9 GB | | `ThinkingCap-Qwen3.6-27B-Q8_0.gguf` | Q8_0 | 27.1 GB | | `ThinkingCap-Qwen3.6-27B-f16.gguf` | f16 | 50.9 GB | | `mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf` | mmproj (vision) | 0.9 GB | `f16` is the unquantized source; `Q8_0` is near-lossless; `Q6_K` is a slightly smaller near-lossless option; `Q4_K_M` is the recommended size/quality balance for most local setups. ## Usage (llama.cpp) ```bash # pull a specific quant straight from the Hub and chat llama-cli -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M -p "Hi" # or download one file and run it huggingface-cli download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf --local-dir . llama-cli -m ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf -p "Hi" ``` ### Speculative decoding (MTP) llama.cpp can run MTP (multi-token-prediction) self-speculative decoding on these GGUFs for a decode speed-up — no separate draft model needed. Add `--spec-type draft-mtp` when serving: ```bash llama-server -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M --spec-type draft-mtp ``` Set the draft length with `--spec-draft-n-max` (e.g. `4`). Requires a recent llama.cpp build with MTP support. ## Vision (image input) ThinkingCap is a **vision-language model**. Image input needs the multimodal projector `mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf` (in this repo) loaded alongside a text GGUF — the single `f16` mmproj pairs with any of the quants above. - **LM Studio / Jan / Ollama, …:** download the `mmproj-*.gguf` from this repo; LM Studio auto-detects it and enables the image (🖼️) button. - **llama.cpp CLI:** ```bash huggingface-cli download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF \ ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf --local-dir . llama-mtmd-cli -m ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf \ --mmproj mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf --image photo.jpg -p "Describe this image." ``` - **llama-server:** add `--mmproj mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf` to expose an OpenAI-compatible vision endpoint. ## Expected performance From our internal serving-validation harness (llama.cpp, single-stream, temperature 0) on a fast **N=100/dataset** subset of MMLU-Pro (reasoning) and RealWorldQA (vision) — a quick **quant-parity + decode-speed** check, *not* the headline accuracy evals (for the multi-seed, statistically-tested results see the [main model card](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B)). Our four quants (`f16`/`Q8_0`/`Q6_K`/`Q4_K_M`) stay within subset noise of `f16` on accuracy, and MTP self-speculative decoding (`--spec-type draft-mtp`, `n=4`) accepts ≈3.75 tokens per verify step — a ≈1.4–1.7× per-token decode speed-up on top of the finetune's ≈50% token savings. The two **bolded** rows are our picks: `Q8_0` + MTP is the **fastest per task** on our hardware (it out-decodes `Q4_K_M` here) and near-lossless; `Q4_K_M` + MTP is the smaller **size/quality balance** for tighter memory budgets. `Q6_K` sits between them on size and is near-lossless like `Q8_0`, but note it does *not* decode faster than `Q8_0` here — `Q8_0` has llama.cpp's simplest, fastest dequant kernel — so pick `Q6_K` only when its ~6 GB size saving over `Q8_0` matters. For reference we also list **[unsloth](https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF)'s Dynamic GGUFs of the *base* model** (`UD-*`): same llama.cpp path, but base-model quants — so they match base accuracy and reason ≈2× longer (none of the finetune's token savings). `median tokens` = median completion length; `task s` = median tokens ÷ single-stream tok/s (real per-request time); `speedup` is vs the unquantized base model (bf16 GGUF) in standard decoding — same llama.cpp path as every row, so the comparison is apples-to-apples. **MMLU-Pro (reasoning)** | config | acc | median tokens | tok/s | task s | speedup | accept_len (n=4) | |---|---|---|---|---|---|---| | Qwen3.6-27B base (bf16 GGUF) · standard | 0.83 | 1999 | 50.4 | 39.6 | 1.00× | — | | f16 · standard | 0.89 | 884 | 50.4 | 17.5 | 2.26× | — | | f16 · MTP | 0.88 | 870 | 86.7 | 10.0 | 3.96× | 3.78 | | Q8_0 · standard | 0.88 | 890 | 57.2 | 15.6 | 2.54× | — | | **Q8_0 · MTP** | **0.86** | **856** | **99.4** | **8.6** | **4.60×** | **3.77** | | Q6_K · standard | 0.86 | 865 | 50.0 | 17.3 | 2.29× | — | | Q6_K · MTP | 0.85 | 896 | 99.8 | 9.0 | 4.40× | 3.80 | | Q4_K_M · standard | 0.86 | 814 | 61.8 | 13.2 | 3.00× | — | | **Q4_K_M · MTP** | **0.85** | **848** | **89.2** | **9.5** | **4.17×** | **3.74** | | unsloth UD-Q8_K_XL (base) · standard | 0.85 | 1896 | 54.5 | 34.8 | 1.14× | — | | unsloth UD-Q8_K_XL (base) · MTP | 0.86 | 1925 | 98.2 | 19.6 | 2.02× | 3.74 | | unsloth UD-Q4_K_XL (base) · standard | 0.84 | 1976 | 62.1 | 31.8 | 1.25× | — | | unsloth UD-Q4_K_XL (base) · MTP | 0.83 | 1928 | 87.1 | 22.1 | 1.79× | 3.72 | **RealWorldQA (vision)** | config | acc | median tokens | tok/s | task s | speedup | accept_len (n=4) | |---|---|---|---|---|---|---| | Qwen3.6-27B base (bf16 GGUF) · standard | 0.66 | 612 | 50.4 | 12.1 | 1.00× | — | | f16 · standard | 0.79 | 271 | 50.4 | 5.4 | 2.24× | — | | f16 · MTP | 0.79 | 271 | 86.7 | 3.1 | 3.90× | 3.78 | | Q8_0 · standard | 0.79 | 270 | 57.2 | 4.7 | 2.57× | — | | **Q8_0 · MTP** | **0.78** | **273** | **99.4** | **2.7** | **4.48×** | **3.77** | | Q6_K · standard | 0.80 | 270 | 50.0 | 5.4 | 2.24× | — | | Q6_K · MTP | 0.79 | 267 | 99.8 | 2.7 | 4.48× | 3.80 | | Q4_K_M · standard | 0.78 | 283 | 61.8 | 4.6 | 2.63× | — | | **Q4_K_M · MTP** | **0.78** | **274** | **89.2** | **3.1** | **3.90×** | **3.74** | | unsloth UD-Q8_K_XL (base) · standard | 0.68 | 530 | 54.5 | 9.7 | 1.25× | — | | unsloth UD-Q8_K_XL (base) · MTP | 0.69 | 550 | 98.2 | 5.6 | 2.16× | 3.74 | | unsloth UD-Q4_K_XL (base) · standard | 0.65 | 655 | 62.1 | 10.5 | 1.15× | — | | unsloth UD-Q4_K_XL (base) · MTP | 0.70 | 564 | 87.1 | 6.5 | 1.86× | 3.72 |