Unofficial guide
Inkling 2 Mini GGUF
Last updated
Inkling 2 Mini GGUF files are quantized copies for llama.cpp and Unsloth. They are not the 20 GB NVFP4 checkpoint. Unsloth publishes the size, the quality delta, and a llama.cpp build that can read the loop. The Hugging Face release post publishes a second, smaller set under ggml-org and a different command, llama serve.
This page is unofficial. Every command below is from one of those two guides. The memory numbers are Unsloth's starting ranges, not a lab measurement of a named graphics card.
Two GGUF packs
| Pack | What it is for | Command the docs use |
|---|---|---|
| unsloth/Inkling-2-Mini-GGUF | UD quants from UD-IQ1_S through UD-Q8_K_XL, plus mmproj and a DSpark drafter | Unsloth's llama-server from release b12588-mix-c0c3de8, or unsloth run |
| ggml-org/Inkling-2-Mini-GGUF | IQ4_NL, Q8_0, and BF16, plus BF16 and Q8_0 projectors | llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL |
Unsloth is explicit about layout. Root GGUFs use their looped-layer layout. Upstream llama.cpp reads a different one. If you want mainline llama.cpp, they say to use the Mainline_Support files and follow that README. Mixing a root Unsloth GGUF with an old upstream binary is the failure mode behind most "it will not load" reports. The fixes are on the troubleshooting page.
Which quant
Unsloth calls UD-Q8_K_XL full precision and recommends UD-Q4_K_XL. Their comparison keeps the original NVFP4 expert weights and stores the remaining weights in BF16 for the reference. They also say UD-Q4_K_XL is nearly lossless when MXFP8 weights are dequantized to Q8_0, and that UD-Q8_K_XL upcasts MXFP8 to BF16 so KL divergence against itself is zero.
| Quant | Size (GB) | Mean KL vs UD-Q8_K_XL | Same top token |
|---|---|---|---|
| UD-IQ1_S | 7.76 | 0.0911 | 89.4% |
| UD-IQ1_M | 8.02 | 0.0867 | 90.2% |
| UD-IQ2_XXS | 8.49 | 0.0726 | 90.6% |
| UD-Q2_K_XL | 9.14 | 0.0595 | 91.6% |
| UD-IQ3_XXS | 11.66 | 0.0269 | 94.7% |
| UD-Q3_K_XL | 13.63 | 0.0174 | 96.3% |
| UD-IQ4_XS | 15.73 | 0.0109 | 97.7% |
| UD-Q4_K_M | 16.61 | 0.0108 | 97.9% |
| UD-Q4_K_XL | 17.63 | 0.0107 | 98.0% |
| UD-Q8_K_XL | 22.87 | reference | reference |
UD-Q4_K_M is close to the recommended XL file (16.61 GB, 97.9% agreement). The 2-bit UD-IQ2_XXS file is 8.49 GB at 90.6% agreement. Those gaps are Unsloth's llama.cpp comparison under identical settings. They are not SWE-bench scores.
Memory table
Unsloth titles this "RAM + VRAM, or unified memory" and says to treat the ranges as a starting point. The mmproj-F16.gguf projector adds 639 MB on top of the quant. The drafter in their server command is a separate 0.4 GB file.
| 1-bit | 2-bit | 3-bit | 4-bit | 8-bit (they say lossless) |
|---|---|---|---|---|
| 9 GB | 10–11 GB | 13–15 GB | 17–18 GB | 24 GB |
The same page also says you can run on 16 GB of RAM, or 24 GB of RAM for full precision. Read that next to the table, not instead of it: 24 GB matches their 8-bit column, and 16 GB sits between the 3-bit and 4-bit columns. A 4-bit load at 17–18 GB does not fit in 16 GB with room for context.
The official checkpoint is a different row. Thinking Machines says about 20 GB on disk and recommends 32 GB or more of VRAM on GPUs other than the machines they name (Apple silicon at 32 GB or more, and one DGX Spark). A GGUF that starts in the teens is how people get under that line, with the quality cost in the table above.
llama.cpp: install a build that knows the loop
Run these from the folder where you will download the GGUF. Each unpacks llama-server into a llama.cpp directory. The Apple archive is the one Unsloth unpacks with --strip-components=1. The NVIDIA, AMD, and DGX Spark commands do not use that flag. If the binary is not at ./llama.cpp/llama-server, list the folder before you edit the command.
curl -L -o llama.tar.gz https://github.com/unslothai/llama.cpp/releases/download/b12588-mix-c0c3de8/llama-b12588-mix-c0c3de8-bin-macos-arm64.tar.gz
mkdir -p llama.cpp
tar -xzf llama.tar.gz -C llama.cpp --strip-components=1
./llama.cpp/llama-server --version curl -L -o llama.tar.gz https://github.com/unslothai/llama.cpp/releases/download/b12588-mix-c0c3de8/app-b12588-mix-c0c3de8-linux-x64-cuda12-portable.tar.gz
mkdir -p llama.cpp
tar -xzf llama.tar.gz -C llama.cpp
./llama.cpp/llama-server --version
Unsloth says the CUDA 12 build covers RTX 20 through RTX 50, V100, T4, A100, H100, and B200, and that the CUDA toolkit has to be installed first. For CUDA 13 they say swap cuda12-portable for cuda13-portable. For GTX 10 series and older, use cuda12-legacy.
curl -L -o llama.tar.gz https://github.com/unslothai/llama.cpp/releases/download/b12588-mix-c0c3de8/app-b12588-mix-c0c3de8-linux-x64-rocm-gfx120X.tar.gz
mkdir -p llama.cpp
tar -xzf llama.tar.gz -C llama.cpp
./llama.cpp/llama-server --version
That ROCm build targets gfx120X and bundles the libraries, so they say a full ROCm install is not required. For other GPUs, replace gfx120X with gfx1151 (Ryzen AI Max), gfx1150 (Ryzen AI 300), gfx110X (RX 7000), gfx103X (RX 6000), gfx90a (MI200), or gfx908 (MI100). Anything else, they say, should use the Vulkan build.
curl.exe -L -o llama.zip https://github.com/unslothai/llama.cpp/releases/download/b12588-mix-c0c3de8/app-b12588-mix-c0c3de8-windows-x64-cuda12-portable.zip
Expand-Archive llama.zip -DestinationPath llama.cpp
.\llama.cpp\llama-server.exe --version curl -L -o llama.tar.gz https://github.com/unslothai/llama.cpp/releases/download/b12588-mix-c0c3de8/app-b12588-mix-c0c3de8-linux-arm64-cuda13-portable.tar.gz
mkdir -p llama.cpp
tar -xzf llama.tar.gz -C llama.cpp
./llama.cpp/llama-server --version
Intel Mac, CPU-only, Vulkan, and Linux ARM64 builds are on that same release, per Unsloth. Keep llama-server next to the libraries it shipped with.
Download the GGUF and start the server
pip install -U huggingface_hub
hf download unsloth/Inkling-2-Mini-GGUF \
Inkling-2-Mini-UD-Q4_K_XL.gguf mmproj-F16.gguf dspark-Inkling-2-Mini-Q8_0.gguf \
--local-dir Inkling-2-Mini-GGUF ./llama.cpp/llama-server \
--model Inkling-2-Mini-GGUF/Inkling-2-Mini-UD-Q4_K_XL.gguf \
--mmproj Inkling-2-Mini-GGUF/mmproj-F16.gguf \
--model-draft Inkling-2-Mini-GGUF/dspark-Inkling-2-Mini-Q8_0.gguf \
--spec-type draft-dspark --spec-draft-n-max 3 --n-gpu-layers-draft 99 \
--jinja --ctx-size 32768 --n-gpu-layers 99 --parallel 1 \
--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \
--host 127.0.0.1 --port 8080 .\llama.cpp\llama-server.exe `
--model Inkling-2-Mini-GGUF/Inkling-2-Mini-UD-Q4_K_XL.gguf `
--mmproj Inkling-2-Mini-GGUF/mmproj-F16.gguf `
--model-draft Inkling-2-Mini-GGUF/dspark-Inkling-2-Mini-Q8_0.gguf `
--spec-type draft-dspark --spec-draft-n-max 3 --n-gpu-layers-draft 99 `
--jinja --ctx-size 32768 --n-gpu-layers 99 --parallel 1 `
--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 `
--host 127.0.0.1 --port 8080
Open http://127.0.0.1:8080. Unsloth says --model-draft and --spec-type draft-dspark turn on speculative decoding with the drafter, and that --spec-draft-n-max 3 is a good default. To run without it, omit the drafter from the download and delete the --model-draft and --spec-type lines. For text only, omit --mmproj. For CPU only, set --n-gpu-layers 0 and --n-gpu-layers-draft 0.
Sampler settings they publish, and use in that command: temperature 1.0, top-p 1.0, top-k 0, min-p 0.0. --jinja selects the chat template embedded in the GGUF. The same samplers are what unsloth run sets:
unsloth run --model unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_XL \
--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0
Change the suffix after the colon to pick another quant in the repo. Unsloth's install options for the desktop app are the shell script at https://unsloth.ai/install.sh, brew install --cask unsloth, and winget install --id UnslothAI.Unsloth -e.
The other CLI: llama serve
The Hugging Face post is aimed at Llama App and at people who want the llama CLI rather than Unsloth's llama-server binary. For the fixed-loop model:
llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL
Swap the suffix to :Q8_0 for the 8-bit file. The variable-loop research weights, if you mean to study passes rather than install the release model, are llama serve -hf ggml-org/TML-d24-varloop-research-GGUF:IQ4_NL in that post. Do not use that repo when you wanted Inkling 2 Mini itself.
llama serve and llama-server are different programs. Flags from one command do not automatically exist on the other.
When the GGUF is the wrong download
If you have a 32 GB Apple silicon Mac or a DGX Spark and you want the weights the lab trained, the model card's checkpoint is the Hugging Face repo thinkingmachines/Inkling-2-Mini, and on a Mac the Ollama tag is the NVFP4 build. GGUF is the route for less memory, for llama.cpp, and for machines the card does not list by name. The trade is the KL column in the table, which is small at Q4 and obvious at 1-bit and 2-bit.
Common questions
Which Inkling 2 Mini GGUF should I download?
Unsloth recommends UD-Q4_K_XL (17.63 GB, 0.0107 mean KL divergence against their UD-Q8_K_XL reference, 98.0% top-token agreement). They call UD-Q8_K_XL full precision. The Hugging Face post's llama serve example uses ggml-org's IQ4_NL, or Q8_0 if you change the suffix. Those are different files from different repositories.
Which llama.cpp build loads Inkling 2 Mini?
Unsloth says the GGUFs in the repository root need their b12588-mix-c0c3de8 build, because that build understands the looped-layer layout and per-expert NVFP4 scales. Upstream llama.cpp uses a different layout. For mainline llama.cpp they say to use the Mainline_Support files and that folder's README.
How do I enable vision in llama.cpp?
Unsloth's server command passes --mmproj pointing at mmproj-F16.gguf. They say that projector adds 639 MB. For text-only chat, leave the projector out of the download and drop --mmproj. The model card still wants image sides between 40 and 4096 pixels.
Sources
- Unsloth, Inkling 2 Mini run guide GGUF sizes, memory ranges, sampler settings, and llama.cpp commands.
- unsloth/Inkling-2-Mini-GGUF
- ggml-org/Inkling-2-Mini-GGUF IQ4_NL, Q8_0, and BF16 GGUFs used by the Hugging Face llama serve example.
- Hugging Face blog, “Welcome Inkling 2 Mini” Published 20 November 2026. Transformers, llama serve, SGLang, and MLX commands.
- Inkling 2 Mini model card Hardware, license, architecture, evaluations, and limitations.
- Inkling 2 Mini config.json Hidden size, layer count, loop bounds, experts, and context length.