Unofficial guide
Inkling 2 Mini errors and troubleshooting
Last updated
Most Inkling 2 Mini errors come from three documented facts: local loop support is new, the native checkpoint wants about 20 GB plus headroom, and Unsloth's GGUF layout does not match every llama.cpp binary. The fixes below are the ones those docs already give. This page does not invent extra error strings.
The guide is unofficial. If a runtime's error text is not quoted in the source, the heading describes the situation instead of pretending to quote a dialog box.
The GGUF will not load
Unsloth writes that the files in the repository root use their looped-layer layout, and that you need build b12588-mix-c0c3de8, which supports that layout and per-expert NVFP4 scales. Upstream llama.cpp reads a different layout. For mainline llama.cpp they tell you to use the Mainline_Support files and that directory's README.
Check these in order:
- You unpacked the archive from the release id in their guide, not a generic llama.cpp you already had installed.
llama-serveris the binary from that unpack, and it is still sitting next to the libraries that came with it. They say to keep those together.- The GGUF is from
unsloth/Inkling-2-Mini-GGUFif you are using their server command. The Hugging Facellama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NLline is a different pack and a different CLI. Flags do not transfer. - You did not pass the vision projector as
--model. The weights andmmproj-F16.ggufare separate arguments.
The install commands are on the GGUF page. The Apple tarball is the one their guide extracts with --strip-components=1. The Linux tarballs are not. If ./llama.cpp/llama-server is missing, look inside the folder before you rewrite the command.
The process is killed, or the GPU reports it is out of memory
Match the file to the machine using the published ranges, then shrink the context. Do not start at 1 million tokens.
- Official checkpoint and the Ollama library build: about 20 GB. Supported floor is 32 GB of Apple silicon memory, or 32 GB of VRAM recommended on other GPUs. Ollama's page names a 32 GB Mac or a DGX Spark for that library build.
- Unsloth 4-bit: they associate 17–18 GB of RAM plus VRAM, or unified memory, with the 4-bit column. The recommended file is 17.63 GB before context.
- Their server flags, if memory is tight: reduce
--ctx-sizefrom 32768, reduce--n-gpu-layers, drop--mmproj, drop--model-draft. CPU only is--n-gpu-layers 0and--n-gpu-layers-draft 0.
A 16 GB machine is not in Thinking Machines' supported list for the native file. Unsloth's 3-bit range is 13–15 GB and their 2-bit range is 10–11 GB, as starting points, with worse KL divergence. The quality table is on the GGUF page. Leaving "a bit of VRAM free" is their warning about context and images, not a license to ignore the column.
The NVIDIA llama.cpp build will not start
Unsloth says to install the CUDA toolkit first. The default URL they print is the CUDA 12 portable build, for RTX 20 series through RTX 50 series, plus V100, T4, A100, H100, and B200. CUDA 13 means changing cuda12-portable to cuda13-portable in that URL. GTX 10 series and older means cuda12-legacy. A CUDA 13 driver with the CUDA 12 archive, or the reverse, is the sort of mismatch their three filenames exist to avoid. DGX Spark has its own ARM64 CUDA 13 archive on the GGUF page.
It loads and then crawls
The model card says local support for looped models is new and that speed varies by runtime and hardware. It also says compute per token is higher than the 3.4 billion active count, because the middle layers run three times. There is no official tokens-per-second number for an M-series chip in the sources this guide checked, so none is given here.
Things the docs do offer:
- Unsloth's DSpark flags (
--spec-type draft-dspark,--spec-draft-n-max 3) are their speed path for the GGUF server. The drafter is optional and about 0.4 GB. - The Hugging Face SGLang section uses
--speculative-algorithm DSPARKwhen draft weights are present. Their launch example also says to run on SM103. If SGLang refuses to start, that prerequisite is the first line of their section, not an afterthought. - The variable-loop research model can run fewer than three passes. The Hugging Face post says one or two passes are faster and may be lower quality. That is a different checkpoint,
TML-d24-varloop-research, not a flag on the release model. Mini's config setsis_varloopto false. - A 1-bit or 2-bit quant is smaller, not free speed, and Unsloth's top-token agreement falls to about 89–92% there.
It reasons in circles
This one is the model, not the loader. The model card says that in internal testing Mini sometimes stuck in long reasoning loops or kept re-checking its work without progress, particularly when debugging code. Their recommendation is to limit output tokens and agent turns. The announcement's coding scores were measured inside named harnesses (mini-swe-agent, pi, and others), which already impose a harness. A raw chat window does not.
Reasoning effort does not remove the loop. Levels in the Hugging Face post run from none at 0.0 through high at 0.9, the default, up to 0.99. The model card says the minimum still does not fully disable reasoning. If an agent is burning the context window on apologies and re-reads, stop the turn. Do not raise the context to 1 million to "give it room".
The same card says tool use is less reliable than on larger models: it may ignore project conventions, delete or overwrite files without asking, and misreport what it did. Sandbox the directory. That is a requirements-level warning, and it shows up as a "bug" the first time an agent cleans a repo.
Images do nothing, or they look wrong
- llama.cpp: add
--mmprojand the F16 projector from the same GGUF repo as the weights. Text-only instructions from Unsloth say to omit both. - MLX: the blog command passes
--imageand a prompt that includes control tokens. Dropping those tokens is a change this guide has not verified. The community 4-bit card sends images asimage_urlparts tomlx_lm.server. - SGLang:
--enable-multimodalis a separate switch in the Hugging Face post. - Pixel size: the model card says each side should be between 40 and 4096 pixels for best results. A screenshot outside that range is a documented reason for a weak read.
- Ollama marks the library model as image-capable and does not print an attach flag. Use the vision flow your Ollama build documents, on the 20 GB tag, on a machine they support.
The answers are much worse than the benchmark posts
Confirm the repository name. thinkingmachines/Inkling-2-Mini is the release model. TML-d24-varloop-research is the research model trained mostly at three passes and less at one, two, four, and five. The announcement says performance at the less-trained pass counts may understate a model trained for that depth.
Then confirm you did not expect Inkling 2 Small. The Mini card says the largest gap versus Small is factual recall. AA-Omniscience for Mini is -8.6, with a 63.4% non-hallucination rate, behind Qwen3.8-27B on both of those cells. Looping will not fill that in. For facts, the lab's own story is search and tools (BrowseComp 79.0% and DeepSearchQA 85.3% are the "with compaction" rows), not a closed-book quiz.
A 1-bit GGUF will also miss the blog posts. UD-IQ1_S is 7.76 GB with KL 0.0911 and 89.4% top-token agreement against Unsloth's 8-bit reference. That is a different model in practice. UD-Q4_K_XL is the file they recommend if you want the write-up people are quoting.
SGLang will not come up
The Hugging Face post says a prerequisite is to run the server on SM103, and it prints a python -m sglang.launch_server command with --attention-backend fa4. If you are not on that GPU generation, their example is not the command for your card. Use the GGUF path or the recipe linked from the model card for the stack you actually have. The command itself is on the download page.
It is clumsy in a long chat
The announcement says Mini is not the strongest small model for multi-turn chat or for instructions with many constraints, and gives MultiChallenge at 50.0%. They describe that result as behind Qwen3.8-27B. On ComplexConstraints v1.1 the limitations paragraph also says "behind", while the score table lists Mini at 33.4% and Qwen3.8-27B at 30.1%. If a multi-constraint instruction fails, you are in the area the lab already flagged. Split the instruction, or use a model that leads those rows, instead of raising the quant and hoping.
Languages other than English can be weaker. The card says Chinese testing showed clipped phrasing and mixed languages. Sensitive mental-health topics draw more caution than the situation calls for, in their words. Neither is a loader bug.
Common questions
Why will llama.cpp not load an Inkling 2 Mini GGUF?
Unsloth's root GGUFs use their looped-layer layout and need the b12588-mix-c0c3de8 build, which also understands per-expert NVFP4 scales. Upstream llama.cpp expects a different layout. Use that build for the root files, or switch to the Mainline_Support files if you are on mainline llama.cpp. Do not mix the two.
Inkling 2 Mini is killed or runs out of memory. What should I change?
The official checkpoint is about 20 GB and is recommended with 32 GB or more of VRAM, or 32 GB or more of Apple silicon memory. For GGUF, Unsloth says to lower --ctx-size and --n-gpu-layers, drop --mmproj if you do not need images (639 MB), and drop the 0.4 GB drafter. Their example context is 32768, not 1 million.
The model never stops thinking, or it repeats itself while debugging.
The model card says internal testing saw long reasoning loops and repeated self-checks, especially on code. They tell you to limit output tokens and agent turns. Setting reasoning effort to the minimum does not fully disable reasoning, because the middle layers still run three times.
It ignores images.
In Unsloth's llama.cpp command, vision requires --mmproj and mmproj-F16.gguf. Text-only runs omit both. The model card wants each image side between 40 and 4096 pixels. SGLang needs --enable-multimodal, which the Hugging Face post states separately from the base server command.
Sources
- Unsloth, Inkling 2 Mini run guide GGUF sizes, memory ranges, sampler settings, and llama.cpp commands.
- Inkling 2 Mini model card Hardware, license, architecture, evaluations, and limitations.
- Hugging Face blog, “Welcome Inkling 2 Mini” Published 20 November 2026. Transformers, llama serve, SGLang, and MLX commands.
- Ollama library page for inkling-2-mini
- Thinking Machines Lab, “Introducing Inkling 2 Mini” Announcement dated 20 November 2026.
- Inkling 2 Mini config.json Hidden size, layer count, loop bounds, experts, and context length.