Field guide Inkling 2 Mini

Checked

Unofficial guide

Inkling 2 Mini errors and troubleshooting

Last updated

Most Inkling 2 Mini errors come from three documented facts: local loop support is new, the native checkpoint wants about 20 GB plus headroom, and Unsloth's GGUF layout does not match every llama.cpp binary. The fixes below are the ones those docs already give. This page does not invent extra error strings.

The guide is unofficial. If a runtime's error text is not quoted in the source, the heading describes the situation instead of pretending to quote a dialog box.

The GGUF will not load

Unsloth writes that the files in the repository root use their looped-layer layout, and that you need build b12588-mix-c0c3de8, which supports that layout and per-expert NVFP4 scales. Upstream llama.cpp reads a different layout. For mainline llama.cpp they tell you to use the Mainline_Support files and that directory's README.

Check these in order:

  1. You unpacked the archive from the release id in their guide, not a generic llama.cpp you already had installed.
  2. llama-server is the binary from that unpack, and it is still sitting next to the libraries that came with it. They say to keep those together.
  3. The GGUF is from unsloth/Inkling-2-Mini-GGUF if you are using their server command. The Hugging Face llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL line is a different pack and a different CLI. Flags do not transfer.
  4. You did not pass the vision projector as --model. The weights and mmproj-F16.gguf are separate arguments.

The install commands are on the GGUF page. The Apple tarball is the one their guide extracts with --strip-components=1. The Linux tarballs are not. If ./llama.cpp/llama-server is missing, look inside the folder before you rewrite the command.

The process is killed, or the GPU reports it is out of memory

Match the file to the machine using the published ranges, then shrink the context. Do not start at 1 million tokens.

A 16 GB machine is not in Thinking Machines' supported list for the native file. Unsloth's 3-bit range is 13–15 GB and their 2-bit range is 10–11 GB, as starting points, with worse KL divergence. The quality table is on the GGUF page. Leaving "a bit of VRAM free" is their warning about context and images, not a license to ignore the column.

The NVIDIA llama.cpp build will not start

Unsloth says to install the CUDA toolkit first. The default URL they print is the CUDA 12 portable build, for RTX 20 series through RTX 50 series, plus V100, T4, A100, H100, and B200. CUDA 13 means changing cuda12-portable to cuda13-portable in that URL. GTX 10 series and older means cuda12-legacy. A CUDA 13 driver with the CUDA 12 archive, or the reverse, is the sort of mismatch their three filenames exist to avoid. DGX Spark has its own ARM64 CUDA 13 archive on the GGUF page.

It loads and then crawls

The model card says local support for looped models is new and that speed varies by runtime and hardware. It also says compute per token is higher than the 3.4 billion active count, because the middle layers run three times. There is no official tokens-per-second number for an M-series chip in the sources this guide checked, so none is given here.

Things the docs do offer:

It reasons in circles

This one is the model, not the loader. The model card says that in internal testing Mini sometimes stuck in long reasoning loops or kept re-checking its work without progress, particularly when debugging code. Their recommendation is to limit output tokens and agent turns. The announcement's coding scores were measured inside named harnesses (mini-swe-agent, pi, and others), which already impose a harness. A raw chat window does not.

Reasoning effort does not remove the loop. Levels in the Hugging Face post run from none at 0.0 through high at 0.9, the default, up to 0.99. The model card says the minimum still does not fully disable reasoning. If an agent is burning the context window on apologies and re-reads, stop the turn. Do not raise the context to 1 million to "give it room".

The same card says tool use is less reliable than on larger models: it may ignore project conventions, delete or overwrite files without asking, and misreport what it did. Sandbox the directory. That is a requirements-level warning, and it shows up as a "bug" the first time an agent cleans a repo.

Images do nothing, or they look wrong

The answers are much worse than the benchmark posts

Confirm the repository name. thinkingmachines/Inkling-2-Mini is the release model. TML-d24-varloop-research is the research model trained mostly at three passes and less at one, two, four, and five. The announcement says performance at the less-trained pass counts may understate a model trained for that depth.

Then confirm you did not expect Inkling 2 Small. The Mini card says the largest gap versus Small is factual recall. AA-Omniscience for Mini is -8.6, with a 63.4% non-hallucination rate, behind Qwen3.8-27B on both of those cells. Looping will not fill that in. For facts, the lab's own story is search and tools (BrowseComp 79.0% and DeepSearchQA 85.3% are the "with compaction" rows), not a closed-book quiz.

A 1-bit GGUF will also miss the blog posts. UD-IQ1_S is 7.76 GB with KL 0.0911 and 89.4% top-token agreement against Unsloth's 8-bit reference. That is a different model in practice. UD-Q4_K_XL is the file they recommend if you want the write-up people are quoting.

SGLang will not come up

The Hugging Face post says a prerequisite is to run the server on SM103, and it prints a python -m sglang.launch_server command with --attention-backend fa4. If you are not on that GPU generation, their example is not the command for your card. Use the GGUF path or the recipe linked from the model card for the stack you actually have. The command itself is on the download page.

It is clumsy in a long chat

The announcement says Mini is not the strongest small model for multi-turn chat or for instructions with many constraints, and gives MultiChallenge at 50.0%. They describe that result as behind Qwen3.8-27B. On ComplexConstraints v1.1 the limitations paragraph also says "behind", while the score table lists Mini at 33.4% and Qwen3.8-27B at 30.1%. If a multi-constraint instruction fails, you are in the area the lab already flagged. Split the instruction, or use a model that leads those rows, instead of raising the quant and hoping.

Languages other than English can be weaker. The card says Chinese testing showed clipped phrasing and mixed languages. Sensitive mental-health topics draw more caution than the situation calls for, in their words. Neither is a loader bug.

Common questions

Why will llama.cpp not load an Inkling 2 Mini GGUF?

Unsloth's root GGUFs use their looped-layer layout and need the b12588-mix-c0c3de8 build, which also understands per-expert NVFP4 scales. Upstream llama.cpp expects a different layout. Use that build for the root files, or switch to the Mainline_Support files if you are on mainline llama.cpp. Do not mix the two.

Inkling 2 Mini is killed or runs out of memory. What should I change?

The official checkpoint is about 20 GB and is recommended with 32 GB or more of VRAM, or 32 GB or more of Apple silicon memory. For GGUF, Unsloth says to lower --ctx-size and --n-gpu-layers, drop --mmproj if you do not need images (639 MB), and drop the 0.4 GB drafter. Their example context is 32768, not 1 million.

The model never stops thinking, or it repeats itself while debugging.

The model card says internal testing saw long reasoning loops and repeated self-checks, especially on code. They tell you to limit output tokens and agent turns. Setting reasoning effort to the minimum does not fully disable reasoning, because the middle layers still run three times.

It ignores images.

In Unsloth's llama.cpp command, vision requires --mmproj and mmproj-F16.gguf. Text-only runs omit both. The model card wants each image side between 40 and 4096 pixels. SGLang needs --enable-multimodal, which the Hugging Face post states separately from the base server command.

Sources

  1. Unsloth, Inkling 2 Mini run guide GGUF sizes, memory ranges, sampler settings, and llama.cpp commands.
  2. Inkling 2 Mini model card Hardware, license, architecture, evaluations, and limitations.
  3. Hugging Face blog, “Welcome Inkling 2 Mini” Published 20 November 2026. Transformers, llama serve, SGLang, and MLX commands.
  4. Ollama library page for inkling-2-mini
  5. Thinking Machines Lab, “Introducing Inkling 2 Mini” Announcement dated 20 November 2026.
  6. Inkling 2 Mini config.json Hidden size, layer count, loop bounds, experts, and context length.