> ## Content Index
> Fetch the complete content index at: https://claudefox.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Inkling 2 Mini: how to run Thinking Machines' 26B open model locally, and which quant fits your RAM
- URL: https://claudefox.com/inkling-2-mini-run-locally/
- Published: 2026-11-20T21:10:16.000Z
- Updated: 2026-11-20T21:10:25.000Z
- Description: Inkling 2 Mini is Thinking Machines' Apache 2.0 looped MoE model: 26.4B parameters, 3.4B active, about 20GB. Which GGUF quant fits your memory, the commands for llama.cpp, MLX and vLLM, launch-day gaps, and how it compares with Qwen3.8-27B.
- Author: hupicake
- Tags: News

Thinking Machines Lab released Inkling 2 Mini on November 20, 2026: an open-weights model under the Apache 2.0 license with 26.4B total parameters, 3.4B active, and a checkpoint of about 20GB. The company says it runs on any Apple silicon Mac with at least 32GB of memory, on an NVIDIA DGX Spark, and on other GPUs with 32GB or more of VRAM. Weights are on Hugging Face as thinkingmachines/Inkling-2-Mini.

This guide is for people who want to run it on their own machine. It covers which download fits how much memory, the commands each runtime documents, where the launch-day support is still rough, and how the benchmarks compare with Qwen3.8-27B, one of four models in Thinking Machines' comparison table and the one Hugging Face used in its own vibe tests. Everything here comes from the Thinking Machines announcement and model card, the Hugging Face launch post, Unsloth's GGUF page and guide, the vLLM recipe and Artificial Analysis, read on November 21, 2026 (Singapore time). We did not run the model ourselves, so every command and number here is second-hand.

## Quick facts

- License: Apache 2.0, with a separate Thinking Machines acceptable use policy linked from the model card.
- Size: 26.4B total parameters, 3.4B active. Because the middle layers run three times, Thinking Machines counts 5.4B effective active parameters per token.
- Checkpoint: about 20GB, mostly NVFP4 with the last four layers in MXFP8, the precision it was trained in.
- Inputs: text and images in, text out. Context window up to 1M tokens.
- Hardware Thinking Machines lists: 1x DGX Spark, Apple silicon with 32GB or more, or other GPUs with 32GB or more of VRAM recommended.
- Runtimes named at launch: Transformers, llama.cpp and Llama App, MLX, Ollama and Exo Labs on Apple silicon, Unsloth, vLLM, SGLang, TokenSpeed and NVIDIA Dynamo. Hosted inference and fine-tuning are on Tinker.
- A second checkpoint, TML-d24-varloop-research, can run with a different number of loop passes. It is a research artifact.

## What a looped transformer means for you

The practical effect: you download 24 layers of weights, but the model does the work of 48\. Per the model card, layers 6 to 17 run three times per token with shared weights, so the checkpoint stays at about 20GB while each token gets more compute. The vLLM recipe notes the cost on the cache side: every pass through a looped layer keeps its own KV and convolution state, 48 of each. Each token is routed to 8 of 128 experts per layer.

The trade-off is knowledge. Thinking Machines says looping improves reasoning but does not let the weights store more facts, and that Inkling 2 Mini's factuality score is in line with other models its size. Its fix is search and tools: its table lists 79.0% on BrowseComp and 85.3% on DeepSearchQA (both with compaction) and 82.7% on MCP Atlas. If you plan to use it offline for factual questions, keep that in mind.

## Which quant fits your machine

Unsloth's memory table (RAM plus VRAM, or unified memory) lists 24GB for 8-bit, 17 to 18GB for 4-bit, 13 to 15GB for 3-bit, 10 to 11GB for 2-bit and 9GB for 1-bit. Unsloth labels its 8-bit UD-Q8\_K\_XL file "full precision"; the original checkpoint itself is NVFP4 and MXFP8\. These are Unsloth's published file sizes and its quality measurements against the 8-bit file. KLD is the divergence from the 8-bit model's output, so lower is better:

- UD-Q8\_K\_XL: 22.87GB (the reference).
- UD-Q4\_K\_XL: 17.63GB, KLD 0.0107, same top token as the 8-bit model 98.0% of the time. Unsloth's recommended quant.
- UD-Q4\_K\_M: 16.61GB, KLD 0.0108, 97.9%.
- UD-IQ4\_XS: 15.73GB, KLD 0.0109, 97.7%.
- UD-Q3\_K\_XL: 13.63GB, KLD 0.0174, 96.3%.
- UD-IQ3\_XXS: 11.66GB, KLD 0.0269, 94.7%.
- UD-Q2\_K\_XL: 9.14GB, KLD 0.0595, 91.6%.
- UD-IQ1\_S: 7.76GB, KLD 0.0911, 89.4%.

What the numbers say: the three 4-bit files are almost identical in quality, while the drop below 4 bits is steep. Going from Q4\_K\_M to IQ3\_XXS saves about 5GB but raises the divergence about 2.5 times (0.0108 to 0.0269). Going from 8-bit to Q4\_K\_XL saves 5.2GB, and the top token still matches 98% of the time. Unsloth's memory guide lists 17 to 18GB for 4-bit and 24GB for 8-bit, plus 639MB more for the mmproj-F16.gguf vision file if you want image input. It adds that longer contexts and image inputs use more. So a 16GB machine falls in the 3-bit range (13 to 15GB) by Unsloth's own table, even though its guide opens with "Run on 16GB RAM". For 4-bit, plan on 17 to 18GB.

One inconsistency to watch: Unsloth's guide gives UD-Q4\_K\_XL as 17.6GB in one place and 16.61GB in another. The quantization table on the model page lists 16.61GB for UD-Q4\_K\_M and 17.63GB for UD-Q4\_K\_XL. Check the file size on Hugging Face before you download.

## How to run it

Hugging Face's launch post shows the official GGUFs from ggml-org, 4-bit IQ4\_NL and 8-bit Q8\_0, served with Llama App's llama command:

```
llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL
# 8-bit: change the suffix to :Q8_0
```

Unsloth's guide says to use a llama.cpp build with looped-layer support and per-expert NVFP4 scales, and points to its unslothai/llama.cpp prebuilt binaries. (We expect an older build to fail on these files, but have not tested it.) Its full path downloads the 4-bit file plus the vision projector and starts llama-server with a 32K context and its recommended sampling settings:

```
pip install -U huggingface_hub
hf download unsloth/Inkling-2-Mini-GGUF \
    Inkling-2-Mini-UD-Q4_K_XL.gguf mmproj-F16.gguf \
    --local-dir Inkling-2-Mini-GGUF

./llama.cpp/llama-server \
    --model Inkling-2-Mini-GGUF/Inkling-2-Mini-UD-Q4_K_XL.gguf \
    --mmproj Inkling-2-Mini-GGUF/mmproj-F16.gguf \
    --jinja --ctx-size 32768 --n-gpu-layers 99 --parallel 1 \
    --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \
    --host 127.0.0.1 --port 8080
```

For text-only chat, Unsloth says to skip the projector and drop --mmproj. If memory is tight, lower --ctx-size or --n-gpu-layers. If you prefer an app, Unsloth Desktop can search for and download the same GGUF.

On a Mac, the Hugging Face post says MLX support is upstream in mlx-lm, with quants shared through the mlx-community organization, and that MLX can load the original checkpoint without converting it, keeping the NVFP4 and MXFP8 weights as trained. Its example, for an image prompt:

```
mlx_lm generate --model thinkingmachines/Inkling-2-Mini --image image.jpg --prompt "Describe this image.<|end_message|><|message_user|><|content_image|><|unused_200054|>" -m 1000
```

Ollama is the less certain path. Thinking Machines says Ollama supports Macs with 32GB or more, and the model card links ollama.com/library/inkling-2-mini, but when we checked that page twice early on November 21 (Singapore time) it returned a 404\. Unsloth's page lists ollama run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4\_K\_M as a way to pull its GGUF directly. If that fails, use llama.cpp or MLX until the library entry is live.

On NVIDIA servers, the vLLM recipe lists vLLM 0.35.0 or later, currently a nightly build, and says to use the vllm/vllm-openai:inkling-2-mini Docker image until support is in a vLLM release. It lists Blackwell (B200, GB200, B300, GB300), Hopper (H100, H200) and DGX Spark. One GPU handles contexts up to 262,144 tokens; the full 1M needs four GPUs, which the recipe says gives lower output throughput per GPU. So the 1M figure is not a local laptop number: Unsloth's own llama.cpp example uses 32K. The recipe's command, which runs the container with --privileged:

```
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:inkling-2-mini thinkingmachines/Inkling-2-Mini \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

The --speculative-config line turns on DSpark, a small draft model that the recipe says ships in the same checkpoint and proposes tokens for the main model to verify. Hugging Face's post describes the SGLang version as a three-layer draft. SGLang's own example needs SM103 hardware, per the Hugging Face post.

Reasoning effort is adjustable. In Transformers you pass reasoning\_effort to the chat template. The levels are none (0.0), minimal (0.1), low (0.2), medium (0.7), high (0.9, the default), xhigh (0.99) and max (0.99), or a number from 0.0 to 0.99\. The SGLang example sends "reasoning\_effort": "low" in the request body.

## How it compares with Qwen3.8-27B

On coding, Inkling 2 Mini leads most rows in Thinking Machines' table, which compares it with Gemma 4 31B, Muse Glimmer, Qwen3.8-27B and Nemotron 3.5 Super VL, but Qwen leads on Terminal Bench v2.1 and AutomationBench-AA:

- DeepSWE v1.1: 48.2% vs 42.2% for Qwen3.8-27B.
- SWE-bench Pro v2 (Hard): 77.9% vs 73.8%.
- SWE-bench Verified (bash only): 79.2% vs 77.8%.
- Terminal Bench v2.1: 71.6% vs 73.0%, one of the rows where Qwen leads.
- Terminal Bench v4.0: 5.6% for both.
- AutomationBench-AA: 41.8% vs 48.2%, another Qwen lead.
- AA-Omniscience non-hallucination rate: 63.4% vs 71.4%, which fits the company's warning about factual recall.

These are vendor-reported numbers, and the table notes that evaluation setups vary by model. Hugging Face's own vibe tests found Inkling 2 Mini "at least comparable" with Qwen 3.8 27B and said more validation is needed. Artificial Analysis, which runs its own index, scores it 20 on its Intelligence Index, against a median of 10 for comparable models. It also calls the model very verbose: it generated 161M tokens across the index, against a median of 85.7M, about 1.9 times as many.

Artificial Analysis lists Thinking Machines' hosted price as $0.15 per million input tokens and $0.50 per million output tokens, and an average of $0.59 per index task.

## Copy-paste prompt: check your machine before you download

**Try it with your AI.** Give this to a coding assistant that can run read-only commands on your computer. It checks your hardware and tells you which Inkling 2 Mini download fits, without installing or downloading anything until you approve.

```
I want to run Inkling 2 Mini (open weights, Thinking Machines) locally. Do not install
or download anything yet.
1. Check my machine with read-only commands: OS, CPU or chip, total RAM or unified
   memory, GPU and VRAM if any, and free disk space. Tell me what you ran.
2. Check which of these I already have and their versions: llama.cpp (llama-server or
   llama), Ollama, mlx-lm, Docker. Do not upgrade anything.
3. Pick a quant using Unsloth's memory table (RAM plus VRAM, or unified memory), not
   file size: 8-bit 24GB, 4-bit 17-18GB, 3-bit 13-15GB, 2-bit 10-11GB, 1-bit 9GB. Files:
   UD-Q8_K_XL 22.87GB, UD-Q4_K_XL 17.63GB, UD-Q4_K_M 16.61GB, UD-IQ4_XS 15.73GB,
   UD-Q3_K_XL 13.63GB, UD-IQ3_XXS 11.66GB. Add 639MB for mmproj-F16.gguf if I want
   images. Leave room for the OS and other apps, and say how much you assumed.
4. Unsloth says llama.cpp needs a build with looped-layer support and per-expert NVFP4
   scales. Tell me how to get such a build (Unsloth prebuilts or a recent source build),
   and do not assume my current build has it.
5. Show me the exact hf download command and a llama-server command with --ctx-size
   32768 and Unsloth's sampling settings (--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0),
   for [QUANT] on my machine. Wait for my confirmation before running anything.
6. If I later use it in a coding agent, set limits on output tokens and agent turns, run
   it in a sandbox, and ask me before any delete or overwrite.
```

## Known limits

- Thinking Machines warns that the model is more likely than larger models to state incorrect information with confidence, with the biggest gaps on factual recall.
- It says the model sometimes gets stuck in long reasoning loops or keeps re-checking its work, especially when debugging code, and recommends limits on output tokens and agent turns. If you put it inside a coding agent, set those limits.
- Launch-day tooling is uneven: vLLM needs a nightly build or a dedicated Docker image, llama.cpp needs a build with looped-layer support, and the Ollama library page returned a 404 when we checked on November 21.
- The model card warns that in agentic use it is less reliable than larger models at following detailed instructions, may delete or overwrite files without confirmation, and may not report its results accurately. It recommends running agents in a sandbox and requiring confirmation before destructive or irreversible actions.
- The SGLang example in the Hugging Face post requires SM103 hardware, and the vLLM recipe targets datacenter GPUs and DGX Spark.
- Benchmark comparisons are self-reported, and setups differ by model.

## FAQ

Can Inkling 2 Mini run on a 16GB Mac? By Unsloth's memory table, only at 3 bits or below: 3-bit needs 13 to 15GB and 4-bit needs 17 to 18GB. Thinking Machines itself supports Macs with 32GB or more.

Is Inkling 2 Mini free for commercial use? The weights are Apache 2.0\. The model card also links a separate acceptable use policy, so read that before you ship it in a product.

Which quant should I use with 24GB of memory? Unsloth's table lists 24GB for its 8-bit UD-Q8\_K\_XL file (22.87GB). If other apps need memory too, its recommended 4-bit UD-Q4\_K\_XL (17.63GB) is close in quality: the top token matches the 8-bit file 98.0% of the time.

Can I use the full 1M-token context locally? Not on one consumer machine, going by the published setups. Unsloth's llama.cpp example uses 32K, and the vLLM recipe handles up to 262,144 tokens on one datacenter GPU and needs four GPUs for 1M.

Does it support images and tool calling? Yes. It accepts text and images, and the Hugging Face post shows a tool-calling example in Transformers. vLLM uses an inkling tool-call parser.

What is the variable-loop model? TML-d24-varloop-research is a research checkpoint trained to run with a different number of loop passes. Hugging Face says one or two passes run faster with slightly lower quality, and that it was not fully evaluated.

## Related on claudefox

- [llama-modchip runs the 284B DeepSeek-V4-Flash on a 64 GB PC with no GPU, using patched llama.cpp](https://claudefox.com/llama-modchip-deepseek-v4-flash-64gb-cpu/)
- [Researcher says released weights show six Chinese models were started on Nvidia Blackwell GPUs](https://claudefox.com/checkpoint-forensics-chinese-models-blackwell-fingerprint/)
- [Hugging Face Inference Providers now route through OpenRouter: what changes for your code, keys and Claude Code](https://claudefox.com/hugging-face-inference-providers-openrouter-routing-custom-keys/)
- [Claude Code vs OpenCode (2026): cost, Claude access, which to use](https://claudefox.com/claude-code-vs-opencode/)

Source: [Thinking Machines: Introducing Inkling 2 Mini](https://thinkingmachines.ai/news/inkling-2-mini/?ref=claudefox.com), [Hugging Face: thinkingmachines/Inkling-2-Mini model card](https://huggingface.co/thinkingmachines/Inkling-2-Mini?ref=claudefox.com), [Thinking Machines: Inkling 2 Mini model card](https://thinkingmachines.ai/model-card/inkling-2-mini/?ref=claudefox.com), [Hugging Face blog: Welcome Inkling 2 Mini](https://huggingface.co/blog/inkling2mini?ref=claudefox.com), [Unsloth: Inkling-2-Mini-GGUF](https://huggingface.co/unsloth/Inkling-2-Mini-GGUF?ref=claudefox.com), [Unsloth: Inkling 2 Mini run locally guide](https://unsloth.ai/docs/models/inkling-2-mini?ref=claudefox.com), [vLLM recipe: Inkling 2 Mini](https://recipes.vllm.ai/thinkingmachines/Inkling-2-Mini?ref=claudefox.com), [Artificial Analysis: Inkling 2 Mini](https://artificialanalysis.ai/models/inkling-2-mini?ref=claudefox.com)

This story first appeared in the [claudefox daily briefing for November 21, 2026](https://claudefox.com/daily-2026-11-21/).