llama-modchip runs the 284B DeepSeek-V4-Flash on a 64 GB PC with no GPU, using patched llama.cpp
llama-modchip (synchopate/llama-modchip) is an Apache-2.0 toolkit of llama.cpp patches, custom quantization formats and launch scripts that runs the 284B DeepSeek-V4-Flash inside 64 GB of RAM on a CPU with no GPU, plus 30B mixture-of-experts models on 16-32 GB machines.
llama-modchip (GitHub: synchopate/llama-modchip) is an Apache-2.0 set of llama.cpp patches, quantized models and a wrapper CLI for running large mixture-of-experts (MoE) models on the CPU alone. Its headline target is DeepSeek-V4-Flash-0731, a 284-billion-parameter model with about 13 billion parameters active per token, squeezed into GGUF files of 56.4 to 58 GiB so it stays fully resident on a 64 GB machine with no disk streaming. It also ships Nemotron-3.5-30B-A3B builds for 16-32 GB laptops. Created October 29, 2026, it had 1,271 stars and 113 forks as of November 16, 2026.
Quick facts
- Licence and price: code under Apache-2.0, free. The DeepSeek-V4-Flash weights it quantizes come from deepseek-ai/DeepSeek-V4-Flash-0731 under the MIT licence; the Nemotron-3.5 builds carry NVIDIA's OpenMDW-1.1 licence, per the Hugging Face model cards. The older Fractal and VQ3 284B builds reuse Unsloth UD-IQ1_S expert tensors; MaxQ-v4 uses none.
- Upstream: patches applied to ggml-org/llama.cpp at pinned commit d2f83055, not a GitHub fork; llama.cpp is credited as the upstream runtime.
- Platforms: prebuilt binaries for Linux x86_64 only (AVX-512 and AVX2 builds, glibc 2.30 or newer, needs libgomp). macOS on Apple silicon and Windows 10/11 build the patched engine from source with the project's scripts.
- Version: v2.5.5 (November 15, 2026), the tenth release since November 2. Release assets across all ten releases had 1,157 downloads as of November 16.
Hardware requirements, as the README lists them
- Laptop or mini-PC, 16-32 GB RAM: Nemotron-3.5-30B-A3B in the 13.30 GiB VQ3 build, about 14.6-16 GiB resident.
- 64 GB desktop or server: DeepSeek-V4-Flash 284B in the MaxQ-v4 (57.99 GiB, the default), MaxQ-Fractal (56.93 GiB) or VQ3 (56.40 GiB) builds, using 57.5-59.1 GiB of RAM.
- Hybrid desktop, 32-64 GB RAM plus a 16 GB GPU: 30B models with attention and SSM layers on the GPU and the routed experts left in CPU RAM.
- 8-channel workstation with 128 GB or more (Threadripper): both models.
- CPU features: the AVX-512 binary targets AMD Zen 4/5 and Intel Ice Lake or Sapphire Rapids (it needs avx512f and avx512vbmi); the AVX2 binary covers older chips.
What it claims, and on what hardware
All figures here are the project's own or community measurements. The 284B numbers come from the author's AMD Ryzen 9 9955HX (16-core Zen 5, 64 GB dual-channel DDR5-5600, no GPU, --cpu-range 0-15, transparent huge pages on). There the README reports 36.1-41.1 tokens per second of cold prompt processing (44.4 warm), 8.66-9.06 tokens per second of plain one-token-at-a-time generation with all six routed experts active, and 8.8-25.3 tokens per second with its speculative decoding layer. The README presents about 9 tokens per second as the envelope for unpruned one-token generation on dual-channel DDR5; optional Top-4 expert pruning reaches 10.12-10.74 at a +2.85% perplexity cost.
Community entries add 27.0 tokens per second for the 284B model on a 64 GB Apple M5 Max (llama-bench tg128, from Bobby Grayson) and a 111.62 tokens per second peak for Nemotron-30B on a 64-core Threadripper PRO 7985WX with 8-channel DDR5.
On the project's own 51-chunk wiki-alt evaluation, the default MaxQ-v4 build scores a mean perplexity of 6.86 against 4.33 for the full-precision reference logs, and its top-1 token matches the reference 73.3% of the time. MaxQ-Fractal scores 70 of 80 on the project's functional gate (30/30 tool calls, 20/30 code) and the older VQ3 and Fractal builds 64 of 80; the README table gives no gate score for the MaxQ-v4 default. For 16-32 GB machines, the Nemotron VQ3 build scores 100/100 on the 30B gate at 24.2-26.1 tokens per second served.
How it works, as the project explains it
The README and FAQ argue that CPU inference of MoE models is limited by memory bandwidth per token, so the job is to shrink the bytes read per token. The project uses per-tensor mixed precision: in the 284B builds the roughly 265 billion routed-expert parameters are stored at about 1.56-2.06 bits per weight with llama.cpp's IQ1_S, IQ1_M and IQ2_XXS types, while attention and shared experts get more bits. The file averages about 1.7 bits per weight.
It adds two custom formats: Fractal-64 (3.75 bits per weight) and LUT-VQ3-64 (3.00 bits), which encodes three weights as one of 128 codebook points looked up in AVX-512 registers. The newest default, MaxQ-v4, drops both and instead narrows expert width in 34 of 43 layers from 2048 to 1792 (ConeCollapse), so it uses only standard ggml types. Its RouteCrack and Stockfish-PVS layers draft up to six tokens ahead, and the model verifies them in one batched pass.
With a GPU, the Windows pack and FAQ use expert offload for 30B models: the attention or SSM trunk goes to the GPU (-ngl 99) and routed experts stay in system RAM, sized with --n-cpu-moe. The Windows guide says this does not help the 284B model on a 12 GB card: its custom kernels are CPU-only, and five offloaded expert layers gave 2.65 tokens per second versus 6.35 on CPU alone (Ryzen 7 9800X3D).
How to install
The README's main route on Linux and macOS is to clone the repo and let the CLI set things up:
git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chatThe installer picks MaxQ-v4 at 60 GB of RAM or more and a Nemotron build below that. On Linux x86_64, ./modchip download-bin fetches the matching prebuilt binary and checks its SHA-256. install.sh takes --preset, --prebuilt, --source and --skip-model. On Windows, the patched engine is compiled with windows/build_patched_engine.ps1 -Target dsv4. Expect a download of about 57 GiB for the 284B model.
First setup
The README's local steps ask for no account or API key. On a 64 GB desktop, the launch script holds the 284B model to 32K context (-c 32768 -ub 512); the full 256K is meant for headless 64 GB machines with zram or swap, or 96 GB and up. For DeepSeek-V4, the guides warn never to set DSV4_MOE_TOPK or LLAMA_MOE_TOP_K in the environment, because it breaks loading; pruning goes through the per-user config file instead. ./modchip status checks CPU readiness and huge pages, ./modchip bench runs a 30-second speed test with a shareable scorecard, ./modchip web opens a local web chat, and ./modchip claude starts Claude Code against the local server.
What stands out
- It shows perplexity and KL divergence next to tokens per second, so you can see the quality cost before a 58 GiB download.
- A self-reported community leaderboard of 31 entries, from a 2012 dual Xeon to a 15 W laptop chip, links most entries to the issue they came from; tests and models vary by row.
How it was built with Claude
Of 97 commits on main, the author's 85 carry no AI co-author lines. On the Apple Silicon side, six commits by Austin Dixson (Metal Q2_0 kernels, tests, a build script change and docs) carry a Co-Authored-By line for Claude Fable 5.2, and five by Bobby Grayson (Metal kernels, GPU tensor mapping and the M5 Max DeepSeek-V4 build) carry one for Claude Opus 5.5.
Try it with your AI. Give this to your AI coding assistant to check your machine before a 57 GiB download.
I want to try synchopate/llama-modchip (v2.5.5) on my [Linux / macOS / Windows] machine.
Work step by step and do not download any model yet.
1. Check prerequisites with read-only commands and show me the results: CPU model, total
RAM, and free disk space where the model will go (the 284B model needs about 57 GiB).
On Linux also check glibc is 2.30 or newer, libgomp is installed, and whether grep
-Eq "avx512_?vbmi" /proc/cpuinfo succeeds (that decides -avx512 vs -avx2). On macOS
or Windows, tell me the README route for my OS instead (Apple Silicon scripts, or
windows\build_patched_engine.ps1).
2. Pick a preset from the README: dsv4-maxq at 60 GB RAM or more, otherwise
nemotron-vq3, or nemotron-fractal on AVX2-only CPUs and Apple Silicon. Never pick an
abliterated preset.
3. Clone https://github.com/synchopate/llama-modchip.git and run ./modchip status. Show
me the output.
4. Show me the exact install command (on Linux, for example ./install.sh --prebuilt
--preset [preset]) and the download size, then wait for my confirmation.
5. For the 284B model on a 64 GB desktop, keep -c 32768 -ub 512. Do not set
DSV4_MOE_TOPK or LLAMA_MOE_TOP_K.
6. After it starts, run ./modchip bench and summarize the scorecard.
Rules: only use commands from the repo's README, docs, install.sh and modchip. Do not
pipe remote scripts into a shell. Do not use sudo, change system or network settings,
touch credential files or delete anything without asking me first.Known limits
- Compatibility: vanilla llama.cpp, Ollama, vLLM and LM Studio cannot load the Fractal-64, VQ3 or deepseek4 files.
- Memory: the 284B model holds 58-59 GiB of non-evictable heap; enable zram or close browsers.
- Uncensored variants: opt-in abliterated models with refusals removed are offered; the installer asks first, and the README limits them to authorized security research and warns against exposing them on shared networks.
- Security: early releases used shared /tmp and /dev/shm control files; v2.5.1 and v2.5.3 moved them to per-user, permission-checked paths, and v2.5.1 removed --dangerously-skip-permissions from ./modchip claude defaults. Use a current release.
- Remote scripts: the readiness check and one-line installer pipe GitHub scripts into python3 or bash; read them first.
- Evidence: the repo was 18 days old on November 16; the headline 284B numbers are the author's own, the leaderboard is self-reported, and there is no independent benchmark.
- Headroom: the fast path needs AVX-512 VBMI and DDR5 bandwidth, and 57.5-59.1 GiB resident leaves little room on a typical 64 GB desktop.
Related on claudefox
- PhoneBox turns an old Android phone into a desk panel and control deck for Windows, Linux and macOS
- Edge0 open-sources a 145B model it says runs on a laptop; its own speed table uses RTX cards
- Researcher says released weights show six Chinese models were started on Nvidia Blackwell GPUs
- CleanYourTV: debloat an Android TV or Google TV over Wi-Fi with Claude Code and get an Apple TV-style home screen
Problems and feedback
Use GitHub issues (31 open issues and pull requests on November 16) and Discussions; benchmarks have their own issue template.
FAQ
Q: Do I need a GPU? A: No. The main results are CPU-only. A GPU helps in the hybrid setups, and Apple silicon runs the 284B model through Metal.
Q: Why DeepSeek-V4-Flash-0731 and not a newer DeepSeek release? A: The README says V4.1-Flash-1017 adds lookup tables that push it past 340 GiB even at 2-bit, so 0731 is the largest V4 Flash checkpoint that stays resident in 64 GB.
Q: Which 284B file should I download? A: MaxQ-v4 is the installer's default and has the best perplexity in the README; MaxQ-Fractal is the build with the published 70/80 functional gate score. Both need about 57-58 GiB.
Built by synchopate with community contributors, on top of ggml-org/llama.cpp
Source: GitHub: synchopate/llama-modchip, llama-modchip README, llama-modchip release notes, llama-modchip releases, llama-modchip engineering FAQ, DeepSeek-V4 optimization guide, Hardware optimization guide, Windows port guide, install.sh, Linux build script (pinned llama.cpp commit), llama-modchip commit history, llama-modchip issues, Hugging Face: DeepSeek-V4-Flash-0731-MaxQ-v4 GGUF, Hugging Face: Nemotron-3.5-30B-A3B VQ3-64 GGUF, Hugging Face: deepseek-ai/DeepSeek-V4-Flash-0731, ggml-org/llama.cpp
This story first appeared in the claudefox daily briefing for November 16, 2026.