inference-optimisation: hill-climb a local LLM server for faster decode and prefill
inference-optimisation is an agent skill that hill-climbs the speed of a local LLM server on any hardware and any quantization, starting from a theoretical ceiling and testing one change at a time.
sybil-solutions/inference-optimisation-skill gives you a repeatable way to make a local language-model server faster, instead of changing settings at random and hoping the next run is quicker.
What is inference-optimisation?
inference-optimisation is an agent skill that hill-climbs the speed of a local LLM server on any hardware and any quantization.
It covers single-user and concurrent decode, speculative decoding, prefill, KV cache size, startup time and time to first token, and it works under a memory cap, with every result having to survive a full-context stability check.
Its workflow is to compute the theoretical ceiling, change one thing at a time, measure against interleaved controls, log every result including the failures, gate on long-context retrieval, and qualify a clean, pinned build before publishing.
The README reports results from the hill-climbs the skill was distilled from, measured on the hardware named. On an Intel Arc B390 laptop, single-user decode rose from 43.6 to 53 tokens per second, prefill at 256K rose from 297 to 628 tokens per second, startup fell from 113 to 43 seconds, and KV cache capacity rose from 264K tokens to four streams of 256K tokens. On four H200 NVL cards, a DeepSeek-V4.1-Flash server that crashed under load reached 229.1 tokens per second for decode and 17,546 tokens per second for prefill, with all six gates passing. Those are the README's reported results for those runs, not a promise for a different machine, model or engine.
Who is inference-optimisation for?
inference-optimisation is for people who run a local LLM server and want to tune it with measured experiments, rather than guessing which setting helps.
Its skill description covers any hardware, including NVIDIA, AMD, Intel and Apple hardware, unified-memory laptops and multi-GPU machines, and any quantization, including GGUF, EXL2 and EXL3, AWQ and GPTQ, FP8, MLX and BF16.
It is written for servers such as vLLM, SGLang, llama.cpp, ExLlama and TabbyAPI, MLX and TensorRT-LLM, and it works with Claude Code and any agent that loads SKILL.md-style skills, while any model can also follow it from PROMPT.md.
Is inference-optimisation free and open source?
inference-optimisation is free and open source under the MIT License.
- Platform: any hardware, including NVIDIA, AMD, Intel and Apple hardware, unified-memory laptops and multi-GPU machines.
- Language: Python, with scripts in standard-library Python plus bash.
- License: MIT, copyright 2026 Sybil Solutions contributors.
- Cost: the license grants permission free of charge.
- Popularity: about 950 GitHub stars as of November 18, 2026.
- Latest release: no releases were listed as of November 18, 2026.
What stands out
- Ceilings first: ceiling.py computes the theoretical maximum for each metric from measured bandwidth and FLOPs plus component sizes, so a target can be checked against the physics before tuning starts.
- One change at a time: each experiment changes one thing, takes at least six decode samples and uses the median, runs control, candidate, control, and is accepted only if it beats both neighbouring controls and regresses nothing by more than about 3 percent.
- Full-context stability gates: a config counts as best only after long-context needle retrieval at about 50 percent and then about 100 percent of the context, a concurrent full-length soak where the KV pool is sized for several users, and a greedy equivalence check where a change claims to be lossless.
- An engine-agnostic harness: bench.py benchmarks any OpenAI-compatible server, measuring startup and short-prompt time to first token, single-user and concurrent decode, cold prefill at several lengths, long-context needle retrieval and peak memory, with token counts from server usage statistics and every run logged to ledger.jsonl.
- Qualification before publishing: publishing.md requires pinning weights, the engine and kernels, building from a clean checkout, running six acceptance gates covering load, chat, reasoning, tools, context and speed, and proving the clean build matches the best build.
How do I install inference-optimisation?
You install inference-optimisation by cloning the repository and copying its skill folder into Claude Code's user-level skills folder.
From the project's README at https://github.com/sybil-solutions/inference-optimisation-skill/blob/main/README.md, the commands are as follows.
git clone https://github.com/sybil-solutions/inference-optimisation-skillmkdir -p ~/.claude/skillscp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/Then the README suggests asking, for example, to hill-climb decode and prefill for the model on port 8000 under a 40 GiB cap, or to ask what the theoretical maximum tokens per second is for the model on the machine.
Other agents can be pointed at skills/inference-optimisation/SKILL.md, or given PROMPT.md as the task prompt.
How it was built with Claude
The repository's commit carries a Co-Authored-By trailer naming Claude Opus 5.5.
Use this prompt to try the skill on your own server:
Try it with your AI. Try inference-optimisation on your own local server.
Use the inference-optimisation skill in skills/inference-optimisation/SKILL.md to
hill-climb the speed of my local LLM server. My model is [model name and quantization,
for example EXL3 4 bpw], my engine is [vLLM, SGLang, llama.cpp, or another engine named
in the skill], and my server is at [base URL, for example http://127.0.0.1:8000]. My
memory cap is [for example 40 GiB], my context length is [for example 262144 tokens],
and my priority is [single-user decode, concurrent decode, prefill, KV capacity, startup
time, or time to first token]. First compute the theoretical ceilings with
skills/inference-optimisation/scripts/ceiling.py, then measure the baseline with
skills/inference-optimisation/scripts/bench.py, change one thing at a time, run each
candidate between controls with skills/inference-optimisation/scripts/queue.sh, and log
every result including failures. Use only the official commands and interfaces named in
the README. Show risky steps first and wait for me to confirm before running them. Do
not install anything from unknown sources, do not touch credential files, do not delete
anything, and do not use keys or tokens.What are inference-optimisation's limits?
inference-optimisation's limits are measurement noise, hardware-dependent results, and safety rules that stop a tuning run from damaging a server or leaking a secret.
- Noise is plus or minus 5 to 10 percent, and most single-run wins are drift, so the skill interleaves controls and uses medians.
- Speculative decoding can lose on bandwidth-poor devices or hard prompts: on an Apple M4, three MTP drafts took decode from 6.35 to 3.25 tokens per second, and llama.cpp MTP on hard prompts ran at 36.7 tokens per second against 45.4 without it.
- The lever numbers come from real hill-climbs, and the skill says to use them as priors, not promises.
- Safety rules require one GPU job at a time, staying on AC power on a fixed performance profile, a memory watchdog, and draining requests before stopping a server, never force-killing mid-batch, because a forced stop mid-prefill has caused GPU driver faults.
- Secrets must never be printed, and must never be copied, or their path copied, into anything published, and publishing.md requires grepping the bundle for secrets, credential-file paths, tokens, personal emails and home-directory usernames before pushing.
- A second engine on the same GPU, or a stray benchmark still running, silently halves results or crashes both, so the skill locks the GPU with scripts/queue.sh.
Related on claudefox
- llama-modchip runs the 284B DeepSeek-V4-Flash on a 64 GB PC with no GPU, using patched llama.cpp
- CleanYourTV: debloat an Android TV or Google TV over Wi-Fi with Claude Code and get an Apple TV-style home screen
- Kanzo puts a kanban board and an MCP server for coding agents in one 4.7 MB binary
- cinematic-skill: turn a script and a voice into a narrated cinematic film built from code
Problems and feedback
Report problems on the project's Issues page at https://github.com/sybil-solutions/inference-optimisation-skill/issues, and send improvements as pull requests with new levers, gotchas or results.
A contribution needs the hardware, model, quantization, engine version, before and after numbers, how they were measured, and a link to raw data.
Source: inference-optimisation-skill repository, README, SKILL.md, PROMPT.md, LICENSE, Levers reference, Gotchas reference, Publishing reference, Releases, Issues
This story first appeared in the claudefox daily briefing for November 18, 2026.