rea-video: an open-source CLI that reverse-engineers a video shot by shot
rea-video is an open-source CLI and agent skill that breaks a video into shots and judges whether each was filmed, AI-generated, 3D-rendered or built in code, then writes a remake plan and a VIDEO.md file for agents.
rea-video (GitHub: JuneYaooo/rea-video) is an open-source Python CLI and agent skill that takes one video at a time and works out, shot by shot, how it was made. Its skill file sorts shots into 11 types, from live footage and AI video to 3D renders, code-built motion graphics and screen recordings. It then writes a remake plan for each shot. The repo was created on 2026-10-29 and had about 9,000 stars and 1,300 forks on 2026-11-15, under the Apache-2.0 licence.
It is not a video generator. The output is a report for people (report.html, with a PDF via rea-video pdf and a local Chrome) and a VIDEO.md package (VIDEO.md plus a video/ folder), an open format meant for AI agents to read layer by layer. It is meant to be run by Claude Code or Codex.
Quick facts
- What it is: a command-line tool (rea-video) plus a skill file for coding agents (skills/rea-video/SKILL.md) and a Claude Code plugin manifest.
- Platforms: needs Python 3.10 to 3.14 and ffmpeg; the docs give ffmpeg install commands for macOS (brew) and Ubuntu (apt) and none for Windows.
- Licence and cost: Apache-2.0, free. Tools-only mode runs offline. An optional vision model (any OpenAI-compatible endpoint, such as Gemini or Doubao) is billed by its provider.
- Languages: reports in Chinese or English; the detailed docs are Chinese only for now.
How to install
The README's intended route is to give your agent one sentence: install the repo from GitHub, then break down a given video file. The agent installs the CLI and follows the skill. If you prefer to install it yourself, the official options are:
pip install "rea-video[ocr] @ git+https://github.com/JuneYaooo/rea-video"
# or, inside a clone of the repo:
bash scripts/install_skill.sh
rea-video --helpinstall_skill.sh also links the skill into the usual agent folders. In Claude Code you can also add it as a plugin with /plugin marketplace add JuneYaooo/rea-video and /plugin install rea-video@rea-video, but the skill still needs the pip-installed CLI. The only tagged release is v0.1.0, published 2026-11-11. Installing from git tracks the main branch, which was still receiving commits on 2026-11-15, so results can change between installs.
Three install traps are documented. If onnxruntime has no matching wheel, your Python is too new; use a 3.10 to 3.14 interpreter. On Homebrew or newer Ubuntu system Python, pip can refuse with externally-managed-environment; the setup notes say to use pipx or the install script instead of --break-system-packages. And the OCR extra on Python 3.12 or older pulls in a second OpenCV build that clashes with the headless one; the script fixes this automatically, and the docs give the manual pip commands.
First setup
No account or key is needed for a first run. The basic command is rea-video decompose ref.mp4 -o out/ref, which writes breakdown.json, report.html, a storyboard image, per-shot frames, separated audio stems, subtitle files (SRT and styled ASS) and the VIDEO.md package. The first breakdown downloads sound-separation models to ~/.cache/rea-video/models: about 67 MB for voice and music (UVR MDX-Net) and about 149 MB for sound effects (BandIt v2). Optional extras download more: about 160 MB for speech recognition, about 35 MB for speaker separation, and 10 to 14 MB for Japanese or Korean OCR. The CLI docs say a 30-second clip takes about a minute without a vision model.
What you miss without a vision model: the setup notes say the live versus AI versus 3D split gets noticeably weaker, and sound effects are reduced to low-confidence signal candidates. You can set REA_VLM_BASE_URL, REA_VLM_API_KEY and REA_VLM_MODEL (or pass --env-file) and add --vlm. Or, if your agent can look at images, run rea-video review and rea-video apply-review so the agent's per-shot answers are merged with the measured evidence. Check the VLM: line of rea-video summary: failed or not_configured means the model was not used. By default at most 60 VLM calls are made per video, spent on the least certain and longest shots first; --max-vlm-shots raises it. If a verdict is wrong, rea-video correct fixes the report and remembers the correction for that file.
What stands out
- Evidence before looks. The pipeline measures frame cadence (24fps generator output shows repeated frames in a 30fps file), sensor noise (a real camera leaves it), whether the whole image moves as one flat plane, and AIGC tags in metadata, and lets that hard evidence beat a visual impression; conflicting shots are flagged for review.
- Published accuracy, with the hard cases included. eval/RESULTS.md reports tools-only accuracy of 98.3% on a 15-video tuning set and 99.6% and 98.9% on two holdout sets, weighted by shot duration, and says the holdouts are not fully independent. The same file is blunter about new material: on two Douyin product videos the rules had never seen, tools alone scored 46.4% (63.2% in a later from-scratch retest), and on film-clip edits about 15%, because the tools cannot detect film clips, 2D animation or stop-motion at all. Agent visual review lifted those sets to about 97 to 99%.
- It plans the remake per shot type. Live shots get a shooting plan with framing, camera position, lighting and a gear list; AI shots get a reverse-engineered prompt and are batched by what the target video model can do; code-like shots are described for rebuilding as a Remotion project.
- VIDEO.md keeps agent context small. According to the docs, a 3-minute talking-head breakdown is about 2k tokens at the overview level versus about 190k tokens for the full breakdown.json, and inferred fields are marked separately from what is visible on screen. Other tools can validate a package with pip install videomd and videomd check.
- Captions, styled text and stickers are timed separately so they can be laid back over regenerated shots instead of being baked into prompts.
How it was built with Claude
The visible footprint is small. Of 139 commits on main as of 2026-11-15, 6 carry Co-Authored-By trailers for Claude models (3 Opus 5.5, including outside contributions adding Japanese and Korean OCR, and 3 Fable 5.1 on README and case-study edits). There is no CLAUDE.md; AGENTS.md tells agents to read the skill first, run pytest and the eval sets after rule changes, and never commit keys.
Try it with your AI. Replace the [bracketed] parts, then paste this into Claude Code, Codex or another agent that can run shell commands, to break down a video you own or have the rights to study.
I want to use rea-video (github.com/JuneYaooo/rea-video, Apache-2.0) to break down one
video: [path to my video file].
First check: run `python3 --version` and `ffmpeg -version` and tell me the results.
rea-video needs Python 3.10 or newer and ffmpeg. If either is missing, stop and tell me
what to install; do not install system packages yourself.
Then show me the install command you plan to use and wait for my confirmation. Use one
of the official methods only:
pip install "rea-video[ocr] @ git+https://github.com/JuneYaooo/rea-video"
or, if pip reports externally-managed-environment:
pipx install "rea-video[ocr] @ git+https://github.com/JuneYaooo/rea-video"
This installs code from the main branch; show me the repo URL before running it. Do not
use --break-system-packages. Confirm with `rea-video --help`.
Tell me that the first run downloads sound-separation models (roughly 216 MB, more if
OCR or speech extras are used) before you start.
Run the offline breakdown without a vision model:
rea-video decompose [path to my video file] -o out/[short name] --lang en
Then run `rea-video summary out/[short name]/breakdown.json`.
Read out/[short name]/VIDEO.md first, not breakdown.json. Tell me, per shot, how it was
most likely made and which shots are flagged as low confidence. If you can view images,
run `rea-video review out/[short name]`, follow the instructions in review_tasks.json,
check high-priority shots first, fill label_probs and evidence in review_answers.json,
and run `rea-video apply-review out/[short name] --by [your name]`.
Do not configure any vision-model API key, do not call paid generation tools, and do not
delete any files. Finish with the remake plan for each shot type and point me to
out/[short name]/report.html.Known limits
- Copyright and likeness: the tool analyses whatever video you give it, including other people's work, and its remake path is designed to keep an original's camera, motion and pacing. The README says it is for learning structure and technique and remaking with material you have rights to, not reusing others' footage, music, characters or likeness, and asks users to follow platform rules and AI-content labelling requirements. The repo also ships re-compressed third-party sample clips and comparison videos that include OiiOii remakes from its WeChat articles, which it says are shown for comparison only. Those clips are not offered for reuse.
- Privacy: with --vlm, short clips and keyframes of each shot are sent to whichever model endpoint you configure. Don't send other people's or sensitive footage to a third-party model. User corrections and preferences are stored locally in ~/.rea-video.
- Platform downloads are harder: recompression wipes out noise evidence, and the README says a visual check is needed to get back above 99%. Static shots are the main error source: in RESULTS.md, fixed-camera shots of re-encoded phone footage are often called 3D or code. The bundled samples are re-compressed too, so running them gives different results from the case reports.
- Sound is the weak layer without a vision model: signal features alone could not separate effects from speech or music onsets in one test mix; music source (AI-generated versus library) was right on 2 of 5 test videos.
- Release notes list open issues: swipes and page-turn animations in phone screen recordings can still be counted as cuts, and finer cutting sometimes misjudges short shots of dim talking-heads over screenshots or 3D plush characters.
- Scope: automatic generation still calls a model once per shot (the cheaper multi-shot prompts in the report are manual only), and the detailed docs are only in Chinese.
Related on claudefox
- AG-UI: connect an AI agent to a frontend app with a shared event protocol
- Kanzo puts a kanban board and an MCP server for coding agents in one 4.7 MB binary
- cinematic-skill: turn a script and a voice into a narrated cinematic film built from code
- ex-USSR Kinship Research: an evidence-first genealogy skill for Claude Code and Codex
Problems and feedback
Bugs go to GitHub Issues, which also has a request-breakdown template. Questions and remake ideas go to GitHub Discussions, and the README links a Discord server and the project site, reavideo.ai. Multi-episode dramas are handled by the sibling project rea-toon. On 2026-11-15 there were 4 open issues and 1 open pull request.
FAQ
Does rea-video need an API key? No. The offline tools-only mode is free and, per the README, already accurate on original files. A vision model is optional and mainly helps with recompressed platform downloads.
Will it work on a TikTok or WeChat download? Yes, but expect more flagged shots. On 10 re-encoded WeChat videos the original tools-only score was 88.6% (97.2% in the later retest), and 99.6% with Gemini. Product videos and film edits need a visual review.
Can it remake the video automatically? Partly. With the author's separate capsule-cinema project installed, AI shots can be generated and placed back on the original timeline; a free dry run comes first and paid model calls happen only after you confirm. Live shots you film yourself.
Source: JuneYaooo/rea-video on GitHub, rea-video v0.1.0 release notes, rea-video CLI reference (docs/cli.md), rea-video install and VLM setup, rea-video eval results (eval/RESULTS.md), VIDEO.md package docs, VIDEO.md open format
This story first appeared in the claudefox daily briefing for November 15, 2026.