diff --git a/README.md b/README.md index 102e9dd..00b5c21 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,34 @@ # OpenShot-ComfyUI -OpenShot-ComfyUI provides production-focused ComfyUI nodes built for OpenShot integration, with a strong focus on reliable SAM2 workflows for longer videos. +OpenShot-ComfyUI is a focused set of ComfyUI nodes built for [OpenShot](https://www.openshot.org/). It exists to bring useful modern AI models into real editing workflows with simpler, more reliable, OpenShot-friendly integrations than most demo-oriented community graphs. -The goal is simple: make advanced segmentation and video analysis features feel native inside OpenShot's UI, while keeping the underlying Comfy graphs stable, predictable, and memory-safe. +## Requirements -## Why this exists +- ComfyUI +- PyTorch +- `ffmpeg` / `ffprobe` +- `git` +- Python `3.10` or `3.11` recommended -OpenShot needs SAM2 pipelines that can handle real-world clips, not just short demos. +## Quick install -Many SAM2 custom-node workflows process or retain full-video state in ways that become fragile or memory-heavy as clip length grows. In practice, that can lead to slowdowns, failures, or OOM behavior on longer timelines. +```bash +python -m pip install -r requirements.txt +python -m pip install --no-build-isolation git+https://github.com/facebookresearch/sam2.git -This project addresses that gap with chunk-oriented processing designed specifically for OpenShot's planned UI integration path. +# Validate the install +python validate.py +``` + +Restart ComfyUI after install. ## How this works -- Keep node interfaces close to standard ComfyUI types and patterns. -- Process video segmentation in bounded chunks instead of retaining full-video mask history. -- Return outputs that are easier for OpenShot to consume and orchestrate in larger editing workflows. -- Include practical companion nodes (GroundingDINO + TransNetV2) that support automated, timeline-aware tooling. +- Wrap useful upstream models in simpler OpenShot-friendly nodes. +- Prefer practical, reliable workflows over demo-style graphs. +- Keep installs and first-use model downloads as simple as possible. -## What this includes (V1) +## What this includes - `OpenShotDownloadAndLoadSAM2Model` - `OpenShotSam2Segmentation` (single-image) @@ -30,62 +39,16 @@ This project addresses that gap with chunk-oriented processing designed specific - `OpenShotDeepFilterNetDenoiseAudio` (file-path based audio denoise -> FLAC path) - `OpenShotLavaSRSpeechClarity` (LavaSR speech runner -> FLAC path) -## Attribution +## SAM2 nodes -This project is inspired by and partially based on ideas and APIs from: +These nodes are intended for practical OpenShot segmentation workflows, including image segmentation, promptable video segmentation, and chunk-friendly processing for longer clips. -- `kijai/ComfyUI-segment-anything-2` -- Meta SAM2 research/code +- `OpenShotDownloadAndLoadSAM2Model` downloads and loads supported SAM2 checkpoints +- `OpenShotSam2Segmentation` handles single-image segmentation +- `OpenShotSam2VideoSegmentationAddPoints` adds prompt points for video workflows +- `OpenShotSam2VideoSegmentationChunked` is designed for chunked processing of longer videos -Please see upstream projects for full original implementations and credits. - -## Requirements - -- ComfyUI -- PyTorch (as used by your Comfy install) -- `ffmpeg` and `ffprobe` available on your `PATH` -- `git` available on your `PATH` for installing LavaSR from `requirements.txt` - -Install this node pack into `ComfyUI/custom_nodes/OpenShot-ComfyUI` and restart ComfyUI. - -## Quick install (copy/paste) - -Install this node pack's Python dependencies: - -```bash -python -m pip install -r requirements.txt -``` - -Install SAM2 separately: - -```bash -python -m pip install --no-build-isolation git+https://github.com/facebookresearch/sam2.git -``` - -Validate the environment: - -```bash -python validate.py -``` - -Restart ComfyUI after install. - -`validate.py` supports two cases: - -- On a regular laptop/dev environment, it validates the Python packages plus `ffmpeg`/`ffprobe` -- Inside the actual ComfyUI Python environment, it also validates Comfy imports and node registration - -If you run it in the ComfyUI environment and it passes, restart ComfyUI and the nodes should load. - -SAM2 is installed separately on purpose. Keeping it out of `requirements.txt` makes the normal dependency install much more reliable and avoids long hangs during pip's build-isolation step. - -## First-use model downloads - -- `OpenShotDownloadAndLoadSAM2Model` downloads supported SAM2 checkpoints into `ComfyUI/models/sam2` on first use. -- `OpenShotGroundingDinoDetect` downloads model weights from Hugging Face on first use and uses the normal HF cache. -- `OpenShotDeepFilterNetDenoiseAudio` downloads the default `DeepFilterNet3` model on first use using DeepFilterNet's cache directory. -- `OpenShotLavaSRSpeechClarity` downloads the `YatharthS/LavaSR` model snapshot from Hugging Face on first run. -- `OpenShotTransNetSceneDetect` does not require a separate manual weight download from this node pack. +GroundingDINO and TransNetV2 are included alongside the SAM2 nodes to support object detection and scene boundary workflows that are useful in larger OpenShot pipelines. ## Audio denoise node @@ -111,22 +74,22 @@ The node accepts typical audio formats that `ffmpeg` can decode and always write LavaSR is installed through `requirements.txt` in the main Comfy environment. The first `Clarity -> Speech` run will still take longer because LavaSR downloads its model snapshot from Hugging Face on demand. -## Validation script +## Troubleshooting -Run: +If `python -m pip install -r requirements.txt` fails on `deepfilterlib` with a Rust / Cargo error, you are most likely using Python `3.12+`. + +Simplest fix: + +- use a Python `3.10` or `3.11` Comfy environment + +If you want to keep Python `3.12`, install Rust first and rerun the install: ```bash -python validate.py +curl https://sh.rustup.rs -sSf | sh +source "$HOME/.cargo/env" +python -m pip install -r requirements.txt ``` -It checks: - -- required Python imports -- DeepFilterNet compatibility through the bundled runner shim -- `ffmpeg` and `ffprobe` -- ComfyUI-side imports needed for node registration, when ComfyUI is available -- that the expected node classes are present, when ComfyUI is available - ## Notes - `OpenShotSam2VideoSegmentationChunked` returns only the requested chunk range (bounded memory) instead of collecting whole-video masks. @@ -134,6 +97,22 @@ It checks: - `torchaudio` is listed explicitly because DeepFilterNet imports it internally and some environments do not pull it in automatically. - LavaSR preserves the original channel count by processing each channel independently before recombining the output. +## Acknowledgements + +OpenShot-ComfyUI builds on several excellent open-source projects and model releases. A big thank you to the maintainers, contributors, and researchers behind these repos and models. This node pack would not exist without their work. + +Core upstream repos used by these nodes: + +- [`facebookresearch/sam2`](https://github.com/facebookresearch/sam2) for SAM 2 model code and checkpoints +- [`IDEA-Research/GroundingDINO`](https://github.com/IDEA-Research/GroundingDINO) for GroundingDINO open-set object detection +- [`soCzech/TransNetV2`](https://github.com/soCzech/TransNetV2) and [`transnetv2-pytorch`](https://github.com/soCzech/TransNetV2/tree/master/inference-pytorch) for scene / shot boundary detection +- [`Rikorose/deepfilternet`](https://github.com/Rikorose/DeepFilterNet) for DeepFilterNet audio denoising +- [`ysharma3501/LavaSR`](https://github.com/ysharma3501/LavaSR) for LavaSR speech enhancement +- [`langtech-bsc/vocos`](https://github.com/langtech-bsc/vocos) via LavaSR for the underlying Vocos-based enhancement stack +- [`kijai/ComfyUI-segment-anything-2`](https://github.com/kijai/ComfyUI-segment-anything-2) for integration ideas and surrounding ComfyUI ecosystem work + +Please see the upstream repositories for full original licenses, credits, papers, and model details. + --- Copyright (C) 2026 OpenShot Studios, LLC