You've already forked OpenShot-ComfyUI
mirror of
https://github.com/OpenShot/OpenShot-ComfyUI.git
synced 2026-06-08 22:18:13 -07:00
Large rewrite of our README.md
This commit is contained in:
Jonathan Thomas
@@ -1,25 +1,34 @@
|
||||
# OpenShot-ComfyUI
|
||||
|
||||
OpenShot-ComfyUI provides production-focused ComfyUI nodes built for OpenShot integration, with a strong focus on reliable SAM2 workflows for longer videos.
|
||||
OpenShot-ComfyUI is a focused set of ComfyUI nodes built for [OpenShot](https://www.openshot.org/). It exists to bring useful modern AI models into real editing workflows with simpler, more reliable, OpenShot-friendly integrations than most demo-oriented community graphs.
|
||||
|
||||
The goal is simple: make advanced segmentation and video analysis features feel native inside OpenShot's UI, while keeping the underlying Comfy graphs stable, predictable, and memory-safe.
|
||||
## Requirements
|
||||
|
||||
## Why this exists
|
||||
- ComfyUI
|
||||
- PyTorch
|
||||
- `ffmpeg` / `ffprobe`
|
||||
- `git`
|
||||
- Python `3.10` or `3.11` recommended
|
||||
|
||||
OpenShot needs SAM2 pipelines that can handle real-world clips, not just short demos.
|
||||
## Quick install
|
||||
|
||||
Many SAM2 custom-node workflows process or retain full-video state in ways that become fragile or memory-heavy as clip length grows. In practice, that can lead to slowdowns, failures, or OOM behavior on longer timelines.
|
||||
```bash
|
||||
python -m pip install -r requirements.txt
|
||||
python -m pip install --no-build-isolation git+https://github.com/facebookresearch/sam2.git
|
||||
|
||||
This project addresses that gap with chunk-oriented processing designed specifically for OpenShot's planned UI integration path.
|
||||
# Validate the install
|
||||
python validate.py
|
||||
```
|
||||
|
||||
Restart ComfyUI after install.
|
||||
|
||||
## How this works
|
||||
|
||||
- Keep node interfaces close to standard ComfyUI types and patterns.
|
||||
- Process video segmentation in bounded chunks instead of retaining full-video mask history.
|
||||
- Return outputs that are easier for OpenShot to consume and orchestrate in larger editing workflows.
|
||||
- Include practical companion nodes (GroundingDINO + TransNetV2) that support automated, timeline-aware tooling.
|
||||
- Wrap useful upstream models in simpler OpenShot-friendly nodes.
|
||||
- Prefer practical, reliable workflows over demo-style graphs.
|
||||
- Keep installs and first-use model downloads as simple as possible.
|
||||
|
||||
## What this includes (V1)
|
||||
## What this includes
|
||||
|
||||
- `OpenShotDownloadAndLoadSAM2Model`
|
||||
- `OpenShotSam2Segmentation` (single-image)
|
||||
@@ -30,62 +39,16 @@ This project addresses that gap with chunk-oriented processing designed specific
|
||||
- `OpenShotDeepFilterNetDenoiseAudio` (file-path based audio denoise -> FLAC path)
|
||||
- `OpenShotLavaSRSpeechClarity` (LavaSR speech runner -> FLAC path)
|
||||
|
||||
## Attribution
|
||||
## SAM2 nodes
|
||||
|
||||
This project is inspired by and partially based on ideas and APIs from:
|
||||
These nodes are intended for practical OpenShot segmentation workflows, including image segmentation, promptable video segmentation, and chunk-friendly processing for longer clips.
|
||||
|
||||
- `kijai/ComfyUI-segment-anything-2`
|
||||
- Meta SAM2 research/code
|
||||
- `OpenShotDownloadAndLoadSAM2Model` downloads and loads supported SAM2 checkpoints
|
||||
- `OpenShotSam2Segmentation` handles single-image segmentation
|
||||
- `OpenShotSam2VideoSegmentationAddPoints` adds prompt points for video workflows
|
||||
- `OpenShotSam2VideoSegmentationChunked` is designed for chunked processing of longer videos
|
||||
|
||||
Please see upstream projects for full original implementations and credits.
|
||||
|
||||
## Requirements
|
||||
|
||||
- ComfyUI
|
||||
- PyTorch (as used by your Comfy install)
|
||||
- `ffmpeg` and `ffprobe` available on your `PATH`
|
||||
- `git` available on your `PATH` for installing LavaSR from `requirements.txt`
|
||||
|
||||
Install this node pack into `ComfyUI/custom_nodes/OpenShot-ComfyUI` and restart ComfyUI.
|
||||
|
||||
## Quick install (copy/paste)
|
||||
|
||||
Install this node pack's Python dependencies:
|
||||
|
||||
```bash
|
||||
python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Install SAM2 separately:
|
||||
|
||||
```bash
|
||||
python -m pip install --no-build-isolation git+https://github.com/facebookresearch/sam2.git
|
||||
```
|
||||
|
||||
Validate the environment:
|
||||
|
||||
```bash
|
||||
python validate.py
|
||||
```
|
||||
|
||||
Restart ComfyUI after install.
|
||||
|
||||
`validate.py` supports two cases:
|
||||
|
||||
- On a regular laptop/dev environment, it validates the Python packages plus `ffmpeg`/`ffprobe`
|
||||
- Inside the actual ComfyUI Python environment, it also validates Comfy imports and node registration
|
||||
|
||||
If you run it in the ComfyUI environment and it passes, restart ComfyUI and the nodes should load.
|
||||
|
||||
SAM2 is installed separately on purpose. Keeping it out of `requirements.txt` makes the normal dependency install much more reliable and avoids long hangs during pip's build-isolation step.
|
||||
|
||||
## First-use model downloads
|
||||
|
||||
- `OpenShotDownloadAndLoadSAM2Model` downloads supported SAM2 checkpoints into `ComfyUI/models/sam2` on first use.
|
||||
- `OpenShotGroundingDinoDetect` downloads model weights from Hugging Face on first use and uses the normal HF cache.
|
||||
- `OpenShotDeepFilterNetDenoiseAudio` downloads the default `DeepFilterNet3` model on first use using DeepFilterNet's cache directory.
|
||||
- `OpenShotLavaSRSpeechClarity` downloads the `YatharthS/LavaSR` model snapshot from Hugging Face on first run.
|
||||
- `OpenShotTransNetSceneDetect` does not require a separate manual weight download from this node pack.
|
||||
GroundingDINO and TransNetV2 are included alongside the SAM2 nodes to support object detection and scene boundary workflows that are useful in larger OpenShot pipelines.
|
||||
|
||||
## Audio denoise node
|
||||
|
||||
@@ -111,22 +74,22 @@ The node accepts typical audio formats that `ffmpeg` can decode and always write
|
||||
LavaSR is installed through `requirements.txt` in the main Comfy environment.
|
||||
The first `Clarity -> Speech` run will still take longer because LavaSR downloads its model snapshot from Hugging Face on demand.
|
||||
|
||||
## Validation script
|
||||
## Troubleshooting
|
||||
|
||||
Run:
|
||||
If `python -m pip install -r requirements.txt` fails on `deepfilterlib` with a Rust / Cargo error, you are most likely using Python `3.12+`.
|
||||
|
||||
Simplest fix:
|
||||
|
||||
- use a Python `3.10` or `3.11` Comfy environment
|
||||
|
||||
If you want to keep Python `3.12`, install Rust first and rerun the install:
|
||||
|
||||
```bash
|
||||
python validate.py
|
||||
curl https://sh.rustup.rs -sSf | sh
|
||||
source "$HOME/.cargo/env"
|
||||
python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
It checks:
|
||||
|
||||
- required Python imports
|
||||
- DeepFilterNet compatibility through the bundled runner shim
|
||||
- `ffmpeg` and `ffprobe`
|
||||
- ComfyUI-side imports needed for node registration, when ComfyUI is available
|
||||
- that the expected node classes are present, when ComfyUI is available
|
||||
|
||||
## Notes
|
||||
|
||||
- `OpenShotSam2VideoSegmentationChunked` returns only the requested chunk range (bounded memory) instead of collecting whole-video masks.
|
||||
@@ -134,6 +97,22 @@ It checks:
|
||||
- `torchaudio` is listed explicitly because DeepFilterNet imports it internally and some environments do not pull it in automatically.
|
||||
- LavaSR preserves the original channel count by processing each channel independently before recombining the output.
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
OpenShot-ComfyUI builds on several excellent open-source projects and model releases. A big thank you to the maintainers, contributors, and researchers behind these repos and models. This node pack would not exist without their work.
|
||||
|
||||
Core upstream repos used by these nodes:
|
||||
|
||||
- [`facebookresearch/sam2`](https://github.com/facebookresearch/sam2) for SAM 2 model code and checkpoints
|
||||
- [`IDEA-Research/GroundingDINO`](https://github.com/IDEA-Research/GroundingDINO) for GroundingDINO open-set object detection
|
||||
- [`soCzech/TransNetV2`](https://github.com/soCzech/TransNetV2) and [`transnetv2-pytorch`](https://github.com/soCzech/TransNetV2/tree/master/inference-pytorch) for scene / shot boundary detection
|
||||
- [`Rikorose/deepfilternet`](https://github.com/Rikorose/DeepFilterNet) for DeepFilterNet audio denoising
|
||||
- [`ysharma3501/LavaSR`](https://github.com/ysharma3501/LavaSR) for LavaSR speech enhancement
|
||||
- [`langtech-bsc/vocos`](https://github.com/langtech-bsc/vocos) via LavaSR for the underlying Vocos-based enhancement stack
|
||||
- [`kijai/ComfyUI-segment-anything-2`](https://github.com/kijai/ComfyUI-segment-anything-2) for integration ideas and surrounding ComfyUI ecosystem work
|
||||
|
||||
Please see the upstream repositories for full original licenses, credits, papers, and model details.
|
||||
|
||||
---
|
||||
|
||||
Copyright (C) 2026 OpenShot Studios, LLC
|
||||
|
||||
Reference in New Issue
Block a user