openshot/libopenshot
0
0
Fork 0
mirror of https://github.com/OpenShot/libopenshot.git synced 2026-06-08 22:17:28 -07:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/develop' into qt6-support

Browse Source
This commit is contained in:
Jonathan Thomas
2026-03-12 23:03:07 -05:00
parent 6c1f687afc d43f71c045
commit 867473ab59
55 changed files with 4531 additions and 580 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitlab-ci.yml
+3 -2
View File
@@ -37,6 +37,7 @@ linux-builder:
- cmake -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -D"CMAKE_INSTALL_PREFIX:PATH=$CI_PROJECT_DIR/build/install-x64" -D"PYTHON_MODULE_PATH=python" -D"RUBY_MODULE_PATH=ruby" -DCMAKE_BUILD_TYPE:STRING=Release -DAPPIMAGE_BUILD=1 -DUSE_SYSTEM_JSONCPP=0 ../
- make -j 4
- make install
- ctest --output-on-failure -VV
- make doc
- ~/auto-update-docs "$CI_PROJECT_DIR/build" "$CI_COMMIT_REF_NAME"
- PROJECT_VERSION=$(grep -E '^set\(PROJECT_VERSION_FULL "(.*)' ../CMakeLists.txt | awk '{print $2}' | tr -d '")')
@@ -66,7 +67,7 @@ mac-builder:
- cmake -DCMAKE_EXE_LINKER_FLAGS="-stdlib=libc++" -DCMAKE_SHARED_LINKER_FLAGS="-stdlib=libc++" -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -D"CMAKE_INSTALL_PREFIX:PATH=$CI_PROJECT_DIR/build/install-x64" -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_OSX_SYSROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk" -D"CMAKE_OSX_DEPLOYMENT_TARGET=10.12" -DCMAKE_PREFIX_PATH=/usr/local/qt5.15.X/qt5.15/5.15.0/clang_64/ -D"CMAKE_INSTALL_RPATH_USE_LINK_PATH=1" -D"ENABLE_RUBY=0" ../
- make -j 9
- make install
- make test
- ctest --output-on-failure -VV
- PROJECT_VERSION=$(grep -E '^set\(PROJECT_VERSION_FULL "(.*)' ../CMakeLists.txt | awk '{print $2}' | tr -d '")')
- PROJECT_SO=$(grep -E '^set\(PROJECT_SO_VERSION (.*)' ../CMakeLists.txt | awk '{print $2}' | tr -d ')')
- echo -e "CI_PROJECT_NAME:$CI_PROJECT_NAME\nCI_COMMIT_REF_NAME:$CI_COMMIT_REF_NAME\nCI_COMMIT_SHA:$CI_COMMIT_SHA\nCI_JOB_ID:$CI_JOB_ID\nCI_PIPELINE_ID:$CI_PIPELINE_ID\nVERSION:$PROJECT_VERSION\nSO:$PROJECT_SO" > "install-x64/share/$CI_PROJECT_NAME.env"
@@ -91,7 +92,7 @@ windows-builder-x64:
- $env:MSYSTEM = "MINGW64"
- cmake -B build -S . -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -D"babl_DIR=C:/msys64/mingw64" -D"CMAKE_INSTALL_PREFIX:PATH=$CI_PROJECT_DIR\build\install-x64" -D"OpenShotAudio_ROOT=$CI_PROJECT_DIR\build\install-x64" -D"PYTHON_MODULE_PATH=python" -D"RUBY_MODULE_PATH=ruby" -G "MinGW Makefiles" -D"CMAKE_BUILD_TYPE:STRING=Release"
- cmake --build build -j 4
- cmake --build build --target coverage
- ctest --test-dir build --output-on-failure -VV
- cmake --install build
- $PROJECT_VERSION = (Select-String -Path "CMakeLists.txt" -Pattern '^set\(PROJECT_VERSION_FULL "(.*)\"' | %{$_.Matches.Groups[1].value})
- $PROJECT_SO = (Select-String -Path "CMakeLists.txt" -Pattern '^set\(PROJECT_SO_VERSION (.*)\)' | %{$_.Matches.Groups[1].value})
doc/HW-ACCEL.md
+228 -78
View File
@@ -6,55 +6,187 @@ SPDX-License-Identifier: LGPL-3.0-or-later
## Hardware Acceleration
OpenShot now has experimental support for hardware acceleration, which uses 1 (or more)
graphics cards to offload some of the work for both decoding and encoding. This is
very new and experimental (as of May 2019), but we look forward to "accelerating"
our support for this in the future!
Hardware acceleration in libopenshot allows FFmpeg to use platform-specific GPU
APIs for video decode and encode when available. In practice, this means some of
the work that would otherwise be done by the CPU can be offloaded to the GPU or
to dedicated media blocks on the GPU.
The following table summarizes our current level of support:
This document focuses on what hardware acceleration in libopenshot does today,
how it fits into the current processing pipeline, and what users and developers
should expect from it.
| | Linux Decode | Linux Encode | Mac Decode | Mac Encode | Windows Decode | Windows Encode | Notes |
|--------------------|:---------------:|:--------------:|:----------:|:--------------:|:--------------:|:--------------:|------------------|
| VA-API | ✔️   | ✔️   | - | - | - | - | *Linux Only* |
| VDPAU | ✔️ <sup>1</sup> | ✅ <sup>2</sup> | - | - | - | - | *Linux Only* |
| CUDA (NVDEC/NVENC) | ❌ <sup>3</sup> | ✔️ &nbsp; | - | - | - | ✔️ &nbsp; | *Cross Platform* |
| VideoToolBox | - | - | ✔️ &nbsp; | ❌ <sup>4</sup> | - | - | *Mac Only* |
| DXVA2 | - | - | - | - | ❌ <sup>3</sup> | - | *Windows Only* |
| D3D11VA | - | - | - | - | ❌ <sup>3</sup> | - | *Windows Only* |
| QSV | ❌ <sup>3</sup> | ❌ &nbsp; | ❌ &nbsp; | ❌ &nbsp; | ❌ &nbsp; | ❌ &nbsp; | *Cross Platform* |
## Backend Overview
The following table summarizes the historically supported hardware-acceleration
backends in libopenshot. Actual behavior still depends on FFmpeg build options,
driver availability, operating system support, and the runtime environment.
| | Linux Decode | Linux Encode | macOS Decode | macOS Encode | Windows Decode | Windows Encode | Notes |
|--------------------|:------------:|:------------:|:------------:|:------------:|:--------------:|:--------------:|-------|
| VA-API | ✔️ | ✔️ | - | - | - | - | Linux only |
| VDPAU | ✔️ <sup>1</sup> | ✅ <sup>2</sup> | - | - | - | - | Linux only |
| CUDA (NVDEC/NVENC) | ❌ <sup>3</sup> | ✔️ | - | - | - | ✔️ | Backend availability depends on the FFmpeg build |
| VideoToolbox | - | - | ✔️ | ❌ <sup>4</sup> | - | - | macOS only |
| DXVA2 | - | - | - | - | ❌ <sup>3</sup> | - | Windows only |
| D3D11VA | - | - | - | - | ❌ <sup>3</sup> | - | Windows only |
| QSV | ❌ <sup>3</sup> | ❌ | ❌ | ❌ | ❌ | ❌ | Backend availability depends on the FFmpeg build |
#### Notes
1. VDPAU for some reason needs a card number one higher than it really is
2. VDPAU is a decoder only
3. Green frames (pixel data not correctly tranferred back to system memory)
4. Crashes and burns
1. VDPAU historically needed a card number one higher than expected.
2. VDPAU is decode-only.
3. Historically associated with failed transfers, corrupt frames, or unusable output on some setups.
4. Historically unstable.
This table should be read as a support map, not a guarantee that every backend
is fully validated on every current OS/driver combination.
## Why Hardware Acceleration Exists
Hardware acceleration is useful for two main reasons:
* It can reduce CPU load during decode and encode.
* It can improve throughput for some media, especially on systems with strong
hardware video support.
However, hardware acceleration is not automatically faster for every file or on
every system. The real result depends on codec support, driver quality, stream
format, pixel format, resolution, frame rate, and how much CPU-side work still
needs to happen after decode.
## What libopenshot Uses Hardware Acceleration For
Today, hardware acceleration in libopenshot is primarily used for:
* video decode
* video encode
It is not currently used to keep the entire edit/render pipeline on the GPU.
Decoded frames usually still need to be copied back into system memory for
colorspace conversion, scaling, caching, effect processing, compositing, and
timeline rendering.
That detail is important because it explains why hardware decode does not always
produce a speedup.
## Decode Flow in libopenshot
The current decode flow looks roughly like this:
1. A hardware decoder may be requested through `Settings::HARDWARE_DECODER`.
2. FFmpeg opens the requested hardware decode path if the backend and driver
support it.
3. The decoder produces a frame, either:
* directly as a software-readable frame, or
* as a hardware frame that must be transferred to system memory.
4. libopenshot converts that frame into the CPU-side image representation used
by the rest of the pipeline.
If hardware decode fails during startup decode or frame transfer, libopenshot
falls back to software decode for that reader instead of returning corrupt,
green, or black frames.
## Fallback Behavior
Hardware decode is best-effort, not all-or-nothing.
If a hardware decoder is requested and one of the following happens early in the
decode path:
* repeated startup decode failures
* failed hardware-frame transfer
* invalid transferred frame data
* failed software conversion of a transferred frame
libopenshot reopens that reader in software decode mode and continues decoding.
This behavior is intentionally conservative. The priority is correctness and
stability:
* valid frames are better than corrupt frames
* software fallback is better than black or green output
* a file that cannot be decoded by one hardware backend should still decode if
CPU decoding can handle it
For diagnostics and UI checks, this means there is a difference between:
* decode succeeded
* hardware decode actually produced frames
* hardware decode failed and software fallback was used
`FFmpegReader::HardwareDecodeSuccessful()` exists to expose that distinction.
## Performance Expectations
Hardware decode is not guaranteed to be faster than software decode.
In libopenshot's current pipeline, decoded frames are brought back to
system memory immediately after decode. That introduces costs that can erase or
outweigh the raw decode benefit:
* hardware device setup overhead
* frame transfer overhead between GPU and CPU memory
* colorspace conversion and scaling after decode
* caching and image wrapping in CPU memory
* container/seek behavior and stream structure
Because of that, hardware decode performance is workload-dependent.
General guidance:
* some files benefit from hardware decode
* some files are effectively neutral
* some files are slower with hardware decode
* files with similar codec and resolution can still behave differently
Hardware acceleration should be treated as a capability that may help, not as a
guarantee of better performance.
## Why Some Files Fail on Hardware Decode
A file can fail on hardware decode for several reasons:
* unsupported codec profile
* unsupported chroma format or pixel format
* unsupported bit depth or color range
* driver/backend limitations
* FFmpeg/backend integration quirks on a specific platform
For example, consumer hardware decode paths often handle H.264 4:2:0 very well,
but may not support H.264 4:2:2 decode reliably. In those cases, software decode
may work perfectly while hardware decode fails.
## Supported FFmpeg Versions
* HW accel is supported from FFmpeg version 3.4
* HW accel was removed for nVidia drivers in Ubuntu for FFmpeg 4+
* Hardware acceleration support requires FFmpeg versions new enough to expose the
relevant hardware APIs to libopenshot.
* In practice, decode support in libopenshot relies on FFmpeg's modern send/receive
decode API and hardware-frame APIs.
* Actual backend availability depends on how FFmpeg was compiled on the target system.
**Notice:** The FFmpeg versions of Ubuntu and PPAs for Ubuntu show the
same behaviour. FFmpeg 3 has working nVidia hardware acceleration while
FFmpeg 4+ has no support for nVidia hardware acceleration
included.
Older historical note:
* Some Ubuntu/FFmpeg/NVIDIA combinations behaved differently between FFmpeg 3.x
and FFmpeg 4.x, especially for NVIDIA decode support.
Because backend support has changed over time, always validate against the
actual FFmpeg build and driver stack in use.
## OpenShot Settings
The following settings are use by libopenshot to enable, disable, and control
the various hardware acceleration features.
The following settings are used by libopenshot to enable, disable, and control
hardware acceleration features.
```{cpp}
```cpp
/// Use video codec for faster video decoding (if supported)
int HARDWARE_DECODER = 0;
/* 0 - No acceleration
1 - Linux VA-API
2 - nVidia NVDEC
2 - NVIDIA NVDEC
3 - Windows D3D9
4 - Windows D3D11
5 - MacOS / VideoToolBox
5 - macOS / VideoToolbox
6 - Linux VDPAU
7 - Intel QSV */
@@ -70,73 +202,91 @@ int DE_LIMIT_HEIGHT_MAX = 1100;
/// Maximum columns that hardware decode can handle
int DE_LIMIT_WIDTH_MAX = 1950;
/// Which GPU to use to decode (0 is the first, LINUX ONLY)
/// Which GPU to use to decode (0 is the first, Linux only)
int HW_DE_DEVICE_SET = 0;
/// Which GPU to use to encode (0 is the first, LINUX ONLY)
/// Which GPU to use to encode (0 is the first, Linux only)
int HW_EN_DEVICE_SET = 0;
```
## Libva / VA-API (Video Acceleration API)
## Platform Notes
The correct version of libva is needed (libva in Ubuntu 16.04 or libva2
in Ubuntu 18.04) for the AppImage to work with hardware acceleration.
An AppImage that works on both systems (supporting libva and libva2),
might be possible when no libva is included in the AppImage.
### Linux / VA-API
* vaapi is working for intel and AMD
* vaapi is working for decode only for nouveau
* nVidia driver is working for export only
VA-API is one of the primary Linux hardware-decode paths used by libopenshot.
On supported Intel and AMD systems it can work well, but not every file format,
codec profile, or pixel format is supported by every driver.
## AMD Graphics Cards (RadeonOpenCompute/ROCm)
### Linux / VDPAU
Decoding and encoding on the (AMD) GPU is possible with the default drivers.
On systems where ROCm is installed and run a future use for GPU acceleration
of effects could be implemented (contributions welcome).
VDPAU support exists historically, but behavior can vary with driver and FFmpeg
stack. Treat it as backend-dependent rather than universally reliable.
### NVIDIA
NVIDIA hardware encode support has historically been more reliable than decode
support in libopenshot, depending on FFmpeg build and driver stack. Validate the
actual runtime environment before assuming support.
### macOS / VideoToolbox
VideoToolbox support exists, but stability and feature coverage should be tested
carefully on the target FFmpeg/macOS version.
### Windows / DXVA2 / D3D11VA
Windows decode backends are highly dependent on FFmpeg build options and device
support. They should be treated as runtime-validated features, not assumptions.
## Multiple Graphics Cards
If the computer has multiple graphics cards installed, you can choose which
should be used by libopenshot. Also, you can optionally use one card for
decoding and the other for encoding (if both cards support acceleration).
This is currently only supported on Linux, due to the device name FFmpeg
expects (i.e. **/dev/dri/render128**). Contributions welcome if anyone can
determine what string format to pass for Windows and Mac.
If the computer has multiple graphics cards installed, libopenshot can choose
which device should be used for decode and encode. This is currently practical
mainly on Linux, where FFmpeg expects device names such as `/dev/dri/renderD128`.
## Help Us Improve Hardware Support
Contributions are welcome for improving cross-platform device enumeration and
selection.
This information might be wrong, and we would love to continue improving
our support for hardware acceleration in OpenShot. Please help us update
this document if you find an error or discover new and/or useful information.
## Testing and Validation
**FFmpeg 4 + nVidia** The manual at:
https://www.tal.org/tutorials/ffmpeg_nvidia_encode
works pretty well. We could compile and install a version of FFmpeg 4.1.3
on Mint 19.1 that supports the GPU on nVidia cards. A version of openshot
with hardware support using these libraries could use the nVidia GPU.
When validating hardware decode, check both:
**BUG:** Hardware supported decoding still has some bugs (as you can see from
the chart above). Also, the speed gains with decoding are not as great
as with encoding. Currently, if hardware decoding fails, there is no
fallback (you either get green frames or an "invalid file" error in OpenShot).
This needs to be improved to successfully fall-back to software decoding.
* correctness of the decoded output
* whether hardware decode actually succeeded
**Needed:**
* A way to get options and limits of the GPU, such as
supported dimensions (width and height).
* A way to list the actual Graphic Cards available to FFmpeg (for the
user to choose which card for decoding and encoding, as opposed
to "Graphics Card X")
A frame that looks correct is not enough to prove that hardware acceleration
worked, because software fallback may have rescued the decode.
**Further improvement:** Right now the frame can be decoded on the GPU, but the
frame is then copied to CPU memory for modifications. It is then copied back to
GPU memory for encoding. Using the GPU for both decoding and modifications
will make it possible to do away with these two copies. A possible solution would
be to use Vulkan compute which would be available on Linux and Windows natively
and on MacOS via MoltenVK.
Recommended validation:
* compare output against a software-decode baseline
* track whether hardware decode actually produced frames
* test both a known-good hardware sample and a known-failing fallback sample
## Future Improvements
The biggest architectural limitation today is that decoded frames are generally
copied back to CPU memory for the rest of the pipeline.
Longer-term improvements could include:
* better hardware capability probing
* better device enumeration for users
* broader backend validation across platforms
* keeping more of the pipeline on GPU memory
* GPU-native effects/compositing paths
Avoiding repeated GPU-to-CPU and CPU-to-GPU copies would make hardware
acceleration much more effective for end-to-end editing and export workflows.
## Help Improve This Document
Hardware acceleration support changes with FFmpeg, drivers, operating systems,
and GPU generations. If you find incorrect information or validate a backend on
a newer stack, please update this document.
## Credit
A big thanks to Peter M (https://github.com/eisneinechse) for all his work
on integrating hardware acceleration into libopenshot! The community thanks
you for this major contribution!
A big thanks to Peter M (https://github.com/eisneinechse) for all his work on
integrating hardware acceleration into libopenshot. The community thanks you for
this major contribution.
examples/test_aac_preroll.mp4
BIN
View File
Binary file not shown.
src/Clip.cpp
+38 -15
View File
@@ -25,6 +25,7 @@
#include <algorithm>
#include <cmath>
#include <sstream>
#include <QPainter>
#ifdef USE_IMAGEMAGICK
#include "MagickUtilities.h"
@@ -512,18 +513,36 @@ std::shared_ptr<Frame> Clip::GetFrame(std::shared_ptr<openshot::Frame> backgroun
final_cache.Add(frame);
}
if (!background_frame) {
// Create missing background_frame w/ transparent color (if needed)
background_frame = std::make_shared<Frame>(frame->number, frame->GetWidth(), frame->GetHeight(),
"#00000000", frame->GetAudioSamplesCount(),
frame->GetAudioChannelsCount());
}
const bool has_external_background = (background_frame != nullptr);
// Apply background canvas (i.e. flatten this image onto previous layer image)
apply_background(frame, background_frame);
// Timeline path.
if (options) {
if (!background_frame) {
// Create a transparent background if missing.
background_frame = std::make_shared<Frame>(frame->number, frame->GetWidth(), frame->GetHeight(),
"#00000000", frame->GetAudioSamplesCount(),
frame->GetAudioChannelsCount());
}
if (options->force_safe_composite) {
// Edit mode: composite without mutating cached frame pixels.
apply_background(frame, background_frame, false);
return frame;
}
// Return processed 'frame'
return frame;
// Playback mode: keep original fast path.
apply_background(frame, background_frame, true);
return frame;
}
// No background: return the frame directly.
if (!has_external_background) {
return frame;
}
// External background: composite on a copy.
auto output = std::make_shared<Frame>(*frame.get());
apply_background(output, background_frame, true);
return output;
}
else
// Throw error if reader not initialized
@@ -1263,7 +1282,9 @@ void Clip::RemoveEffect(EffectBase* effect)
}
// Apply background image to the current clip image (i.e. flatten this image onto previous layer)
void Clip::apply_background(std::shared_ptr<openshot::Frame> frame, std::shared_ptr<openshot::Frame> background_frame) {
void Clip::apply_background(std::shared_ptr<openshot::Frame> frame,
std::shared_ptr<openshot::Frame> background_frame,
bool update_frame_image) {
// Add background canvas
std::shared_ptr<QImage> background_canvas = background_frame->GetImage();
QPainter painter(background_canvas.get());
@@ -1273,8 +1294,10 @@ void Clip::apply_background(std::shared_ptr<openshot::Frame> frame, std::shared_
painter.drawImage(0, 0, *frame->GetImage());
painter.end();
// Add new QImage to frame
frame->AddImage(background_canvas);
// Standalone clip requests update frame->image, but timeline composition
// draws onto the timeline-owned background frame only.
if (update_frame_image)
frame->AddImage(background_canvas);
}
// Apply effects to the source frame (if any)
@@ -1284,9 +1307,9 @@ void Clip::apply_effects(std::shared_ptr<Frame> frame, int64_t timeline_frame_nu
{
// Apply the effect to this frame
if (effect->info.apply_before_clip && before_keyframes) {
effect->GetFrame(frame, frame->number);
effect->ProcessFrame(frame, frame->number);
} else if (!effect->info.apply_before_clip && !before_keyframes) {
effect->GetFrame(frame, frame->number);
effect->ProcessFrame(frame, frame->number);
}
}
src/Clip.h
+3 -1
View File
@@ -128,7 +128,9 @@ namespace openshot {
int64_t adjust_frame_number_minimum(int64_t frame_number);
/// Apply background image to the current clip image (i.e. flatten this image onto previous layer)
void apply_background(std::shared_ptr<openshot::Frame> frame, std::shared_ptr<openshot::Frame> background_frame);
void apply_background(std::shared_ptr<openshot::Frame> frame,
std::shared_ptr<openshot::Frame> background_frame,
bool update_frame_image = true);
/// Apply effects to the source frame (if any)
void apply_effects(std::shared_ptr<openshot::Frame> frame, int64_t timeline_frame_number, TimelineInfoStruct* options, bool before_keyframes);
src/EffectBase.cpp
+354
View File
@@ -12,11 +12,24 @@
#include <iostream>
#include <iomanip>
#include <algorithm>
#include <cmath>
#include "EffectBase.h"
#include "Exceptions.h"
#include "Clip.h"
#include "Timeline.h"
#include "ReaderBase.h"
#include "ChunkReader.h"
#include "FFmpegReader.h"
#include "QtImageReader.h"
#include "ZmqLogger.h"
#include <omp.h>
#ifdef USE_IMAGEMAGICK
#include "ImageReader.h"
#endif
using namespace openshot;
@@ -31,6 +44,10 @@ void EffectBase::InitEffectInfo()
Order(0);
ParentClip(NULL);
parentEffect = NULL;
mask_invert = false;
mask_reader = NULL;
mask_time_mode = MASK_TIME_SOURCE_FPS;
mask_loop_mode = MASK_LOOP_PLAY_ONCE;
info.has_video = false;
info.has_audio = false;
@@ -89,6 +106,13 @@ Json::Value EffectBase::JsonValue() const {
root["has_tracked_object"] = info.has_tracked_object;
root["apply_before_clip"] = info.apply_before_clip;
root["order"] = Order();
root["mask_invert"] = mask_invert;
root["mask_time_mode"] = mask_time_mode;
root["mask_loop_mode"] = mask_loop_mode;
if (mask_reader)
root["mask_reader"] = mask_reader->JsonValue();
else
root["mask_reader"] = Json::objectValue;
// return JsonValue
return root;
@@ -141,6 +165,13 @@ void EffectBase::SetJsonValue(const Json::Value root) {
my_root = root;
}
// Legacy compatibility: older shared-mask JSON stored source trim
// separately from the effect trim. Canonical trim now uses ClipBase.
if (my_root["start"].isNull() && !my_root["mask_start"].isNull())
my_root["start"] = my_root["mask_start"];
if (my_root["end"].isNull() && !my_root["mask_end"].isNull())
my_root["end"] = my_root["mask_end"];
// Set parent data
ClipBase::SetJsonValue(my_root);
@@ -151,6 +182,32 @@ void EffectBase::SetJsonValue(const Json::Value root) {
if (!my_root["apply_before_clip"].isNull())
info.apply_before_clip = my_root["apply_before_clip"].asBool();
if (!my_root["mask_invert"].isNull())
mask_invert = my_root["mask_invert"].asBool();
if (!my_root["mask_time_mode"].isNull()) {
const int time_mode = my_root["mask_time_mode"].asInt();
mask_time_mode = (time_mode == MASK_TIME_TIMELINE || time_mode == MASK_TIME_SOURCE_FPS)
? time_mode : MASK_TIME_SOURCE_FPS;
}
if (!my_root["mask_loop_mode"].isNull()) {
const int loop_mode = my_root["mask_loop_mode"].asInt();
if (loop_mode >= MASK_LOOP_PLAY_ONCE && loop_mode <= MASK_LOOP_PING_PONG)
mask_loop_mode = loop_mode;
else
mask_loop_mode = MASK_LOOP_PLAY_ONCE;
}
const Json::Value mask_reader_json =
!my_root["mask_reader"].isNull() ? my_root["mask_reader"] : my_root["reader"];
if (!mask_reader_json.isNull()) {
if (!mask_reader_json["type"].isNull()) {
MaskReader(CreateReaderFromJson(mask_reader_json));
} else if (mask_reader_json.isObject() && mask_reader_json.empty()) {
MaskReader(NULL);
}
}
if (!my_root["parent_effect_id"].isNull()){
info.parent_effect_id = my_root["parent_effect_id"].asString();
if (info.parent_effect_id.size() > 0 && info.parent_effect_id != "" && parentEffect == NULL)
@@ -194,9 +251,300 @@ Json::Value EffectBase::BasePropertiesJSON(int64_t requested_frame) const {
// Set the parent effect which properties this effect will inherit
root["parent_effect_id"] = add_property_json("Parent", 0.0, "string", info.parent_effect_id, NULL, -1, -1, false, requested_frame);
if (info.has_video) {
root["mask_invert"] = add_property_json("Mask: Invert", mask_invert, "int", "", NULL, 0, 1, false, requested_frame);
root["mask_invert"]["choices"].append(add_property_choice_json("Yes", true, mask_invert));
root["mask_invert"]["choices"].append(add_property_choice_json("No", false, mask_invert));
root["mask_time_mode"] = add_property_json("Mask: Time Mode", mask_time_mode, "int", "", NULL, 0, 1, false, requested_frame);
root["mask_time_mode"]["choices"].append(add_property_choice_json("Timeline", MASK_TIME_TIMELINE, mask_time_mode));
root["mask_time_mode"]["choices"].append(add_property_choice_json("Source FPS", MASK_TIME_SOURCE_FPS, mask_time_mode));
root["mask_loop_mode"] = add_property_json("Mask: Loop", mask_loop_mode, "int", "", NULL, 0, 2, false, requested_frame);
root["mask_loop_mode"]["choices"].append(add_property_choice_json("Play Once", MASK_LOOP_PLAY_ONCE, mask_loop_mode));
root["mask_loop_mode"]["choices"].append(add_property_choice_json("Repeat", MASK_LOOP_REPEAT, mask_loop_mode));
root["mask_loop_mode"]["choices"].append(add_property_choice_json("Ping-Pong", MASK_LOOP_PING_PONG, mask_loop_mode));
if (mask_reader)
root["mask_reader"] = add_property_json("Mask: Source", 0.0, "reader", mask_reader->Json(), NULL, 0, 1, false, requested_frame);
else
root["mask_reader"] = add_property_json("Mask: Source", 0.0, "reader", "{}", NULL, 0, 1, false, requested_frame);
}
return root;
}
ReaderBase* EffectBase::CreateReaderFromJson(const Json::Value& reader_json) const {
if (reader_json["type"].isNull())
return NULL;
ReaderBase* reader = NULL;
const std::string type = reader_json["type"].asString();
if (type == "FFmpegReader") {
reader = new FFmpegReader(reader_json["path"].asString());
reader->SetJsonValue(reader_json);
// Mask readers are video-only sources. Disabling audio avoids FFmpeg
// A/V readiness fallbacks that can repeat stale video frames.
reader->info.has_audio = false;
reader->info.audio_stream_index = -1;
} else if (type == "QtImageReader") {
reader = new QtImageReader(reader_json["path"].asString());
reader->SetJsonValue(reader_json);
} else if (type == "ChunkReader") {
reader = new ChunkReader(reader_json["path"].asString(),
static_cast<ChunkVersion>(reader_json["chunk_version"].asInt()));
reader->SetJsonValue(reader_json);
#ifdef USE_IMAGEMAGICK
} else if (type == "ImageReader") {
reader = new ImageReader(reader_json["path"].asString());
reader->SetJsonValue(reader_json);
#endif
}
return reader;
}
void EffectBase::MaskReader(ReaderBase* new_reader) {
if (mask_reader == new_reader)
return;
if (mask_reader) {
mask_reader->Close();
delete mask_reader;
}
mask_reader = new_reader;
cached_single_mask_image.reset();
cached_single_mask_width = 0;
cached_single_mask_height = 0;
if (mask_reader)
mask_reader->ParentClip(clip);
}
double EffectBase::ResolveMaskHostFps() {
if (clip) {
Clip* parent_clip = dynamic_cast<Clip*>(clip);
if (parent_clip && parent_clip->info.fps.num > 0 && parent_clip->info.fps.den > 0)
return parent_clip->info.fps.ToDouble();
}
Timeline* parent_timeline = dynamic_cast<Timeline*>(ParentTimeline());
if (parent_timeline && parent_timeline->info.fps.num > 0 && parent_timeline->info.fps.den > 0)
return parent_timeline->info.fps.ToDouble();
if (mask_reader && mask_reader->info.fps.num > 0 && mask_reader->info.fps.den > 0)
return mask_reader->info.fps.ToDouble();
return 30.0;
}
double EffectBase::ResolveMaskSourceDuration() const {
if (!mask_reader)
return 0.0;
if (mask_reader->info.duration > 0.0f)
return static_cast<double>(mask_reader->info.duration);
if (mask_reader->info.video_length > 0 &&
mask_reader->info.fps.num > 0 && mask_reader->info.fps.den > 0) {
return static_cast<double>(mask_reader->info.video_length) / mask_reader->info.fps.ToDouble();
}
return 0.0;
}
int64_t EffectBase::MapMaskFrameNumber(int64_t frame_number) {
if (!mask_reader)
return frame_number;
int64_t requested_index = std::max(int64_t(0), frame_number - 1);
if (!clip && ParentTimeline()) {
const double host_fps = ResolveMaskHostFps();
if (host_fps > 0.0) {
const int64_t start_offset = static_cast<int64_t>(std::llround(std::max(0.0f, Start()) * host_fps));
requested_index = std::max(int64_t(0), requested_index - start_offset);
}
}
int64_t mapped_index = requested_index;
if (mask_time_mode == MASK_TIME_SOURCE_FPS &&
mask_reader->info.fps.num > 0 && mask_reader->info.fps.den > 0) {
const double host_fps = ResolveMaskHostFps();
const double source_fps = mask_reader->info.fps.ToDouble();
if (host_fps > 0.0 && source_fps > 0.0) {
const double seconds = static_cast<double>(requested_index) / host_fps;
mapped_index = static_cast<int64_t>(std::llround(seconds * source_fps));
}
}
const int64_t source_len = mask_reader->info.video_length;
const double source_fps = (mask_reader->info.fps.num > 0 && mask_reader->info.fps.den > 0)
? mask_reader->info.fps.ToDouble() : 30.0;
const double source_duration = ResolveMaskSourceDuration();
const double start_sec = std::min<double>(std::max(0.0f, Start()), source_duration);
const double end_sec = std::min<double>(std::max(0.0f, End()), source_duration);
const int64_t range_start = std::max(int64_t(1), static_cast<int64_t>(std::llround(start_sec * source_fps)) + 1);
int64_t range_end = (end_sec > 0.0)
? static_cast<int64_t>(std::llround(end_sec * source_fps)) + 1
: source_len;
if (source_len > 0)
range_end = std::min(range_end, source_len);
if (range_end < range_start)
range_end = range_start;
const int64_t range_len = std::max(int64_t(1), range_end - range_start + 1);
int64_t range_index = mapped_index;
switch (mask_loop_mode) {
case MASK_LOOP_REPEAT:
range_index = mapped_index % range_len;
break;
case MASK_LOOP_PING_PONG:
if (range_len > 1) {
const int64_t cycle_len = (range_len * 2) - 2;
int64_t phase = mapped_index % cycle_len;
if (phase >= range_len)
phase = cycle_len - phase;
range_index = phase;
} else {
range_index = 0;
}
break;
case MASK_LOOP_PLAY_ONCE:
default:
if (mapped_index < 0)
range_index = 0;
else if (mapped_index >= range_len)
range_index = range_len - 1;
else
range_index = mapped_index;
break;
}
int64_t mapped_frame = range_start + range_index;
if (source_len > 0)
mapped_frame = std::min(std::max(int64_t(1), mapped_frame), source_len);
return std::max(int64_t(1), mapped_frame);
}
std::shared_ptr<QImage> EffectBase::GetMaskImage(std::shared_ptr<QImage> target_image, int64_t frame_number) {
if (!mask_reader || !target_image || target_image->isNull())
return {};
std::shared_ptr<QImage> source_mask;
bool used_cached_scaled = false;
#pragma omp critical (open_effect_mask_reader)
{
try {
if (!mask_reader->IsOpen())
mask_reader->Open();
if (mask_reader->info.has_single_image &&
cached_single_mask_image &&
cached_single_mask_width == target_image->width() &&
cached_single_mask_height == target_image->height()) {
source_mask = cached_single_mask_image;
used_cached_scaled = true;
}
else {
const int64_t mapped_frame = MapMaskFrameNumber(frame_number);
auto source_frame = mask_reader->GetFrame(mapped_frame);
if (source_frame && source_frame->GetImage() && !source_frame->GetImage()->isNull())
source_mask = std::make_shared<QImage>(*source_frame->GetImage());
}
} catch (const std::exception& e) {
ZmqLogger::Instance()->Log(
std::string("EffectBase::GetMaskImage unable to read mask frame: ") + e.what());
source_mask.reset();
}
}
if (!source_mask || source_mask->isNull())
return {};
if (used_cached_scaled)
return source_mask;
auto scaled_mask = std::make_shared<QImage>(
source_mask->scaled(
target_image->width(), target_image->height(),
Qt::IgnoreAspectRatio, Qt::SmoothTransformation));
if (mask_reader->info.has_single_image) {
cached_single_mask_image = scaled_mask;
cached_single_mask_width = target_image->width();
cached_single_mask_height = target_image->height();
}
return scaled_mask;
}
void EffectBase::BlendWithMask(std::shared_ptr<QImage> original_image, std::shared_ptr<QImage> effected_image,
std::shared_ptr<QImage> mask_image) const {
if (!original_image || !effected_image || !mask_image)
return;
if (original_image->size() != effected_image->size() || effected_image->size() != mask_image->size())
return;
unsigned char* original_pixels = reinterpret_cast<unsigned char*>(original_image->bits());
unsigned char* effected_pixels = reinterpret_cast<unsigned char*>(effected_image->bits());
unsigned char* mask_pixels = reinterpret_cast<unsigned char*>(mask_image->bits());
const int pixel_count = effected_image->width() * effected_image->height();
#pragma omp parallel for schedule(static)
for (int i = 0; i < pixel_count; ++i) {
const int idx = i * 4;
int gray = qGray(mask_pixels[idx], mask_pixels[idx + 1], mask_pixels[idx + 2]);
if (mask_invert)
gray = 255 - gray;
const float factor = static_cast<float>(gray) / 255.0f;
const float inverse = 1.0f - factor;
effected_pixels[idx] = static_cast<unsigned char>(
(original_pixels[idx] * inverse) + (effected_pixels[idx] * factor));
effected_pixels[idx + 1] = static_cast<unsigned char>(
(original_pixels[idx + 1] * inverse) + (effected_pixels[idx + 1] * factor));
effected_pixels[idx + 2] = static_cast<unsigned char>(
(original_pixels[idx + 2] * inverse) + (effected_pixels[idx + 2] * factor));
effected_pixels[idx + 3] = static_cast<unsigned char>(
(original_pixels[idx + 3] * inverse) + (effected_pixels[idx + 3] * factor));
}
}
std::shared_ptr<openshot::Frame> EffectBase::ProcessFrame(std::shared_ptr<openshot::Frame> frame, int64_t frame_number) {
// Audio-only effects skip common mask handling.
if (!info.has_video || !mask_reader)
return GetFrame(frame, frame_number);
// Effects that already apply masks inside GetFrame() should bypass common blend handling.
if (HandlesMaskInternally())
return GetFrame(frame, frame_number);
auto pre_image = frame->GetImage();
if (!pre_image || pre_image->isNull())
return GetFrame(frame, frame_number);
const auto original_image = std::make_shared<QImage>(pre_image->copy());
auto output_frame = GetFrame(frame, frame_number);
if (!output_frame)
return output_frame;
auto effected_image = output_frame->GetImage();
if (!effected_image || effected_image->isNull() ||
effected_image->size() != original_image->size())
return output_frame;
auto mask_image = GetMaskImage(effected_image, frame_number);
if (!mask_image || mask_image->isNull())
return output_frame;
if (UseCustomMaskBlend(frame_number))
ApplyCustomMaskBlend(original_image, effected_image, mask_image, frame_number);
else
BlendWithMask(original_image, effected_image, mask_image);
return output_frame;
}
/// Parent clip object of this reader (which can be unparented and NULL)
openshot::ClipBase* EffectBase::ParentClip() {
return clip;
@@ -205,6 +553,8 @@ openshot::ClipBase* EffectBase::ParentClip() {
/// Set parent clip object of this reader
void EffectBase::ParentClip(openshot::ClipBase* new_clip) {
clip = new_clip;
if (mask_reader)
mask_reader->ParentClip(new_clip);
}
// Set the parent effect from which this properties will be set to
@@ -239,3 +589,7 @@ std::string EffectBase::ParentClipId() const{
else
return "";
}
EffectBase::~EffectBase() {
MaskReader(NULL);
}
src/EffectBase.h
+67 -1
View File
@@ -18,12 +18,15 @@
#include "Json.h"
#include "TrackedObjectBase.h"
#include <QImage>
#include <memory>
#include <map>
#include <string>
namespace openshot
{
class ReaderBase;
/**
* @brief This struct contains info about an effect, such as the name, video or audio effect, etc...
*
@@ -54,10 +57,48 @@ namespace openshot
{
private:
int order; ///< The order to evaluate this effect. Effects are processed in this order (when more than one overlap).
ReaderBase* mask_reader = nullptr; ///< Optional common reader-based mask source.
std::shared_ptr<QImage> cached_single_mask_image; ///< Cached scaled mask for still-image mask sources.
int cached_single_mask_width = 0; ///< Cached mask width.
int cached_single_mask_height = 0; ///< Cached mask height.
/// Build or refresh a mask image that matches target_image dimensions.
std::shared_ptr<QImage> GetMaskImage(std::shared_ptr<QImage> target_image, int64_t frame_number);
/// Blend original and effected images using mask values.
void BlendWithMask(std::shared_ptr<QImage> original_image, std::shared_ptr<QImage> effected_image,
std::shared_ptr<QImage> mask_image) const;
protected:
openshot::ClipBase* clip; ///< Pointer to the parent clip instance (if any)
/// Create a reader instance from reader JSON.
ReaderBase* CreateReaderFromJson(const Json::Value& reader_json) const;
/// Convert an effect frame number to a mask source frame number.
int64_t MapMaskFrameNumber(int64_t frame_number);
/// Determine host FPS used to convert timeline frames to mask source FPS.
double ResolveMaskHostFps();
/// Determine mask source duration in seconds.
double ResolveMaskSourceDuration() const;
/// Resolve a cached/scaled mask image for the target frame dimensions.
std::shared_ptr<QImage> ResolveMaskImage(std::shared_ptr<QImage> target_image, int64_t frame_number) {
return GetMaskImage(target_image, frame_number);
}
/// Optional override for effects that need custom mask behavior.
virtual bool UseCustomMaskBlend(int64_t frame_number) const { return false; }
/// Optional override for effects with custom mask implementation.
virtual void ApplyCustomMaskBlend(std::shared_ptr<QImage> original_image, std::shared_ptr<QImage> effected_image,
std::shared_ptr<QImage> mask_image, int64_t frame_number) const {}
/// Optional override for effects that apply mask processing inside GetFrame().
virtual bool HandlesMaskInternally() const { return false; }
public:
/// Parent effect (which properties will set this effect properties)
EffectBase* parentEffect;
@@ -67,6 +108,21 @@ namespace openshot
/// Information about the current effect
EffectInfoStruct info;
bool mask_invert = false; ///< Invert grayscale mask values before blending.
enum MaskTimeMode {
MASK_TIME_TIMELINE = 0,
MASK_TIME_SOURCE_FPS = 1
};
enum MaskLoopMode {
MASK_LOOP_PLAY_ONCE = 0,
MASK_LOOP_REPEAT = 1,
MASK_LOOP_PING_PONG = 2
};
int mask_time_mode = MASK_TIME_SOURCE_FPS; ///< How effect frames map to mask source frames.
int mask_loop_mode = MASK_LOOP_PLAY_ONCE; ///< Behavior when mask range reaches the end.
/// Display effect information in the standard output stream (stdout)
void DisplayInfo(std::ostream* out=&std::cout);
@@ -112,13 +168,23 @@ namespace openshot
/// Generate JSON object of base properties (recommended to be used by all effects)
Json::Value BasePropertiesJSON(int64_t requested_frame) const;
/// Apply effect processing with common mask support (if enabled).
std::shared_ptr<openshot::Frame> ProcessFrame(std::shared_ptr<openshot::Frame> frame, int64_t frame_number);
/// Get the common mask reader.
ReaderBase* MaskReader() { return mask_reader; }
const ReaderBase* MaskReader() const { return mask_reader; }
/// Set or replace the common mask reader.
void MaskReader(ReaderBase* new_reader);
/// Get the order that this effect should be executed.
int Order() const { return order; }
/// Set the order that this effect should be executed.
void Order(int new_order) { order = new_order; }
virtual ~EffectBase() = default;
virtual ~EffectBase();
};
}
src/Enums.h
+2 -1
View File
@@ -168,7 +168,8 @@ enum ChromaKeyMethod
CHROMAKEY_CIE_LCH_H, ///< Difference between CIE LCH(ab) hues
CHROMAKEY_CIE_DISTANCE, ///< CIEDE2000 perceptual difference
CHROMAKEY_YCBCR, ///< YCbCr vector difference of CbCr
CHROMAKEY_LAST_METHOD = CHROMAKEY_YCBCR
CHROMAKEY_BASIC_SOFT, ///< BASIC metric + optional halo feathering
CHROMAKEY_LAST_METHOD = CHROMAKEY_BASIC_SOFT
};
} // namespace openshot
src/Exceptions.h
+12 -6
View File
@@ -62,15 +62,21 @@ namespace openshot {
{
public:
std::string file_path;
std::string full_message;
FileExceptionBase(std::string message, std::string file_path="")
: ExceptionBase(message), file_path(file_path) { }
virtual std::string py_message() const override {
// return complete message for Python exception handling
std::string out_msg(m_message +
: ExceptionBase(message),
file_path(file_path),
full_message(message +
(file_path != ""
? " for file " + file_path
: ""));
return out_msg;
: "")) { }
virtual const char* what() const noexcept override {
// Include file path in native C++ exception output (stderr / terminate).
return full_message.c_str();
}
virtual std::string py_message() const override {
// Keep Python exception output consistent with what().
return full_message;
}
};
src/FFmpegReader.cpp
+293 -47
View File
File diff suppressed because it is too large Load Diff
src/FFmpegReader.h
+10
View File
@@ -165,6 +165,10 @@ namespace openshot {
AVFrame *pFrameRGB_cached = nullptr; ///< Temporary frame used for video conversion
int hw_de_supported = 0; // Is set by FFmpegReader
bool force_sw_decode = false;
bool hw_decode_failed = false;
int hw_decode_error_count = 0;
bool hw_decode_succeeded = false;
#if USE_HW_ACCEL
AVPixelFormat hw_de_av_pix_fmt = AV_PIX_FMT_NONE;
AVHWDeviceType hw_de_av_device_type = AV_HWDEVICE_TYPE_NONE;
@@ -198,6 +202,9 @@ namespace openshot {
/// Get an AVFrame (if any)
bool GetAVFrame();
/// Reopen the current reader with software decode after hardware decode fails
bool ReopenWithoutHardwareDecode(int64_t requested_frame);
/// Get the next packet (if any)
int GetNextPacket();
@@ -285,6 +292,9 @@ namespace openshot {
/// Determine if reader is open or closed
bool IsOpen() override { return is_open; };
/// Return true if hardware decode was requested and successfully produced at least one frame
bool HardwareDecodeSuccessful() const override;
/// Return the type name of the class
std::string Name() override { return "FFmpegReader"; };
src/FFmpegWriter.cpp
+13 -4
View File
@@ -79,7 +79,7 @@ FFmpegWriter::FFmpegWriter(const std::string& path) :
initial_audio_input_frame_size(0), img_convert_ctx(NULL),
video_codec_ctx(NULL), audio_codec_ctx(NULL), is_writing(false), video_timestamp(0), audio_timestamp(0),
original_sample_rate(0), original_channels(0), avr(NULL), avr_planar(NULL), is_open(false), prepare_streams(false),
write_header(false), write_trailer(false), audio_encoder_buffer_size(0), audio_encoder_buffer(NULL) {
write_header(false), write_trailer(false), allow_b_frames(false), audio_encoder_buffer_size(0), audio_encoder_buffer(NULL) {
// Disable audio & video (so they can be independently enabled)
info.has_audio = false;
@@ -368,7 +368,7 @@ void FFmpegWriter::SetOption(StreamType stream, std::string name, std::string va
// Was option found?
if (option || (name == "g" || name == "qmin" || name == "qmax" || name == "max_b_frames" || name == "mb_decision" ||
name == "level" || name == "profile" || name == "slices" || name == "rc_min_rate" || name == "rc_max_rate" ||
name == "rc_buffer_size" || name == "crf" || name == "cqp" || name == "qp")) {
name == "rc_buffer_size" || name == "crf" || name == "cqp" || name == "qp" || name == "allow_b_frames")) {
// Check for specific named options
if (name == "g")
// Set gop_size
@@ -386,6 +386,11 @@ void FFmpegWriter::SetOption(StreamType stream, std::string name, std::string va
// Maximum number of B-frames between non-B-frames
convert >> c->max_b_frames;
else if (name == "allow_b_frames")
// Preserve configured B-frames for codecs that support them.
// Values: 1/true/yes/on to enable, everything else disables.
allow_b_frames = (value == "1" || value == "true" || value == "yes" || value == "on");
else if (name == "mb_decision")
// Macroblock decision mode
convert >> c->mb_decision;
@@ -1468,8 +1473,12 @@ void FFmpegWriter::open_video(AVFormatContext *oc, AVStream *st) {
if (!codec)
throw InvalidCodec("Could not find codec", path);
/* Force max_b_frames to 0 in some cases (i.e. for mjpeg image sequences */
if (video_codec_ctx->max_b_frames && video_codec_ctx->codec_id != AV_CODEC_ID_MPEG4 && video_codec_ctx->codec_id != AV_CODEC_ID_MPEG1VIDEO && video_codec_ctx->codec_id != AV_CODEC_ID_MPEG2VIDEO)
/* Legacy behavior: force max_b_frames to 0 for many codecs.
* This can be disabled via SetOption(VIDEO_STREAM, "allow_b_frames", "1"). */
if (!allow_b_frames && video_codec_ctx->max_b_frames &&
video_codec_ctx->codec_id != AV_CODEC_ID_MPEG4 &&
video_codec_ctx->codec_id != AV_CODEC_ID_MPEG1VIDEO &&
video_codec_ctx->codec_id != AV_CODEC_ID_MPEG2VIDEO)
video_codec_ctx->max_b_frames = 0;
// Init options
src/FFmpegWriter.h
+1
View File
@@ -124,6 +124,7 @@ namespace openshot {
bool prepare_streams;
bool write_header;
bool write_trailer;
bool allow_b_frames;
AVFormatContext* oc;
AVStream *audio_st, *video_st;
src/FrameMapper.cpp
-7
View File
@@ -800,13 +800,6 @@ void FrameMapper::SetJsonValue(const Json::Value root) {
// Set parent data
ReaderBase::SetJsonValue(root);
// Re-Open path, and re-init everything (if needed)
if (reader) {
Close();
Open();
}
}
// Change frame rate or audio mapping details
src/ImageReader.cpp
+1 -1
View File
@@ -49,7 +49,7 @@ void ImageReader::Open()
}
catch (const Magick::Exception& e) {
// raise exception
throw InvalidFile("File could not be opened.", path);
throw InvalidFile("ImageReader could not open image file.", path);
}
// Update image properties
src/Profiles.cpp
+2 -2
View File
@@ -129,13 +129,13 @@ Profile::Profile(std::string path) {
catch (const std::exception& e)
{
// Error parsing profile file
throw InvalidFile("Profile could not be found or loaded (or is invalid).", path);
throw InvalidFile("Profile file could not be parsed (invalid format or values).", path);
}
// Throw error if file was not read
if (!read_file)
// Error parsing profile file
throw InvalidFile("Profile could not be found or loaded (or is invalid).", path);
throw InvalidFile("Profile file could not be found or opened.", path);
}
// Return a formatted FPS
src/Qt/PlayerPrivate.cpp
+28 -5
View File
@@ -74,9 +74,13 @@ namespace openshot
// Pausing Code (which re-syncs audio/video times)
// - If speed is zero or speed changes
// - If pre-roll is not ready (This should allow scrubbing of the timeline without waiting on pre-roll)
if ((speed == 0 && video_position == last_video_position) ||
(speed != 0 && last_speed != speed) ||
(speed != 0 && !is_dirty && !videoCache->isReady()))
bool wait_paused_hold = (speed == 0 && video_position == last_video_position);
bool wait_speed_change = (speed != 0 && last_speed != speed);
bool cache_ready = videoCache->isReady();
bool wait_preroll = (speed != 0 && !is_dirty && !cache_ready);
bool should_wait = (wait_paused_hold || wait_speed_change || wait_preroll);
if (should_wait)
{
// Sleep for a fraction of frame duration
std::this_thread::sleep_for(frame_duration / 4);
@@ -97,7 +101,16 @@ namespace openshot
// Set the video frame on the video thread and render frame
videoPlayback->frame = frame;
videoPlayback->rendered.reset();
videoPlayback->render.signal();
// Keep decode/position advancement aligned with actual preview updates.
// This avoids occasional "silent advance then jump" behavior when
// preroll transitions and rendering are briefly out-of-sync.
const int render_wait_ms = std::max(
1,
static_cast<int>(frame_duration.count() / 1000.0 * 2.0)
);
videoPlayback->rendered.wait(render_wait_ms);
// Keep track of the last displayed frame
last_video_position = video_position;
@@ -116,6 +129,12 @@ namespace openshot
// Protect against invalid or too-long sleep times
std::this_thread::sleep_for(max_sleep);
}
} else {
// If we're behind schedule (e.g. preroll/render stall), do not
// burst through delayed frames. Resync timing baseline so
// playback continues smoothly at normal cadence.
start_time = std::chrono::time_point_cast<micro_sec>(current_time);
playback_frames = 0;
}
}
}
@@ -150,8 +169,8 @@ namespace openshot
// Increment playback frames (always in the positive direction)
playback_frames += std::abs(speed);
// Update cache on which frame was retrieved
videoCache->Seek(video_position);
// Update playhead hint for cache window tracking without triggering seek behavior.
videoCache->NotifyPlaybackPosition(video_position);
// return frame from reader
return reader->GetFrame(video_position);
@@ -170,6 +189,10 @@ namespace openshot
{
video_position = new_position;
last_video_position = 0;
// Drop local frame reference so same-frame refreshes cannot reuse stale
// content after timeline/clip property updates.
frame.reset();
// Always force immediate refresh after seek/update, even while playing.
is_dirty = true;
}
src/Qt/VideoCacheThread.cpp
+274 -27
View File
@@ -30,6 +30,8 @@ namespace openshot
, last_dir(1) // assume forward (+1) on first launch
, userSeeked(false)
, preroll_on_next_fill(false)
, clear_cache_on_next_fill(false)
, scrub_active(false)
, requested_display_frame(1)
, current_display_frame(1)
, cached_frame_count(0)
@@ -38,6 +40,8 @@ namespace openshot
, reader(nullptr)
, force_directional_cache(false)
, last_cached_index(0)
, seen_timeline_cache_epoch(0)
, timeline_cache_epoch_initialized(false)
{
}
@@ -59,7 +63,7 @@ namespace openshot
}
const int64_t cached_index = last_cached_index.load();
const int64_t playhead = requested_display_frame.load();
int64_t playhead = requested_display_frame.load();
int dir = computeDirection();
// Near timeline boundaries, don't require more pre-roll than can exist.
@@ -73,6 +77,7 @@ namespace openshot
if (max_frame < 1) {
return false;
}
playhead = clampToTimelineRange(playhead, max_frame);
int64_t required_ahead = ready_min;
int64_t available_ahead = (dir > 0)
@@ -92,6 +97,8 @@ namespace openshot
if (new_speed != 0) {
last_speed.store(new_speed);
last_dir.store(new_speed > 0 ? 1 : -1);
// Leaving paused/scrub context: resume normal cache behavior.
scrub_active.store(false);
}
speed.store(new_speed);
}
@@ -126,45 +133,176 @@ namespace openshot
return !isThreadRunning();
}
void VideoCacheThread::Reader(ReaderBase* new_reader)
{
std::lock_guard<std::mutex> guard(seek_state_mutex);
reader = new_reader;
seen_timeline_cache_epoch = 0;
timeline_cache_epoch_initialized = false;
Play();
}
void VideoCacheThread::Seek(int64_t new_position, bool start_preroll)
{
const int64_t timeline_end = resolveTimelineEnd();
const int64_t clamped_new_position = clampToTimelineRange(new_position, timeline_end);
const int64_t current_requested = requested_display_frame.load();
bool should_mark_seek = false;
bool should_preroll = false;
int64_t new_cached_count = cached_frame_count.load();
bool entering_scrub = false;
bool leaving_scrub = false;
bool cache_contains = false;
bool should_clear_cache = false;
CacheBase* cache = reader ? reader->GetCache() : nullptr;
const bool same_frame_refresh = (new_position == current_requested);
if (cache) {
cache_contains = cache->Contains(clamped_new_position);
}
if (start_preroll) {
should_mark_seek = true;
CacheBase* cache = reader ? reader->GetCache() : nullptr;
if (cache && !cache->Contains(new_position))
{
// If user initiated seek, and current frame not found (
if (Timeline* timeline = dynamic_cast<Timeline*>(reader)) {
timeline->ClearAllCache();
if (same_frame_refresh) {
const bool is_paused = (speed.load() == 0);
if (is_paused) {
const bool was_scrubbing = scrub_active.load();
if (was_scrubbing && cache && cache_contains) {
// Preserve in-range cache for paused scrub preview -> same-frame commit.
should_mark_seek = false;
should_preroll = false;
should_clear_cache = false;
new_cached_count = cache->Count();
} else {
// Paused same-frame edit refresh: force full cache refresh.
if (Timeline* timeline = dynamic_cast<Timeline*>(reader)) {
timeline->ClearAllCache();
}
new_cached_count = 0;
should_mark_seek = true;
should_preroll = true;
should_clear_cache = false;
}
} else {
// Same-frame refresh during playback should stay lightweight.
should_mark_seek = false;
should_preroll = false;
should_clear_cache = false;
if (cache && cache_contains) {
cache->Remove(clamped_new_position);
}
if (cache) {
new_cached_count = cache->Count();
}
}
} else {
if (cache && !cache_contains) {
should_mark_seek = true;
// Uncached commit seek: defer cache clear to cache thread loop.
new_cached_count = 0;
should_preroll = true;
should_clear_cache = true;
}
else if (cache)
{
// In-range commit seek preserves cache window/baseline.
should_mark_seek = false;
should_preroll = false;
should_clear_cache = false;
new_cached_count = cache->Count();
} else {
// No cache object to query: use normal seek behavior.
should_mark_seek = true;
}
new_cached_count = 0;
should_preroll = true;
}
else if (cache)
{
new_cached_count = cache->Count();
leaving_scrub = true;
}
else {
// Non-preroll seeks cover paused scrubbing and live playback refresh.
const bool is_paused = (speed.load() == 0);
if (is_paused && same_frame_refresh) {
// Same-frame paused refresh updates only that frame.
should_mark_seek = false;
should_preroll = false;
should_clear_cache = false;
if (cache && cache_contains) {
cache->Remove(clamped_new_position);
}
if (cache) {
new_cached_count = cache->Count();
}
leaving_scrub = true;
}
else if (is_paused) {
if (cache && !cache_contains) {
should_mark_seek = true;
new_cached_count = 0;
should_clear_cache = true;
}
else if (cache) {
// In-range paused seek preserves cache continuity.
should_mark_seek = false;
new_cached_count = cache->Count();
} else {
should_mark_seek = true;
}
entering_scrub = true;
} else {
// During playback, keep seek/scrub side effects minimal.
should_mark_seek = false;
should_preroll = false;
should_clear_cache = false;
if (cache) {
new_cached_count = cache->Count();
}
leaving_scrub = true;
}
}
{
std::lock_guard<std::mutex> guard(seek_state_mutex);
// Reset readiness baseline only when rebuilding cache.
const int dir = computeDirection();
if (should_mark_seek || should_preroll || should_clear_cache) {
last_cached_index.store(clamped_new_position - dir);
}
requested_display_frame.store(new_position);
cached_frame_count.store(new_cached_count);
if (start_preroll) {
preroll_on_next_fill.store(should_preroll);
userSeeked.store(should_mark_seek);
preroll_on_next_fill.store(should_preroll);
// Clear behavior follows the latest seek intent.
clear_cache_on_next_fill.store(should_clear_cache);
userSeeked.store(should_mark_seek);
if (entering_scrub) {
scrub_active.store(true);
}
if (leaving_scrub) {
scrub_active.store(false);
}
}
}
void VideoCacheThread::Seek(int64_t new_position)
{
Seek(new_position, false);
NotifyPlaybackPosition(new_position);
}
void VideoCacheThread::NotifyPlaybackPosition(int64_t new_position)
{
if (new_position <= 0) {
return;
}
if (scrub_active.load()) {
return;
}
int64_t new_cached_count = cached_frame_count.load();
if (CacheBase* cache = reader ? reader->GetCache() : nullptr) {
new_cached_count = cache->Count();
}
{
std::lock_guard<std::mutex> guard(seek_state_mutex);
requested_display_frame.store(new_position);
cached_frame_count.store(new_cached_count);
}
}
int VideoCacheThread::computeDirection() const
@@ -216,11 +354,39 @@ namespace openshot
return min_frames;
}
int64_t VideoCacheThread::resolveTimelineEnd() const
{
if (!reader) {
return 0;
}
int64_t timeline_end = reader->info.video_length;
if (auto* timeline = dynamic_cast<Timeline*>(reader)) {
const int64_t timeline_max = timeline->GetMaxFrame();
if (timeline_max > 0) {
timeline_end = timeline_max;
}
}
return timeline_end;
}
int64_t VideoCacheThread::clampToTimelineRange(int64_t frame, int64_t timeline_end) const
{
if (timeline_end < 1) {
return frame;
}
return std::clamp<int64_t>(frame, 1, timeline_end);
}
bool VideoCacheThread::clearCacheIfPaused(int64_t playhead,
bool paused,
CacheBase* cache)
{
if (paused && !cache->Contains(playhead)) {
const int64_t timeline_end = resolveTimelineEnd();
int64_t cache_playhead = playhead;
if (reader) {
cache_playhead = clampToTimelineRange(playhead, timeline_end);
}
if (paused && !cache->Contains(cache_playhead)) {
// If paused and playhead not in cache, clear everything
if (Timeline* timeline = dynamic_cast<Timeline*>(reader)) {
timeline->ClearAllCache();
@@ -257,10 +423,12 @@ namespace openshot
int64_t window_begin,
int64_t window_end,
int dir,
ReaderBase* reader)
ReaderBase* reader,
int64_t max_frames_to_fetch)
{
bool window_full = true;
int64_t next_frame = last_cached_index.load() + dir;
int64_t fetched_this_pass = 0;
// Advance from last_cached_index toward window boundary
while ((dir > 0 && next_frame <= window_end) ||
@@ -280,6 +448,7 @@ namespace openshot
auto framePtr = reader->GetFrame(next_frame);
cache->Add(framePtr);
cached_frame_count.store(cache->Count());
++fetched_this_pass;
}
catch (const OutOfBoundsFrame&) {
break;
@@ -292,6 +461,12 @@ namespace openshot
last_cached_index.store(next_frame);
next_frame += dir;
// In active playback, avoid long uninterrupted prefetch bursts
// that can delay player thread frame retrieval.
if (max_frames_to_fetch > 0 && fetched_this_pass >= max_frames_to_fetch) {
break;
}
}
return window_full;
@@ -305,6 +480,22 @@ namespace openshot
while (!threadShouldExit()) {
Settings* settings = Settings::Instance();
CacheBase* cache = reader ? reader->GetCache() : nullptr;
Timeline* timeline = dynamic_cast<Timeline*>(reader);
// Process deferred clears even when caching is currently disabled
// (e.g. active scrub mode), so stale ranges are removed promptly.
bool should_clear_cache = clear_cache_on_next_fill.exchange(false);
if (should_clear_cache && timeline) {
const int dir_on_clear = computeDirection();
const int64_t clear_playhead = clampToTimelineRange(
requested_display_frame.load(), resolveTimelineEnd());
timeline->ClearAllCache();
cached_frame_count.store(0);
// Reset ready baseline immediately after clear. Otherwise a
// stale last_cached_index from the old cache window can make
// isReady() report true before new preroll is actually filled.
last_cached_index.store(clear_playhead - dir_on_clear);
}
// If caching disabled or no reader, mark cache as ready and sleep briefly
if (!settings->ENABLE_PLAYBACK_CACHING || !cache) {
@@ -317,13 +508,13 @@ namespace openshot
// init local vars
min_frames_ahead.store(settings->VIDEO_CACHE_MIN_PREROLL_FRAMES);
Timeline* timeline = dynamic_cast<Timeline*>(reader);
if (!timeline) {
std::this_thread::sleep_for(double_micro_sec(50000));
continue;
}
int64_t timeline_end = timeline->GetMaxFrame();
int64_t playhead = requested_display_frame.load();
int64_t timeline_end = resolveTimelineEnd();
int64_t raw_playhead = requested_display_frame.load();
int64_t playhead = clampToTimelineRange(raw_playhead, timeline_end);
bool paused = (speed.load() == 0);
int64_t preroll_frames = computePrerollFrames(settings);
@@ -335,6 +526,27 @@ namespace openshot
last_dir.store(dir);
}
// If timeline-side cache invalidation occurred (e.g. ApplyJsonDiff / SetJson),
// restart fill from the active playhead window so invalidated gaps self-heal.
if (timeline) {
bool epoch_changed = false;
{
std::lock_guard<std::mutex> guard(seek_state_mutex);
const uint64_t timeline_epoch = timeline->CacheEpoch();
if (!timeline_cache_epoch_initialized) {
seen_timeline_cache_epoch = timeline_epoch;
timeline_cache_epoch_initialized = true;
}
else if (timeline_epoch != seen_timeline_cache_epoch) {
seen_timeline_cache_epoch = timeline_epoch;
epoch_changed = true;
}
}
if (epoch_changed) {
handleUserSeek(playhead, dir);
}
}
// Compute bytes_per_frame, max_bytes, and capacity once
int64_t bytes_per_frame = getBytes(
(timeline->preview_width ? timeline->preview_width : reader->info.width),
@@ -357,7 +569,8 @@ namespace openshot
bool use_preroll = false;
{
std::lock_guard<std::mutex> guard(seek_state_mutex);
playhead = requested_display_frame.load();
raw_playhead = requested_display_frame.load();
playhead = clampToTimelineRange(raw_playhead, timeline_end);
did_user_seek = userSeeked.load();
use_preroll = preroll_on_next_fill.load();
if (did_user_seek) {
@@ -366,7 +579,10 @@ namespace openshot
}
}
if (did_user_seek) {
if (use_preroll) {
// During active playback, prioritize immediate forward readiness
// from the playhead. Use directional preroll offset only while
// paused/scrubbing contexts.
if (use_preroll && paused) {
handleUserSeekWithPreroll(playhead, dir, timeline_end, preroll_frames);
}
else {
@@ -395,6 +611,22 @@ namespace openshot
}
}
// If a clear was requested by a seek that arrived after the loop
// began, apply it now before any additional prefetch work. This
// avoids "build then suddenly clear" behavior during playback.
bool should_clear_mid_loop = clear_cache_on_next_fill.exchange(false);
if (should_clear_mid_loop && timeline) {
timeline->ClearAllCache();
cached_frame_count.store(0);
last_cached_index.store(playhead - dir);
}
// While user is dragging/scrubbing, skip cache prefetch work.
if (scrub_active.load()) {
std::this_thread::sleep_for(double_micro_sec(10000));
continue;
}
// If capacity is insufficient, sleep and retry
if (capacity < 1) {
std::this_thread::sleep_for(double_micro_sec(50000));
@@ -411,7 +643,8 @@ namespace openshot
ready_target = 0;
}
int64_t configured_min = settings->VIDEO_CACHE_MIN_PREROLL_FRAMES;
min_frames_ahead.store(std::min<int64_t>(configured_min, ready_target));
const int64_t required_ahead = std::min<int64_t>(configured_min, ready_target);
min_frames_ahead.store(required_ahead);
// If paused and playhead is no longer in cache, clear everything
bool did_clear = clearCacheIfPaused(playhead, paused, cache);
@@ -429,7 +662,21 @@ namespace openshot
window_end);
// Attempt to fill any missing frames in that window
bool window_full = prefetchWindow(cache, window_begin, window_end, dir, reader);
int64_t max_frames_to_fetch = -1;
if (!paused) {
// Keep cache thread responsive during playback seeks so player
// can start as soon as pre-roll is met instead of waiting for a
// full cache window pass.
max_frames_to_fetch = 8;
}
bool window_full = prefetchWindow(
cache,
window_begin,
window_end,
dir,
reader,
max_frames_to_fetch
);
// If paused and window was already full, keep playhead fresh
if (paused && window_full) {
src/Qt/VideoCacheThread.h
+18 -3
View File
@@ -18,6 +18,7 @@
#include <AppConfig.h>
#include <juce_audio_basics/juce_audio_basics.h>
#include <atomic>
#include <cstdint>
#include <mutex>
#include <memory>
@@ -61,7 +62,7 @@ namespace openshot
/// @return The current speed (1=normal, 2=fast, –1=rewind, etc.)
int getSpeed() const { return speed.load(); }
/// Seek to a specific frame (no preroll).
/// Backward-compatible alias for playback position updates (no seek side effects).
void Seek(int64_t new_position);
/**
@@ -71,6 +72,9 @@ namespace openshot
*/
void Seek(int64_t new_position, bool start_preroll);
/// Update playback position without triggering seek behavior or cache invalidation.
void NotifyPlaybackPosition(int64_t new_position);
/// Start the cache thread at high priority. Returns true if it’s actually running.
bool StartThread();
@@ -81,7 +85,7 @@ namespace openshot
* @brief Attach a ReaderBase (e.g. Timeline, FFmpegReader) and begin caching.
* @param new_reader
*/
void Reader(ReaderBase* new_reader) { reader = new_reader; Play(); }
void Reader(ReaderBase* new_reader);
protected:
/// Thread entry point: loops until threadShouldExit() is true.
@@ -125,6 +129,12 @@ namespace openshot
/// @brief Compute preroll frame count from settings.
int64_t computePrerollFrames(const Settings* settings) const;
/// @brief Resolve timeline end frame from reader/timeline metadata.
int64_t resolveTimelineEnd() const;
/// @brief Clamp frame index to [1, timeline_end] when timeline_end is valid.
int64_t clampToTimelineRange(int64_t frame, int64_t timeline_end) const;
/**
* @brief When paused and playhead is outside current cache, clear all frames.
* @param playhead Current requested_display_frame
@@ -171,7 +181,8 @@ namespace openshot
int64_t window_begin,
int64_t window_end,
int dir,
ReaderBase* reader);
ReaderBase* reader,
int64_t max_frames_to_fetch = -1);
//---------- Internal state ----------
@@ -182,6 +193,8 @@ namespace openshot
std::atomic<int> last_dir; ///< Last direction sign (+1 forward, –1 backward).
std::atomic<bool> userSeeked; ///< True if Seek(..., true) was called (forces a cache reset).
std::atomic<bool> preroll_on_next_fill; ///< True if next cache rebuild should include preroll offset.
std::atomic<bool> clear_cache_on_next_fill; ///< True if next cache loop should clear existing cache ranges.
std::atomic<bool> scrub_active; ///< True while user is dragging/scrubbing the playhead.
std::atomic<int64_t> requested_display_frame; ///< Frame index the user requested.
int64_t current_display_frame; ///< Currently displayed frame (unused here, reserved).
@@ -192,6 +205,8 @@ namespace openshot
ReaderBase* reader; ///< The source reader (e.g., Timeline, FFmpegReader).
bool force_directional_cache; ///< (Reserved for future use).
uint64_t seen_timeline_cache_epoch; ///< Last observed Timeline cache invalidation epoch.
bool timeline_cache_epoch_initialized; ///< True once an initial epoch snapshot has been taken.
std::atomic<int64_t> last_cached_index; ///< Index of the most recently cached frame.
mutable std::mutex seek_state_mutex; ///< Protects coherent seek state updates/consumption.
src/QtImageReader.cpp
+1 -8
View File
@@ -77,7 +77,7 @@ void QtImageReader::Open()
if (!loaded) {
// raise exception
throw InvalidFile("File could not be opened.", path.toStdString());
throw InvalidFile("QtImageReader could not open image file.", path.toStdString());
}
// Update image properties
@@ -367,11 +367,4 @@ void QtImageReader::SetJsonValue(const Json::Value root) {
// Set data from Json (if key is found)
if (!root["path"].isNull())
path = QString::fromStdString(root["path"].asString());
// Re-Open path, and re-init everything (if needed)
if (is_open)
{
Close();
Open();
}
}

Some files were not shown because too many files have changed in this diff Show More