53 Commits

Author SHA1 Message Date
jerome delattre
866fbadf1b Copy Delta and report file when copying image comparison report
#rnx
#rb sebastian.lewicki

[CL 33925573 by jerome delattre in ue5-main branch]
2024-05-27 12:42:10 -04:00
jerome delattre
13878a9635 Avoid comparison name reference mismatch between log and report
#rnx
#rb sebastian.lewicki

[CL 33604691 by jerome delattre in ue5-main branch]
2024-05-13 11:12:35 -04:00
jerome delattre
75450c1298 Remove screenshot legacy path check + fix EApprovedFolderOptions values
#rnx
#rb sebastian.lewicki

[CL 33084114 by jerome delattre in ue5-main branch]
2024-04-18 16:23:48 -04:00
jerome delattre
43322bb6a0 Validate active screenshot id before accepting screenshot comparison completion
#jira UE-202925
#rnx
#rb christopher.fiala, sean.sweeney, sebastian.lewicki

[CL 32122480 by jerome delattre in ue5-main branch]
2024-03-08 11:48:08 -05:00
sergei svetlov
c739c0cb35 Clean up screenshot comparison report pipeline
#rb Jerome.Delattre
#jira UE-200919
#rnx

[CL 30687983 by sergei svetlov in ue5-main branch]
2024-01-18 12:22:38 -05:00
christopher fiala
6291d6738b Update LightweightStereoTestVariants mode to run ONLY stereo test variants, no baseline, since they've been moved into their own category. Remove support for specifically exempting tests from running stereo variants. Save screenshots even for passed variants for the time being, since it's now possible to have a variant be the only test running.
Now `Automation.EnableStereoTestVariants=1` (off by default) enables ViewRectOffset, and adding `Automation.LightweightStereoTestVariants=1` causes us to ONLY run ViewRectOffset (on by default).

#jira UE-194361
#rb Jerome.Delattre

[CL 30174576 by christopher fiala in ue5-main branch]
2023-12-06 23:16:08 -05:00
christopher fiala
d5ec52b3cc Add a new CVar, Automation.LightweightStereoTestVariants, which stops variants from running if we've already encounted an error, and skips saving/attaching screenshots for successful variants to save space. Enabled by default. Also moves the "Supports Stereo Test Variants" checkbox to the "Screenshot" section.
#jira UE-194361
#rb jerome.delattre, chris.constantinescu

[CL 29718813 by christopher fiala in ue5-main branch]
2023-11-14 13:31:30 -05:00
christopher fiala
9942c169c2 Add the ViewRectOffset "test variant" to all screenshot functional tests. When a test variant is enabled, the test will take two screenshots, one in the base configuration, and one in the variant configuration, and fail if both are not similar to the ground truth. The ViewRectOffset variant will be enabled if "Support Stereo Test Variants" is checked on the functional test actor (TRUE by default) and the CVar r.EnableStereoScreenshotTestVariants is set to 1. The CVar is 0 by default for the time being, so there should be no changes to ET unless it is manually set to true.
#jira UE-194361
#rb david.harvey

[CL 29341539 by christopher fiala in ue5-main branch]
2023-11-01 20:00:13 -04:00
oleg krasnov
2eb50ce77a SessionFrontend - Make FScreenShotManager::OpenComparisonReports and metadata loading async
#jira UE-195234
#rb Jerome.Delattre

[CL 28908278 by oleg krasnov in ue5-main branch]
2023-10-19 02:01:14 -04:00
jerome delattre
1d5062721c Fix missing #undef LOCTEXT_NAMESPACE
#jira UE-198039
#rnx
#rb trivial

[CL 28802606 by jerome delattre in ue5-main branch]
2023-10-16 12:35:39 -04:00
jerome delattre
4a5957a406 Add frame comparison failure to the test report when comparing image sequence for Movie Pipeline Functional Test
+ Improve messaging when movie pipeline test is failing for reason other than image comparison.
+ Store ScreenshotName inside comparison request and result.

#jira UE-194701
#rnx
#rb Matt.Hoffman, Sean.Sweeney

[CL 28750594 by jerome delattre in ue5-main branch]
2023-10-13 09:42:36 -04:00
Jerome Delattre
404c9a4cec Fix fails to find screenshot reference when Project launcher is opened and the current project selected
#jira UE-197029
#rnx
#rb Sebastian.Lewicki

[CL 28469483 by Jerome Delattre in ue5-main branch]
2023-10-04 15:13:21 -04:00
josh adams
aeb5cb1347 - Restoring a backout after fixing the issue that causes the backout
[Backout] - CL27745134
[FYI] stan.hormell
Original CL Desc
-----------------------------------------------------------------
[Backout] - CL27731288 - CIS / Build Errors
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others

[CL 27923017 by josh adams in ue5-main branch]
2023-09-15 13:39:02 -04:00
stan hormell
37bad2d07b [Backout] - CL27731288 - CIS / Build Errors
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others

[CL 27745141 by stan hormell in ue5-main branch]
2023-09-10 22:58:21 -04:00
josh adams
a8a9a0b759 - Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others

[CL 27731364 by josh adams in ue5-main branch]
2023-09-08 17:58:55 -04:00
christopher waters
5d5fa707fb Adding logging of failed fallback paths to ScreenShot tests
#rb mihnea.balta, jeannoe.morissette

[CL 26291568 by christopher waters in ue5-main branch]
2023-06-28 13:55:26 -04:00
christopher waters
72568f28c9 Fixing EngineTest screenshot fallbacks for non-D3D platforms where the fallback paths aren't of the form RHI_FeatureLevel.
#jira UE-187354
#rb jeannoe.morissette

[CL 26196059 by christopher waters in ue5-main branch]
2023-06-22 18:05:58 -04:00
Jerome Delattre
06389d5027 Support Confidential platforms in Screenshot fallback settings
#jira UE-163243
#preflight 646e5f421417daba1e01a8eb
#rnx
#rb Brian.White

[CL 25643793 by Jerome Delattre in ue5-main branch]
2023-05-26 11:41:50 -04:00
mikalai sukhikh
43d4e1ea52 Fix: UnrealFrontend - Screenshot reference can't be found in Session frontend
#jira UE-183924
#rb adam.miezianko, chris.constantinescu, jerome.delattre
#preflight 646282fcb87b09aadd2b32a6
#preflight 646282c35b775f11fb08be45

[CL 25503473 by mikalai sukhikh in ue5-main branch]
2023-05-17 03:03:29 -04:00
oleg krasnov
3f55ae9120 Fix for not found screenshot reference while running automated tests in UFE
#jira UE-176679
#rb adam.miezianko, chris.constantinescu, jerome.delattre
#preflight 6414204e345ab08524c422ad
#preflight 6414205c3f3d31c94a634e45

[CL 24746909 by oleg krasnov in ue5-main branch]
2023-03-22 10:01:30 -04:00
Joe Kirchoff
109addf6a3 Cpp20 doesn't allow implicit capture of 'this' via '[=]'
#rb Francis.Hurteau
#preflight 64068c936e1dadfcda41bb82

[CL 24548358 by Joe Kirchoff in ue5-main branch]
2023-03-07 16:49:18 -05:00
jerome delattre
80b32adb45 Prevent approved reference images to be copied to html test report for successful test
#jira UE-176364
#preflight 63e3d66d8374ef68dffe3565
#rb Chris.Constantinescu

[CL 24085750 by jerome delattre in ue5-main branch]
2023-02-08 18:20:35 -05:00
cae-pdaigle
387f4f68fb Fix screenshots path names in test reports (standalone/no editor mode) (Contributed by cae-pdaigle)
PR #9680

#preflight 6357eff043784dae7f04fb62
#jira UE-167613
#rb Chris.Constantinescu

[CL 22782535 by cae-pdaigle in ue5-main branch]
2022-10-26 12:24:58 -04:00
Jerome Delattre
f8c58dfd2c Avoid duplicates of Screenshot comparison artifacts when producing automated test html/json report
#jira UE-144056
#preflight 621e630f3e14f0c7e5497b07
#rb Chris.Constantinescu

[CL 19271563 by Jerome Delattre in ue5-main branch]
2022-03-04 14:04:40 -05:00
jerome delattre
6490e355e2 Properly initiate FImageComparisonResult::CreationTime member
Fix originaly proposed through PR #8632. But the fix is not to exclude the class member to be tested but to properly intiate it.

#jira UE-135340
#rb Julien.StJean

#ROBOMERGE-AUTHOR: jerome.delattre
#ROBOMERGE-SOURCE: CL 18286341 in //UE5/Release-5.0/... via CL 18286368
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18286382 by jerome delattre in ue5-release-engine-test branch]
2021-11-24 13:36:04 -05:00