Commit Graph

47 Commits

Author SHA1 Message Date
christopher fiala
d5ec52b3cc Add a new CVar, Automation.LightweightStereoTestVariants, which stops variants from running if we've already encounted an error, and skips saving/attaching screenshots for successful variants to save space. Enabled by default. Also moves the "Supports Stereo Test Variants" checkbox to the "Screenshot" section.
#jira UE-194361
#rb jerome.delattre, chris.constantinescu

[CL 29718813 by christopher fiala in ue5-main branch]
2023-11-14 13:31:30 -05:00
christopher fiala
9942c169c2 Add the ViewRectOffset "test variant" to all screenshot functional tests. When a test variant is enabled, the test will take two screenshots, one in the base configuration, and one in the variant configuration, and fail if both are not similar to the ground truth. The ViewRectOffset variant will be enabled if "Support Stereo Test Variants" is checked on the functional test actor (TRUE by default) and the CVar r.EnableStereoScreenshotTestVariants is set to 1. The CVar is 0 by default for the time being, so there should be no changes to ET unless it is manually set to true.
#jira UE-194361
#rb david.harvey

[CL 29341539 by christopher fiala in ue5-main branch]
2023-11-01 20:00:13 -04:00
oleg krasnov
2eb50ce77a SessionFrontend - Make FScreenShotManager::OpenComparisonReports and metadata loading async
#jira UE-195234
#rb Jerome.Delattre

[CL 28908278 by oleg krasnov in ue5-main branch]
2023-10-19 02:01:14 -04:00
jerome delattre
1d5062721c Fix missing #undef LOCTEXT_NAMESPACE
#jira UE-198039
#rnx
#rb trivial

[CL 28802606 by jerome delattre in ue5-main branch]
2023-10-16 12:35:39 -04:00
jerome delattre
4a5957a406 Add frame comparison failure to the test report when comparing image sequence for Movie Pipeline Functional Test
+ Improve messaging when movie pipeline test is failing for reason other than image comparison.
+ Store ScreenshotName inside comparison request and result.

#jira UE-194701
#rnx
#rb Matt.Hoffman, Sean.Sweeney

[CL 28750594 by jerome delattre in ue5-main branch]
2023-10-13 09:42:36 -04:00
Jerome Delattre
404c9a4cec Fix fails to find screenshot reference when Project launcher is opened and the current project selected
#jira UE-197029
#rnx
#rb Sebastian.Lewicki

[CL 28469483 by Jerome Delattre in ue5-main branch]
2023-10-04 15:13:21 -04:00
josh adams
aeb5cb1347 - Restoring a backout after fixing the issue that causes the backout
[Backout] - CL27745134
[FYI] stan.hormell
Original CL Desc
-----------------------------------------------------------------
[Backout] - CL27731288 - CIS / Build Errors
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others

[CL 27923017 by josh adams in ue5-main branch]
2023-09-15 13:39:02 -04:00
stan hormell
37bad2d07b [Backout] - CL27731288 - CIS / Build Errors
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others

[CL 27745141 by stan hormell in ue5-main branch]
2023-09-10 22:58:21 -04:00
josh adams
a8a9a0b759 - Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others

[CL 27731364 by josh adams in ue5-main branch]
2023-09-08 17:58:55 -04:00
christopher waters
5d5fa707fb Adding logging of failed fallback paths to ScreenShot tests
#rb mihnea.balta, jeannoe.morissette

[CL 26291568 by christopher waters in ue5-main branch]
2023-06-28 13:55:26 -04:00
christopher waters
72568f28c9 Fixing EngineTest screenshot fallbacks for non-D3D platforms where the fallback paths aren't of the form RHI_FeatureLevel.
#jira UE-187354
#rb jeannoe.morissette

[CL 26196059 by christopher waters in ue5-main branch]
2023-06-22 18:05:58 -04:00
Jerome Delattre
06389d5027 Support Confidential platforms in Screenshot fallback settings
#jira UE-163243
#preflight 646e5f421417daba1e01a8eb
#rnx
#rb Brian.White

[CL 25643793 by Jerome Delattre in ue5-main branch]
2023-05-26 11:41:50 -04:00
mikalai sukhikh
43d4e1ea52 Fix: UnrealFrontend - Screenshot reference can't be found in Session frontend
#jira UE-183924
#rb adam.miezianko, chris.constantinescu, jerome.delattre
#preflight 646282fcb87b09aadd2b32a6
#preflight 646282c35b775f11fb08be45

[CL 25503473 by mikalai sukhikh in ue5-main branch]
2023-05-17 03:03:29 -04:00
oleg krasnov
3f55ae9120 Fix for not found screenshot reference while running automated tests in UFE
#jira UE-176679
#rb adam.miezianko, chris.constantinescu, jerome.delattre
#preflight 6414204e345ab08524c422ad
#preflight 6414205c3f3d31c94a634e45

[CL 24746909 by oleg krasnov in ue5-main branch]
2023-03-22 10:01:30 -04:00
Joe Kirchoff
109addf6a3 Cpp20 doesn't allow implicit capture of 'this' via '[=]'
#rb Francis.Hurteau
#preflight 64068c936e1dadfcda41bb82

[CL 24548358 by Joe Kirchoff in ue5-main branch]
2023-03-07 16:49:18 -05:00
jerome delattre
80b32adb45 Prevent approved reference images to be copied to html test report for successful test
#jira UE-176364
#preflight 63e3d66d8374ef68dffe3565
#rb Chris.Constantinescu

[CL 24085750 by jerome delattre in ue5-main branch]
2023-02-08 18:20:35 -05:00
cae-pdaigle
387f4f68fb Fix screenshots path names in test reports (standalone/no editor mode) (Contributed by cae-pdaigle)
PR #9680

#preflight 6357eff043784dae7f04fb62
#jira UE-167613
#rb Chris.Constantinescu

[CL 22782535 by cae-pdaigle in ue5-main branch]
2022-10-26 12:24:58 -04:00
Jerome Delattre
f8c58dfd2c Avoid duplicates of Screenshot comparison artifacts when producing automated test html/json report
#jira UE-144056
#preflight 621e630f3e14f0c7e5497b07
#rb Chris.Constantinescu

[CL 19271563 by Jerome Delattre in ue5-main branch]
2022-03-04 14:04:40 -05:00
jerome delattre
6490e355e2 Properly initiate FImageComparisonResult::CreationTime member
Fix originaly proposed through PR #8632. But the fix is not to exclude the class member to be tested but to properly intiate it.

#jira UE-135340
#rb Julien.StJean

#ROBOMERGE-AUTHOR: jerome.delattre
#ROBOMERGE-SOURCE: CL 18286341 in //UE5/Release-5.0/... via CL 18286368
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18286382 by jerome delattre in ue5-release-engine-test branch]
2021-11-24 13:36:04 -05:00
Marc Audy
e80ea6b959 Merge from Release-Engine-Staging @ 16444985
This represents UE4/Main @ 16445039 and Dev-PerfTest @ 16444526

[CL 16445122 by Marc Audy in ue5-release-engine-test branch]
2021-05-25 02:43:26 -04:00
ionut matasaru
74bf2e67b9 Fixed errors re format string / parameter list for various UE_LOGs:
- number of supplied parameters less or more than the number of specifiers used in format string
  - invalid format string specifiers used

#rb Matt.Peters

#ROBOMERGE-SOURCE: CL 16315056 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v804-16311228)

[CL 16315068 by ionut matasaru in ue5-release-engine-test branch]
2021-05-13 13:22:48 -04:00
Josh Adams
599aa317c2 - ConfigCacheIni overhaul to allow for multiple platform's "GConfigs" at once.
- FConfigCacheIni::ForPlatform(PlatformName) is the way to get the in-memory (won't write to disk) ini system for a different platform. Can pass it GEngineIni, etc for the filename (  FConfigCacheIni::ForPlatform("IOS")->GetString(....., GEngineIni);  )
- GEngineIni and some others (the "stadnard, runtime ini files") are no longer filenames, but identifiers that will work with other platform Ini systems
- Modified "GETINI" console command to allow for a <Platform>@ prefix, to get another platform's ini value (using new multi-platform config) (GetIni [Platform@]IniFile:Section.SubSection Key)
#rb rolando.caloca
#fyi chris.gagnon,Sebastian.Nordgren

[CL 14473359 by Josh Adams in ue5-main branch]
2020-10-12 13:54:50 -04:00
Josh Adams
94f4be4de1 - Renamed PlatformInfo::FPlatformInfo to PlatformInfo::FTargetPlatformInfo
- Renamed FDataDrivenPlatformRegistry::FPlatformInfo to FDataDrivenPlatformInfo
- Moved some fields from PlatformInfo::FTargetPlatformInfo to FDataDrivenPlatformInfo, and cleaned them up in the process
- Fixed the DataDrivePlatformInfo.ini files to match the previous items
- Removed FVanillaPlatformEntry, and now just using FTargetPlatformInfo to manage flavors under a vanilla PlatformInfo (see PlatformInfo::GetVanillaPlatformInfoArray())
- Cleaned up TPerPlatformValue, as it was often misused (took a group and platform name, but we can get the group from the platform name) [AnimationSharingManager.cpp, *Engine.cpp, Runtime\Engine\*, ]
- Fixed FBlueprintNativeCodeGenPaths::GetDefaultCodeGenPaths() to use IniPlatformName instead of some hacky code [BlueprintNativeCodeGenManifest.cpp]
- Fixed various Turnkey bugs that recent testing exposed
- Enabled AUTOSDKS_ENABLED in both Mac and Linux on the Editor side (not that there are much set up to use it)
- Using Turnkey to get the SDK status instead of ValidatePlatforms (gives more detailed information - may want to go back to ValidatePlatforms so only UBT is needed, but change it to give more info)
- Moved OnDeviceDiscovered and Lost delegates from each platform's TargetPlatform class to a static in ITargetPlatform - there was no need for per-platform implementations
- Started working on allowing for SDK to be installed with editor running and not need to restart editor - Work in Progress! It is not usable yet.
#fyi jack.porter
#rb pete.sauerbrei

[CL 13816905 by Josh Adams in ue5-main branch]
2020-07-01 17:07:12 -04:00
Marc Audy
11f5b21210 Merging //UE5/Release-Engine-Staging @ 13752110 to Main (//UE5/Main)
#rnx

[CL 13753156 by Marc Audy in ue5-main branch]
2020-06-23 18:40:00 -04:00
zabir hoque
8c89dae9ea Enable RenderDoc Captures for failing automation tests.
[FYI] Andrew.Grant; Arciel.Rekman;
#rb: none
#jira: UE-91964

#ROBOMERGE-SOURCE: CL 12941114 in //UE4/Release-4.25/... via CL 12941117 via CL 12941124
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)

[CL 12941128 by zabir hoque in Main branch]
2020-04-20 19:57:16 -04:00