Commit Graph

463 Commits

Author SHA1 Message Date
william ewen
19b11fcb3e Gauntlet: Make DeviceReservations forget problem devices between passes, so that the DeviceService is authoritative
#rb josh.engebretson

[CL 22203062 by william ewen in ue5-main branch]
2022-09-27 06:57:20 -04:00
j baumgartner
df850050eb Compilation fix
#rb william.ewen

[CL 22148167 by j baumgartner in ue5-main branch]
2022-09-22 18:50:21 -04:00
william ewen
fa6b069775 Gauntlet: Make it so multi-pass tests will retain devices between each run
#rb Josh.Engebretson
[FYI] Andrew.Grant

[CL 22147439 by william ewen in ue5-main branch]
2022-09-22 18:17:10 -04:00
brandon schaefer
e08d29da57 Disable splash screen when running unattended
EngineTest simply does not have a splash screen and we warn on Linux. Lets just disable it for these tests

#jira UE-153920
#rb Josh.Adams
#preflight N/A
[FYI] Robert.Seiver

[CL 22042330 by brandon schaefer in ue5-main branch]
2022-09-15 20:02:08 -04:00
chris constantinescu
ddf55c6ac2 Retry save role artifacts once with device reboot on exception containing message "A retry should be performed"
#jira UE-161735
#rb Jerome.Delattre
#preflight skip

[CL 21896605 by chris constantinescu in ue5-main branch]
2022-09-08 12:50:25 -04:00
ben salem
4a662bbbac Fix Gauntlet throwing an InitializationFailed error when a process fails to log in.
#rb brendan.lienau, j.baumgartner

[CL 21864920 by ben salem in ue5-main branch]
2022-09-07 18:12:06 -04:00
Jerome Delattre
040633abd1 Fix Cooked Editor exported path in Windows
+ set CookedEditor role in Gauntlet UE tests when -cookededitor switch is used

#jira none
#rb Nick.bullard
#preflight 631269c6a20b67673b862af4
#rnx

[CL 21849523 by Jerome Delattre in ue5-main branch]
2022-09-07 09:51:36 -04:00
chris constantinescu
b604e668e2 Test runner:
- sleep on init to allow for sync w/ Gauntlet for Xbox platforms
- remove unused --base-global-module #fyi Stephen.Ma
Gauntlet:
- support low level tests build for different configurations (debug etc)
- readable command line arguments (-r to --reporter)
- allow argument names containing "-" e.g. --filenames-as-tags
#preflight 630536b1c00af5e2943540a1
#rb Jerome.Delattre

[CL 21550045 by chris constantinescu in ue5-main branch]
2022-08-24 15:53:16 -04:00
chris constantinescu
f5123e1e5f Re-throw if not a "retry" exception.
#rb Jerome.Delattre
#preflight skip

[CL 21457802 by chris constantinescu in ue5-main branch]
2022-08-19 11:10:00 -04:00
chris constantinescu
5f866f13cf Retry various file system operations that could fail for PS5 file system.
#jira UE-160038
#preflight 62fe598e3d3fb466b22e8b43
#rb Jerome.Delattre

[CL 21445586 by chris constantinescu in ue5-main branch]
2022-08-18 13:38:03 -04:00
andrew grant
4855221f20 Add logging to launch of tests in UnrealSession that display the role type
#preflight https://horde.devtools.epicgames.com/log/62f6b4c4ad3bd8ad648b4107

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21364065 via CL 21387057 via CL 21389907 via CL 21390632
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v975-21357124)

[CL 21392562 by andrew grant in ue5-main branch]
2022-08-15 15:50:44 -04:00
eric mcdaniel
8ccb89a3c8 Xbox: Increase initial wait time on PLM test to account for long startup times experienced by XB1 on Horde
- PLM test is failing sporadically on Horde due to the first suspend attempt being made before any frames have been displayed
    - locally I see ~5 secs before the app is displaying frames but this is ~20 more on Horde for some runs
    - even successful runs are nearing the current timeout (30s) on XB1
    - unclear why some Horde startups are so slow so just increase the initial wait to 45 secs to give us some headroom

#rb David.Harvey
#jira UE-159810
#preflight 62fa6f24ad3bd8ad640f8a99

[CL 21387897 by eric mcdaniel in ue5-main branch]
2022-08-15 12:14:20 -04:00
andrew grant
02e3ed342d When looking for a Gauntlet exit code check for -gauntlet= and not -gauntlet. It's the former that indicates there will be code in the process that is orchestrating the test.
#rb swarm
[REVIEW] [at]eric.knapik [at]daniel.lamb
#preflight https://horde.devtools.epicgames.com/job/62f54b3cbd746abb9987728c

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21343066 via CL 21350278 via CL 21350428 via CL 21350452
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21350672 by andrew grant in ue5-main branch]
2022-08-11 23:42:50 -04:00
Joe Conley
604cae73eb #jira UE-147963 Boot test for Hololens
Support for Hololens gauntlet automation launching to device
#preflight 62f3f9a0b43215cf6c2d98ac

[CL 21324460 by Joe Conley in ue5-main branch]
2022-08-10 15:47:10 -04:00
UnrealBot
73409369c0 Branch snapshot for CL 21319338
[CL 21319338 in ue5-main branch]
2022-08-10 16:03:37 +00:00
Juan Canada
44a0afd9da Enable ODSC in ray tracing tests again
#preflight none
#fyi luke.szarmach, chris.kulla, jason.nadro
#jira UE-159577

[CL 21218386 by Juan Canada in ue5-main branch]
2022-07-22 05:25:29 -04:00
eric knapik
d4be7f7baf #jira: none
Explicitly close the log file reader when an application terminates or has exited. This is make sure all log file handles are closed when another device comes along to potentially delete this log file.

#preflight 62d9b7cd4b6d775f5f076bf6
#preflight 62d9cf5f3122bf80ae18bdb9

#ROBOMERGE-AUTHOR: eric.knapik
#ROBOMERGE-SOURCE: CL 21212603 via CL 21213335 via CL 21214157 via CL 21214231
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21214664 by eric knapik in ue5-main branch]
2022-07-22 00:13:05 -04:00
james singer
8438087f41 Fix UserDir path handling for Gauntlet Linux host
#rnx
#jira none
#review-21186711 @brandon.schaefer
#rb @brandon.schaefer
#preflight 62d971d8bc175ec68c7c5d33

Details
- Previously UserDir was set in such a way that multiple gauntlet processes launched on the same machine would have their log directories collide.  In the observed failure case, the client would launch after the server, stomping the server's log.  Then gauntlet would be unable to find the test results and report as "No tests executed".  This change just makes the path be the same as what Windows uses.

[CL 21201734 by james singer in ue5-main branch]
2022-07-21 11:43:40 -04:00
eric knapik
1252ffa0fb #jira: none
Fix a potential warning when two UAT commands are ran back to back in the build graph. This can result in files potentially not being immediately deleted.

#preflight 62d88523d54af4b9a2de539d
#rb: Chris.Constantinescu

#ROBOMERGE-AUTHOR: eric.knapik
#ROBOMERGE-SOURCE: CL 21195999 via CL 21197116 via CL 21197117 via CL 21197118
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21197500 by eric knapik in ue5-main branch]
2022-07-20 23:51:19 -04:00
Juan Canada
809ae11052 Disable ODSC in ray tracing tests temporary to track down a bug
#preflight none
#fyi luke.szarmach, chris.kulla
#rb jason.nadro
#jira UE-159577

[CL 21182143 by Juan Canada in ue5-main branch]
2022-07-20 12:05:29 -04:00
Jerome Delattre
54ca504c14 Fix Gauntlet test search - Include original test name as first search critiria
#rnx
#preflight 62d71fb4c0180d8071a86426
#rb Chris.Constantinescu

[CL 21178286 by Jerome Delattre in ue5-main branch]
2022-07-20 09:11:05 -04:00
Jerome Delattre
8a01f176d1 Update ImageMagick tps
#jira none
#rnx
#preflight skip
#rb trivial

[CL 21115855 by Jerome Delattre in ue5-main branch]
2022-07-15 15:34:43 -04:00
gary yuan
073328c2e2 #jira: none
Gauntlet implement timestamp in output log messages relative to program start time

[REVIEW] [at]Eric.Knapik [at]Tim.Kennedy
#rb Eric.Knapik Tim.Kennedy
#preflight: 62cef450f30df2b55bfa8bf2

#ROBOMERGE-AUTHOR: gary.yuan
#ROBOMERGE-SOURCE: CL 21076332 via CL 21083608 via CL 21083941 via CL 21084219
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21086545 by gary yuan in ue5-main branch]
2022-07-13 21:25:10 -04:00
andrew grant
1fa8cac756 Made Gauntlet issue with a locked log a warning vs an error
#preflight skip

#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 21066766 via CL 21083532 via CL 21083910 via CL 21084171
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21086518 by andrew grant in ue5-main branch]
2022-07-13 21:24:36 -04:00
jerome delattre
2652ce1b3f Avoid enforcing Horde Simple Report to all UnrealTestNode.
#preflight 62c8accfb04c766648a41525
#rb none

#ROBOMERGE-AUTHOR: jerome.delattre
#ROBOMERGE-SOURCE: CL 21017400 via CL 21017405 via CL 21017409
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21023880 by jerome delattre in ue5-main branch]
2022-07-09 14:05:05 -04:00