Commit Graph

57 Commits

Author SHA1 Message Date
andrew grant
4bc7154601 Split Incremental / Regular EngineTests into groups.
Moved jobs that don't require rendering to general build pool.
Added option for including cooked platforms (currently disabled, need to remove copy from DevStreams.xml).
Added XB1/Switch tests (also currently disabled)
Moved reports and logs to new file structure and site on automation.epicgames.net
Added VerifyDDC option to Gauntlet job.

#jira #rb na
[FYI] ben.marsh,josh.engebretson
#tests preflighted (lots).

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 6547274 in //UE4/Main/...
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR) (v350-6578747)

[CL 6579843 by andrew grant in Dev-VR branch]
2019-05-20 14:40:58 -04:00
josh engebretson
2b8035ed5d Gauntlet: Use temp folder and include worker id for iOS IPA cache
#rnx

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: josh.engebretson
#ROBOMERGE-SOURCE: CL 6487502 via CL 6487583 via CL 6500574 via CL 6500841
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6531422 by josh engebretson in Dev-VR branch]
2019-05-16 03:33:07 -04:00
josh engebretson
46a14e836e Gauntlet: Don't attempt to reserve devices with model from service, reduce some device pool log severity
#rnx

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: josh.engebretson
#ROBOMERGE-SOURCE: CL 6451264 via CL 6452060 via CL 6452197
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6510634 by josh engebretson in Dev-VR branch]
2019-05-15 15:57:08 -04:00
josh engebretson
a9fae99ea2 Gauntlet: Disable Vivox voice chat when running iOS tests, otherwise get blocking microphone permission request
#rnx

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: josh.engebretson
#ROBOMERGE-SOURCE: CL 6447595 via CL 6449338 via CL 6449476
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6508824 by josh engebretson in Dev-VR branch]
2019-05-15 15:41:17 -04:00
josh engebretson
a656ba4b20 Gauntlet: Adding hardware model constraints and worker job id
#rnx
[FYI] andrew.grant

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: josh.engebretson
#ROBOMERGE-SOURCE: CL 6373407 via CL 6383531 via CL 6383639
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6387285 by josh engebretson in Dev-VR branch]
2019-05-08 21:07:27 -04:00
josh engebretson
f68e290c5e Gauntlet: Add platform independent mapping to root drive paths
#rnx
[FYI] ben.salem, clayton.langford

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: josh.engebretson
#ROBOMERGE-SOURCE: CL 6340419 via CL 6341332 via CL 6341339
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6352218 by josh engebretson in Dev-VR branch]
2019-05-07 18:45:13 -04:00
andrew grant
830bc906f1 By default only consider failed artifacts when looking for fatal errors.
#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 6252932 via CL 6253237 via CL 6253249
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6255670 by andrew grant in Dev-VR branch]
2019-05-02 13:06:19 -04:00
andrew grant
56a782d1e0 Fixed missing copyright notice
#jira UE-74085
#rb na

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 6252764 via CL 6253149 via CL 6253168
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6255600 by andrew grant in Dev-VR branch]
2019-05-02 13:05:53 -04:00
andrew grant
e9cb19c9e2 Fix for CrashReportTest errors causing a false positive
#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 6252450 via CL 6252699 via CL 6252706
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6254952 by andrew grant in Dev-VR branch]
2019-05-02 13:02:52 -04:00
andrew grant
e71673cb1c Rewrote RunAutomatedTests Gauntlet node to be more comprehensive in terms of error reporting and to support both Editor and Cooked formats.
Updated Fortnite build scripts to run EngineTests as multiple groups to better diagnose and identify errors.

#rb CRd
[at]josh.engebretson [at]ben.salem [at]clayton.langford

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 6241602 via CL 6246013 via CL 6246065
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6249861 by andrew grant in Dev-VR branch]
2019-05-02 05:02:48 -04:00
andrew grant
1a87b02536 Gauntlet -
ITestNode now defines GetWarnings and GetErrors calls for tests to return instances that occurred during a test.

Split creation of Summary report in UnrealTestNode into Header/Body functions to allow tests to more easily customize these.

Added GetLogChannels() function to UnrealLogParser that will return all lines from multiple channels

Added AutomatonLogParser that specifically parses the output of automation (non-Gauntlet) tests.

#rb CRd
[at]josh.engebretson [at]clayton.langford [at]ben.salem

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 6241133 via CL 6246005 via CL 6246055
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6249793 by andrew grant in Dev-VR branch]
2019-05-02 05:02:21 -04:00
clayton langford
08aeab5e6a Fixed a config format issue and improved logging to more easily find these issues in the future.
#rnx

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: clayton.langford
#ROBOMERGE-SOURCE: CL 6141936 via CL 6141942 via CL 6142990 via CL 6176201
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6205660 by clayton langford in Dev-VR branch]
2019-05-01 17:54:34 -04:00
andrew grant
d31b54cd78 Gauntlet - cleaned up some logging on error cases.
#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 6115333 via CL 6132878 via CL 6174059
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6198795 by andrew grant in Dev-VR branch]
2019-05-01 17:13:57 -04:00
ben salem
3d33b2e20b Pass through -gauntlet when not requiring a test controller to let the client know we are still running via Gauntlet.
#RB josh.engebretson


#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: ben.salem
#ROBOMERGE-SOURCE: CL 6111304 via CL 6111307 via CL 6112798 via CL 6112961
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6132579 by ben salem in Dev-VR branch]
2019-04-26 23:11:50 -04:00
josh engebretson
0a98d57273 Gauntlet: update cached stdout property when pulling artifacts on Android
#rnx

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: josh.engebretson
#ROBOMERGE-SOURCE: CL 6104984 via CL 6105113 via CL 6105520 via CL 6105534
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6132209 by josh engebretson in Dev-VR branch]
2019-04-26 22:56:51 -04:00
andrew grant
2c3594796e Added Win32 support to Gauntlet
Tests with EngineTest project

[at]josh.engebretson [at]ben.salem
#rb na

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 6095648 via CL 6096480 via CL 6096540
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6128169 by andrew grant in Dev-VR branch]
2019-04-26 22:22:11 -04:00
josh engebretson
72b92aeb5a Gauntlet: Fix issue with adb attempting to get activity log after disposal (and adb server kill)
#rnx

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: josh.engebretson
#ROBOMERGE-SOURCE: CL 6095222 via CL 6095232 via CL 6096459 via CL 6096507
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6128076 by josh engebretson in Dev-VR branch]
2019-04-26 22:21:39 -04:00
josh engebretson
38f1c5e65b Adding logging to help catch android device error
#rnx

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: josh.engebretson
#ROBOMERGE-SOURCE: CL 6079218 via CL 6079219 via CL 6085366 via CL 6085559
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6115750 by josh engebretson in Dev-VR branch]
2019-04-26 21:01:57 -04:00
josh engebretson
b871ea65f7 Suspend EC error parsing while killing adb server
#rnx

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: josh.engebretson
#ROBOMERGE-SOURCE: CL 6039203 via CL 6039204 via CL 6045452 via CL 6045630
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 6047181 by josh engebretson in Dev-VR branch]
2019-04-23 01:03:33 -04:00
andrew grant
bb86629e24 Implemented Gauntlet's "-dev" feature for Mac. This gives the ability to use a locally build executable against network builds by comparing timestamps and using the -basedir argument to point the client to the network data.
[CODEREVIEW] josh.engebretson,michael.trepka
#rb na

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 5871266 via CL 5871937 via CL 5871950 via CL 5872516 via CL 5872524
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 5888384 by andrew grant in Dev-VR branch]
2019-04-12 22:29:31 -04:00
andrew grant
29bf5ab575 Fix for NoMcp xbox tests breaking. Now detect if an XB1 NoMcp client is involved in a test and turn off trust validation.
Removed old -skipxboxauth flag

#rb codereview

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 5749377 via CL 5749558 via CL 5749651 via CL 5749736 via CL 5754588 via CL 5754753
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 5758119 by andrew grant in Dev-VR branch]
2019-04-04 21:09:42 -04:00
andrew grant
550c881140 Fixed issue with parallel tests of the same name trying to use the same artifact folder.
[CODEREVIEW] josh.engebretson,ben.salem

#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 5747120 via CL 5749542 via CL 5749605 via CL 5749696 via CL 5754546 via CL 5754697
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 5758072 by andrew grant in Dev-VR branch]
2019-04-04 21:09:21 -04:00
andrew grant
0f383e4ce5 Fix exception in clean devices
#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 5720355 via CL 5722816 via CL 5722834
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 5729662 by andrew grant in Dev-VR branch]
2019-04-04 01:19:11 -04:00
ben salem
6f4f74f08c Re-enable Gauntlet Account Pool changes now that all Load Bot accounts have been properly profiled on GameDev.
#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: ben.salem
#ROBOMERGE-SOURCE: CL 5660582 via CL 5660583 via CL 5660584 via CL 5660585 via CL 5660625 via CL 5660655
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 5667632 by ben salem in Dev-VR branch]
2019-04-01 09:32:24 -04:00
ben salem
f59b3975ab Temporarily backing out changelist 5659802 to unblock Editor tests until we can bulk stash new accounts.
#ROBOMERGE-OWNER: ryan.vance
#ROBOMERGE-AUTHOR: ben.salem
#ROBOMERGE-SOURCE: CL 5659905 via CL 5659906 via CL 5659907 via CL 5659908 via CL 5659934 via CL 5659943
#ROBOMERGE-BOT: DEVVR (Main -> Dev-VR)

[CL 5666986 by ben salem in Dev-VR branch]
2019-04-01 09:24:11 -04:00