Commit Graph

58 Commits

Author SHA1 Message Date
Nick Darnell
d75cbc192e Core - Updating the TBB copy location so that the dll and pdb end up in the same place to make generating portable symbols easier.
Build - updating the tempstorage build graph script to ignore additonal pdb build products for tbb.

#jira UE-159742
#preflight 62d5e73ec0180d80715a8b8d

[CL 21161802 by Nick Darnell in ue5-main branch]
2022-07-19 10:00:06 -04:00
Ryan Hummer
dcb81492ef Use robocopy/rsync to copy temp storage manifests locally before unzipping them, and use them to copy manifests off to the share at the end of a task.
improves platform reliability, m1 macs struggle to update file attributes when copying to an smb share. rsync ensures the attributes are correctly copied.

#rnx
#jira none
#rb none
#preflight 62d170c93b057e72748b9aa8
#author ryan.durand

[CL 21110668 by Ryan Hummer in ue5-main branch]
2022-07-15 11:15:30 -04:00
ryan hummer
f91cb75ed7 Add retrying to potential network share failure points in temp storage.
#rnx
#jira UE-156108, UE-158088
#rb ryan.hummer
#preflight 62bdeccded35ee71a70ee5a5

#ushell-cherrypick of 20574535 by Ryan.Durand

#ROBOMERGE-AUTHOR: ryan.hummer
#ROBOMERGE-SOURCE: CL 20901486 in //UE5/Release-5.0/... via CL 20902831
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)

[CL 20905677 by ryan hummer in ue5-main branch]
2022-06-30 18:10:24 -04:00
Ryan Hummer
8a3fcafa63 Fix TempStorage Manifest XML Writing
dotnet6 xml serialization nolonger emits new lines, change to using XMLWriter to enable newlines

#rnx
#jira none
#rb ryan.durand
#preflight 62a34bd10251cacef0b2a9e1

[CL 20595487 by Ryan Hummer in ue5-main branch]
2022-06-10 10:45:41 -04:00
Ryan Durand
bf54f10db3 Add retrying to potential network share failure points in temp storage.
#rnx
#jira UE-156108
#rb ryan.hummer
#preflight

[CL 20574535 by Ryan Durand in ue5-main branch]
2022-06-09 08:55:18 -04:00
Ryan Durand
c2a32ddc91 Couple of temp storage clean up changes to let multiple clean up jobs run less painfully.
* Flag folders as delete in progress to allow other clean up jobs to skip a folder and go to the next one.
* Delete from newest to oldest in an attempt to provide a buffer before multiple jobs will attempt to delete the same folder.

#rnx
#jira UE-150924
#rb ryan.hummer,bryan.johnson
#preflight

[CL 20047320 by Ryan Durand in ue5-main branch]
2022-05-04 15:44:17 -04:00
matt johnson
25c86ecac1 UBT: recognize shared libraries with version number suffixes as allowed duplicate build products
Shared libraries on Linux commonly use a scheme of version number suffixes
and/or symlinks to help ensure that client applications can link against the
appropriate version of a library (e.g. "libfoo.so.1.70.0"). This change updates
IsDuplicateBuildProduct() with an additional case to cover these libraries so
that they are not flagged as an error by the build system if multiple steps
produce them.

#jira UE-148232
#rb brandon.schaefer
#preflight 624c76a8712e7c09a6a15d69

[CL 19627733 by matt johnson in ue5-main branch]
2022-04-05 13:17:02 -04:00
Ryan Hummer
e4945b7f53 Clarify error message when scanning temp storage for cleanup
#rnx
#jira none
#rb none
#preflight skip

[CL 19579046 by Ryan Hummer in ue5-main branch]
2022-03-31 17:23:08 -04:00
Ryan Durand
7e494a86dd * Fix spelling on Retrieve
* Copy manifest ZIP files locally before unzipping them.
* Use retries when copying to harden against network blips.

#rnx
#jira UE-139648
#rb ryan.hummer, ben.marsh
#preflight

[CL 19480296 by Ryan Durand in ue5-main branch]
2022-03-23 12:02:11 -04:00
ben marsh
66f37c2bb9 BuildGraph: Take the digest of any *.version files when copying files to temp storage, so their timestamps can be updated without making any content changes.
#preflight none
#jira

#ROBOMERGE-AUTHOR: ben.marsh
#ROBOMERGE-SOURCE: CL 18588704 in //UE5/Release-5.0/... via CL 18588716 via CL 18588728
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)

[CL 18588738 by ben marsh in ue5-main branch]
2022-01-12 15:24:13 -05:00
ryan hummer
c0419eb70e Fixing logging format call to include the file name
#rnx
#jira none

#ROBOMERGE-AUTHOR: ryan.hummer
#ROBOMERGE-SOURCE: CL 18444558 in //UE5/Release-5.0/... via CL 18445395
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18445546 by ryan hummer in ue5-release-engine-test branch]
2021-12-13 13:45:41 -05:00
josh engebretson
ab07ba85cb Add error logging for temp storage zip operations
#jira none
#rnx

#ROBOMERGE-AUTHOR: josh.engebretson
#ROBOMERGE-SOURCE: CL 18421529 in //UE5/Release-5.0/... via CL 18422973
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18423279 by josh engebretson in ue5-release-engine-test branch]
2021-12-09 15:01:49 -05:00
mark lintott
9362edf2fe Catch IO issues when creating or copying ZIP files
#rb josh.engebretson
#jira none

#ROBOMERGE-AUTHOR: mark.lintott
#ROBOMERGE-SOURCE: CL 18420781 in //UE5/Release-5.0/... via CL 18422611
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18422889 by mark lintott in ue5-release-engine-test branch]
2021-12-09 14:52:06 -05:00
ryan durand
a954aa1ab3 Add an explicit error message when we run out of retries and fail to unzip temp storage.
#rnx
#rb none
#jira UE-135258

#ROBOMERGE-AUTHOR: ryan.durand
#ROBOMERGE-SOURCE: CL 18243011 in //UE5/Release-5.0/... via CL 18243061
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18243291 by ryan durand in ue5-release-engine-test branch]
2021-11-18 15:38:54 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
brandon schaefer
c7e8e6a8f3 Add lib/so rules for TempStorage in IsDuplicateBuildProduct for Linux
#jira none
#rb none
[FYI] Ben.Marsh

#ROBOMERGE-AUTHOR: brandon.schaefer
#ROBOMERGE-SOURCE: CL 17561654 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17561661 by brandon schaefer in ue5-release-engine-test branch]
2021-09-17 23:03:56 -04:00
aurel cordonnier
02c0f425e8 Copy up from Release-Engine-Staging @ 16738359
This represents UE4/Main @ 16738161 and Dev-PerfTest @ 16737719

[CL 16738582 by aurel cordonnier in ue5-release-engine-test branch]
2021-06-22 00:27:54 -04:00
ryan durand
e0919db187 Early out of CleanTempStorage if the temp storage path doesn't work.
#rnx
#rb none
#jira none

#ROBOMERGE-SOURCE: CL 16621959 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v830-16605563)

[CL 16621975 by ryan durand in ue5-release-engine-test branch]
2021-06-10 08:48:38 -04:00
jonathan adamczewski
9b69a17dbc TempStorage.cs:
Remove some mono references.

#jira none
#trivial

[CL 16175220 by jonathan adamczewski in ue5-main branch]
2021-04-30 20:58:26 -04:00
Ryan Durand
10bcbb3b86 Harden code for finding shared file manifests.
#rnx
#rb none
#jira none

[CL 15860260 by Ryan Durand in ue5-main branch]
2021-03-30 08:58:10 -04:00
Ryan Durand
74a82706a9 Catch exceptions when scanning folders for folders that need to be deleted.
#rnx
#rb none
#jira none

[CL 15696383 by Ryan Durand in ue5-main branch]
2021-03-15 10:25:55 -04:00
ben marsh
b4553ed744 Fix permission bits being reset by heurisitc that retrieves files from temp storage.
#rb none
#jira UE-109089

#ROBOMERGE-SOURCE: CL 15676777 in //UE5/Release-5.0-EarlyAccess/...
#ROBOMERGE-BOT: STARSHIP (Release-5.0-EarlyAccess -> Main) (v781-15675533)

[CL 15676799 by ben marsh in ue5-main branch]
2021-03-11 11:19:28 -04:00
Ryan Durand
b6eff43263 Fixing compile break.
#rnx
#rb none
#jira none

[CL 15675703 by Ryan Durand in ue5-main branch]
2021-03-11 08:21:47 -04:00
Ryan Durand
517d36846d Catch and log exceptions when enumerating files/folders to delete.
#rnx
#rb none
#jira none

[CL 15675515 by Ryan Durand in ue5-main branch]
2021-03-11 07:27:29 -04:00