Commit Graph

32 Commits

Author SHA1 Message Date
dan engelbrecht
eccdbbbd02 Make attachment names written as temp-files in ZenStoreHttpClient uses a per-process unique salt
#rb zousar.shaker matt.peters
#preflight 63d7e0e7d21dbe1d29486735

[CL 23917462 by dan engelbrecht in ue5-main branch]
2023-01-30 17:38:18 -05:00
dan engelbrecht
e018c47697 Add marker file for project store oplog in Saved/Cooked/<platform> folder so Zen can check if an oplog should be garbage collected.
Companion Zen PR: https://github.com/EpicGames/zen/pull/209

#preflight 63971e55cb2317695e180344

[CL 23510788 by dan engelbrecht in ue5-main branch]
2022-12-14 07:18:38 -05:00
dan engelbrecht
2f8823f7dc Pass along path to project file when zen server is running locally so we can detect when it is removed and garbage-collect any projectstore entries on the zen side.
Allow Cook to remote Zen, if we detect that we are targetting a zenservice on a different machine we change the local file references to attachment based references.
#rb stefan.boberg
#preflight 63491c31ad0f7e2f20230b39

[CL 22525021 by dan engelbrecht in ue5-main branch]
2022-10-14 08:17:43 -04:00
Zousar Shaker
d34d74fad2 Avoid compiling in desktop-only Zen functionality for Stadia due to the use of PLATORM_UNIX instead of PLATFORM_LINUX.
#rb devin.doucette
#fyi martin.ridgers
#preflight 62967ef3261e39f6206b0a82

[CL 20444919 by Zousar Shaker in ue5-main branch]
2022-05-31 20:12:27 -04:00
Zousar Shaker
edf8ae1f0b 2nd submit:
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath.  This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store.  Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.

Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).

#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298
#preflight 629642a1be0ae0b33a46d012

[CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
Zousar Shaker
77f51a02b9 [Backout] - CL20433163
#fyi Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath.  This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store.  Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.

Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).

#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298

[CL 20433363 by Zousar Shaker in ue5-main branch]
2022-05-30 19:32:59 -04:00
Zousar Shaker
2f6cbf5012 Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not.
Tested Zen operation using ShooterGame editor.
Tested Jupiter operation using CitySample editor (empty local cache).

#rb devin.doucette
#fyi stefan.boberg
#preflight 62953e65401169f78c57c298

[CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
devin doucette
b2a07ea03e DDC: Merge from UE5/Main
#preflight 6288ff678828ea88c8af7034
#preflight 628ab5d93246d5019db76ed2
#rb none
#rnx

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: Devin.Doucette
#ROBOMERGE-SOURCE: CL 20353148 via CL 20353832 via CL 20353839
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v948-20297126)

[CL 20355348 by devin doucette in ue5-main branch]
2022-05-24 16:40:25 -04:00
CarlMagnus Nordin
210d67b30e Promoted Zen project id generation to FApp
#rb pj.kack
#preflight 628769032c34da686ac82442

[CL 20316798 by CarlMagnus Nordin in ue5-main branch]
2022-05-23 01:49:25 -04:00
Devin Doucette
748e39e3c0 Replaced uses of _SV with TEXTVIEW
#preflight 6266da740634d0904ce6de29
#rb Steve.Robb
#rnx

[CL 19905760 by Devin Doucette in ue5-main branch]
2022-04-25 13:56:30 -04:00
Per Larsson
ddbf9eb5d2 Zen - cook to custom project/oplog ID with -ZenStoreProject=<ProjectId> and -ZenStorePlatform=<OplogId>
#rb none
#rnx
#preflight 19755222

[CL 19804798 by Per Larsson in ue5-main branch]
2022-04-19 02:37:06 -04:00
Martin Ridgers
db99164288 - Enable Zen on Mac and Linux.
- Updated zenserver binaries to 4ae4f1f

#rnx
#preflight 621ddbb0313c8c01033b88ae

[CL 19195911 by Martin Ridgers in ue5-main branch]
2022-03-01 03:56:00 -05:00
per larsson
24a854e5f7 Support cooking to remote Zen server
#rb none
#preflight 61ea791ae71cba4d253c6919
#rnx

#ROBOMERGE-AUTHOR: per.larsson
#ROBOMERGE-SOURCE: CL 18687394 in //UE5/Release-5.0/... via CL 18687401 via CL 18687404
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v902-18672795)

[CL 18687410 by per larsson in ue5-main branch]
2022-01-21 04:24:08 -05:00
mark lintott
b43fb780c7 Non-Unity Fixes
#jira none
[FYI] zousar.shaker
#rb trival
#preflight 61b72383fc570891337a735f

#ROBOMERGE-AUTHOR: mark.lintott
#ROBOMERGE-SOURCE: CL 18442318 in //UE5/Release-5.0/... via CL 18444991
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18445172 by mark lintott in ue5-release-engine-test branch]
2021-12-13 13:30:16 -05:00
zousar shaker
0832797d28 Ensure that Zen ProjectIds are unique for projects with the same name in differing roots (or in the same root) while attempting to maintain some easily recognizable desriptors for the project.
Staged builds are explicitly assigned a project id via the persistent UECommandline.txt file.  This works on mobile/consoles, but isn't implemented for Windows/Linux/Mac, so staged builds for those platforms don't have a persistent assignment of project id and instead rely on the launch by the editor including the appropriate commandlines.
Unstaged builds on Windows/Mac/Linux now determine project id independently after they have determined the project file path.  This happens after the project file has been determined and has had case corrected.  This ensures it is different/unique for multiple blueprint projects that share the same executable.
Fix a bug where second cook in editor to zen would stall because the queue thinks all additions are finished due to a boolean that isn't reset.

#rb devin.doucette
#rb per.larsson
#preflight 61b39a35a2562c8b1c40f81e

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18432317 in //UE5/Release-5.0/... via CL 18435404
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18435895 by zousar shaker in ue5-release-engine-test branch]
2021-12-10 18:06:39 -05:00
devin doucette
27c1393427 CompressedBuffer: Removed partial decompression from FCompressedBuffer now that FCompressedBufferReader is available
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.

- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.

#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18382377 by devin doucette in ue5-release-engine-test branch]
2021-12-06 10:16:05 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
matt peters
763b1ad106 ZenStoreHttpClient: Expose connection failure to the Oplog owner to handle.
#rb Per.Larsson
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17661528 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)

[CL 17661553 by matt peters in ue5-release-engine-test branch]
2021-09-29 10:51:17 -04:00
per larsson
fa26f9c012 Make content-type/accept headers mandatory
#rb Stefan.Boberg
#rnx

#ROBOMERGE-AUTHOR: per.larsson
#ROBOMERGE-SOURCE: CL 17645289 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)

[CL 17645303 by per larsson in ue5-release-engine-test branch]
2021-09-28 09:53:06 -04:00
zousar shaker
96d0784963 Allow ZenServer to be autolaunched by editor use cases that require its presence without insisting that it always be autolaunched.
#rb matt.peters
#preflight 614b4150ee16e20001b146e3

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 17597162 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17597204 by zousar shaker in ue5-release-engine-test branch]
2021-09-22 13:00:17 -04:00
carlmagnus nordin
a627591ffa IoStore: Changes to allow IoStore to run as part of UnrealPak instead of as a cmdlet
#rb per.larsson
#rnx

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17517167 via CL 17517189 via CL 17517191 via CL 17519703
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v870-17433530)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17519714 by carlmagnus nordin in ue5-release-engine-test branch]
2021-09-15 10:39:23 -04:00
matt peters
56274a7640 ZenStoreWriter - Make CommitPackage take an array of attachments; remove separate variables for TargetDomainDependencies and send them in as an attachment.
ZenStoreHttpClient - Add IsConnected.
#rb CarlMagnus.Nordin, Stefan.Boberg
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17456396 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)
#ROBOMERGE[bot1]: dev-enginemerge-test

[CL 17456510 by matt peters in ue5-release-engine-test branch]
2021-09-08 07:04:39 -04:00
zousar shaker
8c80e5a866 Change Compact Binary Package API to represent binary attachments as compressed buffers identified by their raw hash.
Change Compact Binary Package serialization for binary attachments, compressed binary attachments and objects.

#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 17450533 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17450587 by zousar shaker in ue5-release-engine-test branch]
2021-09-07 18:07:00 -04:00