Commit Graph

49 Commits

Author SHA1 Message Date
paul chipchase 935b554612 Add a new cmdline option to UnrealPak allowing all IAS content to be primed on an endpoint, so that testers can have the same experience as an end users.
#rb Per.Larsson
#rnx

### New Command
- Run UnrealPak.exe with the commandline "-PrimeEndPoint=ABC" where ABC is a path to a IoStoreOnDemand.ini containing the info about the end point to prime.
- The command will mount the correct backend, connect to the end point, download the utoc files there and then attempt to download each chunk present so that they will be cached by the CDN endpoint.
- The command does not use the IAS filecache and should not affect the state of the local machine.
- We also make use of the existing IAS code, so this path can also be used for profiling when the IAS requests are saturated etc.

### Code Changes

- The code for parsing the ini file in FIoStoreOnDemandModule::StartupModule has been moved to common utility code so that it can be reused.

[CL 27026561 by paul chipchase in ue5-main branch]
2023-08-11 09:49:46 -04:00
Matt Peters ec9fef914b Cooking: Iterative: Add -iterativevalidate mode that diffs iteratively unmodified previous packages against their resaved version.
#rb Zousar.Shaker
#rnx

[CL 26327591 by Matt Peters in ue5-main branch]
2023-06-29 16:40:47 -04:00
per larsson 24d9d8349c IAS - support for downloading on demand container file(s) with UnrealPak
Usage: UnrealPak.exe -Download=<RemoteTocPath> -Directory=<LocalPath> -ServiceUrl=<URL> -Bucket=<BucketName>

#rb none
#rnx

[CL 26307209 by per larsson in ue5-main branch]
2023-06-29 05:28:06 -04:00
per larsson 3bed8ef537 I/O - support for uploading on demand I/O store containers to S3 with UnrealPak
- added support for uploading containers marked as on demand directly from UnrealPak.exe
- removed C# based upload logic from automation scripts
- removed on demand I/O store writer since this change reads chunks directly from container files instead of loose files

Example usage:
UnrealPak.exe -Upload=<ContainerPathOrWildcard> -ServiceUrl=<URL> -Bucket=<BucketName> -AccessKey=<Key> -SecretKey=<Key>

Read credentials from an AWS key chain file with the following command line:
-CredentialsFile=<Path> -CredentialsFileKeyName=<EntryName>

Specify -KeepUploadedContainers to prevent UnrealPak to delete on demand containers after the upload has been completed.
Specify -BucketPrefix=<Path> to upload chunks to a specific sub directory within the bucket.

#rb none

[CL 26115169 by per larsson in ue5-main branch]
2023-06-20 04:21:22 -04:00
Matt Peters 88d33c5f37 Cooker and SavePackage: Move calls to BeginCacheForCookedPlatformData and ClearCachedCookedPlatformData from SavePackage into the cooker. Moving the Clear calls allows us to clear only after repeated SavePackage attempts from the diffonly cooker have been completed, preventing a break in contract by having objects being saved after Clear is called. Moving the Begin calls is necessary to track which objects need Clear called on them.
#jira FORT-613920
#rn Minor, Cooking
#rb Zousar.Shaker

[CL 25851419 by Matt Peters in ue5-main branch]
2023-06-07 13:56:57 -04:00
zousar shaker b0d5721dde Initial implementation of DiffOnly mode for ZenServer as Cooked Output Store. Lacking detailed header diffing, but stubbed in for future addition. When using ZenServer as Cooked Output Store the packages use the ZenPackageSummary header format instead of the PackageFileSummary header format. This is how data is stored in ZenServer and the format it is when fetched. Data produced in process is transformed into this format ahead of diffing.
The code for parsing a view of the ZenPackageSummary has been extracted from AsyncLoading2.cpp into a private h/cpp and is possible to use from the header diffing code when printing out detailed diffs, but is not used yet.

#rb matt.peters
#rb pj.kack
#jira UE-175237

[CL 25549055 by zousar shaker in ue5-main branch]
2023-05-19 14:43:02 -04:00
lukasz furman 574becb64b exposed IoStore extract loop for in memory processing and added optional task throttle
added build scripts for content validations

[CL 25473614 by lukasz furman in ue5-main branch]
2023-05-15 10:30:13 -04:00
dan engelbrecht db5e568754 Game client no longer needs -zenstoreproject or -zenstorehost arguments to correctly run with a COTF server which uses Zen.
Game client now accepts only -filehostip for connection to a CTOF server, -cookonthefly is no longer necessary and is ignored.
Game client will now receive ProjectName (ProjectId), Platform (OplogId), Zen server host name and port from COTF server if running with zenstore.
Fixed issues with generating the ProjectId from a path where drive letters where lower case.

#rb zousar.shaker pj.kack
#preflight

[CL 24889513 by dan engelbrecht in ue5-main branch]
2023-04-03 01:51:40 -04:00
carlmagnus nordin 024df6d65c ZenCook: Added support for file regions
#preflight 64269c8491589478cdeb2a6f
#rb pj.kack
#rnx

[CL 24869761 by carlmagnus nordin in ue5-main branch]
2023-03-31 08:36:14 -04:00
carlmagnus nordin 30c0f28fc0 ZenCook: Merge the oplog and package store manifest. Write an oplog instead of a package store manifest when cooking to loose files and read the oplog instead of the package store manifest to find cooked files in UAT when staging from Zen.
#rb per.larsson
#preflight 640efc9bc41a0a2a7799d470
#rnx

[CL 24632735 by carlmagnus nordin in ue5-main branch]
2023-03-14 06:43:23 -04:00
carlmagnus nordin 641eb5b626 ZenCook: Added support for optional package segments
#jira UE-148021
#preflight 63f365ace832ea950a6217e4
#rb per.larsson

[CL 24343344 by carlmagnus nordin in ue5-main branch]
2023-02-21 13:03:12 -05:00
Matt Peters fa59e5fba4 MPCook: ZenStoreWriter: Replicate ZenStoreWriter artifacts data.
#rnx
#rb Zousar.Shaker
#preflight 63dd7dd85c0ce8f11b71f405

[CL 23999102 by Matt Peters in ue5-main branch]
2023-02-03 17:38:06 -05:00
pj kack 5450132470 UnrealPak: Improve diffutils
Add support for diffing pak directories for pak diff.
Add support for different cryptokeys for pak diff and legacy iostore diff.

#jira UE-175144
#rb carlmagnus.nordin
#rnx
#preflight 63d9259e7a39a18021d4f997

[CL 23945218 by pj kack in ue5-main branch]
2023-02-01 03:39:07 -05:00
carlmagnus nordin 9f028b8744 ZenCook: Remove unused fields "guid" and "disksize" from oplog
#preflight 63d8ede38505ea6b1f7e96b6
#rb per.larsson

[CL 23927486 by carlmagnus nordin in ue5-main branch]
2023-01-31 10:39:54 -05:00
Matt Peters 5d2f838486 Cooker: MPCook: Replicate package hashes in LooseCookedPackageWriter.
#rb Zousar.Shaker
#rnx
#preflight 63d431bc5428dc67b168e481

[CL 23899261 by Matt Peters in ue5-main branch]
2023-01-28 14:33:10 -05:00
Matt Peters 178d5856dc MPCook - PackageWriter and IOStore API changes to support MPCook.
Add FCookInfo argument to BeginCook and EndCook, the same data passed to initialize. It is provided in Begin/End as well since Writers need to consider it for Begin/End actions.
Add WriteMPCookMessageForPackage/TryReadMPCookMessageForPackage functions for replicating information from PackageWriters from CookWorker to CookDirector.
Add WritePackage/TryReadPackage to Manifest. Some extra information is collected when MPCook is active to support this replication.
Add CompactBinary serialization functions to FIoChunkId.
#rbx
#preflight 63d1c62a45795e7d5c164b0e

[CL 23872901 by Matt Peters in ue5-main branch]
2023-01-26 14:32:12 -05:00
Zousar Shaker b9548ddad4 Move Zen module public headers into an "Experimental" subfolder to have them reflect the feature status.
Fix locations that include the headers to use the "Experimental" subfolder and re-sort includes.  No functional change, purely a move, search+replace, and sort of includes.

#rb matt.peters
#preflight 63939eebc415e8dc78d35d63

[CL 23468934 by Zousar Shaker in ue5-main branch]
2022-12-09 16:11:41 -05:00
matt peters daff9f8c5d EditorDomain: Implement support for PackageTrailers in the same package as BulkData in a regular BulkData section for PackageWriters, so that EditorDomain can have both references to BulkData in a WorkspaceDomain PackageTrailer and embedded BulkDatas that needed to be resaved.
Because the PackageFooter comes before the PackageTrailer, we need to change accounting for the PackageFooter in PackageWriters to include adjusting the VirtualExportsFileOffset that is passed into the PackageTrailer.
#rb Paul.Chipchase
#rnx
#preflight 638cbb1135192facc13f83e6

[CL 23386395 by matt peters in ue5-main branch]
2022-12-04 12:54:00 -05:00
graeme thornton faf6311ef1 IoStore: Add functionality to sign a container as a separate step
[FYI] carlmagnus.nordin

[CL 23312469 by graeme thornton in ue5-main branch]
2022-11-29 12:35:12 -05:00
carlmagnus nordin ffd1e2abfb ZenCook: Use the oplog for package store entries instead of writing a container header
#preflight 63720da8ee4d25f90aacf4dd
#rb per.larsson

[CL 23116204 by carlmagnus nordin in ue5-main branch]
2022-11-14 06:35:53 -05:00
dan engelbrecht 2f8823f7dc Pass along path to project file when zen server is running locally so we can detect when it is removed and garbage-collect any projectstore entries on the zen side.
Allow Cook to remote Zen, if we detect that we are targetting a zenservice on a different machine we change the local file references to attachment based references.
#rb stefan.boberg
#preflight 63491c31ad0f7e2f20230b39

[CL 22525021 by dan engelbrecht in ue5-main branch]
2022-10-14 08:17:43 -04:00
bryan sefcik de1956f47b Ran IWYU on Public headers under Engine/Source/Developer/...
Headers are updated to contain any missing #includes needed to compile and #includes are sorted.  Nothing is removed.

#ushell-cherrypick of 21064294 by bryan.sefcik
#jira
#preflight 62d5c2111062f2e63015e598

#ROBOMERGE-OWNER: bryan.sefcik
#ROBOMERGE-AUTHOR: bryan.sefcik
#ROBOMERGE-SOURCE: CL 21155249 via CL 21158121 via CL 21161259
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21182053 by bryan sefcik in ue5-main branch]
2022-07-20 12:03:45 -04:00
Matt Peters 02ba0b5b70 ZenStoreWriter: Fix assertion "Trying to add package that is already pending" if the cooker commits a package a second time while the first commit of the package is queued for asynchronous completion aand has not yet reached CommitPackageInternal.
This is necessary because as of CL 20151503, the cooker can commit the same package multiple times if the first save failed due to a timeout.

#fyi CarlMagnus.Nordin
#rnx
#rb Per.Larsson
#preflight 62853ba90f0d5076dee74ced

[CL 20265738 by Matt Peters in ue5-main branch]
2022-05-18 14:49:11 -04:00
CarlMagnus Nordin 63b8cbc9dd AsyncLoading2: Implemented loading of optional package segments
#rnx
#preflight 62627548ac29fcf6a213fe33
#rb francis.hurteau

[CL 19900368 by CarlMagnus Nordin in ue5-main branch]
2022-04-25 07:37:07 -04:00
Per Larsson ddbf9eb5d2 Zen - cook to custom project/oplog ID with -ZenStoreProject=<ProjectId> and -ZenStorePlatform=<OplogId>
#rb none
#rnx
#preflight 19755222

[CL 19804798 by Per Larsson in ue5-main branch]
2022-04-19 02:37:06 -04:00