Commit Graph

300 Commits

Author SHA1 Message Date
dan thompson
3226e61821 Include shader sizes in plugin size reporting.
Add shader information to the cook metadata file.
Update cook metadata dependency counts to uint32.
Separate out plugin graph generation from usage.
Add "other" size classification
#rb arciel.rekman
#rb fabian.giesen

[CL 27805498 by dan thompson in ue5-main branch]
2023-09-12 15:58:42 -04:00
stan hormell
37bad2d07b [Backout] - CL27731288 - CIS / Build Errors
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others

[CL 27745141 by stan hormell in ue5-main branch]
2023-09-10 22:58:21 -04:00
eric knapik
7c24cc7a5c #jira: FORT-601192
Fixup cookmetadata.ucookmeta after we join the cook outputs. Update the ucookedmeta file to have the same hash as the newly made combined devAR.
Added Save and Load functions to the cooked meta file

[REVIEW] [at]Dan.Thompson [at]Matt.Breindel

[CL 27734479 by eric knapik in ue5-main branch]
2023-09-08 19:51:52 -04:00
josh adams
a8a9a0b759 - Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others

[CL 27731364 by josh adams in ue5-main branch]
2023-09-08 17:58:55 -04:00
dan thompson
7db58e5967 Bumping schema version since i changed it.
[FYI] mark.lintott
#rb none

[CL 27540389 by dan thompson in ue5-main branch]
2023-08-31 17:50:42 -04:00
paul chipchase
a9e7ca0ec2 Improve the 'PrimeEndPoint' command to run many times faster and add additional options.
#rb Per.Larsson
#rnx

- Moved the priming code to it's own code files.
- Priming no longer uses FOnDemandIoBackend but manages its own requests.
- Bumped up the number of connections to the largest supported by FEventLoop (63) to speed up the priming as much as possible.
- Added additional cmdline options although -PrimeEndPoint is still required:
-- '-PrimeAll' will cause all available CDNs to be primed.
-- '-PrimeUrl=' will override the CDN url with the one provided.
--- '-List' will print all of the available CDN urls to screen along with the average latency to it.
-- '-Help' prints help on the new commands to the screen.
- Fix minor bug where UnrealPak was returning none zero for success and zero for error cases.
- Automatically call ::ResolveDeferredEndpoints as part of FDistributionEndpoints::Flush if needed. Failure to do so can lead to flush waiting forever on work that has never been initiated.

[CL 27514497 by paul chipchase in ue5-main branch]
2023-08-31 04:08:32 -04:00
per larsson
5f7f91f8bc IAS - remove warning about unencrypted PAK's
#rb Paul.Chipchase
#rnx

[CL 27513771 by per larsson in ue5-main branch]
2023-08-31 02:48:02 -04:00
tomasz obrebski
9e09fa9069 Zen header alignment
#jira UE-179443
#rb pj.kack

[CL 27440335 by tomasz obrebski in ue5-main branch]
2023-08-29 07:38:16 -04:00
per larsson
c06a27addc VFS - return error if any chunk fails to write to disk
#rb Dan.Engelbrecht
#rnx

[CL 27407872 by per larsson in ue5-main branch]
2023-08-28 05:49:14 -04:00
mark lintott
f009b64063 Added Skeletal Mesh asset support to IO Store Telemetry
#rb pj.kack
[FYI] dan.thompson

[CL 27338727 by mark lintott in ue5-main branch]
2023-08-24 09:03:18 -04:00
martin ridgers
a2ef72b131 Moved everything Ias-related into its own consistent UE::IO::IAS namespace
#jira
#rnx
#rb

[CL 27299509 by martin ridgers in ue5-main branch]
2023-08-23 02:38:00 -04:00
pj kack
3b92239e69 IoStore: Make sure the FZenPackageHeader array views starting with ImportedPublicExportHashes are 8-byte aligned
#jira UE-192747
#rb per.larsson

[CL 27135261 by pj kack in ue5-main branch]
2023-08-16 06:18:19 -04:00
dan thompson
5899f2b386 Fix for log spew when packaging DLC plugins.
#rb fabian.giesen

[CL 27089868 by dan thompson in ue5-main branch]
2023-08-14 18:05:57 -04:00
dan thompson
a94a34de51 * Add a system for routing plugin custom fields in to the cook metadata file.
* Use that system to upload whether a plugin is installed or not to analytics.
* Compute "unique" size for root plugins to more accurately represent the incremental cost of a plugin in analytics.
* Provide sizes for key asset types in addition to tracking total cost of a plugin.
#rb matt.peters
#rb fabian.giesen

[CL 27084058 by dan thompson in ue5-main branch]
2023-08-14 16:03:52 -04:00
dan thompson
ef526cc050 Fix for asset registry writeback hash not matching when plugin size writeback is disabled.
#rb fabian.giesen

[CL 27032181 by dan thompson in ue5-main branch]
2023-08-11 12:44:50 -04:00
paul chipchase
935b554612 Add a new cmdline option to UnrealPak allowing all IAS content to be primed on an endpoint, so that testers can have the same experience as an end users.
#rb Per.Larsson
#rnx

### New Command
- Run UnrealPak.exe with the commandline "-PrimeEndPoint=ABC" where ABC is a path to a IoStoreOnDemand.ini containing the info about the end point to prime.
- The command will mount the correct backend, connect to the end point, download the utoc files there and then attempt to download each chunk present so that they will be cached by the CDN endpoint.
- The command does not use the IAS filecache and should not affect the state of the local machine.
- We also make use of the existing IAS code, so this path can also be used for profiling when the IAS requests are saturated etc.

### Code Changes

- The code for parsing the ini file in FIoStoreOnDemandModule::StartupModule has been moved to common utility code so that it can be reused.

[CL 27026561 by paul chipchase in ue5-main branch]
2023-08-11 09:49:46 -04:00
zousar shaker
98800ff6a4 When writing file manifest to zenstore, don't attempt to set both the "data" field on a file entry and the "serverpath" field on the file entry, as zenserver will treat the two as mutually exclusive, and setting the "data" field will mean we no loger get "serverpath" out of zenserver. This causes issues when doing an iterative cook and trying to read the file manifest back from zenserver.
#rb matt.peters

[CL 26980097 by zousar shaker in ue5-main branch]
2023-08-09 19:55:19 -04:00
bob tellez
de0ac0a5c5 #UE Only assert that you have package data to write when you are intending to write data or computehash. The new SkipSave cook option does not write data intentionally
[FYI] Matt.Peters

[CL 26837731 by bob tellez in ue5-main branch]
2023-08-03 23:00:11 -04:00
per larsson
8d6ce7dbb0 I/O - minor change to summary stats
#rb trivial
#rnx

[CL 26704583 by per larsson in ue5-main branch]
2023-07-31 09:40:50 -04:00
Matt Peters
69174cd749 Cooker: ZenStore: Downgrade ZenFileSystemManifest's per-plugin log messages to verbose to make it easier to read other end of cook logs.
#rnx
#rb Dan.Engelbrecht

[CL 26463793 by Matt Peters in ue5-main branch]
2023-07-19 12:34:11 -04:00
Matt Peters
13643c4f10 ZenWriter: Log the fatal assertion about invalid CommitPackage call on the CommitPackage thread in addition to the async thread so it is easier to debug.
#rb Zousar.Shaker
#rnx

[CL 26355924 by Matt Peters in ue5-main branch]
2023-06-30 17:07:55 -04:00
dan thompson
54f13432d6 Misc CookMetadataFixes:
* Restaging a build no longer just constantly increases size
* Inclusive size actually looks at valid data
* Horde URL environment variable corrected.
* Add a new root pseudo plugin json event that holds the size of all plugins that don't belong to a root plugin.
#rb charles.bloom

[CL 26335038 by dan thompson in ue5-main branch]
2023-06-29 23:20:46 -04:00
Matt Peters
ec9fef914b Cooking: Iterative: Add -iterativevalidate mode that diffs iteratively unmodified previous packages against their resaved version.
#rb Zousar.Shaker
#rnx

[CL 26327591 by Matt Peters in ue5-main branch]
2023-06-29 16:40:47 -04:00
per larsson
24d9d8349c IAS - support for downloading on demand container file(s) with UnrealPak
Usage: UnrealPak.exe -Download=<RemoteTocPath> -Directory=<LocalPath> -ServiceUrl=<URL> -Bucket=<BucketName>

#rb none
#rnx

[CL 26307209 by per larsson in ue5-main branch]
2023-06-29 05:28:06 -04:00
dan thompson
e43078eb06 Add different size types to development asset registry.
Put the tag names in a global place to prevent string copy pasta.
#rb fabian.giesen

[CL 26272535 by dan thompson in ue5-main branch]
2023-06-27 17:50:24 -04:00