Commit Graph

322 Commits

Author SHA1 Message Date
per larsson
f1db8a6191 UnrealPak - dump bulk meta data to Json or Csv
Writes a package bulk data instances with offset, size and flags to the specified destination file.

Usage:
UnrealPak.exe -ListContainerBulkData=<Path/To/PakFolder/OrContainerFile> -cryptokeys=<Path> -Out=<Filename[.json|.csv]>

#rb Paul.Chipchase

[CL 30073803 by per larsson in ue5-main branch]
2023-12-04 04:27:29 -05:00
paul chipchase
4220a35232 Remove endpoint priming option (-PrimeEndPoint) from UnrealPak
#rb Per.Larsson
#rnx

- Although this priming did help in certain cases it is not really robust enough for most CDN implementations so removing it so that people do not try to use it thinking that it will help them.

[CL 30073410 by paul chipchase in ue5-main branch]
2023-12-04 03:21:13 -05:00
per larsson
7eb089a219 UnrealPak - expose -List option for dumping container file info to csv
#rb Martin.Ridgers

[CL 30037407 by per larsson in ue5-main branch]
2023-12-01 07:26:40 -05:00
paul chipchase
f604e5f8aa IAS can now be provided with a fallback cdn url to be used if the distributed endpoint cannot be contacted.
#rb PJ.Kack, Martin.Ridgers
#rnx

- If a user is unable to contact the distributed endpoint IAS will continue to retry the connection until it works. However it is possible that some users might never be able to contact the endpoint even if their network connection is okay. This means they would never be able to download IAS data and end up with poor visuals even though they should be able to access the data.
- To fix this we have added the ability to add a fallback url to the IoStoreOnDemand.ini which the user will try connecting to after a certain number of failed attempts at the distributed endpoint.
- To set the url add "-FallbackUrl=..." when uploading IAS data via UnrealPak.exe, although note that this will only be used if you are also providing "-DistributionUrl=..."
- The url can be overriden if needed by setting the cvar "ias.DistributedEndpointFallbackUrl" on startup
- The cvar "ias.DistributedEndpointAttemptCount" controls how many attempts the user should make at resolving the distributed endpoint before trying the fallback url. If the url is not set then this value is ignored and we will continue with the current behavior of retrying as many times as it takes.

[CL 29941432 by paul chipchase in ue5-main branch]
2023-11-27 14:37:57 -05:00
per larsson
c8fa897e90 IAS - separate out trailling host path when writing the configuraiton .ini file
#rb Martin.Ridgers

[CL 29838232 by per larsson in ue5-main branch]
2023-11-20 05:16:43 -05:00
dan thompson
c7e10fc6b3 Adding asset class count to plugin analytics.
#rb matt.breindel

[CL 29801691 by dan thompson in ue5-main branch]
2023-11-16 22:49:14 -05:00
per larsson
dbf2cf9839 IAS - list on demand TOC file(s)
A new command for listing on demand TOC files from an S3 compatible endpoint.
Usage: UnrealPak.exe -ListTocs -Region="us-east-1" -BucketPath=<some/bucket/path/> -CredentialsFile="s3_credentials.txt"

- write json file with -Json="Path/To/File.json".
- dump a specific toc: -TocKey=<IoHash>
- dump a specific build version: -BuildVersion=<Version>
- dump a specific build version and platform: -BuildVersion=<Version> -TargetPlatform=<Platform>
- dump chunk object keys with -ChunkKeys=<path/To/File.json>
- fetch a specific toc from a public CDN: -TocUrl="http://some.cdn.net/bd22076ca12bfc6feb982ffb064d18f28156606c.iochunktoc"

#rb Martin.Ridgers, Paul.Chipchase
#rnx

[CL 29136404 by per larsson in ue5-main branch]
2023-10-26 11:17:20 -04:00
dan thompson
58f2f4782b Rework the custom fields in the cook metadata to make it easier to use.
#rb matt.breindel

[CL 28922724 by dan thompson in ue5-main branch]
2023-10-19 12:37:07 -04:00
dan thompson
57af690b26 Shader dependencies incorrectly were added as base-1.
#rb non

[CL 28773150 by dan thompson in ue5-main branch]
2023-10-13 17:22:11 -04:00
zousar shaker
e95a2b2c30 Change default port for autolaunched ZenServer local service to 8558. When connecting to an existing service (not autolaunched), the default port remains what it was before, but will change in the future.
[CL 28665636 by zousar shaker in ue5-main branch]
2023-10-11 11:18:41 -04:00
dan thompson
1d24c2119d Fix for build warning with missing shader plugin dependencies.
#rb trivial

[CL 28428915 by dan thompson in ue5-main branch]
2023-10-03 16:05:43 -04:00
dan thompson
8c011f8fde Pack the plugin jsons in to one file to avoid nuking network file protocols.
#rb fabian.giesen

[CL 28396458 by dan thompson in ue5-main branch]
2023-10-02 17:34:41 -04:00
nick edwards
220ccc4b3e FIoDirectoryIndexReader::IterateDirectoryIndex: Use FStringViews and TStringBuilders instead of FStrings to avoid potentially unnecessary heap allocations
FPakPlatformFile::ForeachPackageInIostoreWhile: Update usage of IterateDirectoryIndex, and use overload of TryConvertFilenameToLongPackageName using string views and builders to avoid allocations

#rb elias.lozadabenavent
#rb anton.dunchev

[CL 28282945 by nick edwards in ue5-main branch]
2023-09-27 14:10:02 -04:00
mark lintott
ad27a196d8 Added support for Geometry adn Virtual Texture Builder types to telemetry
#rb trivial
[FYI] dan.thompson

[CL 28144192 by mark lintott in ue5-main branch]
2023-09-22 13:27:21 -04:00
matt peters
b6e8c4f2a9 Cooker: ZenStore -diffonly: Fix typo in AppendPathForPublicExportHash that caused an infinite loop.
#rnx
#rb None, trivial

[CL 28032424 by matt peters in ue5-main branch]
2023-09-20 08:41:40 -04:00
mark lintott
777d4cc80b Added Level, NiagaraSystem, MaterialInstance, Blueprint and AnimationSequence to asset telemetry tracking
Oprtimised plugin event data to filter out records where there is a zero inclusive size for a given asset type.
#rb dan.thompson

[CL 28000389 by mark lintott in ue5-main branch]
2023-09-19 12:43:31 -04:00
dan thompson
71d20ba15b Adding some sanity against uninitialized values and fixing the missing serialize.
#robert.millar

[CL 27980999 by dan thompson in ue5-main branch]
2023-09-18 19:32:50 -04:00
mark lintott
68ffbfd30b Added Level asset type to the IO Store telemetry
#rb pj.kack
[FYI] dan.thompson

[CL 27955014 by mark lintott in ue5-main branch]
2023-09-18 08:37:10 -04:00
josh adams
aeb5cb1347 - Restoring a backout after fixing the issue that causes the backout
[Backout] - CL27745134
[FYI] stan.hormell
Original CL Desc
-----------------------------------------------------------------
[Backout] - CL27731288 - CIS / Build Errors
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others

[CL 27923017 by josh adams in ue5-main branch]
2023-09-15 13:39:02 -04:00
robert millar
9fbd6babc7 Save script serialization offsets into packages that use versioned property serialization.
Offset is saved relative to export serial data.
Add new object version entry SCRIPT_SERIALIZATION_OFFSET.

#rb PJ.Kack, Francis.Hurteau
#lockdown marc.audy

[CL 27898807 by robert millar in ue5-main branch]
2023-09-14 18:18:29 -04:00
dan thompson
20e09bb684 Best guess as to the cause of the pak crash. No local repro, and this should probably be done anyway.
#rb none

[CL 27887689 by dan thompson in ue5-main branch]
2023-09-14 14:13:06 -04:00
matt peters
a77af3f0e8 ZenStore structures: expose structures to interpret zenstore headers for use in -diffonly cooks.
PackageStoreOptimizer: Expose a static version of FindScriptObject and functions for calculating PublicExportHash.
FNameMap: Add a foreach iterator to provide a list of all data in the struct.
ZenPackageHeader: Move from private to internal, add a constructor that returns an error string rather than fatally asserting.
#rb Per.Larsson
#rnx

[CL 27855318 by matt peters in ue5-main branch]
2023-09-13 18:11:20 -04:00
dan thompson
3226e61821 Include shader sizes in plugin size reporting.
Add shader information to the cook metadata file.
Update cook metadata dependency counts to uint32.
Separate out plugin graph generation from usage.
Add "other" size classification
#rb arciel.rekman
#rb fabian.giesen

[CL 27805498 by dan thompson in ue5-main branch]
2023-09-12 15:58:42 -04:00
stan hormell
37bad2d07b [Backout] - CL27731288 - CIS / Build Errors
[FYI] Josh.Adams
Original CL Desc
-----------------------------------------------------------------
- Deprecated GetSectionPrivate and FindOrAddSection, and accessors in FConfigFile that could return a non-const FConfigSection (this is so we can track modifications to config values)
- Added AddToSection, RemoveKeyFromSection, etc to replace directly accessing a FConfigSection
- Fixed up Epic code for the deprecations (at least the majority, some projects that aren't built by Horde presubmit may have some that we will address going forward)
#jira UE-194955
#rb david.harvey and various others

[CL 27745141 by stan hormell in ue5-main branch]
2023-09-10 22:58:21 -04:00
eric knapik
7c24cc7a5c #jira: FORT-601192
Fixup cookmetadata.ucookmeta after we join the cook outputs. Update the ucookedmeta file to have the same hash as the newly made combined devAR.
Added Save and Load functions to the cooked meta file

[REVIEW] [at]Dan.Thompson [at]Matt.Breindel

[CL 27734479 by eric knapik in ue5-main branch]
2023-09-08 19:51:52 -04:00