Commit Graph

43 Commits

Author SHA1 Message Date
Per Larsson
7f07f1c443 Made path to package/cooker order and output directory safe to use with command line for IoStore commandlet.
#jira UE-89293
#rb none

[CL 11894592 by Per Larsson in 4.25 branch]
2020-03-04 03:03:56 -05:00
Stefan Boberg
55a8b15f1f Copying //UE4/Dev-Core to Main (//UE4/Main)
#rb none

[CL 11258477 by Stefan Boberg in Main branch]
2020-02-05 14:26:36 -05:00
CarlMagnus Nordin
4b67fd622f Use CookerOpenOrder when building IoDispatcher containers
#rb per.larsson

[CL 11127827 by CarlMagnus Nordin in Dev-Core branch]
2020-01-28 06:49:34 -05:00
Per Larsson
89e82b7e5c IoStoreUtilities: Parallel serialize container files and cpu trace scopes
#rb CarlMagnus.Nordin, PJ.Kack

[CL 11114809 by Per Larsson in Dev-Core branch]
2020-01-27 07:18:53 -05:00
PJ Kack
14840e11d3 AsyncLoading2: Fix replay check failures when acessing shader code library entries before their corresponding external callbacks have been executed.
Add post load arcs for all packages for now.
They are actually only required for circular packages and packages with external read dependencies (but these are currently only known in runtime).

#fyi paul.chipchase
#rb none

[CL 11114572 by PJ Kack in Dev-Core branch]
2020-01-27 03:35:34 -05:00
PJ Kack
8596dbff4f IoStore: Linux compile fixes.
#rb none
#fyi per.larsson

[CL 11104072 by PJ Kack in Dev-Core branch]
2020-01-24 07:05:10 -05:00
Per Larsson
7b8fb88163 IoStoreUtilities: Refactored custom serialization to use standard serialization and added parallel parsing of uasset files
#rb CarlMagnus.Nordin

[CL 11103704 by Per Larsson in Dev-Core branch]
2020-01-24 03:50:37 -05:00
PJ Kack
6795731833 IoStore: Replace individual file GetStatData calls with one directory scan of the cooked directory.
#rb none

[CL 11093594 by PJ Kack in Dev-Core branch]
2020-01-23 10:46:14 -05:00
paul chipchase
d8f0d3753b Fix an issue that was causing occasional duplicate FIoChunkIds when generating them for Bulkdata.
Changed the way we generate the FIoChunkId by reducing the chance of losing data when discarding the top byte of the BulkDataChunkId.
Add a log error if we find we have discarded data

#rb none
#rnx

[CL 11082720 by paul chipchase in Dev-Core branch]
2020-01-22 10:29:25 -05:00
CarlMagnus Nordin
365c9d5723 IoStore:
Refactored to have access to the size of each chunk before the chunk is added to a container
Basic chunk ordering
Optional output of .csv container manifests

#rb none

[CL 11082359 by CarlMagnus Nordin in Dev-Core branch]
2020-01-22 09:38:57 -05:00
paul chipchase
81e837af05 The BulkData manifest can no longer be broken by changing the values of EIoChunkType.
Note that this will invalidating existing manifests and require a new cook!

- We have had several cases of EIoChunkType having new entries inserted at the start causing the values for bulkdata to change which in turn would invalidate the contents of the BulkData manifest, so to avoid issues in the future the manifest will now save out its own values and we only convert them to EIoChunkType at the last possible moment when building the ucas files.
- Added a custom version to the manifests file archive, this will make it safer to edit the format in the future as we will be able to avoid invalidating the entire file.
 - Unfortunatly adding the version does not help with existing files which are now invalid.

#rb CarlMagnus.Nordin
#rnx

[CL 11042093 by paul chipchase in Dev-Core branch]
2020-01-17 08:18:31 -05:00
paul chipchase
46aa59430f Fix some coding standards mistakes I made when originally submitted FPackageStoreBulkDataManifest.
#rb trivial
#rnx

[CL 11028252 by paul chipchase in Dev-Core branch]
2020-01-16 07:28:59 -05:00
CarlMagnus Nordin
6abf1ca5c9 Create one IoDispatcher container file per pak file and let the pak file system handle mounting of IoStores
#rb per.larsson

[CL 11027862 by CarlMagnus Nordin in Dev-Core branch]
2020-01-16 06:01:45 -05:00
paul chipchase
7631481031 Move FPackageStoreBulkDataManifest to its own code files to help cut down on the complexity of SavePackage.cpp
#rb trivial
#rnx

[CL 11027753 by paul chipchase in Dev-Core branch]
2020-01-16 05:06:40 -05:00
CarlMagnus Nordin
fab9e500ad AsyncLoading2: Common header for loader and IoStoreUtils
#rb PJ.Kack


#ushell-cherrypick of 10950326 by Per.Larsson

[CL 10994443 by CarlMagnus Nordin in Dev-Core branch]
2020-01-15 12:28:12 -05:00
ryan durand
84e0b188d8 Updating copyright for Engine Developer.
#rnx
#rb none

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: ryan.durand
#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902 via CL 10870584
#ROBOMERGE-BOT: CORE (Main -> Dev-Core) (v626-10872990)

[CL 10897060 by ryan durand in Dev-Core branch]
2020-01-07 15:07:57 -05:00
Per Larsson
6223092d73 AsyncLoading2: Runtime filtering of client/server only exports
- needs gc.UseDisregardForGCOnDedicatedServers=True

#rb PJ.Kack

[CL 10890171 by Per Larsson in Dev-Core branch]
2020-01-07 09:08:18 -05:00
PJ Kack
99ff0ee38d AsyncLoading2: Remove all PostLoad arcs for non-circular packages.
Rename Event_StartPostLoad to Event_PostLoad and let it run all Event_Tick logic.
Change processing of ExternalReadQueue to do at most most one blocking wait per iteration.

#rb none


#ushell-cherrypick of 10703723 by PJ.Kack

[CL 10888736 by PJ Kack in Dev-Core branch]
2020-01-07 05:05:23 -05:00
ryan durand
471d972e62 Updating copyright for Engine Developer.
#rnx
#rb none


#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902
#ROBOMERGE-BOT: (v613-10869866)

[CL 10870584 by ryan durand in Main branch]
2019-12-26 15:32:37 -05:00
PJ Kack
47a384fa28 IoStore: Fix IoDispatcher shutdown crash by moving shutdown to end of AppPreExit (i.e. after UObjectBaseShutdown and ShutdownAsyncThread, but before UnloadModulesAtShutdown)
Fix parsing of chunkmanifest to handle options and skip creating chunks without files.
Optimize file exists and recursive find to do one single file system traversal only.
Fix "Serializing..." output to display total number of packages

#rb per.larsson

[CL 10760676 by PJ Kack in Dev-Core branch]
2019-12-17 04:48:08 -05:00
paul chipchase
92db373ee6 The IoStore commandlet now accepts '-targetplatform=*' in the same format that the cook commandlet does. This allows us to request that we create the IoStore data for multiple platforms at a time ie '-targetplatform=A+B+etc'
- Replace the manual command line parsing with the ITargetPlatformManagerModule instead, this will keep the behaviour similar to the cook commandlet.
- Note that this will change functionality, so that running the IoStore commandlet without '-targetplatform=XXX' will now attempt to run on all available platforms.
- Made a few things const.

#rb Per.Larsson

[CL 10760400 by paul chipchase in Dev-Core branch]
2019-12-17 02:47:23 -05:00
Robert Manuszewski
7b6f840f7f Copying //UE4/Dev-Core @ 10708550 to Dev-Main (//UE4/Dev-Main)
#rb none

[CL 10708666 by Robert Manuszewski in Main branch]
2019-12-13 11:07:03 -05:00
paul chipchase
bb86b9afdb BulkData2 will now be able to work with 0 sized BulkData, which maintains the exisiting behaviour of the original BulkData api.
- If the data is missing then a lot of features just won't work but since we were silently allowing it with the old BulkData api we should maintain this for now.
- When saving out a 0 byte BulkData entry to the manifest we instead add a special entry that will be 0 size and at the start of the bulkdata file. We also do not warn if this is a dupe.
- Moved around the code when adding BulkData to the IoStore so that we still create mappings for BulkData entries even in the case that all BulkData for a package is 0 sized and we don't have the file at all.
- Check for the size when creating the FIoChunkID at runtime and if 0 then use the special id instead of the offset.

#rnx
#rb none
#jira UE-85581
#lockdown robert.manuszewski

[CL 10672514 by paul chipchase in Dev-Core branch]
2019-12-12 10:11:17 -05:00
CarlMagnus Nordin
f5e4762b22 IoDispatcher:
Split container files based on pak chunking

#rb per.larsson

[CL 10669979 by CarlMagnus Nordin in Dev-Core branch]
2019-12-12 06:39:05 -05:00
CarlMagnus Nordin
424b967ff7 IoStore:
Tactical submit to prepare for pakchunking
#rb none

[CL 10634717 by CarlMagnus Nordin in Dev-Core branch]
2019-12-10 04:47:36 -05:00