Commit Graph

189 Commits

Author SHA1 Message Date
carlmagnus nordin
641eb5b626 ZenCook: Added support for optional package segments
#jira UE-148021
#preflight 63f365ace832ea950a6217e4
#rb per.larsson

[CL 24343344 by carlmagnus nordin in ue5-main branch]
2023-02-21 13:03:12 -05:00
carlmagnus nordin
301c2993b7 IoStore: Error checking when reading compressed data from DDC
#preflight 63eb43ccff4fbdc05c4640a3
#rb pj.kack

[CL 24213741 by carlmagnus nordin in ue5-main branch]
2023-02-14 12:03:12 -05:00
anton dunchev
990e8f0d44 MemQuery use PackageStore to resolve dependencies
#tests replay
#rb Robert.Millar
[FYI] andrew.ladenberger, simon.orr

[CL 24202843 by anton dunchev in ue5-main branch]
2023-02-13 22:45:46 -05:00
dan thompson
c0c06db0d5 Adding a bunch of iostore/unrealpak performance stats and fixing a merge error with the reference db on optional containers.
#rb matt.peters
#preflight 63ea85f8ff4fbdc05c00d07e

[CL 24197489 by dan thompson in ue5-main branch]
2023-02-13 20:09:46 -05:00
pj kack
eb2c730a89 UnrealPak: Add -csv parameter to create pak and create ucas commands to list all the chunks.
Output is the same as when executing the -list command separately.
Running with -csv=<dir> will create one <pakchunkfilename>.pak.csv file per generated pak and one <pakchunkfilename>.utoc.csv file per generated container.
Running with -csv=<file> will create one single <file>.pak.csv for all paks and one <file>.utoc.csv for all containers.

Details:
The container output (the big set of data) is generated based on in-memory writer TOC data once the IoStoreWriters have been finalized.
The pak output (smaller set of data) is generated using the existing list command and loads the newly generated files from disk.
Cleanup the remains of three obsolete csv output features for containers (-csvoutput, -writefinalorder and OUTPUT_CHUNKID_DIRECTORY).
Optimize container output generation by extending FIoStoreTocChunkInfo with OffsetOnDisk and NumCompressedBlocks filled directly by EnumerateChunks.
Add a ChunkIdToFileNameMap to FCookedPackageStore.
Use the FCookedPackageStore to output the actual package ids and package names (this is currently broken when running -list as a separate command).

#rb carlmagnus.nordin
#rnx
#preflight 63e5eacb98775169f8dea431

[CL 24118941 by pj kack in ue5-main branch]
2023-02-10 10:13:22 -05:00
carlmagnus nordin
c456d29b73 ZenCook: Add missing generation of package ids when serializing package store entries
#preflight 63e601ba1b05e5a4b6a3ccfa
#rb pj.kack

[CL 24116693 by carlmagnus nordin in ue5-main branch]
2023-02-10 05:39:55 -05:00
dan thompson
5e26b56715 Fix for parsing validate flag.
Set the reference chunk db on the optional writers.
Disable reference chunk use for uncompressed chunks.
Failing to load the reference chunk db is now a warning rather than a failure, since pak can succeed without it.
#rb none
#preflight 63e434bf7467a6960f7d31db

[CL 24110712 by dan thompson in ue5-main branch]
2023-02-09 19:38:49 -05:00
henrik karlsson
c8c3d0b8b2 [Core]
* Made code compatible with modules/header units. This includes following things:
   * Removed static in front of global functions/variables (static indicates it is hidden outside of module/header unit which is not what many places want...)
   * Moved dllexport from type to methods in order to get dllexport of static constexpr fields. This is supposed to work according to microsoft but since it is a good change I don't mind doing it anyway)
   * static constexpr -> static constexpr inline for member variables and methods (this makes them available outside module/headerunit)
   * Added HEADER_UNIT_IGNORE markup to includes that are circular
   * Added HEADER_UNIT_SKIP to files that can't compile by themselves

#preflight 63e2aeba3c44c83044bfcc75
#rb steve.robb

[CL 24066986 by henrik karlsson in ue5-main branch]
2023-02-08 00:25:15 -05:00
pj kack
dc4db38c86 UnrealPak: Skip DDC initialization when running without -compressionEnableDDC.
#rb carlmagnus.nordin
#rnx
#preflight
#preflight 63e261f83c44c83044b52dec

[CL 24051192 by pj kack in ue5-main branch]
2023-02-07 12:06:47 -05:00
pj kack
c1dfe83c61 UnrealPak: Add more fine grained trace cpu scopes for profiling on remote machines.
#rb trivial
#rnx
#preflight 63e10793b923ed4bddea469f

[CL 24031827 by pj kack in ue5-main branch]
2023-02-06 11:39:10 -05:00
per larsson
9931407ef5 I/O - hex to FIoChunkId
#rb Paul.Chipchase
#rnx
#preflight 63de302978716a01e837afea

[CL 24028928 by per larsson in ue5-main branch]
2023-02-06 08:19:43 -05:00
carlmagnus nordin
4f6431a740 Add support for specifying priority when mounting iodispatcher and packagestore backends
#preflight 63d8ed7dd21dbe1d29b81bf4
#rb per.larsson

[CL 23927481 by carlmagnus nordin in ue5-main branch]
2023-01-31 10:39:49 -05:00
martins mozeiko
dddf036247 Reduce amount of malloc calls during pak creation
1) Move few containers outside of inner loop to reuse same memory across iterations in AssignPackagesDiskOrder
2) Change FString to FStringView when list of filenames is prepared during FIoStoreWriter::Finalize

#rb fabian.giesen
#preflight 63d826653656ea96dc1fabfe

[CL 23923937 by martins mozeiko in ue5-main branch]
2023-01-31 01:26:03 -05:00
Matt Peters
200f98dd5f MPCook: Sort the output of IoStore PackageStoreManifest when writing to disk to make it deterministic.
#rnx
#rb Zousar.Shaker
#preflight 63d6b9b15354589b5c9b1668

[CL 23900962 by Matt Peters in ue5-main branch]
2023-01-29 13:46:27 -05:00
Matt Peters
178d5856dc MPCook - PackageWriter and IOStore API changes to support MPCook.
Add FCookInfo argument to BeginCook and EndCook, the same data passed to initialize. It is provided in Begin/End as well since Writers need to consider it for Begin/End actions.
Add WriteMPCookMessageForPackage/TryReadMPCookMessageForPackage functions for replicating information from PackageWriters from CookWorker to CookDirector.
Add WritePackage/TryReadPackage to Manifest. Some extra information is collected when MPCook is active to support this replication.
Add CompactBinary serialization functions to FIoChunkId.
#rbx
#preflight 63d1c62a45795e7d5c164b0e

[CL 23872901 by Matt Peters in ue5-main branch]
2023-01-26 14:32:12 -05:00
dan thompson
74bb7ea87f IoStoreWriterHashDatabase - add a way for the iostore writers to access hashes that have already been calculated for chunks, to save pak time.
#rb pj.kack
#preflight 63d03c01d83c1837b1b6bc9b

[CL 23843548 by dan thompson in ue5-main branch]
2023-01-24 21:44:25 -05:00
pj kack
d7328b7b98 IoDispatcher: Fix shutdown crash by shutting down the backends before releasing the DispatcherEvent
#rb carlmagnus.nordin
#rnx
#preflight 63cfcc336730a500ffa6f0b7

[CL 23829640 by pj kack in ue5-main branch]
2023-01-24 09:01:11 -05:00
dan thompson
e7f23fe480 ParallelFor IoStoreWriter Finalize() calls. Previously there were very few iostore writers as there weren't many containers, but with content on demand there are 4k+, so we need to multi thread this. 130s -> 12s for pak with my testing.
#rb carl-magnus.nordin
#preflight 63c9d5e1e31388156528ab4c

[CL 23796019 by dan thompson in ue5-main branch]
2023-01-20 15:28:59 -05:00
graeme thornton
104a9fbc0f Fix for an error when creating an iostore toc if the max partition size is set to int64_max
Submitting on behalf of PJ Kack

[FYI] pj.kack

[CL 23565335 by graeme thornton in ue5-main branch]
2023-01-03 08:45:49 -05:00
graeme thornton
faf6311ef1 IoStore: Add functionality to sign a container as a separate step
[FYI] carlmagnus.nordin

[CL 23312469 by graeme thornton in ue5-main branch]
2022-11-29 12:35:12 -05:00
per larsson
d793219fad I/O Dispatcher - move chunk ID related types/functions to make IoChunkId.h independant of IoDispatcher.h
#rb CarlMagnus.Nordin
#preflight 6380a19e4004f73f62b88311

[CL 23272264 by per larsson in ue5-main branch]
2022-11-26 05:51:22 -05:00
carlmagnus nordin
3d76ce80b5 IoDispatcher: Fixed shutdown crash
#preflight 63774398f514e1ded9982c1b
#rb pj.kack

[CL 23192024 by carlmagnus nordin in ue5-main branch]
2022-11-18 05:46:42 -05:00
carlmagnus nordin
f815a8a92f IoDispatcher: Fixed crash when requesting reads of 0 bytes
#preflight 6375edf0953c19d43557cb97
#rb per.larsson

[CL 23174498 by carlmagnus nordin in ue5-main branch]
2022-11-17 07:44:38 -05:00
carlmagnus nordin
72c3c99cf1 IoStoreReader: Fixed incorrect data being returned when reading partial chunks
#preflight 63735d0ab663683828ae8c34
#rb per.larsson

[CL 23133948 by carlmagnus nordin in ue5-main branch]
2022-11-15 06:20:34 -05:00
henrik karlsson
280bd598af [Core]
* Moved IoChunkId to its own file

#preflight 6360c1c441625be270a84079
#rb none

[CL 22888769 by henrik karlsson in ue5-main branch]
2022-11-01 15:13:52 -04:00