Commit Graph

134 Commits

Author SHA1 Message Date
graeme thornton
e39117ad7e PakFile: Change a "verify" on the required encryption key being available to a fatal as it is never going to be a recoverable event and we don't want to continue if verify is compiled out
#ROBOMERGE-SOURCE: CL 7648169 via CL 7648171 via CL 7648297
#ROBOMERGE-BOT: (v372-7473910)

[CL 7648372 by graeme thornton in Main branch]
2019-07-29 05:11:01 -04:00
hongyi yu
7f0f04b310 When a chunk is installed, mount optional pak files and other pak files mapped to this chunk.
[CODEREVIEW] Daniel.Lamb


#ROBOMERGE-OWNER: hongyi.yu
#ROBOMERGE-AUTHOR: hongyi.yu
#ROBOMERGE-SOURCE: CL 7616984 via CL 7617471 via CL 7617513 via CL 7625628
#ROBOMERGE-BOT: (v372-7473910)

[CL 7625629 by hongyi yu in Main branch]
2019-07-25 10:40:21 -04:00
justin marcus
9acc36b8b6 If no pak files are found when IPlatformFilePak mounts paks, fallback to mounting all paks.
[FYI] [at]Bob.Tellez


#ROBOMERGE-SOURCE: CL 7477481 via CL 7493553 via CL 7545521
#ROBOMERGE-BOT: (v372-7473910)

[CL 7545795 by justin marcus in Main branch]
2019-07-23 10:52:46 -04:00
gil gribb
4adc370e24 UE4 - Added a panic recovery mode to failures to decompress data. This is a workaround for a mysterious failure here with no known repro.
#rb none


#ROBOMERGE-SOURCE: CL 7265416 via CL 7270974
#ROBOMERGE-BOT: (v369-7254125)

[CL 7271110 by gil gribb in Main branch]
2019-07-10 21:29:29 -04:00
bob tellez
452aeba28e Merging CL#7257773
Back out changelist 7256178. This was a bad merge



#ROBOMERGE-OWNER: bob.tellez
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 7258012 via CL 7266926
#ROBOMERGE-BOT: (v369-7254125)

[CL 7266952 by bob tellez in Main branch]
2019-07-10 17:05:15 -04:00
bob tellez
97e9932675 Merging //Fortnite/Release-10.0 to Dev-Main (//Fortnite/Dev-Main)
Merging CL#7251132 from //Fortnite/Release-Competitive.9.40

Merging in all missing changes from Release-9.40 to Release-Competitive.9.40 to try to make RoboMerge happy.

#rnx
#rb none


#ROBOMERGE-OWNER: bob.tellez
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 7256432 via CL 7266483
#ROBOMERGE-BOT: (v369-7254125)

[CL 7266503 by bob tellez in Main branch]
2019-07-10 16:41:03 -04:00
graeme thornton
51cfbaef7d UnrealPak - Added "TestMemoryOptimization" mode which attempts to unload filenames and shrink pak entries across every pak in a directory, making sure there are no collisons and that everything works correctly
#ROBOMERGE-SOURCE: CL 7227149 via CL 7227151 via CL 7227434
#ROBOMERGE-BOT: (v367-6836689)

[CL 7227472 by graeme thornton in Main branch]
2019-07-08 10:23:53 -04:00
graeme thornton
f657104452 Fix crash while shrinking pak entries for delete records
[FYI] ben.woodhouse


#ROBOMERGE-SOURCE: CL 7207336 via CL 7207337 via CL 7209962
#ROBOMERGE-BOT: (v367-6836689)

[CL 7210072 by graeme thornton in Main branch]
2019-07-01 02:38:47 -04:00
graeme thornton
1ac0af212f Improvements to pak index memory optimizations
- Fix entry shrinking not working on pak files that contain files that are encrypted AND compressed
 - Optimize any pak files that are added after encryption keys are registered
 - Improvements to logging / diagnostics
 - Use a unique starting seed (hash of filename) when doing filename unloading. Reduces cost expected collisions (localization paks / encrypted paks with duplicated assets)
 - In non-shipping builds, decode shrunken entries after the shrinking to check they match the original
 - Fix leaked FPakFile object when it was immediately determined to be invalid

#rb allan.bentham
[FYI] ben.woodhouse, hongyi.yu


#ROBOMERGE-OWNER: graeme.thornton
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 7139776 via CL 7139778 via CL 7167780
#ROBOMERGE-BOT: (v367-6836689)

[CL 7167782 by graeme thornton in Main branch]
2019-06-22 04:31:15 -04:00
gil gribb
0a1eac830c UE4 - Fix rare race condition with cancelled IO requests in the pak precacher.
#rb none


#ROBOMERGE-SOURCE: CL 7039770 via CL 7046935
#ROBOMERGE-BOT: (v367-6836689)

[CL 7047913 by gil gribb in Main branch]
2019-06-17 13:53:22 -04:00
david harvey
58c25dd385 Adding internal -CalcCompressionBlockCRCs command line option to UnrealPak to try and track down the rare FPakAsyncReadFileHandle::DoProcessing issue + a bit more logging when it fails.
#rb gil.gribb


#ROBOMERGE-SOURCE: CL 6953229 via CL 6953341
#ROBOMERGE-BOT: (v366-6836689)

[CL 6953356 by david harvey in Main branch]
2019-06-12 07:11:50 -04:00
graeme thornton
7fcbd57786 Only log pak key registration when something was actually mounted
#rb none

#ROBOMERGE-OWNER: graeme.thornton
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6947364 via CL 6947367 via CL 6947369 via CL 6947371 via CL 6947469
#ROBOMERGE-BOT: (v366-6836689)

[CL 6949111 by graeme thornton in Main branch]
2019-06-12 02:43:02 -04:00
daniel lamb
724dd8a0b0 Clean up missing packages list when we mount pak files.
#ROBOMERGE-SOURCE: CL 6912467 via CL 6912474 via CL 6912475 via CL 6913209
#ROBOMERGE-BOT: (v366-6836689)

[CL 6914251 by daniel lamb in Main branch]
2019-06-10 14:27:11 -04:00
peter knepley
f5d5ddbcd1 Back out changelist 6910520, missing some linker changes it looks like
[FYI] Daniel.Lamb


#ROBOMERGE-SOURCE: CL 6910644 via CL 6910646 via CL 6911055 via CL 6912095
#ROBOMERGE-BOT: (v366-6836689)

[CL 6913768 by peter knepley in Main branch]
2019-06-10 14:20:13 -04:00
daniel lamb
8c99fdf647 Clean up missing packages list when we mount pak files.
#ROBOMERGE-SOURCE: CL 6910520 via CL 6910522 via CL 6911043 via CL 6912049
#ROBOMERGE-BOT: (v366-6836689)

[CL 6913717 by daniel lamb in Main branch]
2019-06-10 14:19:49 -04:00
justin marcus
33403b1102 Fix pak index optimization not retaining all loc directories which could cause loc to fail depending on when it was initailized.
Re-enable triggering pak index optimization from bundle manager so that it works in Kairos.

[REVIEW] [at]Daniel.Lamb


#ROBOMERGE-SOURCE: CL 6855526 via CL 6855527 via CL 6855987 via CL 6856605
#ROBOMERGE-BOT: (v366-6836689)

[CL 6856736 by justin marcus in Main branch]
2019-06-05 18:29:37 -04:00
Stefan Boberg
8607ecb30d Copying //UE4/Dev-Core to Dev-Main (//UE4/Dev-Main)
#rb none

[CL 6815521 by Stefan Boberg in Main branch]
2019-06-03 15:32:00 -04:00
graeme thornton
54c0741d48 Hardening of pak index loading
- If a corrupt index is detected, do a retry but collect and log more information along the way to help diagnose the issue.

#rb stefan.boberg


#ROBOMERGE-SOURCE: CL 6685180 via CL 6685186 via CL 6751170
#ROBOMERGE-BOT: (v365-6733468)

[CL 6751664 by graeme thornton in Main branch]
2019-05-31 18:22:01 -04:00
justin marcus
8d401e597b Bundle Manager now makes calls to optimize pak memory usage after mounting.
[at]Josh.Adams [at]Graeme.Thornton


#ROBOMERGE-OWNER: justin.marcus
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 6658947 via CL 6659206 via CL 6659209 via CL 6659344 via CL 6659918
#ROBOMERGE-BOT: (vundefined-6654089)

[CL 6660722 by justin marcus in Main branch]
2019-05-28 19:48:41 -04:00
graeme thornton
ed1b2b3fc0 Make pak platform file wnbind the RegisterEncryptionKey delegate when it is destroyed
#ROBOMERGE-SOURCE: CL 6616196 via CL 6616584 via CL 6616743
#ROBOMERGE-BOT: (v358-6608238)

[CL 6616759 by graeme thornton in Main branch]
2019-05-22 22:42:46 -04:00
david harvey
d70361928b include a CRC of the compressed data in the fatal error log if decompression fails to help narrow down where in the IO pipline the failure occurred.
#rb ben.woodhouse


#ROBOMERGE-SOURCE: CL 6605801 via CL 6605803 via CL 6607800
#ROBOMERGE-BOT: (v357-6606154)

[CL 6607835 by david harvey in Main branch]
2019-05-22 13:14:29 -04:00
graeme thornton
c0cc487d44 Add support to pak platform file for a custom decryption function that a game can override the way decryption is performed at runtime
[FYI] joe.barrett


#ROBOMERGE-SOURCE: CL 6440730 via CL 6440731 via CL 6440813

[CL 6443772 by graeme thornton in Main branch]
2019-05-14 09:19:53 -04:00
graeme thornton
0ec13c7d73 Improvements to pak file check command
- Add timing information to the log
 - If pak is signed, use a faster method where we just read one byte from each signing block
 - Still invoked with the -checkpak command line parameter

[FYI] bart.hawthorne, steve.white

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6251572 via CL 6251573 via CL 6251574 via CL 6251688 via CL 6251693
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 6269944 by graeme thornton in Dev-Core branch]
2019-05-03 05:17:11 -04:00
justin marcus
cc903b7468 Add DumpEarlyPakFileReads command to dump all pak reads that occur before it's possible to install patched content to CSV/Log.
This is disabled in shipping.
[at]Daniel.Lamb


#ROBOMERGE-SOURCE: CL 6258473 via CL 6258479 via CL 6258480 via CL 6266092

[CL 6266227 by justin marcus in Main branch]
2019-05-02 19:21:35 -04:00
graeme thornton
4b586ac237 Improvements to pak file check command
- Add timing information to the log
 - If pak is signed, use a faster method where we just read one byte from each signing block
 - Still invoked with the -checkpak command line parameter

[FYI] bart.hawthorne, steve.white


#ROBOMERGE-SOURCE: CL 6251572 via CL 6251573 via CL 6251574 via CL 6251688

[CL 6251693 by graeme thornton in Main branch]
2019-05-02 08:05:18 -04:00