Commit Graph

146 Commits

Author SHA1 Message Date
gil gribb
c9f811c0e8 UE4 - Added a panic recovery mode to failures to decompress data. This is a workaround for a mysterious failure here with no known repro.
#rb none

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: gil.gribb
#ROBOMERGE-SOURCE: CL 7265416 via CL 7270974 via CL 7271110
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v371-7306989)

[CL 7344148 by gil gribb in Dev-Build branch]
2019-07-17 00:23:25 -04:00
bob tellez
b2b36b0fa0 Merging CL#7257773
Back out changelist 7256178. This was a bad merge


#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 7258012 via CL 7266926 via CL 7266952
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v371-7306989)

[CL 7339329 by bob tellez in Dev-Build branch]
2019-07-16 23:57:23 -04:00
bob tellez
6c3ed3403a Merging //Fortnite/Release-10.0 to Dev-Main (//Fortnite/Dev-Main)
Merging CL#7251132 from //Fortnite/Release-Competitive.9.40

Merging in all missing changes from Release-9.40 to Release-Competitive.9.40 to try to make RoboMerge happy.

#rnx
#rb none

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 7256432 via CL 7266483 via CL 7266503
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v371-7306989)

[CL 7338361 by bob tellez in Dev-Build branch]
2019-07-16 23:50:12 -04:00
graeme thornton
b84498eb83 UnrealPak - Added "TestMemoryOptimization" mode which attempts to unload filenames and shrink pak entries across every pak in a directory, making sure there are no collisons and that everything works correctly
#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 7227149 via CL 7227151 via CL 7227434 via CL 7227472
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v367-6836689)

[CL 7236800 by graeme thornton in Dev-Build branch]
2019-07-09 00:17:52 -04:00
graeme thornton
ca16d71f63 Fix crash while shrinking pak entries for delete records
[FYI] ben.woodhouse

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 7207336 via CL 7207337 via CL 7209962 via CL 7210072
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v367-6836689)

[CL 7213476 by graeme thornton in Dev-Build branch]
2019-07-01 08:46:52 -04:00
graeme thornton
d760805450 Improvements to pak index memory optimizations
- Fix entry shrinking not working on pak files that contain files that are encrypted AND compressed
 - Optimize any pak files that are added after encryption keys are registered
 - Improvements to logging / diagnostics
 - Use a unique starting seed (hash of filename) when doing filename unloading. Reduces cost expected collisions (localization paks / encrypted paks with duplicated assets)
 - In non-shipping builds, decode shrunken entries after the shrinking to check they match the original
 - Fix leaked FPakFile object when it was immediately determined to be invalid

#rb allan.bentham
[FYI] ben.woodhouse, hongyi.yu

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 7139776 via CL 7139778 via CL 7167780 via CL 7167782
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v367-6836689)

[CL 7188952 by graeme thornton in Dev-Build branch]
2019-06-23 08:23:06 -04:00
gil gribb
538bac6f18 UE4 - Fix rare race condition with cancelled IO requests in the pak precacher.
#rb none

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: gil.gribb
#ROBOMERGE-SOURCE: CL 7039770 via CL 7046935 via CL 7047913
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v367-6836689)

[CL 7120451 by gil gribb in Dev-Build branch]
2019-06-21 01:20:53 -04:00
david harvey
8c060c1b50 Adding internal -CalcCompressionBlockCRCs command line option to UnrealPak to try and track down the rare FPakAsyncReadFileHandle::DoProcessing issue + a bit more logging when it fails.
#rb gil.gribb

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: david.harvey
#ROBOMERGE-SOURCE: CL 6953229 via CL 6953341 via CL 6953356
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v367-6836689)

[CL 7090578 by david harvey in Dev-Build branch]
2019-06-19 09:23:54 -04:00
graeme thornton
489fcaf40d Only log pak key registration when something was actually mounted
#rb none

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6947364 via CL 6947367 via CL 6947369 via CL 6947371 via CL 6947469 via CL 6949111
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v367-6836689)

[CL 7090538 by graeme thornton in Dev-Build branch]
2019-06-19 09:21:03 -04:00
Ben Marsh
cfe09c649f Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none
#rnx

[CL 6922670 by Ben Marsh in Dev-Build branch]
2019-06-10 19:47:29 -04:00
justin marcus
db8a33f3e4 Fix pak index optimization not retaining all loc directories which could cause loc to fail depending on when it was initailized.
Re-enable triggering pak index optimization from bundle manager so that it works in Kairos.

[REVIEW] [at]Daniel.Lamb

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 6855526 via CL 6855527 via CL 6855987 via CL 6856605 via CL 6856736
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v366-6836689)

[CL 6916365 by justin marcus in Dev-Build branch]
2019-06-10 15:58:36 -04:00
stefan boberg
285a908f86 Copying //UE4/Dev-Core to Dev-Main (//UE4/Dev-Main)
#rb none

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: stefan.boberg
#ROBOMERGE-SOURCE: CL 6815521 in //UE4/Main/...
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v365-6733468)

[CL 6821265 by stefan boberg in Dev-Build branch]
2019-06-03 19:09:16 -04:00
graeme thornton
f48b80f826 Hardening of pak index loading
- If a corrupt index is detected, do a retry but collect and log more information along the way to help diagnose the issue.

#rb stefan.boberg

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6685180 via CL 6685186 via CL 6751170 via CL 6751664
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v365-6733468)

[CL 6762248 by graeme thornton in Dev-Build branch]
2019-06-01 11:00:51 -04:00
justin marcus
db961bb45c Bundle Manager now makes calls to optimize pak memory usage after mounting.
[at]Josh.Adams [at]Graeme.Thornton

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 6658947 via CL 6659206 via CL 6659209 via CL 6659344 via CL 6659918 via CL 6660722
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v363-6677109)

[CL 6704599 by justin marcus in Dev-Build branch]
2019-05-31 13:21:15 -04:00
Ben Marsh
5f3ac5bc9e Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none
#rnx

[CL 6620114 by Ben Marsh in Dev-Build branch]
2019-05-23 13:55:56 -04:00
graeme thornton
fe31dd6c09 Add support to pak platform file for a custom decryption function that a game can override the way decryption is performed at runtime
[FYI] joe.barrett

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6440730 via CL 6440731 via CL 6440813 via CL 6443772
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6478138 by graeme thornton in Dev-Build branch]
2019-05-15 10:20:04 -04:00
justin marcus
47b034dd8b Add DumpEarlyPakFileReads command to dump all pak reads that occur before it's possible to install patched content to CSV/Log.
This is disabled in shipping.
[at]Daniel.Lamb

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 6258473 via CL 6258479 via CL 6258480 via CL 6266092 via CL 6266227
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6280536 by justin marcus in Dev-Build branch]
2019-05-03 17:18:24 -04:00
graeme thornton
54a8813899 Improvements to pak file check command
- Add timing information to the log
 - If pak is signed, use a faster method where we just read one byte from each signing block
 - Still invoked with the -checkpak command line parameter

[FYI] bart.hawthorne, steve.white

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6251572 via CL 6251573 via CL 6251574 via CL 6251688 via CL 6251693
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6254611 by graeme thornton in Dev-Build branch]
2019-05-02 13:01:44 -04:00
graeme thornton
9ffb7aebf9 Minor off-by-one correction to chunk hash failure warning message
#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6113928 via CL 6113932 via CL 6115047 via CL 6115246
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6131956 by graeme thornton in Dev-Build branch]
2019-04-26 22:42:54 -04:00
graeme thornton
a782a573ec Improvements to loading/validation/logging of pak signature files. Better logging of errors.
#jira UE-73250

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6095198 via CL 6095203 via CL 6096453 via CL 6096500
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6122197 by graeme thornton in Dev-Build branch]
2019-04-26 21:43:40 -04:00
graeme thornton
727d99e5fa Make ChunkHashToString() an inline function in IPlatformFilePak.h so that it can be used by other modules without requiring a link dependency
#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6092711 via CL 6092712 via CL 6092999 via CL 6093032
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6120714 by graeme thornton in Dev-Build branch]
2019-04-26 21:35:50 -04:00
graeme thornton
ca133d2f11 Centralize the loading of pak signature data so that the same object can be reused between the SignedArchiveReader and FPakPrecacher implementations. Stops the same data being loaded and verified twice
[FYI] joe.barrett

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6067905 via CL 6067908 via CL 6069306 via CL 6069330
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6073221 by graeme thornton in Dev-Build branch]
2019-04-23 21:20:30 -04:00
graeme thornton
84f9a6a029 Improve logging of pak signature failures.
Treat a missing .sig file like a signature failure. Log it and ignore the pak file.


#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5993822 via CL 5994244 via CL 5995533 via CL 5995549 via CL 5997449 via CL 5997477
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6011395 by graeme thornton in Dev-Build branch]
2019-04-19 08:22:39 -04:00
allan bentham
14f3d66ca5 UnloadPakEntryFilenames uses 64bit hashes.
[FYI] Graeme.Thornton
#rb Jack.porter
#rnx

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: allan.bentham
#ROBOMERGE-SOURCE: CL 5937139 via CL 5952597 via CL 5956705 via CL 5957127
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5962860 by allan bentham in Dev-Build branch]
2019-04-17 21:04:00 -04:00
graeme thornton
569a413697 Don't mount pak files that fail the initial signature check
#rb joe.barrett, robert.manuszewski

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5897890 via CL 5897891 via CL 5898565 via CL 5898684
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5900073 by graeme thornton in Dev-Build branch]
2019-04-15 12:59:29 -04:00