Commit Graph

95 Commits

Author SHA1 Message Date
dan phillips
a803f35fce Allow a pak file to be made using the subset of another pak file.
Allow more control over the pak cache, cache per pak, indvidual caches for all paks.
Increase the number of cached blocks for XboxOne.
Enable cache per pak for XboxOne.

[FYI] ben.woodhouse
gil.gribb, graeme.thornton


#ROBOMERGE-OWNER: dan.phillips
#ROBOMERGE-AUTHOR: dan.phillips
#ROBOMERGE-SOURCE: CL 7681509 via CL 7689295
#ROBOMERGE-BOT: (v383-7686620)

[CL 7689301 by dan phillips in Main branch]
2019-08-01 13:07:35 -04:00
graeme thornton
51cfbaef7d UnrealPak - Added "TestMemoryOptimization" mode which attempts to unload filenames and shrink pak entries across every pak in a directory, making sure there are no collisons and that everything works correctly
#ROBOMERGE-SOURCE: CL 7227149 via CL 7227151 via CL 7227434
#ROBOMERGE-BOT: (v367-6836689)

[CL 7227472 by graeme thornton in Main branch]
2019-07-08 10:23:53 -04:00
graeme thornton
1ac0af212f Improvements to pak index memory optimizations
- Fix entry shrinking not working on pak files that contain files that are encrypted AND compressed
 - Optimize any pak files that are added after encryption keys are registered
 - Improvements to logging / diagnostics
 - Use a unique starting seed (hash of filename) when doing filename unloading. Reduces cost expected collisions (localization paks / encrypted paks with duplicated assets)
 - In non-shipping builds, decode shrunken entries after the shrinking to check they match the original
 - Fix leaked FPakFile object when it was immediately determined to be invalid

#rb allan.bentham
[FYI] ben.woodhouse, hongyi.yu


#ROBOMERGE-OWNER: graeme.thornton
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 7139776 via CL 7139778 via CL 7167780
#ROBOMERGE-BOT: (v367-6836689)

[CL 7167782 by graeme thornton in Main branch]
2019-06-22 04:31:15 -04:00
daniel lamb
724dd8a0b0 Clean up missing packages list when we mount pak files.
#ROBOMERGE-SOURCE: CL 6912467 via CL 6912474 via CL 6912475 via CL 6913209
#ROBOMERGE-BOT: (v366-6836689)

[CL 6914251 by daniel lamb in Main branch]
2019-06-10 14:27:11 -04:00
peter knepley
f5d5ddbcd1 Back out changelist 6910520, missing some linker changes it looks like
[FYI] Daniel.Lamb


#ROBOMERGE-SOURCE: CL 6910644 via CL 6910646 via CL 6911055 via CL 6912095
#ROBOMERGE-BOT: (v366-6836689)

[CL 6913768 by peter knepley in Main branch]
2019-06-10 14:20:13 -04:00
daniel lamb
8c99fdf647 Clean up missing packages list when we mount pak files.
#ROBOMERGE-SOURCE: CL 6910520 via CL 6910522 via CL 6911043 via CL 6912049
#ROBOMERGE-BOT: (v366-6836689)

[CL 6913717 by daniel lamb in Main branch]
2019-06-10 14:19:49 -04:00
Stefan Boberg
8607ecb30d Copying //UE4/Dev-Core to Dev-Main (//UE4/Dev-Main)
#rb none

[CL 6815521 by Stefan Boberg in Main branch]
2019-06-03 15:32:00 -04:00
graeme thornton
54c0741d48 Hardening of pak index loading
- If a corrupt index is detected, do a retry but collect and log more information along the way to help diagnose the issue.

#rb stefan.boberg


#ROBOMERGE-SOURCE: CL 6685180 via CL 6685186 via CL 6751170
#ROBOMERGE-BOT: (v365-6733468)

[CL 6751664 by graeme thornton in Main branch]
2019-05-31 18:22:01 -04:00
justin marcus
8d401e597b Bundle Manager now makes calls to optimize pak memory usage after mounting.
[at]Josh.Adams [at]Graeme.Thornton


#ROBOMERGE-OWNER: justin.marcus
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 6658947 via CL 6659206 via CL 6659209 via CL 6659344 via CL 6659918
#ROBOMERGE-BOT: (vundefined-6654089)

[CL 6660722 by justin marcus in Main branch]
2019-05-28 19:48:41 -04:00
graeme thornton
c0cc487d44 Add support to pak platform file for a custom decryption function that a game can override the way decryption is performed at runtime
[FYI] joe.barrett


#ROBOMERGE-SOURCE: CL 6440730 via CL 6440731 via CL 6440813

[CL 6443772 by graeme thornton in Main branch]
2019-05-14 09:19:53 -04:00
graeme thornton
66206b90d8 Improvements to loading/validation/logging of pak signature files. Better logging of errors.
#jira UE-73250


#ROBOMERGE-SOURCE: CL 6095198 via CL 6095203 via CL 6096453

[CL 6096500 by graeme thornton in Main branch]
2019-04-25 14:11:23 -04:00
graeme thornton
3b2758065f Make ChunkHashToString() an inline function in IPlatformFilePak.h so that it can be used by other modules without requiring a link dependency
#ROBOMERGE-SOURCE: CL 6092711 via CL 6092712 via CL 6092999

[CL 6093032 by graeme thornton in Main branch]
2019-04-25 09:10:49 -04:00
graeme thornton
47bcaf79cf Centralize the loading of pak signature data so that the same object can be reused between the SignedArchiveReader and FPakPrecacher implementations. Stops the same data being loaded and verified twice
[FYI] joe.barrett


#ROBOMERGE-SOURCE: CL 6067905 via CL 6067908 via CL 6069306

[CL 6069330 by graeme thornton in Main branch]
2019-04-23 18:56:29 -04:00
graeme thornton
8632a76237 Improve logging of pak signature failures.
Treat a missing .sig file like a signature failure. Log it and ignore the pak file.



#ROBOMERGE-SOURCE: CL 5993822 via CL 5994244 via CL 5995533 via CL 5995549 via CL 5997449

[CL 5997477 by graeme thornton in Main branch]
2019-04-18 18:24:11 -04:00
allan bentham
2a4ca05717 UnloadPakEntryFilenames uses 64bit hashes.
[FYI] Graeme.Thornton
#rb Jack.porter
#rnx


#ROBOMERGE-SOURCE: CL 5937139 via CL 5952597 via CL 5956705

[CL 5957127 by allan bentham in Main branch]
2019-04-17 17:03:40 -04:00
graeme thornton
5f6c7468ed Don't mount pak files that fail the initial signature check
#rb joe.barrett, robert.manuszewski


#ROBOMERGE-SOURCE: CL 5897890 via CL 5897891 via CL 5898565

[CL 5898684 by graeme thornton in Main branch]
2019-04-15 10:22:26 -04:00
graeme thornton
f715649fdd Fix debug logging of chunk hashes for pak file signature check failures
#ROBOMERGE-SOURCE: CL 5834330 via CL 5834331 via CL 5838008

[CL 5838453 by graeme thornton in Main branch]
2019-04-12 10:40:24 -04:00
david harvey
399e0488ad Delete record logging changed to Verbose.
#rb anthony.bills


#ROBOMERGE-SOURCE: CL 5773263 via CL 5773264 via CL 5773265 via CL 5773386

[CL 5773389 by david harvey in Main branch]
2019-04-08 07:55:16 -04:00
peter sauerbrei
d3433d5fe0 turn pak entry unloading back on with a better algorithm courtesy of GraemeT
doesn't solve thread safety, but better collision detection and 30MB on FN
#rb graeme.thornton, josh.adams


#ROBOMERGE-SOURCE: CL 5764474 via CL 5764475 via CL 5764476 via CL 5764477 via CL 5765310

[CL 5765340 by peter sauerbrei in Main branch]
2019-04-05 11:28:48 -04:00
daniel lamb
9e6a72c368 Made the pak file binary compatible with previous version to reduce patch sizes.
Reenabled oodle compression in 8.30 for android, ios and switch.
Cleaned up unrealpak commandline when run from UAT.
#rb Josh.Adams
[FYI] Ben.Marsh, Andrew.Grant


#ROBOMERGE-SOURCE: CL 5619788 via CL 5619815 via CL 5619821 via CL 5619827 via CL 5623747

[CL 5623814 by daniel lamb in Main branch]
2019-03-27 21:08:44 -04:00
andrew grant
145e17cf26 Added warning about the dangers of modifying pak format.
[FYI] daniel.lamb :)


#ROBOMERGE-SOURCE: CL 5590393 via CL 5590405 via CL 5590410 via CL 5590416 via CL 5593650

[CL 5593700 by andrew grant in Main branch]
2019-03-27 12:26:48 -04:00
graeme thornton
9505003326 Created a new interface to RSA functionality that is independent of key size (unlike the old TBigInt version). Internally, RSA functions are implemented by OpenSSL on Windows/Mac/Linux an falling back to the TBigInt implementation for other platforms
- Future plan would be not to use OpenSSL directly, but to go via PlatformCrypto or something like it which could provide platform specific RSA functionality if it exists
- Also contains some compensation code for platforms that are still stuck on an older version of OpenSSL (everything except windows)
- On platforms that use OpenSSL < 1.1.1, register a locking callback so we can provide mutexes and stop it crashing horribly when the same RSA key is used concurrently on different threads

Changed default key size in the CryptoKeys plugin to 4096 bits.

#jira UE-71377
#rb ian.fox, robert.manuszewski, ryan.gerleve

#ROBOMERGE-OWNER: graeme.thornton
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5447433 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)

[CL 5447485 by graeme thornton in Main branch]
2019-03-19 07:53:22 -04:00
graeme thornton
7848a4a68f Disable SHA1 hashes for pak signing. Still experimental, and not indended to be enabled by default
#rb none
#jira UE-71377

#ROBOMERGE-SOURCE: CL 5383199 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)

[CL 5383328 by graeme thornton in Main branch]
2019-03-13 11:42:05 -04:00
graeme thornton
a0e55b062e Improvements to RSA key generation and pak signing
- CryptoKeys plugin can now be told how many bits to use for keys. Defaults to 2048 as a happy medium between security and performance
- After creating the OpenSSL RSA key, use the exp/mod values directly rather than recomputing from primes
- Added RSA.h with some TBigInt based helpers for RSA keys
- Added FPakSignatureFile to wrap the contents of the pak .sig file, including determining that it is of a new format that supports versioning. Wraps the encryption and decryption of he master table hash
- Changed pak master signature hash from crc to SHA1
- Modified access to executable embedded public key data so that it is just an array of bytes. Will allow custom key sizes in the future.

#rb joe.barrett, stefan.boberg
#jira UE-71280, UE-71377

#ROBOMERGE-OWNER: graeme.thornton
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5381335 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)

[CL 5383057 by graeme thornton in Main branch]
2019-03-13 11:12:27 -04:00
justin marcus
8c97e3c469 #CodeReview: justin.marcus,jeff.williams
Merge conflict.
justin.marcus, please merge this change by hand.
More info at https://robomerge.epicgames.net#FORTNITE

/src/robomerge_ue4_main_01/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericPlatformMisc.cpp - merging //Fortnite/Dev-EngineMerge/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericPlatformMisc.cpp#3
/src/robomerge_ue4_main_01/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericWidePlatformString.cpp - merging //Fortnite/Dev-EngineMerge/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericWidePlatformString.cpp#4
/src/robomerge_ue4_main_01/Engine/Source/Runtime/Core/Public/GenericPlatform/GenericPlatformMisc.h - merging //Fortnite/Dev-EngineMerge/Engine/Source/Runtime/Core/Public/GenericPlatform/GenericPlatformMisc.h#3


--------------------------------------
Merging //Fortnite/Dev-Kairos to Main (//Fortnite/Main)

Get Install Bundle Manager and dependcies building in main.



#ROBOMERGE-SOURCE: CL 4949345 via CL 4952070

#rb none

[CL 4961624 by justin marcus in Main branch]
2019-02-11 09:22:06 -05:00