Commit Graph

141 Commits

Author SHA1 Message Date
peter sauerbrei
ba932a53cf allow turning pak unload off from commandline with -nounloadpakentries, this allows all items to be properly loaded in PSO gen
#rb trivial


#ROBOMERGE-SOURCE: CL 5772245 via CL 5795091 via CL 5795113 via CL 5795129 via CL 5799583

[CL 5799699 by peter sauerbrei in Main branch]
2019-04-08 18:22:07 -04:00
david harvey
399e0488ad Delete record logging changed to Verbose.
#rb anthony.bills


#ROBOMERGE-SOURCE: CL 5773263 via CL 5773264 via CL 5773265 via CL 5773386

[CL 5773389 by david harvey in Main branch]
2019-04-08 07:55:16 -04:00
peter sauerbrei
d3433d5fe0 turn pak entry unloading back on with a better algorithm courtesy of GraemeT
doesn't solve thread safety, but better collision detection and 30MB on FN
#rb graeme.thornton, josh.adams


#ROBOMERGE-SOURCE: CL 5764474 via CL 5764475 via CL 5764476 via CL 5764477 via CL 5765310

[CL 5765340 by peter sauerbrei in Main branch]
2019-04-05 11:28:48 -04:00
josh adams
9e32b0aec2 - Replaced -nommio with a CVar: mmio.enable. Can use commandline to disable with: -dpcvars=mmio.enable=0
- This allows it to be hotfixed off
#rb chris.babcock


#ROBOMERGE-SOURCE: CL 5714583 via CL 5714586 via CL 5714590 via CL 5714986 via CL 5717097

[CL 5717275 by josh adams in Main branch]
2019-04-03 20:19:45 -04:00
david harvey
f7bf258dcd [LLM] add missing FileSystem tag to file request. moves about ~2MB on average out of Untagged.
#rb anthony.bills


#ROBOMERGE-SOURCE: CL 5630089 via CL 5630203 via CL 5630220 via CL 5630238 via CL 5632396

[CL 5632706 by david harvey in Main branch]
2019-03-28 13:51:25 -04:00
daniel lamb
9e6a72c368 Made the pak file binary compatible with previous version to reduce patch sizes.
Reenabled oodle compression in 8.30 for android, ios and switch.
Cleaned up unrealpak commandline when run from UAT.
#rb Josh.Adams
[FYI] Ben.Marsh, Andrew.Grant


#ROBOMERGE-SOURCE: CL 5619788 via CL 5619815 via CL 5619821 via CL 5619827 via CL 5623747

[CL 5623814 by daniel lamb in Main branch]
2019-03-27 21:08:44 -04:00
andrew grant
145e17cf26 Added warning about the dangers of modifying pak format.
[FYI] daniel.lamb :)


#ROBOMERGE-SOURCE: CL 5590393 via CL 5590405 via CL 5590410 via CL 5590416 via CL 5593650

[CL 5593700 by andrew grant in Main branch]
2019-03-27 12:26:48 -04:00
graeme thornton
5c9b91860c Initialize signature checking hashcache array when using CRCs. Static analysis warning fix.
#rb none


#ROBOMERGE-SOURCE: CL 5516109 via CL 5516121 via CL 5518542

[CL 5535898 by graeme thornton in Main branch]
2019-03-25 14:05:37 -04:00
graeme thornton
17730913e6 Accidentally forced pak filename/entry compression on. Disabling again.
#rb none


#ROBOMERGE-SOURCE: CL 5500872 via CL 5500873 via CL 5500937

[CL 5500964 by graeme thornton in Main branch]
2019-03-22 07:20:03 -04:00
hongyi yu
36b439a63b Fixed the issue where unrealpak failed to read from old pak files with no content.
#rb Graeme.Thornton


#ROBOMERGE-OWNER: hongyi.yu
#ROBOMERGE-AUTHOR: hongyi.yu
#ROBOMERGE-SOURCE: CL 5484250 via CL 5487362

[CL 5487860 by hongyi yu in Main branch]
2019-03-20 18:55:06 -04:00
graeme thornton
c491de975a Edigrated rollup of all engine RSA changes that were done for 4.22. Pak signature checking is now much more competent.
#rb none
[FYI] joe.barrett
#rnx


#ROBOMERGE-SOURCE: CL 5469067 via CL 5469119 via CL 5470585

[CL 5470597 by graeme thornton in Main branch]
2019-03-20 12:03:26 -04:00
graeme thornton
9505003326 Created a new interface to RSA functionality that is independent of key size (unlike the old TBigInt version). Internally, RSA functions are implemented by OpenSSL on Windows/Mac/Linux an falling back to the TBigInt implementation for other platforms
- Future plan would be not to use OpenSSL directly, but to go via PlatformCrypto or something like it which could provide platform specific RSA functionality if it exists
- Also contains some compensation code for platforms that are still stuck on an older version of OpenSSL (everything except windows)
- On platforms that use OpenSSL < 1.1.1, register a locking callback so we can provide mutexes and stop it crashing horribly when the same RSA key is used concurrently on different threads

Changed default key size in the CryptoKeys plugin to 4096 bits.

#jira UE-71377
#rb ian.fox, robert.manuszewski, ryan.gerleve

#ROBOMERGE-OWNER: graeme.thornton
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5447433 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)

[CL 5447485 by graeme thornton in Main branch]
2019-03-19 07:53:22 -04:00
graeme thornton
7848a4a68f Disable SHA1 hashes for pak signing. Still experimental, and not indended to be enabled by default
#rb none
#jira UE-71377

#ROBOMERGE-SOURCE: CL 5383199 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)

[CL 5383328 by graeme thornton in Main branch]
2019-03-13 11:42:05 -04:00
graeme thornton
a0e55b062e Improvements to RSA key generation and pak signing
- CryptoKeys plugin can now be told how many bits to use for keys. Defaults to 2048 as a happy medium between security and performance
- After creating the OpenSSL RSA key, use the exp/mod values directly rather than recomputing from primes
- Added RSA.h with some TBigInt based helpers for RSA keys
- Added FPakSignatureFile to wrap the contents of the pak .sig file, including determining that it is of a new format that supports versioning. Wraps the encryption and decryption of he master table hash
- Changed pak master signature hash from crc to SHA1
- Modified access to executable embedded public key data so that it is just an array of bytes. Will allow custom key sizes in the future.

#rb joe.barrett, stefan.boberg
#jira UE-71280, UE-71377

#ROBOMERGE-OWNER: graeme.thornton
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5381335 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)

[CL 5383057 by graeme thornton in Main branch]
2019-03-13 11:12:27 -04:00
Allan Bentham
926555ac14 Fix comments somehow lost during merge
#jira UE-70963
#rb none

[CL 5294300 by Allan Bentham in Main branch]
2019-03-05 07:15:10 -05:00
allan bentham
60be956374 Optimize deserialization of compressed pak files.
Reuse previously decompressed buffer where possible to avoid repeatedly deserialising and decompressing the same block when serializing data < compressed block size.
#rb Graeme.Thornton
[FYI] Graeme.Thornton


#ROBOMERGE-OWNER: allan.bentham
#ROBOMERGE-AUTHOR: allan.bentham
#ROBOMERGE-SOURCE: CL 5237090 via CL 5240033

[CL 5240035 by allan bentham in Main branch]
2019-02-28 12:30:10 -05:00
peter sauerbrei
6ade612778 addition of better hash algorithm for removal of file names from hash
#rb josh.adams


#ROBOMERGE-SOURCE: CL 5075321 via CL 5075990 via CL 5081503

[CL 5081850 by peter sauerbrei in Main branch]
2019-02-19 18:59:08 -05:00
josh adams
ccb6baf53b - Reenabled encryption on bulk files with memory mapping - and then disallow mapping at runtime
#rb gil.bribb
[FYI] gil.gribb


#ROBOMERGE-OWNER: josh.adams
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 4998359 via CL 4998360 via CL 4999252

[CL 5001374 by josh adams in Main branch]
2019-02-14 17:25:52 -05:00
josh adams
b7f8902fb9 - Memory Mapped IO enabled on IOS for Audio (via ini and platform property)
- -nommio on the commandline will disble it for testing
#rb gil.gribb


#ROBOMERGE-OWNER: josh.adams
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 4963115 via CL 4963118 via CL 4963812

[CL 4969283 by josh adams in Main branch]
2019-02-12 10:27:29 -05:00
justin marcus
8c97e3c469 #CodeReview: justin.marcus,jeff.williams
Merge conflict.
justin.marcus, please merge this change by hand.
More info at https://robomerge.epicgames.net#FORTNITE

/src/robomerge_ue4_main_01/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericPlatformMisc.cpp - merging //Fortnite/Dev-EngineMerge/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericPlatformMisc.cpp#3
/src/robomerge_ue4_main_01/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericWidePlatformString.cpp - merging //Fortnite/Dev-EngineMerge/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericWidePlatformString.cpp#4
/src/robomerge_ue4_main_01/Engine/Source/Runtime/Core/Public/GenericPlatform/GenericPlatformMisc.h - merging //Fortnite/Dev-EngineMerge/Engine/Source/Runtime/Core/Public/GenericPlatform/GenericPlatformMisc.h#3


--------------------------------------
Merging //Fortnite/Dev-Kairos to Main (//Fortnite/Main)

Get Install Bundle Manager and dependcies building in main.



#ROBOMERGE-SOURCE: CL 4949345 via CL 4952070

#rb none

[CL 4961624 by justin marcus in Main branch]
2019-02-11 09:22:06 -05:00
peter knepley
ae04c536e6 Remove PakFile layer from the platform file chain upon shutdown (FORT-145923)
- Prevents CRC from crashing when writing the config files on exit.



#rnx


#rb Josh.Adams, Brandon.Schaefer


#ROBOMERGE-AUTHOR: Arciel.Rekman
#ROBOMERGE-SOURCE: CL 4846937 via CL 4846952 via CL 4848372 via CL 4850934

[CL 4851162 by peter knepley in Main branch]
2019-01-30 18:56:57 -05:00
Josh Adams
0cd6ef2a9c - Fixed conflict from a merge to main
#rb trivial
#jira none

[CL 4846991 by Josh Adams in Main branch]
2019-01-30 15:27:53 -05:00
josh adams
87bcd24cc3 [Gil.Gribb and maybe others did most of the work, I just did the IOS parts, but I am checking it in, since IOS was the last stop on the code train]
- Added memory mapped IO concept to the engine
- Implemented and tested on Windows and IOS
- Not in use by anything yet, however, as the use cases are narrow and special (bulk data that does not need massaging after load - so it can be swapped out and back in silently by OS)
[FYI] gil.gribb,chris.babcock


#ROBOMERGE-SOURCE: CL 4842051 via CL 4845281

[CL 4845299 by josh adams in Main branch]
2019-01-30 13:45:30 -05:00
graeme thornton
8c534fe278 When a shader pak reader is first created for a thread, return the FThreadCheckingArchiveProxy wrapped version rather than the raw archive pointer.
#ROBOMERGE-SOURCE: CL 4829903 via CL 4829916 via CL 4830765

[CL 4830771 by graeme thornton in Main branch]
2019-01-29 08:11:08 -05:00
graeme thornton
c0bae37e42 Pak file handles no longer store a pointer to a shared reader archive. Those readers are thread specific and means handles can't be read from different threads without causing problems. By storing a function that looks up the pak reader dynamically on each file operation, the requests will work properly from any thread.
#rb gil.gribb


#ROBOMERGE-SOURCE: CL 4829513 via CL 4829541 via CL 4830764

[CL 4830770 by graeme thornton in Main branch]
2019-01-29 08:11:07 -05:00