Commit Graph

96 Commits

Author SHA1 Message Date
graeme thornton
67c7c2e278 Minor off-by-one correction to chunk hash failure warning message
#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6113928 via CL 6113932 via CL 6115047 via CL 6115246
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6131955 by graeme thornton in Dev-Anim branch]
2019-04-26 22:42:53 -04:00
graeme thornton
aa6ff0a059 Improvements to loading/validation/logging of pak signature files. Better logging of errors.
#jira UE-73250

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6095198 via CL 6095203 via CL 6096453 via CL 6096500
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6122177 by graeme thornton in Dev-Anim branch]
2019-04-26 21:43:34 -04:00
graeme thornton
2912cfb4d4 Make ChunkHashToString() an inline function in IPlatformFilePak.h so that it can be used by other modules without requiring a link dependency
#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6092711 via CL 6092712 via CL 6092999 via CL 6093032
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6120691 by graeme thornton in Dev-Anim branch]
2019-04-26 21:35:44 -04:00
graeme thornton
6350f02730 Centralize the loading of pak signature data so that the same object can be reused between the SignedArchiveReader and FPakPrecacher implementations. Stops the same data being loaded and verified twice
[FYI] joe.barrett

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6067905 via CL 6067908 via CL 6069306 via CL 6069330
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6073109 by graeme thornton in Dev-Anim branch]
2019-04-23 21:20:07 -04:00
graeme thornton
98254d7acf Improve logging of pak signature failures.
Treat a missing .sig file like a signature failure. Log it and ignore the pak file.


#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5993822 via CL 5994244 via CL 5995533 via CL 5995549 via CL 5997449 via CL 5997477
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6002765 by graeme thornton in Dev-Anim branch]
2019-04-18 21:42:18 -04:00
allan bentham
f3f51590ee UnloadPakEntryFilenames uses 64bit hashes.
[FYI] Graeme.Thornton
#rb Jack.porter
#rnx

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: allan.bentham
#ROBOMERGE-SOURCE: CL 5937139 via CL 5952597 via CL 5956705 via CL 5957127
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5962911 by allan bentham in Dev-Anim branch]
2019-04-17 21:04:09 -04:00
graeme thornton
4259589301 Don't mount pak files that fail the initial signature check
#rb joe.barrett, robert.manuszewski

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5897890 via CL 5897891 via CL 5898565 via CL 5898684
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5900137 by graeme thornton in Dev-Anim branch]
2019-04-15 12:59:51 -04:00
graeme thornton
0f00662a66 Fix debug logging of chunk hashes for pak file signature check failures
#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5834330 via CL 5834331 via CL 5838008 via CL 5838453
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5880380 by graeme thornton in Dev-Anim branch]
2019-04-12 21:31:07 -04:00
peter sauerbrei
7386fae5db allow turning pak unload off from commandline with -nounloadpakentries, this allows all items to be properly loaded in PSO gen
#rb trivial

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: peter.sauerbrei
#ROBOMERGE-SOURCE: CL 5772245 via CL 5795091 via CL 5795113 via CL 5795129 via CL 5799583 via CL 5799699
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5802129 by peter sauerbrei in Dev-Anim branch]
2019-04-08 21:05:27 -04:00
peter sauerbrei
9aa5d73f27 turn pak entry unloading back on with a better algorithm courtesy of GraemeT
doesn't solve thread safety, but better collision detection and 30MB on FN
#rb graeme.thornton, josh.adams

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: peter.sauerbrei
#ROBOMERGE-SOURCE: CL 5764474 via CL 5764475 via CL 5764476 via CL 5764477 via CL 5765310 via CL 5765340
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5778018 by peter sauerbrei in Dev-Anim branch]
2019-04-08 13:12:12 -04:00
josh adams
3963d79efb - Replaced -nommio with a CVar: mmio.enable. Can use commandline to disable with: -dpcvars=mmio.enable=0
- This allows it to be hotfixed off
#rb chris.babcock

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 5714583 via CL 5714586 via CL 5714590 via CL 5714986 via CL 5717097 via CL 5717275
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5727028 by josh adams in Dev-Anim branch]
2019-04-04 01:00:10 -04:00
david harvey
fd3244ae2f [LLM] add missing FileSystem tag to file request. moves about ~2MB on average out of Untagged.
#rb anthony.bills

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: david.harvey
#ROBOMERGE-SOURCE: CL 5630089 via CL 5630203 via CL 5630220 via CL 5630238 via CL 5632396 via CL 5632706
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5636297 by david harvey in Dev-Anim branch]
2019-03-28 17:04:07 -04:00
graeme thornton
915b1f8c26 Initialize signature checking hashcache array when using CRCs. Static analysis warning fix.
#rb none

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5516109 via CL 5516121 via CL 5518542 via CL 5535898
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5542248 by graeme thornton in Dev-Anim branch]
2019-03-25 17:12:24 -04:00
graeme thornton
d5fc2b4bd0 Accidentally forced pak filename/entry compression on. Disabling again.
#rb none

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5500872 via CL 5500873 via CL 5500937 via CL 5500964
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5514788 by graeme thornton in Dev-Anim branch]
2019-03-22 17:57:29 -04:00
hongyi yu
1f74432e13 Fixed the issue where unrealpak failed to read from old pak files with no content.
#rb Graeme.Thornton

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: hongyi.yu
#ROBOMERGE-SOURCE: CL 5484250 via CL 5487362 via CL 5487860
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5507694 by hongyi yu in Dev-Anim branch]
2019-03-22 17:13:01 -04:00
graeme thornton
4f06e16149 Edigrated rollup of all engine RSA changes that were done for 4.22. Pak signature checking is now much more competent.
#rb none
[FYI] joe.barrett
#rnx

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5469067 via CL 5469119 via CL 5470585 via CL 5470597
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5505293 by graeme thornton in Dev-Anim branch]
2019-03-22 16:59:44 -04:00
graeme thornton
cf5dbb9860 Created a new interface to RSA functionality that is independent of key size (unlike the old TBigInt version). Internally, RSA functions are implemented by OpenSSL on Windows/Mac/Linux an falling back to the TBigInt implementation for other platforms
- Future plan would be not to use OpenSSL directly, but to go via PlatformCrypto or something like it which could provide platform specific RSA functionality if it exists
- Also contains some compensation code for platforms that are still stuck on an older version of OpenSSL (everything except windows)
- On platforms that use OpenSSL < 1.1.1, register a locking callback so we can provide mutexes and stop it crashing horribly when the same RSA key is used concurrently on different threads

Changed default key size in the CryptoKeys plugin to 4096 bits.

#jira UE-71377
#rb ian.fox, robert.manuszewski, ryan.gerleve

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5447433 in //UE4/Release-4.22/... via CL 5447485
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5447738 by graeme thornton in Dev-Anim branch]
2019-03-19 09:01:12 -04:00
graeme thornton
8036e13e06 Improvements to RSA key generation and pak signing
- CryptoKeys plugin can now be told how many bits to use for keys. Defaults to 2048 as a happy medium between security and performance
- After creating the OpenSSL RSA key, use the exp/mod values directly rather than recomputing from primes
- Added RSA.h with some TBigInt based helpers for RSA keys
- Added FPakSignatureFile to wrap the contents of the pak .sig file, including determining that it is of a new format that supports versioning. Wraps the encryption and decryption of he master table hash
- Changed pak master signature hash from crc to SHA1
- Modified access to executable embedded public key data so that it is just an array of bytes. Will allow custom key sizes in the future.

#rb joe.barrett, stefan.boberg
#jira UE-71280, UE-71377

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5381335 in //UE4/Release-4.22/... via CL 5383057
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5384942 by graeme thornton in Dev-Anim branch]
2019-03-13 13:02:40 -04:00
allan bentham
af9a0e89ac Fix comments somehow lost during merge
#jira UE-70963
#rb none

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: allan.bentham
#ROBOMERGE-SOURCE: CL 5294300 in //UE4/Main/...
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5302006 by allan bentham in Dev-Anim branch]
2019-03-05 15:30:46 -05:00
allan bentham
0f2f4e0ffd Optimize deserialization of compressed pak files.
Reuse previously decompressed buffer where possible to avoid repeatedly deserialising and decompressing the same block when serializing data < compressed block size.
#rb Graeme.Thornton
[FYI] Graeme.Thornton

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: allan.bentham
#ROBOMERGE-SOURCE: CL 5237090 via CL 5240033 via CL 5240035
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5247417 by allan bentham in Dev-Anim branch]
2019-02-28 17:22:19 -05:00
peter sauerbrei
71988fa557 addition of better hash algorithm for removal of file names from hash
#rb josh.adams

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: peter.sauerbrei
#ROBOMERGE-SOURCE: CL 5075321 via CL 5075990 via CL 5081503 via CL 5081850
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5144260 by peter sauerbrei in Dev-Anim branch]
2019-02-22 11:06:22 -05:00
josh adams
e94a3775b4 - Reenabled encryption on bulk files with memory mapping - and then disallow mapping at runtime
#rb gil.bribb
[FYI] gil.gribb

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 4998359 via CL 4998360 via CL 4999252 via CL 5001374
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5046982 by josh adams in Dev-Anim branch]
2019-02-18 15:08:12 -05:00
josh adams
a487e89fda - Memory Mapped IO enabled on IOS for Audio (via ini and platform property)
- -nommio on the commandline will disble it for testing
#rb gil.gribb

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 4963115 via CL 4963118 via CL 4963812 via CL 4969283
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5027870 by josh adams in Dev-Anim branch]
2019-02-16 04:08:25 -05:00
justin marcus
c9e2bc0814 [CODEREVIEW] justin.marcus,jeff.williams
Merge conflict.
justin.marcus, please merge this change by hand.
More info at https://robomerge.epicgames.net#FORTNITE

/src/robomerge_ue4_main_01/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericPlatformMisc.cpp - merging //Fortnite/Dev-EngineMerge/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericPlatformMisc.cpp#3
/src/robomerge_ue4_main_01/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericWidePlatformString.cpp - merging //Fortnite/Dev-EngineMerge/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericWidePlatformString.cpp#4
/src/robomerge_ue4_main_01/Engine/Source/Runtime/Core/Public/GenericPlatform/GenericPlatformMisc.h - merging //Fortnite/Dev-EngineMerge/Engine/Source/Runtime/Core/Public/GenericPlatform/GenericPlatformMisc.h#3


--------------------------------------
Merging //Fortnite/Dev-Kairos to Main (//Fortnite/Main)

Get Install Bundle Manager and dependcies building in main.




#rb none

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 4949345 via CL 4952070 via CL 4961624
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5026992 by justin marcus in Dev-Anim branch]
2019-02-16 03:55:35 -05:00
Lina Halper
f7baa58bc9 Merging //UE4/Dev-Main to Dev-Anim (//UE4/Dev-Anim)
#fyi: Laurent.Delayen, James.Hopkin
#rb: none

[CL 4923303 by Lina Halper in Dev-Anim branch]
2019-02-06 14:35:56 -05:00