Commit Graph

139 Commits

Author SHA1 Message Date
david harvey
d70361928b include a CRC of the compressed data in the fatal error log if decompression fails to help narrow down where in the IO pipline the failure occurred.
#rb ben.woodhouse


#ROBOMERGE-SOURCE: CL 6605801 via CL 6605803 via CL 6607800
#ROBOMERGE-BOT: (v357-6606154)

[CL 6607835 by david harvey in Main branch]
2019-05-22 13:14:29 -04:00
graeme thornton
c0cc487d44 Add support to pak platform file for a custom decryption function that a game can override the way decryption is performed at runtime
[FYI] joe.barrett


#ROBOMERGE-SOURCE: CL 6440730 via CL 6440731 via CL 6440813

[CL 6443772 by graeme thornton in Main branch]
2019-05-14 09:19:53 -04:00
graeme thornton
0ec13c7d73 Improvements to pak file check command
- Add timing information to the log
 - If pak is signed, use a faster method where we just read one byte from each signing block
 - Still invoked with the -checkpak command line parameter

[FYI] bart.hawthorne, steve.white

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6251572 via CL 6251573 via CL 6251574 via CL 6251688 via CL 6251693
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 6269944 by graeme thornton in Dev-Core branch]
2019-05-03 05:17:11 -04:00
justin marcus
cc903b7468 Add DumpEarlyPakFileReads command to dump all pak reads that occur before it's possible to install patched content to CSV/Log.
This is disabled in shipping.
[at]Daniel.Lamb


#ROBOMERGE-SOURCE: CL 6258473 via CL 6258479 via CL 6258480 via CL 6266092

[CL 6266227 by justin marcus in Main branch]
2019-05-02 19:21:35 -04:00
graeme thornton
4b586ac237 Improvements to pak file check command
- Add timing information to the log
 - If pak is signed, use a faster method where we just read one byte from each signing block
 - Still invoked with the -checkpak command line parameter

[FYI] bart.hawthorne, steve.white


#ROBOMERGE-SOURCE: CL 6251572 via CL 6251573 via CL 6251574 via CL 6251688

[CL 6251693 by graeme thornton in Main branch]
2019-05-02 08:05:18 -04:00
graeme thornton
a25f6f4f34 Minor off-by-one correction to chunk hash failure warning message
#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6113928 via CL 6113932 via CL 6115047 via CL 6115246
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 6131899 by graeme thornton in Dev-Core branch]
2019-04-26 22:42:29 -04:00
graeme thornton
8569da66c9 Minor off-by-one correction to chunk hash failure warning message
#ROBOMERGE-SOURCE: CL 6113928 via CL 6113932 via CL 6115047

[CL 6115246 by graeme thornton in Main branch]
2019-04-26 19:44:52 -04:00
graeme thornton
66206b90d8 Improvements to loading/validation/logging of pak signature files. Better logging of errors.
#jira UE-73250


#ROBOMERGE-SOURCE: CL 6095198 via CL 6095203 via CL 6096453

[CL 6096500 by graeme thornton in Main branch]
2019-04-25 14:11:23 -04:00
graeme thornton
3b2758065f Make ChunkHashToString() an inline function in IPlatformFilePak.h so that it can be used by other modules without requiring a link dependency
#ROBOMERGE-SOURCE: CL 6092711 via CL 6092712 via CL 6092999

[CL 6093032 by graeme thornton in Main branch]
2019-04-25 09:10:49 -04:00
graeme thornton
47bcaf79cf Centralize the loading of pak signature data so that the same object can be reused between the SignedArchiveReader and FPakPrecacher implementations. Stops the same data being loaded and verified twice
[FYI] joe.barrett


#ROBOMERGE-SOURCE: CL 6067905 via CL 6067908 via CL 6069306

[CL 6069330 by graeme thornton in Main branch]
2019-04-23 18:56:29 -04:00
graeme thornton
8632a76237 Improve logging of pak signature failures.
Treat a missing .sig file like a signature failure. Log it and ignore the pak file.



#ROBOMERGE-SOURCE: CL 5993822 via CL 5994244 via CL 5995533 via CL 5995549 via CL 5997449

[CL 5997477 by graeme thornton in Main branch]
2019-04-18 18:24:11 -04:00
allan bentham
2a4ca05717 UnloadPakEntryFilenames uses 64bit hashes.
[FYI] Graeme.Thornton
#rb Jack.porter
#rnx


#ROBOMERGE-SOURCE: CL 5937139 via CL 5952597 via CL 5956705

[CL 5957127 by allan bentham in Main branch]
2019-04-17 17:03:40 -04:00
graeme thornton
5f6c7468ed Don't mount pak files that fail the initial signature check
#rb joe.barrett, robert.manuszewski


#ROBOMERGE-SOURCE: CL 5897890 via CL 5897891 via CL 5898565

[CL 5898684 by graeme thornton in Main branch]
2019-04-15 10:22:26 -04:00
graeme thornton
f715649fdd Fix debug logging of chunk hashes for pak file signature check failures
#ROBOMERGE-SOURCE: CL 5834330 via CL 5834331 via CL 5838008

[CL 5838453 by graeme thornton in Main branch]
2019-04-12 10:40:24 -04:00
peter sauerbrei
ba932a53cf allow turning pak unload off from commandline with -nounloadpakentries, this allows all items to be properly loaded in PSO gen
#rb trivial


#ROBOMERGE-SOURCE: CL 5772245 via CL 5795091 via CL 5795113 via CL 5795129 via CL 5799583

[CL 5799699 by peter sauerbrei in Main branch]
2019-04-08 18:22:07 -04:00
peter sauerbrei
d3433d5fe0 turn pak entry unloading back on with a better algorithm courtesy of GraemeT
doesn't solve thread safety, but better collision detection and 30MB on FN
#rb graeme.thornton, josh.adams


#ROBOMERGE-SOURCE: CL 5764474 via CL 5764475 via CL 5764476 via CL 5764477 via CL 5765310

[CL 5765340 by peter sauerbrei in Main branch]
2019-04-05 11:28:48 -04:00
josh adams
9e32b0aec2 - Replaced -nommio with a CVar: mmio.enable. Can use commandline to disable with: -dpcvars=mmio.enable=0
- This allows it to be hotfixed off
#rb chris.babcock


#ROBOMERGE-SOURCE: CL 5714583 via CL 5714586 via CL 5714590 via CL 5714986 via CL 5717097

[CL 5717275 by josh adams in Main branch]
2019-04-03 20:19:45 -04:00
david harvey
f7bf258dcd [LLM] add missing FileSystem tag to file request. moves about ~2MB on average out of Untagged.
#rb anthony.bills


#ROBOMERGE-SOURCE: CL 5630089 via CL 5630203 via CL 5630220 via CL 5630238 via CL 5632396

[CL 5632706 by david harvey in Main branch]
2019-03-28 13:51:25 -04:00
graeme thornton
5c9b91860c Initialize signature checking hashcache array when using CRCs. Static analysis warning fix.
#rb none


#ROBOMERGE-SOURCE: CL 5516109 via CL 5516121 via CL 5518542

[CL 5535898 by graeme thornton in Main branch]
2019-03-25 14:05:37 -04:00
graeme thornton
17730913e6 Accidentally forced pak filename/entry compression on. Disabling again.
#rb none


#ROBOMERGE-SOURCE: CL 5500872 via CL 5500873 via CL 5500937

[CL 5500964 by graeme thornton in Main branch]
2019-03-22 07:20:03 -04:00
hongyi yu
36b439a63b Fixed the issue where unrealpak failed to read from old pak files with no content.
#rb Graeme.Thornton


#ROBOMERGE-OWNER: hongyi.yu
#ROBOMERGE-AUTHOR: hongyi.yu
#ROBOMERGE-SOURCE: CL 5484250 via CL 5487362

[CL 5487860 by hongyi yu in Main branch]
2019-03-20 18:55:06 -04:00
graeme thornton
c491de975a Edigrated rollup of all engine RSA changes that were done for 4.22. Pak signature checking is now much more competent.
#rb none
[FYI] joe.barrett
#rnx


#ROBOMERGE-SOURCE: CL 5469067 via CL 5469119 via CL 5470585

[CL 5470597 by graeme thornton in Main branch]
2019-03-20 12:03:26 -04:00
graeme thornton
9505003326 Created a new interface to RSA functionality that is independent of key size (unlike the old TBigInt version). Internally, RSA functions are implemented by OpenSSL on Windows/Mac/Linux an falling back to the TBigInt implementation for other platforms
- Future plan would be not to use OpenSSL directly, but to go via PlatformCrypto or something like it which could provide platform specific RSA functionality if it exists
- Also contains some compensation code for platforms that are still stuck on an older version of OpenSSL (everything except windows)
- On platforms that use OpenSSL < 1.1.1, register a locking callback so we can provide mutexes and stop it crashing horribly when the same RSA key is used concurrently on different threads

Changed default key size in the CryptoKeys plugin to 4096 bits.

#jira UE-71377
#rb ian.fox, robert.manuszewski, ryan.gerleve

#ROBOMERGE-OWNER: graeme.thornton
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5447433 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)

[CL 5447485 by graeme thornton in Main branch]
2019-03-19 07:53:22 -04:00
graeme thornton
a0e55b062e Improvements to RSA key generation and pak signing
- CryptoKeys plugin can now be told how many bits to use for keys. Defaults to 2048 as a happy medium between security and performance
- After creating the OpenSSL RSA key, use the exp/mod values directly rather than recomputing from primes
- Added RSA.h with some TBigInt based helpers for RSA keys
- Added FPakSignatureFile to wrap the contents of the pak .sig file, including determining that it is of a new format that supports versioning. Wraps the encryption and decryption of he master table hash
- Changed pak master signature hash from crc to SHA1
- Modified access to executable embedded public key data so that it is just an array of bytes. Will allow custom key sizes in the future.

#rb joe.barrett, stefan.boberg
#jira UE-71280, UE-71377

#ROBOMERGE-OWNER: graeme.thornton
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5381335 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)

[CL 5383057 by graeme thornton in Main branch]
2019-03-13 11:12:27 -04:00
Allan Bentham
926555ac14 Fix comments somehow lost during merge
#jira UE-70963
#rb none

[CL 5294300 by Allan Bentham in Main branch]
2019-03-05 07:15:10 -05:00