Commit Graph

130 Commits

Author SHA1 Message Date
justin marcus
47b034dd8b Add DumpEarlyPakFileReads command to dump all pak reads that occur before it's possible to install patched content to CSV/Log.
This is disabled in shipping.
[at]Daniel.Lamb

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 6258473 via CL 6258479 via CL 6258480 via CL 6266092 via CL 6266227
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6280536 by justin marcus in Dev-Build branch]
2019-05-03 17:18:24 -04:00
graeme thornton
54a8813899 Improvements to pak file check command
- Add timing information to the log
 - If pak is signed, use a faster method where we just read one byte from each signing block
 - Still invoked with the -checkpak command line parameter

[FYI] bart.hawthorne, steve.white

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6251572 via CL 6251573 via CL 6251574 via CL 6251688 via CL 6251693
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6254611 by graeme thornton in Dev-Build branch]
2019-05-02 13:01:44 -04:00
graeme thornton
9ffb7aebf9 Minor off-by-one correction to chunk hash failure warning message
#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6113928 via CL 6113932 via CL 6115047 via CL 6115246
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6131956 by graeme thornton in Dev-Build branch]
2019-04-26 22:42:54 -04:00
graeme thornton
a782a573ec Improvements to loading/validation/logging of pak signature files. Better logging of errors.
#jira UE-73250

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6095198 via CL 6095203 via CL 6096453 via CL 6096500
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6122197 by graeme thornton in Dev-Build branch]
2019-04-26 21:43:40 -04:00
graeme thornton
727d99e5fa Make ChunkHashToString() an inline function in IPlatformFilePak.h so that it can be used by other modules without requiring a link dependency
#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6092711 via CL 6092712 via CL 6092999 via CL 6093032
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6120714 by graeme thornton in Dev-Build branch]
2019-04-26 21:35:50 -04:00
graeme thornton
ca133d2f11 Centralize the loading of pak signature data so that the same object can be reused between the SignedArchiveReader and FPakPrecacher implementations. Stops the same data being loaded and verified twice
[FYI] joe.barrett

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6067905 via CL 6067908 via CL 6069306 via CL 6069330
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6073221 by graeme thornton in Dev-Build branch]
2019-04-23 21:20:30 -04:00
graeme thornton
84f9a6a029 Improve logging of pak signature failures.
Treat a missing .sig file like a signature failure. Log it and ignore the pak file.


#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5993822 via CL 5994244 via CL 5995533 via CL 5995549 via CL 5997449 via CL 5997477
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6011395 by graeme thornton in Dev-Build branch]
2019-04-19 08:22:39 -04:00
allan bentham
14f3d66ca5 UnloadPakEntryFilenames uses 64bit hashes.
[FYI] Graeme.Thornton
#rb Jack.porter
#rnx

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: allan.bentham
#ROBOMERGE-SOURCE: CL 5937139 via CL 5952597 via CL 5956705 via CL 5957127
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5962860 by allan bentham in Dev-Build branch]
2019-04-17 21:04:00 -04:00
graeme thornton
569a413697 Don't mount pak files that fail the initial signature check
#rb joe.barrett, robert.manuszewski

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5897890 via CL 5897891 via CL 5898565 via CL 5898684
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5900073 by graeme thornton in Dev-Build branch]
2019-04-15 12:59:29 -04:00
graeme thornton
476c5a6d91 Fix debug logging of chunk hashes for pak file signature check failures
#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5834330 via CL 5834331 via CL 5838008 via CL 5838453
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5892789 by graeme thornton in Dev-Build branch]
2019-04-14 07:11:11 -04:00
peter sauerbrei
500229e8f6 allow turning pak unload off from commandline with -nounloadpakentries, this allows all items to be properly loaded in PSO gen
#rb trivial

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: peter.sauerbrei
#ROBOMERGE-SOURCE: CL 5772245 via CL 5795091 via CL 5795113 via CL 5795129 via CL 5799583 via CL 5799699
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5802098 by peter sauerbrei in Dev-Build branch]
2019-04-08 21:05:19 -04:00
david harvey
2aa16d2ab4 Delete record logging changed to Verbose.
#rb anthony.bills

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: david.harvey
#ROBOMERGE-SOURCE: CL 5773263 via CL 5773264 via CL 5773265 via CL 5773386 via CL 5773389
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5784142 by david harvey in Dev-Build branch]
2019-04-08 13:41:36 -04:00
peter sauerbrei
4c329140b9 turn pak entry unloading back on with a better algorithm courtesy of GraemeT
doesn't solve thread safety, but better collision detection and 30MB on FN
#rb graeme.thornton, josh.adams

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: peter.sauerbrei
#ROBOMERGE-SOURCE: CL 5764474 via CL 5764475 via CL 5764476 via CL 5764477 via CL 5765310 via CL 5765340
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5778092 by peter sauerbrei in Dev-Build branch]
2019-04-08 13:12:30 -04:00
josh adams
d9069e227e - Replaced -nommio with a CVar: mmio.enable. Can use commandline to disable with: -dpcvars=mmio.enable=0
- This allows it to be hotfixed off
#rb chris.babcock

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 5714583 via CL 5714586 via CL 5714590 via CL 5714986 via CL 5717097 via CL 5717275
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5726978 by josh adams in Dev-Build branch]
2019-04-04 00:59:52 -04:00
david harvey
3d330e403e [LLM] add missing FileSystem tag to file request. moves about ~2MB on average out of Untagged.
#rb anthony.bills

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: david.harvey
#ROBOMERGE-SOURCE: CL 5630089 via CL 5630203 via CL 5630220 via CL 5630238 via CL 5632396 via CL 5632706
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5636276 by david harvey in Dev-Build branch]
2019-03-28 17:04:02 -04:00
daniel lamb
fd8abed8e0 Made the pak file binary compatible with previous version to reduce patch sizes.
Reenabled oodle compression in 8.30 for android, ios and switch.
Cleaned up unrealpak commandline when run from UAT.
#rb Josh.Adams
[FYI] Ben.Marsh, Andrew.Grant

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: daniel.lamb
#ROBOMERGE-SOURCE: CL 5619788 via CL 5619815 via CL 5619821 via CL 5619827 via CL 5623747 via CL 5623814
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5627517 by daniel lamb in Dev-Build branch]
2019-03-28 01:11:43 -04:00
andrew grant
af166dd161 Added warning about the dangers of modifying pak format.
[FYI] daniel.lamb :)

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 5590393 via CL 5590405 via CL 5590410 via CL 5590416 via CL 5593650 via CL 5593700
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5608589 by andrew grant in Dev-Build branch]
2019-03-27 16:59:11 -04:00
graeme thornton
6412f95dc4 Initialize signature checking hashcache array when using CRCs. Static analysis warning fix.
#rb none

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5516109 via CL 5516121 via CL 5518542 via CL 5535898
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5542310 by graeme thornton in Dev-Build branch]
2019-03-25 17:12:34 -04:00
graeme thornton
ba9528355a Accidentally forced pak filename/entry compression on. Disabling again.
#rb none

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5500872 via CL 5500873 via CL 5500937 via CL 5500964
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5514739 by graeme thornton in Dev-Build branch]
2019-03-22 17:57:16 -04:00
hongyi yu
6f77c05607 Fixed the issue where unrealpak failed to read from old pak files with no content.
#rb Graeme.Thornton

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: hongyi.yu
#ROBOMERGE-SOURCE: CL 5484250 via CL 5487362 via CL 5487860
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5507695 by hongyi yu in Dev-Build branch]
2019-03-22 17:13:01 -04:00
graeme thornton
890f80422d Edigrated rollup of all engine RSA changes that were done for 4.22. Pak signature checking is now much more competent.
#rb none
[FYI] joe.barrett
#rnx

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5469067 via CL 5469119 via CL 5470585 via CL 5470597
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5505285 by graeme thornton in Dev-Build branch]
2019-03-22 16:59:41 -04:00
graeme thornton
917f827e1f Created a new interface to RSA functionality that is independent of key size (unlike the old TBigInt version). Internally, RSA functions are implemented by OpenSSL on Windows/Mac/Linux an falling back to the TBigInt implementation for other platforms
- Future plan would be not to use OpenSSL directly, but to go via PlatformCrypto or something like it which could provide platform specific RSA functionality if it exists
- Also contains some compensation code for platforms that are still stuck on an older version of OpenSSL (everything except windows)
- On platforms that use OpenSSL < 1.1.1, register a locking callback so we can provide mutexes and stop it crashing horribly when the same RSA key is used concurrently on different threads

Changed default key size in the CryptoKeys plugin to 4096 bits.

#jira UE-71377
#rb ian.fox, robert.manuszewski, ryan.gerleve

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5447433 in //UE4/Release-4.22/... via CL 5447485
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5458239 by graeme thornton in Dev-Build branch]
2019-03-19 15:02:46 -04:00
Ben Marsh
07e535411b Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none
#rnx

[CL 5435186 by Ben Marsh in Dev-Build branch]
2019-03-18 16:38:36 -04:00
allan bentham
12b62c0252 Fix comments somehow lost during merge
#jira UE-70963
#rb none

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: allan.bentham
#ROBOMERGE-SOURCE: CL 5294300 in //UE4/Main/...
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5302038 by allan bentham in Dev-Build branch]
2019-03-05 15:30:55 -05:00
allan bentham
91d15ec587 Optimize deserialization of compressed pak files.
Reuse previously decompressed buffer where possible to avoid repeatedly deserialising and decompressing the same block when serializing data < compressed block size.
#rb Graeme.Thornton
[FYI] Graeme.Thornton

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: allan.bentham
#ROBOMERGE-SOURCE: CL 5237090 via CL 5240033 via CL 5240035
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5251967 by allan bentham in Dev-Build branch]
2019-02-28 17:49:28 -05:00