Commit Graph

137 Commits

Author SHA1 Message Date
graeme thornton
c2c39eea8d Hardening of pak index loading
- If a corrupt index is detected, do a retry but collect and log more information along the way to help diagnose the issue.

#rb stefan.boberg

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6685180 via CL 6685186 via CL 6751170 via CL 6751664
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim) (v365-6733468)

[CL 6795050 by graeme thornton in Dev-Anim branch]
2019-06-03 08:35:15 -04:00
justin marcus
b0b96aebc3 Bundle Manager now makes calls to optimize pak memory usage after mounting.
[at]Josh.Adams [at]Graeme.Thornton

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 6658947 via CL 6659206 via CL 6659209 via CL 6659344 via CL 6659918 via CL 6660722
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim) (v365-6733468)

[CL 6760667 by justin marcus in Dev-Anim branch]
2019-05-31 20:20:19 -04:00
graeme thornton
90f88d1fc6 Make pak platform file wnbind the RegisterEncryptionKey delegate when it is destroyed
#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6616196 via CL 6616584 via CL 6616743 via CL 6616759
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim) (v365-6733468)

[CL 6739721 by graeme thornton in Dev-Anim branch]
2019-05-31 16:47:42 -04:00
david harvey
0059ebee4e include a CRC of the compressed data in the fatal error log if decompression fails to help narrow down where in the IO pipline the failure occurred.
#rb ben.woodhouse

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: david.harvey
#ROBOMERGE-SOURCE: CL 6605801 via CL 6605803 via CL 6607800 via CL 6607835
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim) (v365-6733468)

[CL 6734961 by david harvey in Dev-Anim branch]
2019-05-31 16:11:44 -04:00
graeme thornton
9edad54185 Add support to pak platform file for a custom decryption function that a game can override the way decryption is performed at runtime
[FYI] joe.barrett

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6440730 via CL 6440731 via CL 6440813 via CL 6443772
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6472220 by graeme thornton in Dev-Anim branch]
2019-05-15 07:17:59 -04:00
justin marcus
f0cbb9e128 Add DumpEarlyPakFileReads command to dump all pak reads that occur before it's possible to install patched content to CSV/Log.
This is disabled in shipping.
[at]Daniel.Lamb

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: justin.marcus
#ROBOMERGE-SOURCE: CL 6258473 via CL 6258479 via CL 6258480 via CL 6266092 via CL 6266227
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6281896 by justin marcus in Dev-Anim branch]
2019-05-03 17:25:18 -04:00
graeme thornton
0d1e40c73d Improvements to pak file check command
- Add timing information to the log
 - If pak is signed, use a faster method where we just read one byte from each signing block
 - Still invoked with the -checkpak command line parameter

[FYI] bart.hawthorne, steve.white

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6251572 via CL 6251573 via CL 6251574 via CL 6251688 via CL 6251693
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6254683 by graeme thornton in Dev-Anim branch]
2019-05-02 13:01:58 -04:00
graeme thornton
67c7c2e278 Minor off-by-one correction to chunk hash failure warning message
#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6113928 via CL 6113932 via CL 6115047 via CL 6115246
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6131955 by graeme thornton in Dev-Anim branch]
2019-04-26 22:42:53 -04:00
graeme thornton
aa6ff0a059 Improvements to loading/validation/logging of pak signature files. Better logging of errors.
#jira UE-73250

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6095198 via CL 6095203 via CL 6096453 via CL 6096500
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6122177 by graeme thornton in Dev-Anim branch]
2019-04-26 21:43:34 -04:00
graeme thornton
2912cfb4d4 Make ChunkHashToString() an inline function in IPlatformFilePak.h so that it can be used by other modules without requiring a link dependency
#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6092711 via CL 6092712 via CL 6092999 via CL 6093032
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6120691 by graeme thornton in Dev-Anim branch]
2019-04-26 21:35:44 -04:00
graeme thornton
6350f02730 Centralize the loading of pak signature data so that the same object can be reused between the SignedArchiveReader and FPakPrecacher implementations. Stops the same data being loaded and verified twice
[FYI] joe.barrett

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6067905 via CL 6067908 via CL 6069306 via CL 6069330
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6073109 by graeme thornton in Dev-Anim branch]
2019-04-23 21:20:07 -04:00
graeme thornton
98254d7acf Improve logging of pak signature failures.
Treat a missing .sig file like a signature failure. Log it and ignore the pak file.


#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5993822 via CL 5994244 via CL 5995533 via CL 5995549 via CL 5997449 via CL 5997477
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 6002765 by graeme thornton in Dev-Anim branch]
2019-04-18 21:42:18 -04:00
allan bentham
f3f51590ee UnloadPakEntryFilenames uses 64bit hashes.
[FYI] Graeme.Thornton
#rb Jack.porter
#rnx

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: allan.bentham
#ROBOMERGE-SOURCE: CL 5937139 via CL 5952597 via CL 5956705 via CL 5957127
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5962911 by allan bentham in Dev-Anim branch]
2019-04-17 21:04:09 -04:00
graeme thornton
4259589301 Don't mount pak files that fail the initial signature check
#rb joe.barrett, robert.manuszewski

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5897890 via CL 5897891 via CL 5898565 via CL 5898684
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5900137 by graeme thornton in Dev-Anim branch]
2019-04-15 12:59:51 -04:00
graeme thornton
0f00662a66 Fix debug logging of chunk hashes for pak file signature check failures
#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5834330 via CL 5834331 via CL 5838008 via CL 5838453
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5880380 by graeme thornton in Dev-Anim branch]
2019-04-12 21:31:07 -04:00
peter sauerbrei
7386fae5db allow turning pak unload off from commandline with -nounloadpakentries, this allows all items to be properly loaded in PSO gen
#rb trivial

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: peter.sauerbrei
#ROBOMERGE-SOURCE: CL 5772245 via CL 5795091 via CL 5795113 via CL 5795129 via CL 5799583 via CL 5799699
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5802129 by peter sauerbrei in Dev-Anim branch]
2019-04-08 21:05:27 -04:00
david harvey
c34ab81711 Delete record logging changed to Verbose.
#rb anthony.bills

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: david.harvey
#ROBOMERGE-SOURCE: CL 5773263 via CL 5773264 via CL 5773265 via CL 5773386 via CL 5773389
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5784080 by david harvey in Dev-Anim branch]
2019-04-08 13:41:19 -04:00
peter sauerbrei
9aa5d73f27 turn pak entry unloading back on with a better algorithm courtesy of GraemeT
doesn't solve thread safety, but better collision detection and 30MB on FN
#rb graeme.thornton, josh.adams

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: peter.sauerbrei
#ROBOMERGE-SOURCE: CL 5764474 via CL 5764475 via CL 5764476 via CL 5764477 via CL 5765310 via CL 5765340
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5778018 by peter sauerbrei in Dev-Anim branch]
2019-04-08 13:12:12 -04:00
josh adams
3963d79efb - Replaced -nommio with a CVar: mmio.enable. Can use commandline to disable with: -dpcvars=mmio.enable=0
- This allows it to be hotfixed off
#rb chris.babcock

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 5714583 via CL 5714586 via CL 5714590 via CL 5714986 via CL 5717097 via CL 5717275
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5727028 by josh adams in Dev-Anim branch]
2019-04-04 01:00:10 -04:00
david harvey
fd3244ae2f [LLM] add missing FileSystem tag to file request. moves about ~2MB on average out of Untagged.
#rb anthony.bills

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: david.harvey
#ROBOMERGE-SOURCE: CL 5630089 via CL 5630203 via CL 5630220 via CL 5630238 via CL 5632396 via CL 5632706
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5636297 by david harvey in Dev-Anim branch]
2019-03-28 17:04:07 -04:00
daniel lamb
6cddcbed6a Made the pak file binary compatible with previous version to reduce patch sizes.
Reenabled oodle compression in 8.30 for android, ios and switch.
Cleaned up unrealpak commandline when run from UAT.
#rb Josh.Adams
[FYI] Ben.Marsh, Andrew.Grant

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: daniel.lamb
#ROBOMERGE-SOURCE: CL 5619788 via CL 5619815 via CL 5619821 via CL 5619827 via CL 5623747 via CL 5623814
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5627536 by daniel lamb in Dev-Anim branch]
2019-03-28 01:11:51 -04:00
andrew grant
1c81f62245 Added warning about the dangers of modifying pak format.
[FYI] daniel.lamb :)

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: andrew.grant
#ROBOMERGE-SOURCE: CL 5590393 via CL 5590405 via CL 5590410 via CL 5590416 via CL 5593650 via CL 5593700
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5608595 by andrew grant in Dev-Anim branch]
2019-03-27 16:59:13 -04:00
graeme thornton
915b1f8c26 Initialize signature checking hashcache array when using CRCs. Static analysis warning fix.
#rb none

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5516109 via CL 5516121 via CL 5518542 via CL 5535898
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5542248 by graeme thornton in Dev-Anim branch]
2019-03-25 17:12:24 -04:00
graeme thornton
d5fc2b4bd0 Accidentally forced pak filename/entry compression on. Disabling again.
#rb none

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5500872 via CL 5500873 via CL 5500937 via CL 5500964
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5514788 by graeme thornton in Dev-Anim branch]
2019-03-22 17:57:29 -04:00
hongyi yu
1f74432e13 Fixed the issue where unrealpak failed to read from old pak files with no content.
#rb Graeme.Thornton

#ROBOMERGE-OWNER: lina.halper
#ROBOMERGE-AUTHOR: hongyi.yu
#ROBOMERGE-SOURCE: CL 5484250 via CL 5487362 via CL 5487860
#ROBOMERGE-BOT: ANIM (Main -> Dev-Anim)

[CL 5507694 by hongyi yu in Dev-Anim branch]
2019-03-22 17:13:01 -04:00