Commit Graph

26 Commits

Author SHA1 Message Date
graeme thornton
b84498eb83 UnrealPak - Added "TestMemoryOptimization" mode which attempts to unload filenames and shrink pak entries across every pak in a directory, making sure there are no collisons and that everything works correctly
#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 7227149 via CL 7227151 via CL 7227434 via CL 7227472
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v367-6836689)

[CL 7236800 by graeme thornton in Dev-Build branch]
2019-07-09 00:17:52 -04:00
david harvey
8c060c1b50 Adding internal -CalcCompressionBlockCRCs command line option to UnrealPak to try and track down the rare FPakAsyncReadFileHandle::DoProcessing issue + a bit more logging when it fails.
#rb gil.gribb

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: david.harvey
#ROBOMERGE-SOURCE: CL 6953229 via CL 6953341 via CL 6953356
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v367-6836689)

[CL 7090578 by david harvey in Dev-Build branch]
2019-06-19 09:23:54 -04:00
stefan boberg
285a908f86 Copying //UE4/Dev-Core to Dev-Main (//UE4/Dev-Main)
#rb none

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: stefan.boberg
#ROBOMERGE-SOURCE: CL 6815521 in //UE4/Main/...
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v365-6733468)

[CL 6821265 by stefan boberg in Dev-Build branch]
2019-06-03 19:09:16 -04:00
graeme thornton
f48b80f826 Hardening of pak index loading
- If a corrupt index is detected, do a retry but collect and log more information along the way to help diagnose the issue.

#rb stefan.boberg

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 6685180 via CL 6685186 via CL 6751170 via CL 6751664
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build) (v365-6733468)

[CL 6762248 by graeme thornton in Dev-Build branch]
2019-06-01 11:00:51 -04:00
daniel lamb
345541e654 Content build label update now errors if the build is not on prod, instead of trying to upload it again.
Added new content only build node  and now generates pak patch files for pak files not in original build.
Also merges manifest with original build
Also posts build to gamedev and prod
GenerateChunkDBs now takes in two additional parameters specifying how many rollback labels to process and which ones to optimize (preparation for future change where we process more chunkdbs).
Added support for generating buildinfo.ini for ps4.


#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: daniel.lamb
#ROBOMERGE-SOURCE: CL 6278253 via CL 6287210 via CL 6287835
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 6302275 by daniel lamb in Dev-Build branch]
2019-05-03 21:13:25 -04:00
ben woodhouse
74c278fdf3 Disable GUARANTEE_UASSET_AND_UEXP_IN_SAME_PAK for patch pak generation, since it's not necessary with delete records. Lifing this restriction makes pak seek optimization a lot more efficient.
Profiling in suggests it improves streaming performance, and the paks are smaller
#rb none
[FYI] hongyi.yu

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: ben.woodhouse
#ROBOMERGE-SOURCE: CL 5810432 via CL 5810437 via CL 5810648 via CL 5813653 via CL 5813762
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5843878 by ben woodhouse in Dev-Build branch]
2019-04-12 13:04:27 -04:00
ben woodhouse
ad3ac145cb Add -fallbackOrderForNonUassetFiles option for ordering uexp and ubulk files if order is not specified (e.g for the secondary/cooker open order). Previously this was non-optional, but this caused a regression in patch sizes on some platforms
#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: ben.woodhouse
#ROBOMERGE-SOURCE: CL 5772400 via CL 5795133 via CL 5795152 via CL 5795156 via CL 5799600 via CL 5799704
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5802192 by ben woodhouse in Dev-Build branch]
2019-04-08 21:06:01 -04:00
ben woodhouse
58efd1e6ba Fix bug caught by static analysis in AuditPakFiles()
#jira UE-72242
[FYI] marc.audy

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: ben.woodhouse
#ROBOMERGE-SOURCE: CL 5708883 via CL 5708886 via CL 5708887 via CL 5708891 via CL 5709641 via CL 5709794
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5718826 by ben woodhouse in Dev-Build branch]
2019-04-03 21:05:48 -04:00
daniel lamb
d1a57dbc96 Added additional logging to unrealpak for compressor settings.
#test Unrealpak

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: daniel.lamb
#ROBOMERGE-SOURCE: CL 5691627 via CL 5691628 via CL 5691630 via CL 5691631 via CL 5695674 via CL 5695740
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5701434 by daniel lamb in Dev-Build branch]
2019-04-02 21:12:04 -04:00
ben woodhouse
417a6a34c9 UnrealPak patch pak seek optimization v2
- Add support for incremental gap filling. Does gap filling in multiple passes, allows us to limit patch size growth to some fixed percentage
- Fixed cooker order UEXP files (previously these were unordered, which could actually result in patches increasing in size after gap filling)
- Added better method for measuring fragmentation - reports as a percentage, takes order index into account
- Refactored OrderMapping via new FPakOrderMap class which knows about primary/secondary mappings
- Added various schemes for defragmentation, including OnePass method which matches legacy behaviour, and various incremental schemes. Incremental_PrimaryFirst seems to give best results
#rb hongyi.yu

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: ben.woodhouse
#ROBOMERGE-SOURCE: CL 5688530 via CL 5688536 via CL 5688538 via CL 5690113 via CL 5694330 via CL 5694501
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5699568 by ben woodhouse in Dev-Build branch]
2019-04-02 21:04:55 -04:00
daniel lamb
fd8abed8e0 Made the pak file binary compatible with previous version to reduce patch sizes.
Reenabled oodle compression in 8.30 for android, ios and switch.
Cleaned up unrealpak commandline when run from UAT.
#rb Josh.Adams
[FYI] Ben.Marsh, Andrew.Grant

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: daniel.lamb
#ROBOMERGE-SOURCE: CL 5619788 via CL 5619815 via CL 5619821 via CL 5619827 via CL 5623747 via CL 5623814
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5627517 by daniel lamb in Dev-Build branch]
2019-03-28 01:11:43 -04:00
graeme thornton
917f827e1f Created a new interface to RSA functionality that is independent of key size (unlike the old TBigInt version). Internally, RSA functions are implemented by OpenSSL on Windows/Mac/Linux an falling back to the TBigInt implementation for other platforms
- Future plan would be not to use OpenSSL directly, but to go via PlatformCrypto or something like it which could provide platform specific RSA functionality if it exists
- Also contains some compensation code for platforms that are still stuck on an older version of OpenSSL (everything except windows)
- On platforms that use OpenSSL < 1.1.1, register a locking callback so we can provide mutexes and stop it crashing horribly when the same RSA key is used concurrently on different threads

Changed default key size in the CryptoKeys plugin to 4096 bits.

#jira UE-71377
#rb ian.fox, robert.manuszewski, ryan.gerleve

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5447433 in //UE4/Release-4.22/... via CL 5447485
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5458239 by graeme thornton in Dev-Build branch]
2019-03-19 15:02:46 -04:00
Ben Marsh
07e535411b Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none
#rnx

[CL 5435186 by Ben Marsh in Dev-Build branch]
2019-03-18 16:38:36 -04:00
graeme thornton
03983935c3 Minor unrealpak refactor
- Added a "-sign" parameter to enable signing for the given pak. The cached settings in the crypto.json file aren't used to enable signing any more.
 - Encapsulated all signing and encryption keys in a single keychain object which is passed around, rather than individual keys
 - Added some validation to UBT/UAT encryption config parsing so that if you enable any signing or encryption stuff but haven't generated a key, it'll warn and disable everything.

#jira

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 5237725 via CL 5237726 via CL 5240337 via CL 5240343
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5252097 by graeme thornton in Dev-Build branch]
2019-02-28 17:50:33 -05:00
josh adams
b3cc809913 - Merging Dev-Kairos/Engine/... to Main/Engine/...
- Brings over the necessary engine changes for embedding UE4 mobile as a dylib/so in native mobile app
- Various changes for facial animation, screen recording, others
- ARKit and ARCore plugins were removed, as deemed "not ready"
#rb many people

#ROBOMERGE-OWNER: ben.marsh
#ROBOMERGE-AUTHOR: josh.adams
#ROBOMERGE-SOURCE: CL 5201138 via CL 5203024 via CL 5226277
#ROBOMERGE-BOT: BUILD (Main -> Dev-Build)

[CL 5243833 by josh adams in Dev-Build branch]
2019-02-28 17:02:41 -05:00
Ben Marsh
dece8de2d2 Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none
#rnx

[CL 5095547 by Ben Marsh in Dev-Build branch]
2019-02-20 15:06:37 -05:00
Ben Marsh
0f79d0e564 Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none
#rnx

[CL 4979967 by Ben Marsh in Dev-Build branch]
2019-02-12 21:08:26 -05:00
Ben Marsh
84420696fe Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none
#rnx

[CL 4830875 by Ben Marsh in Dev-Build branch]
2019-01-29 08:56:46 -05:00
Ben Marsh
a3e87004d6 Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none
#rnx

[CL 4783611 by Ben Marsh in Dev-Build branch]
2019-01-23 10:59:53 -05:00
Ben Marsh
c09b6d7f18 Merge from //UE4/Main.
#rb none
#rnx

[CL 4701155 by Ben Marsh in Dev-Build branch]
2019-01-10 11:10:29 -05:00
Ben Marsh
0c830ebcdb Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none
#rnx

[CL 4671723 by Ben Marsh in Dev-Build branch]
2018-12-20 08:33:57 -05:00
Ben Marsh
530369c613 Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none
#rnx

[CL 4662695 by Ben Marsh in Dev-Build branch]
2018-12-14 14:49:12 -05:00
Ben Marsh
b0da1f4fe7 Merging //UE4/Dev-Main to Dev-Build (//UE4/Dev-Build)
#rb none

[CL 4577636 by Ben Marsh in Dev-Build branch]
2018-11-19 10:12:17 -05:00
Marc Audy
af90b7bcd4 Copying //UE4/Fortnite-Staging to Dev-Main (//UE4/Dev-Main) @ 4395008
#rb
#rnx
#lockdown Nick.Penwarden

[CL 4395058 by Marc Audy in Main branch]
2018-09-25 10:11:35 -04:00
Ben Marsh
61597540e2 UnrealPak: Prevent non-deterministic padding data when compression is disabled.
#rb none

[CL 4325607 by Ben Marsh in Dev-Build branch]
2018-08-29 09:13:06 -04:00