Commit Graph

59 Commits

Author SHA1 Message Date
ben marsh
c60b615a40 Fix static analysis warning.
#rb none
[FYI] daniel.lamb
#jira


#ROBOMERGE-SOURCE: CL 8288568 via CL 8288569 via CL 8288836
#ROBOMERGE-BOT: (v401-8057353)

[CL 8288864 by ben marsh in Main branch]
2019-08-25 16:33:56 -04:00
daniel lamb
20429a57b7 Fixed alignment of non compressed files in pak files.
Also removed DDC config file.  Will need to figure out another way to do this when we enable DDC.
Graeme.Thornton


#ROBOMERGE-SOURCE: CL 8240468 via CL 8240483 via CL 8243394
#ROBOMERGE-BOT: (v401-8057353)

[CL 8243494 by daniel lamb in Main branch]
2019-08-23 19:57:51 -04:00
daniel lamb
cee443cc76 Fixed crash in unrealpak in uncompressed non async code path.
#ROBOMERGE-SOURCE: CL 8223513 via CL 8223514 via CL 8229913
#ROBOMERGE-BOT: (v401-8057353)

[CL 8230750 by daniel lamb in Main branch]
2019-08-23 12:04:28 -04:00
bob tellez
696d945a82 Merging CL#8215904 from //Fortnite/Release-10.30
Multithreaded file compression in unrealpak CreatePak.  Local testing shows 50% improvement.
Added compressed files to the DDC this doesn't improve performance when using Oodle Optimal2 but does improve when using Oodle Optimal3.

#test preflight switch builds.
Graeme.Thornton


#ROBOMERGE-AUTHOR: Daniel.Lamb
#ROBOMERGE-SOURCE: CL 8216871 via CL 8229351
#ROBOMERGE-BOT: (v401-8057353)

[CL 8230264 by bob tellez in Main branch]
2019-08-23 11:56:05 -04:00
daniel lamb
e59eda4aec Added validation to patch generation to detect when unsupported files are changed.
#ROBOMERGE-SOURCE: CL 7937403 via CL 7937406 via CL 7937407 via CL 7940721
#ROBOMERGE-BOT: (v391-7919777)

[CL 7940816 by daniel lamb in Main branch]
2019-08-09 20:38:31 -04:00
graeme thornton
9a3d7dcf99 UnrealPak: Only print out "adding entries" message when we're actually creating a pak file
#ROBOMERGE-SOURCE: CL 7648160 via CL 7648161 via CL 7648295
#ROBOMERGE-BOT: (v372-7473910)

[CL 7648369 by graeme thornton in Main branch]
2019-07-29 05:10:56 -04:00
bob tellez
452aeba28e Merging CL#7257773
Back out changelist 7256178. This was a bad merge



#ROBOMERGE-OWNER: bob.tellez
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 7258012 via CL 7266926
#ROBOMERGE-BOT: (v369-7254125)

[CL 7266952 by bob tellez in Main branch]
2019-07-10 17:05:15 -04:00
bob tellez
97e9932675 Merging //Fortnite/Release-10.0 to Dev-Main (//Fortnite/Dev-Main)
Merging CL#7251132 from //Fortnite/Release-Competitive.9.40

Merging in all missing changes from Release-9.40 to Release-Competitive.9.40 to try to make RoboMerge happy.

#rnx
#rb none


#ROBOMERGE-OWNER: bob.tellez
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 7256432 via CL 7266483
#ROBOMERGE-BOT: (v369-7254125)

[CL 7266503 by bob tellez in Main branch]
2019-07-10 16:41:03 -04:00
graeme thornton
51cfbaef7d UnrealPak - Added "TestMemoryOptimization" mode which attempts to unload filenames and shrink pak entries across every pak in a directory, making sure there are no collisons and that everything works correctly
#ROBOMERGE-SOURCE: CL 7227149 via CL 7227151 via CL 7227434
#ROBOMERGE-BOT: (v367-6836689)

[CL 7227472 by graeme thornton in Main branch]
2019-07-08 10:23:53 -04:00
david harvey
58c25dd385 Adding internal -CalcCompressionBlockCRCs command line option to UnrealPak to try and track down the rare FPakAsyncReadFileHandle::DoProcessing issue + a bit more logging when it fails.
#rb gil.gribb


#ROBOMERGE-SOURCE: CL 6953229 via CL 6953341
#ROBOMERGE-BOT: (v366-6836689)

[CL 6953356 by david harvey in Main branch]
2019-06-12 07:11:50 -04:00
Stefan Boberg
8607ecb30d Copying //UE4/Dev-Core to Dev-Main (//UE4/Dev-Main)
#rb none

[CL 6815521 by Stefan Boberg in Main branch]
2019-06-03 15:32:00 -04:00
graeme thornton
54c0741d48 Hardening of pak index loading
- If a corrupt index is detected, do a retry but collect and log more information along the way to help diagnose the issue.

#rb stefan.boberg


#ROBOMERGE-SOURCE: CL 6685180 via CL 6685186 via CL 6751170
#ROBOMERGE-BOT: (v365-6733468)

[CL 6751664 by graeme thornton in Main branch]
2019-05-31 18:22:01 -04:00
daniel lamb
04d57e57cf Content build label update now errors if the build is not on prod, instead of trying to upload it again.
Added new content only build node  and now generates pak patch files for pak files not in original build.
Also merges manifest with original build
Also posts build to gamedev and prod
GenerateChunkDBs now takes in two additional parameters specifying how many rollback labels to process and which ones to optimize (preparation for future change where we process more chunkdbs).
Added support for generating buildinfo.ini for ps4.



#ROBOMERGE-SOURCE: CL 6278253 via CL 6287210

[CL 6287835 by daniel lamb in Main branch]
2019-05-03 17:47:05 -04:00
ben woodhouse
f442eb9f19 Disable GUARANTEE_UASSET_AND_UEXP_IN_SAME_PAK for patch pak generation, since it's not necessary with delete records. Lifing this restriction makes pak seek optimization a lot more efficient.
Profiling in suggests it improves streaming performance, and the paks are smaller
#rb none
[FYI] hongyi.yu

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: ben.woodhouse
#ROBOMERGE-SOURCE: CL 5810432 via CL 5810437 via CL 5810648 via CL 5813653 via CL 5813762
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 5843937 by ben woodhouse in Dev-Core branch]
2019-04-12 13:04:38 -04:00
ben woodhouse
e741450957 Add -fallbackOrderForNonUassetFiles option for ordering uexp and ubulk files if order is not specified (e.g for the secondary/cooker open order). Previously this was non-optional, but this caused a regression in patch sizes on some platforms
#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: ben.woodhouse
#ROBOMERGE-SOURCE: CL 5772400 via CL 5795133 via CL 5795152 via CL 5795156 via CL 5799600 via CL 5799704
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 5818691 by ben woodhouse in Dev-Core branch]
2019-04-10 03:07:09 -04:00
ben woodhouse
76d99dbc1f Disable GUARANTEE_UASSET_AND_UEXP_IN_SAME_PAK for patch pak generation, since it's not necessary with delete records. Lifing this restriction makes pak seek optimization a lot more efficient.
Profiling in suggests it improves streaming performance, and the paks are smaller
#rb none
[FYI] hongyi.yu


#ROBOMERGE-SOURCE: CL 5810432 via CL 5810437 via CL 5810648 via CL 5813653

[CL 5813762 by ben woodhouse in Main branch]
2019-04-09 17:12:14 -04:00
ben woodhouse
5d0b3cfc55 Add -fallbackOrderForNonUassetFiles option for ordering uexp and ubulk files if order is not specified (e.g for the secondary/cooker open order). Previously this was non-optional, but this caused a regression in patch sizes on some platforms
#ROBOMERGE-SOURCE: CL 5772400 via CL 5795133 via CL 5795152 via CL 5795156 via CL 5799600

[CL 5799704 by ben woodhouse in Main branch]
2019-04-08 18:22:18 -04:00
ben woodhouse
b52d87ce16 Fix bug caught by static analysis in AuditPakFiles()
#jira UE-72242
[FYI] marc.audy

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: ben.woodhouse
#ROBOMERGE-SOURCE: CL 5708883 via CL 5708886 via CL 5708887 via CL 5708891 via CL 5709641 via CL 5709794
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 5718872 by ben woodhouse in Dev-Core branch]
2019-04-03 21:05:57 -04:00
ben woodhouse
9a133b9de3 Fix bug caught by static analysis in AuditPakFiles()
#jira UE-72242
[FYI] marc.audy


#ROBOMERGE-SOURCE: CL 5708883 via CL 5708886 via CL 5708887 via CL 5708891 via CL 5709641

[CL 5709794 by ben woodhouse in Main branch]
2019-04-03 10:50:33 -04:00
daniel lamb
302f8e067e Added additional logging to unrealpak for compressor settings.
#test Unrealpak

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: daniel.lamb
#ROBOMERGE-SOURCE: CL 5691627 via CL 5691628 via CL 5691630 via CL 5691631 via CL 5695674 via CL 5695740
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 5701481 by daniel lamb in Dev-Core branch]
2019-04-02 21:12:14 -04:00
ben woodhouse
a069583ffb UnrealPak patch pak seek optimization v2
- Add support for incremental gap filling. Does gap filling in multiple passes, allows us to limit patch size growth to some fixed percentage
- Fixed cooker order UEXP files (previously these were unordered, which could actually result in patches increasing in size after gap filling)
- Added better method for measuring fragmentation - reports as a percentage, takes order index into account
- Refactored OrderMapping via new FPakOrderMap class which knows about primary/secondary mappings
- Added various schemes for defragmentation, including OnePass method which matches legacy behaviour, and various incremental schemes. Incremental_PrimaryFirst seems to give best results
#rb hongyi.yu

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: ben.woodhouse
#ROBOMERGE-SOURCE: CL 5688530 via CL 5688536 via CL 5688538 via CL 5690113 via CL 5694330 via CL 5694501
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 5699633 by ben woodhouse in Dev-Core branch]
2019-04-02 21:05:07 -04:00
daniel lamb
6f34dab1e9 Added additional logging to unrealpak for compressor settings.
#test Unrealpak


#ROBOMERGE-SOURCE: CL 5691627 via CL 5691628 via CL 5691630 via CL 5691631 via CL 5695674

[CL 5695740 by daniel lamb in Main branch]
2019-04-02 18:11:24 -04:00
ben woodhouse
703ef90966 UnrealPak patch pak seek optimization v2
- Add support for incremental gap filling. Does gap filling in multiple passes, allows us to limit patch size growth to some fixed percentage
- Fixed cooker order UEXP files (previously these were unordered, which could actually result in patches increasing in size after gap filling)
- Added better method for measuring fragmentation - reports as a percentage, takes order index into account
- Refactored OrderMapping via new FPakOrderMap class which knows about primary/secondary mappings
- Added various schemes for defragmentation, including OnePass method which matches legacy behaviour, and various incremental schemes. Incremental_PrimaryFirst seems to give best results
#rb hongyi.yu


#ROBOMERGE-SOURCE: CL 5688530 via CL 5688536 via CL 5688538 via CL 5690113 via CL 5694330

[CL 5694501 by ben woodhouse in Main branch]
2019-04-02 17:52:35 -04:00
robert manuszewski
624093cbf4 Made the pak file binary compatible with previous version to reduce patch sizes.
Reenabled oodle compression in 8.30 for android, ios and switch.
Cleaned up unrealpak commandline when run from UAT.
#rb Josh.Adams
[FYI] Ben.Marsh, Andrew.Grant

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: daniel.lamb
#ROBOMERGE-SOURCE: CL 5619788 via CL 5619815 via CL 5619821 via CL 5619827 via CL 5623747 via CL 5623814
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 5628527 by robert manuszewski in Dev-Core branch]
2019-03-28 03:54:25 -04:00
daniel lamb
9e6a72c368 Made the pak file binary compatible with previous version to reduce patch sizes.
Reenabled oodle compression in 8.30 for android, ios and switch.
Cleaned up unrealpak commandline when run from UAT.
#rb Josh.Adams
[FYI] Ben.Marsh, Andrew.Grant


#ROBOMERGE-SOURCE: CL 5619788 via CL 5619815 via CL 5619821 via CL 5619827 via CL 5623747

[CL 5623814 by daniel lamb in Main branch]
2019-03-27 21:08:44 -04:00