168 Commits

Author SHA1 Message Date
CarlMagnus Nordin
6abf1ca5c9 Create one IoDispatcher container file per pak file and let the pak file system handle mounting of IoStores
#rb per.larsson

[CL 11027862 by CarlMagnus Nordin in Dev-Core branch]
2020-01-16 06:01:45 -05:00
ryan durand
471d972e62 Updating copyright for Engine Developer.
#rnx
#rb none


#ROBOMERGE-SOURCE: CL 10869240 via CL 10869516 via CL 10869902
#ROBOMERGE-BOT: (v613-10869866)

[CL 10870584 by ryan durand in Main branch]
2019-12-26 15:32:37 -05:00
marc audy
25ee46da1d Fix error: copying variable of type 'TAtomic<int64>' (aka 'TAtomic<long long>') invokes deleted constructor
#jira
#rnx
[FYI] Graeme.Thornton


#ROBOMERGE-SOURCE: CL 10808963 via CL 10808964
#ROBOMERGE-BOT: (v610-10636431)

[CL 10808966 by marc audy in Main branch]
2019-12-19 09:13:55 -05:00
graeme thornton
180e80ba64 Added an option to unrealpak which forces all compressed files inside a pak to be compressed, even if the compression results in a larger file size
#rb per.larsson


#ROBOMERGE-SOURCE: CL 10808714 via CL 10808715
#ROBOMERGE-BOT: (v610-10636431)

[CL 10808716 by graeme thornton in Main branch]
2019-12-19 08:20:24 -05:00
marc audy
dbcf07a558 Use new inline versions of substring functions
Fix up nearby cases where ESearchCase::CaseSensitive should have been used
#jira
#rnx
#rb

#ROBOMERGE-OWNER: marc.audy
#ROBOMERGE-AUTHOR: marc.audy
#ROBOMERGE-SOURCE: CL 10309793 via CL 10309818
#ROBOMERGE-BOT: (v593-10286020)

[CL 10309932 by marc audy in Main branch]
2019-11-19 13:27:07 -05:00
hongyi yu
b22caf3c66 Handle UInt64 output properly when list file with unrealpak
#ROBOMERGE-SOURCE: CL 9421816 via CL 9422754
#ROBOMERGE-BOT: (v480-9420520)

[CL 9423096 by hongyi yu in Main branch]
2019-10-04 16:50:41 -04:00
Stefan Boberg
1f813eb516 Copying //UE4/Dev-Core to Main (//UE4/Main)
#rb many

[CL 9405827 by Stefan Boberg in Main branch]
2019-10-03 16:26:48 -04:00
hongyi yu
d5615ff31e Add the option to force aligning files in pak files that are larger than block size and enable it
Aligning large files will significantly reduce patch size when doing delta patching on XboxOne.
However, it will also increase package size by about 5%.

#rb Daniel.Lamb


#ROBOMERGE-SOURCE: CL 8334773 via CL 8357932
#ROBOMERGE-BOT: (v401-8057353)

[CL 8359105 by hongyi yu in Main branch]
2019-08-27 21:23:37 -04:00
ben marsh
c60b615a40 Fix static analysis warning.
#rb none
[FYI] daniel.lamb
#jira


#ROBOMERGE-SOURCE: CL 8288568 via CL 8288569 via CL 8288836
#ROBOMERGE-BOT: (v401-8057353)

[CL 8288864 by ben marsh in Main branch]
2019-08-25 16:33:56 -04:00
daniel lamb
20429a57b7 Fixed alignment of non compressed files in pak files.
Also removed DDC config file.  Will need to figure out another way to do this when we enable DDC.
Graeme.Thornton


#ROBOMERGE-SOURCE: CL 8240468 via CL 8240483 via CL 8243394
#ROBOMERGE-BOT: (v401-8057353)

[CL 8243494 by daniel lamb in Main branch]
2019-08-23 19:57:51 -04:00
daniel lamb
cee443cc76 Fixed crash in unrealpak in uncompressed non async code path.
#ROBOMERGE-SOURCE: CL 8223513 via CL 8223514 via CL 8229913
#ROBOMERGE-BOT: (v401-8057353)

[CL 8230750 by daniel lamb in Main branch]
2019-08-23 12:04:28 -04:00
bob tellez
696d945a82 Merging CL#8215904 from //Fortnite/Release-10.30
Multithreaded file compression in unrealpak CreatePak.  Local testing shows 50% improvement.
Added compressed files to the DDC this doesn't improve performance when using Oodle Optimal2 but does improve when using Oodle Optimal3.

#test preflight switch builds.
Graeme.Thornton


#ROBOMERGE-AUTHOR: Daniel.Lamb
#ROBOMERGE-SOURCE: CL 8216871 via CL 8229351
#ROBOMERGE-BOT: (v401-8057353)

[CL 8230264 by bob tellez in Main branch]
2019-08-23 11:56:05 -04:00
daniel lamb
e59eda4aec Added validation to patch generation to detect when unsupported files are changed.
#ROBOMERGE-SOURCE: CL 7937403 via CL 7937406 via CL 7937407 via CL 7940721
#ROBOMERGE-BOT: (v391-7919777)

[CL 7940816 by daniel lamb in Main branch]
2019-08-09 20:38:31 -04:00
graeme thornton
9a3d7dcf99 UnrealPak: Only print out "adding entries" message when we're actually creating a pak file
#ROBOMERGE-SOURCE: CL 7648160 via CL 7648161 via CL 7648295
#ROBOMERGE-BOT: (v372-7473910)

[CL 7648369 by graeme thornton in Main branch]
2019-07-29 05:10:56 -04:00
bob tellez
452aeba28e Merging CL#7257773
Back out changelist 7256178. This was a bad merge



#ROBOMERGE-OWNER: bob.tellez
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 7258012 via CL 7266926
#ROBOMERGE-BOT: (v369-7254125)

[CL 7266952 by bob tellez in Main branch]
2019-07-10 17:05:15 -04:00
bob tellez
97e9932675 Merging //Fortnite/Release-10.0 to Dev-Main (//Fortnite/Dev-Main)
Merging CL#7251132 from //Fortnite/Release-Competitive.9.40

Merging in all missing changes from Release-9.40 to Release-Competitive.9.40 to try to make RoboMerge happy.

#rnx
#rb none


#ROBOMERGE-OWNER: bob.tellez
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 7256432 via CL 7266483
#ROBOMERGE-BOT: (v369-7254125)

[CL 7266503 by bob tellez in Main branch]
2019-07-10 16:41:03 -04:00
graeme thornton
51cfbaef7d UnrealPak - Added "TestMemoryOptimization" mode which attempts to unload filenames and shrink pak entries across every pak in a directory, making sure there are no collisons and that everything works correctly
#ROBOMERGE-SOURCE: CL 7227149 via CL 7227151 via CL 7227434
#ROBOMERGE-BOT: (v367-6836689)

[CL 7227472 by graeme thornton in Main branch]
2019-07-08 10:23:53 -04:00
david harvey
58c25dd385 Adding internal -CalcCompressionBlockCRCs command line option to UnrealPak to try and track down the rare FPakAsyncReadFileHandle::DoProcessing issue + a bit more logging when it fails.
#rb gil.gribb


#ROBOMERGE-SOURCE: CL 6953229 via CL 6953341
#ROBOMERGE-BOT: (v366-6836689)

[CL 6953356 by david harvey in Main branch]
2019-06-12 07:11:50 -04:00
Stefan Boberg
8607ecb30d Copying //UE4/Dev-Core to Dev-Main (//UE4/Dev-Main)
#rb none

[CL 6815521 by Stefan Boberg in Main branch]
2019-06-03 15:32:00 -04:00
graeme thornton
54c0741d48 Hardening of pak index loading
- If a corrupt index is detected, do a retry but collect and log more information along the way to help diagnose the issue.

#rb stefan.boberg


#ROBOMERGE-SOURCE: CL 6685180 via CL 6685186 via CL 6751170
#ROBOMERGE-BOT: (v365-6733468)

[CL 6751664 by graeme thornton in Main branch]
2019-05-31 18:22:01 -04:00
daniel lamb
bfae378499 Content build label update now errors if the build is not on prod, instead of trying to upload it again.
Added new content only build node  and now generates pak patch files for pak files not in original build.
Also merges manifest with original build
Also posts build to gamedev and prod
GenerateChunkDBs now takes in two additional parameters specifying how many rollback labels to process and which ones to optimize (preparation for future change where we process more chunkdbs).
Added support for generating buildinfo.ini for ps4.


#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: daniel.lamb
#ROBOMERGE-SOURCE: CL 6278253 via CL 6287210 via CL 6287835
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 6302329 by daniel lamb in Dev-Core branch]
2019-05-03 21:13:34 -04:00
daniel lamb
04d57e57cf Content build label update now errors if the build is not on prod, instead of trying to upload it again.
Added new content only build node  and now generates pak patch files for pak files not in original build.
Also merges manifest with original build
Also posts build to gamedev and prod
GenerateChunkDBs now takes in two additional parameters specifying how many rollback labels to process and which ones to optimize (preparation for future change where we process more chunkdbs).
Added support for generating buildinfo.ini for ps4.



#ROBOMERGE-SOURCE: CL 6278253 via CL 6287210

[CL 6287835 by daniel lamb in Main branch]
2019-05-03 17:47:05 -04:00
ben woodhouse
f442eb9f19 Disable GUARANTEE_UASSET_AND_UEXP_IN_SAME_PAK for patch pak generation, since it's not necessary with delete records. Lifing this restriction makes pak seek optimization a lot more efficient.
Profiling in suggests it improves streaming performance, and the paks are smaller
#rb none
[FYI] hongyi.yu

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: ben.woodhouse
#ROBOMERGE-SOURCE: CL 5810432 via CL 5810437 via CL 5810648 via CL 5813653 via CL 5813762
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 5843937 by ben woodhouse in Dev-Core branch]
2019-04-12 13:04:38 -04:00
ben woodhouse
e741450957 Add -fallbackOrderForNonUassetFiles option for ordering uexp and ubulk files if order is not specified (e.g for the secondary/cooker open order). Previously this was non-optional, but this caused a regression in patch sizes on some platforms
#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: ben.woodhouse
#ROBOMERGE-SOURCE: CL 5772400 via CL 5795133 via CL 5795152 via CL 5795156 via CL 5799600 via CL 5799704
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)

[CL 5818691 by ben woodhouse in Dev-Core branch]
2019-04-10 03:07:09 -04:00
ben woodhouse
76d99dbc1f Disable GUARANTEE_UASSET_AND_UEXP_IN_SAME_PAK for patch pak generation, since it's not necessary with delete records. Lifing this restriction makes pak seek optimization a lot more efficient.
Profiling in suggests it improves streaming performance, and the paks are smaller
#rb none
[FYI] hongyi.yu


#ROBOMERGE-SOURCE: CL 5810432 via CL 5810437 via CL 5810648 via CL 5813653

[CL 5813762 by ben woodhouse in Main branch]
2019-04-09 17:12:14 -04:00