Commit Graph

152 Commits

Author SHA1 Message Date
Dan Thompson
809c0f9062 Adding support for writing back metadata to an asset registry during iostore staging. Initially this is size information about the chunks.
#rb matt.peters
#rb carl-magnus.nordin
#preflight 62200571943787f1d2199924

[CL 19234043 by Dan Thompson in ue5-main branch]
2022-03-02 19:22:28 -05:00
Martin Ridgers
db99164288 - Enable Zen on Mac and Linux.
- Updated zenserver binaries to 4ae4f1f

#rnx
#preflight 621ddbb0313c8c01033b88ae

[CL 19195911 by Martin Ridgers in ue5-main branch]
2022-03-01 03:56:00 -05:00
CarlMagnus Nordin
52e3f3db01 ZenCook:
Removed several unnecessary files from the file system manifest
Added IoStoreCmdlet utility command for generating the file system manifest
#rnx
#preflight 621772bcdb60b6b59216e62f
#rb per.larsson

[CL 19171203 by CarlMagnus Nordin in ue5-main branch]
2022-02-28 03:11:38 -05:00
pj kack
d3a6effc10 Fix crash in iostore describe command when reading uncompressed data.
Fix broken import lookup logic.

#jira UE-142894
#rb carlmagnus.nordin
#rnx
#preflight 620d0c47b71c717652eeea70
#lockdown mark.lintott

#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 19070380 in //UE5/Release-5.0/... via CL 19090244
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)

[CL 19133386 by pj kack in ue5-main branch]
2022-02-24 19:30:07 -05:00
robert millar
5811293e2a Add TargetRules.bFNameOutlineNumber as an option to reduce the size of FName by 4 bytes by storing the number in the name table.
Adds a define UE_FNAME_OUTLINE_NUMBER.
Removes of FName, FMinimalName from memory image support. Adds of FMemoryImageName.
Removal of FMinimalName operator<<, all fields made private, size made variable.
All fields of FScriptName made private.
Added console commands for dumping numbered/unnumbered names and stats.

#rb johan.torp

#ROBOMERGE-OWNER: robert.millar
#ROBOMERGE-AUTHOR: robert.millar
#ROBOMERGE-SOURCE: CL 19058026 via CL 19058611 via CL 19058656 via CL 19061727 via CL 19061740 via CL 19064047
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v918-19018356)

[CL 19066701 by robert millar in ue5-main branch]
2022-02-21 02:35:48 -05:00
CarlMagnus Nordin
d4a06b766c ZenCOTF:
*Let the game be in full control of what is getting cooked
*Removed artificial delays when waiting for DDC operations
*No longer relies on all requests being urgent
#rnx
#preflight 620d2f80f8655681e051eded
#rb pj.kack

[CL 19032506 by CarlMagnus Nordin in ue5-main branch]
2022-02-17 04:54:33 -05:00
CarlMagnus Nordin
2abbea75a6 Remove Flush/WaitForPendingFlush calls when running COTF, they are no longer needed
#rb per.larsson
#preflight 61fbac05176256ec4f6d4e4b

[CL 18841496 by CarlMagnus Nordin in ue5-main branch]
2022-02-03 06:34:31 -05:00
pj kack
aa6459712e Force disable bPakUsesSecondaryOrder for all projects when staging with iostore, i.e. don't use the non-deterministic CookerOpenOrder.log in the UAT stage and package step.
This improves the determinism in iostore packaged builds with no regression in load times.
Game specific ordering .log files are still enabled, so any files located in [GameName]\Build\[CookPlatform]\FileOpenOrder\ will still be used.

#jira none
#rb carlmagnus.nordin
#rnx
#preflight 61f29f04c0033b39f8b92c8b

#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 18752854 in //UE5/Release-5.0/... via CL 18752859 via CL 18756833
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18757113 by pj kack in ue5-main branch]
2022-01-27 14:06:40 -05:00
graeme thornton
fa985cf523 IOStoreUtilities checks for the existance of a development asset registry before trying to load it
[FYI] carlmagnus.nordin

#ROBOMERGE-AUTHOR: graeme.thornton
#ROBOMERGE-SOURCE: CL 18734136 via CL 18734139 via CL 18734144 via CL 18734148 via CL 18734262 via CL 18734289
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18734293 by graeme thornton in ue5-main branch]
2022-01-26 07:38:57 -05:00
per larsson
24a854e5f7 Support cooking to remote Zen server
#rb none
#preflight 61ea791ae71cba4d253c6919
#rnx

#ROBOMERGE-AUTHOR: per.larsson
#ROBOMERGE-SOURCE: CL 18687394 in //UE5/Release-5.0/... via CL 18687401 via CL 18687404
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v902-18672795)

[CL 18687410 by per larsson in ue5-main branch]
2022-01-21 04:24:08 -05:00
arciel rekman
71c0fe40e8 Compress shaders as groups for cooked games.
- Saves disk size and runtime memory on all platforms that use IoStore-based shader archive (on some - significantly).
- Decompression time is small and seems to get lost in noise given how rarely shaders are actually created.
- Also changed IoStore-based archive to allocate preload entries on demand, and disabled preloading singular shaders (used by the PSO cache) as the API needs more work.

#rb CM.Nordin, Serge.Bernier, Ben.Ingram
[REVIEW] [at]CarlMagnus.Nordin, [at]Serge.Bernier, [at]Ben.Ingram

#ROBOMERGE-AUTHOR: arciel.rekman
#ROBOMERGE-SOURCE: CL 18657587 via CL 18657664 via CL 18657670 via CL 18662749 via CL 18662792 via CL 18662871
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18662941 by arciel rekman in ue5-main branch]
2022-01-19 14:35:49 -05:00
francis hurteau
eb9289d834 Added support to PackageStoreOptimizer to support optional package when generated from legacy cooked header
Support for Zen store will be added separately

#jira UE-129805
#rb CarlMagnus.Nordin, Matt.Peters
#preflight 61e6f8d73778a195debaffa6

#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18642382 in //UE5/Release-5.0/... via CL 18642393 via CL 18642414
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18642448 by francis hurteau in ue5-main branch]
2022-01-18 13:01:31 -05:00
francis hurteau
d262daac13 Added support to Loose File Package Writer to write multiple package output per input
Added similar support to the package store manifest
Zen Store support will be added separately
Added some extra data to linker object resources to be used by the iostore package optimizer to properly support optional package chunks.

Multiple cooking outputs is used to support "optional objects" which are editor data package exports that would be put in separate iostore containers than the normal cooked game content.

#jira UE-129801
#rb Matt.Peters, CarlMagnus.Nordin
#preflight 61e5d3bb873f2ea48f48166c

#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18639742 in //UE5/Release-5.0/... via CL 18639751 via CL 18639758
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18639762 by francis hurteau in ue5-main branch]
2022-01-18 09:56:23 -05:00
pj kack
380bad2a75 Minor optimizations to PackageStoreOptimizer when cooking with ZenStore
#jira none
#rb carlmagnus.nordin
#rnx
#preflight 61bb19cc440d71c977b63c7b

#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 18474617 in //UE5/Release-5.0/... via CL 18480638
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18480720 by pj kack in ue5-release-engine-test branch]
2021-12-16 18:22:15 -05:00
carlmagnus nordin
fe4d9e9c89 Fixed package store manifest not working correctly for iterative cooks
#rnx
#rb per.larsson

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 18465551 in //UE5/Release-5.0/... via CL 18465553
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18465554 by carlmagnus nordin in ue5-release-engine-test branch]
2021-12-15 03:42:08 -05:00
ionut matasaru
1c030f6852 Fixed usage of TCHAR_TO_ANSI in IoStoreUtilities.cpp, PakFileUtilities.cpp and in GeForceNOWWrapper.cpp.
Fixed usage of TCHAR_TO_UTF8 in FbxMainExport (ExportStaticMeshToFbx, ExportSplineMeshToFbx).

#rb CarlMagnus.Nordin
#preflight 61b7589bf42e2f02a8dbedaa

#ROBOMERGE-AUTHOR: ionut.matasaru
#ROBOMERGE-SOURCE: CL 18453463 in //UE5/Release-5.0/... via CL 18453474
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v898-18417669)

[CL 18453477 by ionut matasaru in ue5-release-engine-test branch]
2021-12-14 06:53:29 -05:00
mark lintott
b43fb780c7 Non-Unity Fixes
#jira none
[FYI] zousar.shaker
#rb trival
#preflight 61b72383fc570891337a735f

#ROBOMERGE-AUTHOR: mark.lintott
#ROBOMERGE-SOURCE: CL 18442318 in //UE5/Release-5.0/... via CL 18444991
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18445172 by mark lintott in ue5-release-engine-test branch]
2021-12-13 13:30:16 -05:00
zousar shaker
0832797d28 Ensure that Zen ProjectIds are unique for projects with the same name in differing roots (or in the same root) while attempting to maintain some easily recognizable desriptors for the project.
Staged builds are explicitly assigned a project id via the persistent UECommandline.txt file.  This works on mobile/consoles, but isn't implemented for Windows/Linux/Mac, so staged builds for those platforms don't have a persistent assignment of project id and instead rely on the launch by the editor including the appropriate commandlines.
Unstaged builds on Windows/Mac/Linux now determine project id independently after they have determined the project file path.  This happens after the project file has been determined and has had case corrected.  This ensures it is different/unique for multiple blueprint projects that share the same executable.
Fix a bug where second cook in editor to zen would stall because the queue thinks all additions are finished due to a boolean that isn't reset.

#rb devin.doucette
#rb per.larsson
#preflight 61b39a35a2562c8b1c40f81e

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18432317 in //UE5/Release-5.0/... via CL 18435404
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18435895 by zousar shaker in ue5-release-engine-test branch]
2021-12-10 18:06:39 -05:00
devin doucette
27c1393427 CompressedBuffer: Removed partial decompression from FCompressedBuffer now that FCompressedBufferReader is available
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.

- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.

#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18382377 by devin doucette in ue5-release-engine-test branch]
2021-12-06 10:16:05 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
matt peters
b9be0036df #jira UE-132874
#jira UE-132939
LargeMemoryArchive::ReleaseOwnership() sets TotalSize() to 0, so TotalSize must be called before calling ReleaseOwnership. Calling them both as arguments into a function was succeeding on some compilers and failing on others due to compiler optimization differences.
#rb Zousar.Shaker
#rnx
#[fyi] Francis.Hurteau

#ushell-cherrypick of 17947411 by Matt.Peters

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18008712 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v885-17909292)

[CL 18008728 by matt peters in ue5-release-engine-test branch]
2021-11-01 17:20:22 -04:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
robert millar
14fcaa7623 Add chunk id to output of iostore list command
#rb none
[FYI] carlmagnus.nordin

#ROBOMERGE-AUTHOR: robert.millar
#ROBOMERGE-SOURCE: CL 17738354 via CL 17738362 via CL 17738378 via CL 17738387 via CL 17738435
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17738662 by robert millar in ue5-release-engine-test branch]
2021-10-06 14:36:44 -04:00