Commit Graph

49 Commits

Author SHA1 Message Date
steve robb
fd2a0b1e20 Removal of TEXT literal concatenations, which doesn't work in UTF-8 mode.
#rb devin.doucette
#jira UE-132142
#preflight 62a8748fa76c84377748036b

#ROBOMERGE-OWNER: steve.robb
#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 20647443 via CL 20648283 via CL 20648315
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20651786 by steve robb in ue5-main branch]
2022-06-14 12:46:34 -04:00
Devin Doucette
006e85ab44 CompressedBuffer: Changed RawHash to FIoHash
- Only the first 20 bytes of the RawHash field in the header will be consumed.
- In a subsequent change, the last 12 bytes of the RawHash field will be set to zero for new compressed buffers.

- Leveraged composite buffers to avoid performing an extra copy of compressed blocks.
- Updated FBlockEncoder::Compress to always use ParallelFor.
- Added ParallelForWithPreWorkWithTaskContext.

#preflight 622ba3869b7ce6ea5d44b7ab
#rb Zousar.Shaker
#rnx

[CL 19360767 by Devin Doucette in ue5-main branch]
2022-03-11 16:21:11 -05:00
charles bloom
fabb392aca FBlockEncoder::Compress parallel enhanced
special case 1 block fast path
in multi-block case, run hash on thread too

#preflight 621e3c3ead75a7f58fc2a39e
#rb fabian.giesen

[CL 19207211 by charles bloom in ue5-main branch]
2022-03-01 16:50:24 -05:00
charles bloom
24c44d3173 OodleDataCompression VeryVerbose logging and some cleanup
#preflight 621e5072e15c51d8c5d3ded5
#rb none

[CL 19200528 by charles bloom in ue5-main branch]
2022-03-01 12:09:41 -05:00
charles bloom
43ae97e644 FBlockEncoder::Compress parallel for faster writing of large packages
#preflight 621d9e35ca28c556340c4ce4
#rb fabian.giesen

[CL 19197345 by charles bloom in ue5-main branch]
2022-03-01 09:03:27 -05:00
charles bloom
7d338bc9fe OodleDataCompression OodleScratchBufferCount use one per core in tools
still defaults to 2 for game runtimes
significant speedup in iostore compression at fast compress levels

#preflight 620be31f01253d2e19e2ad2d
#rb pj.kack

[CL 18999615 by charles bloom in ue5-main branch]
2022-02-15 13:42:27 -05:00
aurel cordonnier
ea933b41c4 Merge Dev-EngineMerge [at] 18718262 to FN/Main
this represents UE5/Release-5.0 [at] 18319674
#preflight none

#ROBOMERGE-OWNER: aurel.cordonnier
#ROBOMERGE-AUTHOR: aurel.cordonnier
#ROBOMERGE-SOURCE: CL 18719526 via CL 18719984 via CL 18719985 via CL 18720536 via CL 18720568
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18720577 by aurel cordonnier in ue5-main branch]
2022-01-25 03:06:40 -05:00
charles bloom
13b8772d9a OodleDataCompression: clean up common error cases in OodleDecode
#rb fabian.giesen
#preflight

#ROBOMERGE-AUTHOR: charles.bloom
#ROBOMERGE-SOURCE: CL 18599232 in //UE5/Release-5.0/... via CL 18599279 via CL 18599348
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)

[CL 18599390 by charles bloom in ue5-main branch]
2022-01-13 13:03:56 -05:00
arciel rekman
2a499437fe Change shader compression method to Oodle Selkie6 by default (UE-136845).
- Provides about 8% runtime memory savings (in local tests).
- Also, adds more compression types for shaders.
- Impact on the shader compilation (in SCWs) seems to be negligible in local tests.

#rb Devin.Doucette, Charles.Bloom
[REVIEW] [at]Devin.Doucette, [at]Charles.Bloom, [at]Jason.Nadro
#jira UE-136845

#ROBOMERGE-AUTHOR: arciel.rekman
#ROBOMERGE-SOURCE: CL 18502862 via CL 18503105 via CL 18503112 via CL 18505939 via CL 18505950
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18505961 by arciel rekman in ue5-release-engine-test branch]
2022-01-04 00:58:30 -05:00
devin doucette
d5cd33acde CompressedBuffer: Validate the size of the compressed buffer when validating the header
The size check was removed from the decoders during the addition of FCompressedBufferReader, but is needed to avoid asserts when attempting to decompress a buffer that is too small.

#rb Paul.Chipchase
#rnx
#preflight 61b8a79d2e65a1df0462c95b

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18457919 in //UE5/Release-5.0/... via CL 18457926
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v898-18417669)

[CL 18457953 by devin doucette in ue5-release-engine-test branch]
2021-12-14 14:58:58 -05:00
dan thompson
89f184e47f string helper functions for oodle enums.
#rb Fabien.Giesen
#preflight 61ae76ab2b3384289ad5be91

#ROBOMERGE-AUTHOR: dan.thompson
#ROBOMERGE-SOURCE: CL 18387813 in //UE5/Release-5.0/... via CL 18387899
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18387935 by dan thompson in ue5-release-engine-test branch]
2021-12-06 16:26:55 -05:00
devin doucette
bedf259d3e CompressedBuffer: Added const to params on virtual functions now that it is not an error on certain platforms
#rb none
#rnx
#preflight 61ae2b21fc3f6823e81897dc

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18383097 in //UE5/Release-5.0/... via CL 18383105
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18383123 by devin doucette in ue5-release-engine-test branch]
2021-12-06 11:18:42 -05:00
devin doucette
27c1393427 CompressedBuffer: Removed partial decompression from FCompressedBuffer now that FCompressedBufferReader is available
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.

- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.

#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18382377 by devin doucette in ue5-release-engine-test branch]
2021-12-06 10:16:05 -05:00
mark lintott
50ba1392be Made DecompressToComposite pure virtual as there was no specific definition for the FDecoder class.
#jira none
[FYI] devin.doucette

#ROBOMERGE-AUTHOR: mark.lintott
#ROBOMERGE-SOURCE: CL 18340031 in //UE5/Release-5.0/... via CL 18340044
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18340051 by mark lintott in ue5-release-engine-test branch]
2021-12-01 11:25:33 -05:00
devin doucette
c26bedd2fa CompressedBuffer: Added FCompressedBufferReader to store decoder buffers and allow streaming from seekable archives
#rb Zousar.Shaker
#rnx
#preflight 61a5acc4ee100b68faabeecc

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18325071 in //UE5/Release-5.0/... via CL 18325078
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18325081 by devin doucette in ue5-release-engine-test branch]
2021-11-30 10:34:57 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
per larsson
59ff132987 CompressedBuffer - decompress at offset
#rb Devin.Doucette, Paul.Chipchase
#rnx

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: per.larsson
#ROBOMERGE-COMMAND: _robomerge UE5-Main
#ROBOMERGE-SOURCE: CL 17797770 in //UE5/Release-5.0/... via CL 17797780
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v883-17842818)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17859847 by per larsson in ue5-release-engine-test branch]
2021-10-19 09:38:07 -04:00
devin doucette
ec3fa9680f CompositeBuffer: Renamed Flatten() to ToShared()
#rb none
#rnx
[FYI] Zousar.Shaker
#preflight 613f872b3bbb48000114081d

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17495384 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17495398 by devin doucette in ue5-release-engine-test branch]
2021-09-13 15:16:06 -04:00
charles bloom
4d7bcb6e04 OodleCompressDLL config option needs to be read in CopyBuildToStagingDirectory and passed on the command line
because UnrealPak as a program can't read project config hierarchy

#rb dan.thompson,josh.adams

#ROBOMERGE-AUTHOR: charles.bloom
#ROBOMERGE-SOURCE: CL 17464507 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17464538 by charles bloom in ue5-release-engine-test branch]
2021-09-08 17:53:21 -04:00
carlmagnus nordin
4d924fbd4a Restore Oodle allocator hooks
#rnx
#jira UE-124678
#rb charles.bloom

#ROBOMERGE-SOURCE: CL 17405626 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17405629 by carlmagnus nordin in ue5-release-engine-test branch]
2021-09-02 13:41:10 -04:00
devin doucette
3d262ca7f0 CompressedBuffer: Removed the FormatName functionality and update its last remaining callers
#rb Paul.Chipchase, Zousar.Shaker
#rnx
#preflight 612feb940380e700016fc8e5

#ROBOMERGE-SOURCE: CL 17403021 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17403035 by devin doucette in ue5-release-engine-test branch]
2021-09-02 10:30:25 -04:00
charles bloom
9a5487c8dd OodleDataCompression fix 64 bit size cast to int
#rb none

#ROBOMERGE-SOURCE: CL 17089616 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17089636 by charles bloom in ue5-release-engine-test branch]
2021-08-06 18:35:37 -04:00
devin doucette
996bacd58d CompressedBuffer: Removed calls to Ar.TotalSize() to fix loading from a text-format archive
#rb Paul.Chipchase
#rnx

#ROBOMERGE-SOURCE: CL 17082464 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17082472 by devin doucette in ue5-release-engine-test branch]
2021-08-06 09:16:42 -04:00