Commit Graph

68 Commits

Author SHA1 Message Date
Matt Peters
da707613b5 Cooker IterativeValidate: Fix diff output from DiffWriterArchive that is being logged directly rather than being sent to the MessageCallback.
#rb Zousar.Shaker
#rnx

[CL 26356735 by Matt Peters in ue5-main branch]
2023-06-30 17:32:21 -04:00
fabian giesen
bbb9d0ff10 Change Oodle encoder scratch allocation to accomodate other codecs.
Currently sized for Mermaid VeryFast only, but we use Kraken Fast in certain places,
and bumping it up all the way to Kraken Normal is less than a 2x increase in memory
use (~1.2MB/scratch buffer to ~2.2MB in the editor; not used in game)

#rb dan.thompson

[CL 26138693 by fabian giesen in ue5-main branch]
2023-06-20 19:15:34 -04:00
charles bloom
d8257dfbb6 OodleDataCompression : make CompressParallel output not be machine dependent
#preflight none
#rb none
#rnx

[CL 25727111 by charles bloom in ue5-main branch]
2023-06-01 11:01:36 -04:00
devin doucette
cd050ee69a CompressedBuffer: Added parallel decoding for targets with editor-only data
#jira UE-183996
#preflight 645eaf63cf788a25587d9c0c
#rb Zousar.Shaker, Dan.Thompson

[CL 25474141 by devin doucette in ue5-main branch]
2023-05-15 11:07:39 -04:00
dan thompson
28970d0e6f Adding printf handler to oodle data.
#rb fabian.giesen
#preflight 64553e7a023fe5d3ad9cbf98

[CL 25363351 by dan thompson in ue5-main branch]
2023-05-05 21:38:08 -04:00
keaton stewart
304ee14f1b [Backout] - CL25309015 CIS AutoTest Cook Error
[FYI] Dan.Thompson
Original CL Desc
-----------------------------------------------------------------
Parallelizing FBlockDecoder::TryDecompressTo
#rb charles.bloom
#preflight 645164720e33f2d51d6465dc

[CL 25311278 by keaton stewart in ue5-main branch]
2023-05-02 20:08:37 -04:00
dan thompson
152b670945 Parallelizing FBlockDecoder::TryDecompressTo
#rb charles.bloom
#preflight 645164720e33f2d51d6465dc

[CL 25310355 by dan thompson in ue5-main branch]
2023-05-02 19:02:40 -04:00
charles bloom
42d21f0267 CompressedBuffer : profile scope and note that decoder is serial and should be parallelized
#preflight none
#rb none
#rnx

[CL 25140355 by charles bloom in ue5-main branch]
2023-04-20 19:29:00 -04:00
charles bloom
8057d877d4 OodleDataCompression just comments behavior nop
#rb none
#preflight none

[CL 24604196 by charles bloom in ue5-main branch]
2023-03-11 13:20:53 -05:00
charles bloom
0a198a3d1f OodleDataCompression parallel encode and decode helpers
can be swapped in with existing OodleDataCompression calls
does not add any extra header

#preflight https://horde.devtools.epicgames.com/job/640b9be070639dfc94013653
#rb fabian.giesen

[CL 24601830 by charles bloom in ue5-main branch]
2023-03-10 20:20:03 -05:00
dan thompson
4a6b8c8b00 Removal of the r.Shaders.CompressionFormat CVar, and shader group compression during UnrealPak now inherits the packaging compresison level.
Due to the fact that UnrealPak doesn't load INIs, the cvar was always its default value. This, combined with the fact that shaders don't store what codec they are compressed with means that setting this cvar to anything other than default (oodle) causes a crash in UnrealPak as DecompressShader will try to decompress with Oodle independent of the actual codec. Rather than plumb this information down through ProjectParams, I felt it prudent to just always use Oodle.

Additionally, in order to facilitate faster iteration times for development build farm jobs, the shader compression level used during UnrealPak will inherit from the overall package compression level set via Project Settings (PackageCompressionLevel_*).

This tacitly avoids patch issues as the default shader compression level happens to be the default package compression level, except now it inherits any faster levels for iteration time.

Note that we still use Mermaid as the codec as it's faster for decompressing than Kraken, which we care about for shaders.

#rb Fabian.Giesen
#rb Arciel.Rekman
#preflight 63e2f94c786751d1e0ffb290

[CL 24092953 by dan thompson in ue5-main branch]
2023-02-09 00:35:55 -05:00
martins mozeiko
6a8a5a4a75 Fix static analysis warnings in updated lz4 library code.
#preflight 63e3004c5c0ce8f11beccee5

[CL 24064469 by martins mozeiko in ue5-main branch]
2023-02-07 21:09:07 -05:00
martins mozeiko
a8c418acd4 Upgrade lz4 to 1.9.4 version.
#preflight 63e17d459740e583f439d183
#rb devin.doucette

[CL 24058819 by martins mozeiko in ue5-main branch]
2023-02-07 16:03:20 -05:00
devin doucette
461379fbf0 CompressedBuffer: Added a check to avoid a buffer overflow
#rb Zousar.Shaker
#rnx

[CL 23798852 by devin doucette in ue5-main branch]
2023-01-20 18:05:15 -05:00
charles bloom
fe229cf1e3 OodleDataCompressionUtil : take dest buff size to ensure no overrun
#jira https://jira.it.epicgames.com/browse/FORT-546392
#preflight https://horde.devtools.epicgames.com/job/63c899cfb065224750e38076
#rb none

[CL 23768764 by charles bloom in ue5-main branch]
2023-01-18 20:24:32 -05:00
arciel rekman
1facaa88b2 Add more logging to investigate shader decompression errors (UE-159777).
- Hexdump of the whole compressed shader will be dumped if small enough (which is what we're seeing mostly).

#rb Jason.Nadro
[REVIEW] [at]Jason.Nadro
#jira UE-159777
#preflight 6335b508f76de2d4d55a858c

[CL 22262833 by arciel rekman in ue5-main branch]
2022-09-29 23:45:42 -04:00
Devin Doucette
b58796b7e2 OodleDataCompression: Added memory tracking
#preflight 6335abf9a852bd922ced3cb1
#rb Charles.Bloom
#rnx

[CL 22249838 by Devin Doucette in ue5-main branch]
2022-09-29 12:44:24 -04:00
Devin Doucette
154e40f6bc OodleDataCompression: Changed scratch buffer count and size to match usage by the engine
#preflight 63346f5d1b5c12202420f466
#rb Charles.Bloom

[CL 22225493 by Devin Doucette in ue5-main branch]
2022-09-28 12:59:27 -04:00
bryan sefcik
62489fe2b3 Ran a pass of IWYU on Core.
Things to note regarding this pass:
* No includes are currently being removed from public headers.
* Any private file that has a #if is currently not being updated.

#rb Devin Doucette and Marc Audy
#preflight 62c62086756222ced497ff95
#preflight 62c626d12f2d04691814d331
#preflight 62c630107e606620fdd64e31
#preflight 62c635c6756222ced49a13cf

[CL 20979027 by bryan sefcik in ue5-main branch]
2022-07-06 21:44:18 -04:00
steve robb
fd2a0b1e20 Removal of TEXT literal concatenations, which doesn't work in UTF-8 mode.
#rb devin.doucette
#jira UE-132142
#preflight 62a8748fa76c84377748036b

#ROBOMERGE-OWNER: steve.robb
#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 20647443 via CL 20648283 via CL 20648315
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20651786 by steve robb in ue5-main branch]
2022-06-14 12:46:34 -04:00
Devin Doucette
006e85ab44 CompressedBuffer: Changed RawHash to FIoHash
- Only the first 20 bytes of the RawHash field in the header will be consumed.
- In a subsequent change, the last 12 bytes of the RawHash field will be set to zero for new compressed buffers.

- Leveraged composite buffers to avoid performing an extra copy of compressed blocks.
- Updated FBlockEncoder::Compress to always use ParallelFor.
- Added ParallelForWithPreWorkWithTaskContext.

#preflight 622ba3869b7ce6ea5d44b7ab
#rb Zousar.Shaker
#rnx

[CL 19360767 by Devin Doucette in ue5-main branch]
2022-03-11 16:21:11 -05:00
charles bloom
fabb392aca FBlockEncoder::Compress parallel enhanced
special case 1 block fast path
in multi-block case, run hash on thread too

#preflight 621e3c3ead75a7f58fc2a39e
#rb fabian.giesen

[CL 19207211 by charles bloom in ue5-main branch]
2022-03-01 16:50:24 -05:00
charles bloom
24c44d3173 OodleDataCompression VeryVerbose logging and some cleanup
#preflight 621e5072e15c51d8c5d3ded5
#rb none

[CL 19200528 by charles bloom in ue5-main branch]
2022-03-01 12:09:41 -05:00
charles bloom
43ae97e644 FBlockEncoder::Compress parallel for faster writing of large packages
#preflight 621d9e35ca28c556340c4ce4
#rb fabian.giesen

[CL 19197345 by charles bloom in ue5-main branch]
2022-03-01 09:03:27 -05:00
charles bloom
7d338bc9fe OodleDataCompression OodleScratchBufferCount use one per core in tools
still defaults to 2 for game runtimes
significant speedup in iostore compression at fast compress levels

#preflight 620be31f01253d2e19e2ad2d
#rb pj.kack

[CL 18999615 by charles bloom in ue5-main branch]
2022-02-15 13:42:27 -05:00