Commit Graph

150 Commits

Author SHA1 Message Date
arciel rekman
d61e2cf74c Fix shader code stats being output in gigabytes instead of megabytes.
- Thanks to Dan Elksnitis for noticing!

#rb Dan.Elksnitis

[CL 31147346 by arciel rekman in ue5-main branch]
2024-02-02 18:15:30 -05:00
arciel rekman
9b1b932d44 Add per-shader-type itemization for shader libraries.
- At the end of the cook, produces stats like

Unique shaders itemization (sorted by compressed size):
SF_Compute: 3202 shaders (65.59%), compressed size: 0.06 MB (19.70 KB avg per shader), uncompressed size: 0.09 MB (29.44 KB avg per shader)
SF_Pixel: 1238 shaders (25.36%), compressed size: 0.02 MB (16.65 KB avg per shader), uncompressed size: 0.03 MB (24.99 KB avg per shader)
SF_RayGen: 296 shaders (6.06%), compressed size: 0.01 MB (37.48 KB avg per shader), uncompressed size: 0.02 MB (55.79 KB avg per shader)
SF_Vertex: 115 shaders (2.36%), compressed size: 0.00 MB (6.97 KB avg per shader), uncompressed size: 0.00 MB (10.78 KB avg per shader)
SF_RayHitGroup: 18 shaders (0.37%), compressed size: 0.00 MB (5.12 KB avg per shader), uncompressed size: 0.00 MB (7.84 KB avg per shader)
SF_RayMiss: 9 shaders (0.18%), compressed size: 0.00 MB (7.63 KB avg per shader), uncompressed size: 0.00 MB (11.81 KB avg per shader)
SF_Geometry: 3 shaders (0.06%), compressed size: 0.00 MB (2.63 KB avg per shader), uncompressed size: 0.00 MB (4.78 KB avg per shader)
SF_RayCallable: 1 shaders (0.02%), compressed size: 0.00 MB (1.68 KB avg per shader), uncompressed size: 0.00 MB (2.71 KB avg per shader)

#rb dan.elksnitis
[REVIEW] [at]Dan.Elksnitis

[CL 31141673 by arciel rekman in ue5-main branch]
2024-02-02 16:40:20 -05:00
kenzo terelst
e704efdd1c Remove IsInParallelRenderingThread when creating RHI shaders because this call can happen during PSO precaching as well from worker threads - check was only around stat tracking updates
#rb Arciel.Rekman, mihnea.balta

[CL 30150119 by kenzo terelst in ue5-main branch]
2023-12-06 04:37:17 -05:00
arciel rekman
a589fb7e25 Do not attempt to open plugin shader libraries on the server.
#rb Dan.Elksnitis

[CL 29755472 by arciel rekman in ue5-main branch]
2023-11-15 14:39:51 -05:00
matt peters
026534b97f ShaderMaps in loose cooked builds: Fix not being able to run with loose files when using -zenstore. Fall back to looking for .ushaderbytecode files on disk even when using iostore, since zenstore uses iostore, but can run in non-packaged mode where it needs to read the shadermaps off of disk.
#rb Arciel.Rekman
#rnx

[CL 29716387 by matt peters in ue5-main branch]
2023-11-14 12:11:35 -05:00
arciel rekman
4e7252e2a7 SCL: downgrade a common warning to display.
#rb none
#preflgiht none

[CL 29369021 by arciel rekman in ue5-main branch]
2023-11-02 12:16:42 -04:00
arciel rekman
c944658667 Reduce shader library startup times.
- Plugins that we have cooked with the build will not get their own libraries due to the way the cooker works (all content is cooked at once).
- Do not open any plugins as chunked. Rationale: DLC plugins have chunking turned off (extra code added to validate this), whereas baked-in plugins will not have a separate shader library and will not go through that route.
- Also, do not attempt to load iterative shader library or discover libraries on disk in Test - this is a development feature, whereas Test is used for perf metrics.

#rb Dan.Elksnitis, Eric.Knapik, Justin.Marcus, Daniel.Lamb
[REVIEW] [at]Dan.Elksnitis, [at]Eric.Knapik

[CL 29366205 by arciel rekman in ue5-main branch]
2023-11-02 11:20:07 -04:00
arciel rekman
df6d9fcfac Fix not being able to run with loose files.
#rb Henry.Falconer, Paul.Chipchase
[REVIEW] [at]Henry.Falconer, [at]Paul.Chipchase, [at]PJ.Kack

[CL 29246775 by arciel rekman in ue5-main branch]
2023-10-30 16:05:20 -04:00
arciel rekman
ababa69488 Fix checkerboard materials happening in games that were cooked multi-process.
- New asset associations for already-encountered shadermaps were not being sent to the cook director from the workers,
  since the shadermap wasn't new.
- Now the workers will send all encountered shadermaps after the last send (but not their shader code), so new assets that share the same shadermap are known to the director.


#rb Matt.Peters, Dan.Elksnitis
[REVIEW] [at]Matt.Peters, [at]Dan.Elksnitis

[CL 29202154 by arciel rekman in ue5-main branch]
2023-10-27 18:47:08 -04:00
arciel rekman
668b18e206 Do not try to open the shaderlibrary from a pak file if using IoStore
#rb mickael.gilabert
#rnx

[CL 29067542 by arciel rekman in ue5-main branch]
2023-10-25 00:59:39 -04:00
matt peters
3a74f4fe69 ShaderLibraries in iterative cooking:
Suppress warnings about stable shaders cooked previously that were also cooked in the current cook but have a modified value. Instead, silently ignore the previously cooked value for the shaders.
#rnx
#rb Laura.Hermanns, Eric.Knapik, Dan.Elksnitis

[CL 27320516 by matt peters in ue5-main branch]
2023-08-23 17:07:11 -04:00
eric knapik
b186b9ca4f #jira: Iterative-Cooking
Fix an issue where intermediate shader dirs were not being cleand up at the start of a full cook.
Fixed an issue where chunked shader libraries were not loaded from a previous cook resulting in shaders missing.

[REVIEW] [at]Matt.Peters, [at]Dan.Elksnitis [at]Jason.Nadro

[CL 27259503 by eric knapik in ue5-main branch]
2023-08-21 19:52:50 -04:00
eric knapik
0581e4f72e #Jira: FORT-617867
Add a new engine function to union shader byte code made from different engine cooks
New funciton takes in multiple meta data directories and writes to a desired directory

[REVIEW] [at]Arciel.Rekman [at]Matt.Peters [at]Arturo.Colorado

[CL 26934636 by eric knapik in ue5-main branch]
2023-08-08 18:41:45 -04:00
dave belanger
147b9a0171 - Add ShaderCodeLibrary api to ignore some plugins on mount (so that they can be loaded in parallel for several plugins)
- Add an optional param to OpenLibrary to not look for shader lib chunks (because there's strong thread contention surrounding FMountedPakFileInfo::KnownPakFilesAccessLock)
- Removed reading of shared-cooked override since that feature is obsolete
#rb Arciel.Rekman
#rnx

[CL 26815876 by dave belanger in ue5-main branch]
2023-08-03 12:46:28 -04:00
Matt Peters
e88035918f ShaderLibrary: Logging: Restore LogShaderLibrary stat messages to Display because some farm jobs only capture stdout.
#rb None, trivial
#rnx

[CL 26513514 by Matt Peters in ue5-main branch]
2023-07-21 11:30:30 -04:00
Matt Peters
bd4c3442f6 ShaderLibrary: Logging: Downgrade LogShaderLibrary stat messages to Log to make it easier to scan the display output of cooking.
#rnx
#b Jason.Nadro

[CL 26483347 by Matt Peters in ue5-main branch]
2023-07-20 08:26:32 -04:00
joe pribele
f5481ce835 changed FShaderCodeLibraryPluginUnmountedCallback to close the shader library at the end of the render pipeline
#preflight 646cf60a1b241f0748ae995d
https://p4-swarm.epicgames.net/reviews/25511457
#rb justin.marcus

[CL 25592363 by joe pribele in ue5-main branch]
2023-05-23 18:16:09 -04:00
Matt Peters
a40b62a8f9 MPCook ShaderLibrary replication: add a maximum size for shader data in CopyToCompactBinaryAndClear to avoid overflowing the maximum size per message.
#jira UE-179094
#rnx
#rb Laura.Hermanns
#preflight 6452b19e6538e45f75c990e5

[CL 25324819 by Matt Peters in ue5-main branch]
2023-05-03 16:14:10 -04:00
arciel rekman
2b8a5076d4 Add owner name to the ensure reporting leaked SM resources as well.
#rb none
#preflight 642f106f6d46f6b1bfab1bf7

[CL 24961464 by arciel rekman in ue5-main branch]
2023-04-06 21:11:16 -04:00
Andriy Tylychko
7b5ca318e0 deprecated FCoreDelegates::OnPakFileMounted2 because it's thread-safe but is used concurrently. Added a thread-safe version FCoreDelegates::GetOnPakFileMounted2() and switched all use cases to the new one
#rb francis.hurteau
#preflight 641d99d214423a492d30c75a

[CL 24778298 by Andriy Tylychko in ue5-main branch]
2023-03-24 08:57:53 -04:00
Matt Peters
0027082473 RequestExit Feedback: Add Callsite argument to FPlatformMisc::RequestExit
Add the argument to many but not all callsites.
#jira FORT-578919
#rnx
#rb Devin.Doucette
#preflight 6414ca9d691c5ebc15b30410

[CL 24696053 by Matt Peters in ue5-main branch]
2023-03-17 16:46:34 -04:00
arciel rekman
1ffb07bd99 Do not produce empty output stable shader key.
#rb none
#rnx
#preflight 6409bb02a450329d6c205c7a

[CL 24574858 by arciel rekman in ue5-main branch]
2023-03-09 08:57:33 -05:00
arciel rekman
80c58ff63e Track (first) owners of FShaderMapResource_SharedCode to improve leak investigation.
#rb Jason.Nadro, Rob.Krajcarski, Jeremy.Moore, Eric.Renaud-Houde
#preflight 6402182a59017a559b7df6bd
[REVIEW] [at]Jason.Nadro, [at]Rob.Krajcarski, [at]Jeremy.Moore, [at]Eric.Renaudhoude
#rnx

[CL 24507827 by arciel rekman in ue5-main branch]
2023-03-03 17:02:58 -05:00
arciel rekman
1baa660036 Fix not having materials in DLC cooked against -fastcook builds.
- Fastcook disables chunking of the game content, so when a DLC is cooked against it and its content is split according to regular chunk IDs, the game can no longer find the shader libraries (that get chunk-specific names). To remedy this, chunking of the DLC shader libraries is disabled, especially given that currently the actual DLC pak file isn't split into chunks (if and when it becomes so, we will reconsider).
- Also improved logging to better understand how the libraries are getting opened, given the multitude of possibilities.

#rb Jason.Nadro
[REVIEW] [at]Jason.Nadro
#preflight 63f6ec877064074bd8e15684 , also 63f79cfdef1b24bf940d3624

[CL 24397829 by arciel rekman in ue5-main branch]
2023-02-24 04:00:37 -05:00
christopher waters
1f21b73b25 Ran IWYU on RHI and RenderCore, private only.
#preflight 63d358c85c69f453c1f79c37

[CL 23889591 by christopher waters in ue5-main branch]
2023-01-27 14:54:10 -05:00