arciel rekman e10b7aa5d0 Shader compiler improvements. ...

Local shader compiler:
 - Not making 1 job batches (for High prio jobs) on startup
 - Randomizing pending job selection to reduce chance that multiple 60+ sec jobs get into a single batch

 (the above have the most effect on speed up, below is optional/misc)
 - Pushing the completed jobs back to manager before, and not after, pulling new ones to reduce idle time
 - Processing jobs in FIFO rather than LIFO order (change to LIFO seems like an ODSC regression? Hard to say definitively if it's a regression from the numbers, but seems odd to have the earliest jobs processed last)
 - Parallel processing of input and output files (starts sequential by default to reduce the CPU overhead, but is enabled if we ever see too long write/read)
 - More insights scopes

Distributed:
  - Not avoiding local machine for XGE on startup

Both:
 - Input file compression (disabled by default, need to better figure out when it's beneficial in a general case. Mostly for people with really slow I/O and XGE over VPN).
 - More stats, also more dense stat output. Removed an unimportant one, added stats about the batches.

#rb Jason.Nadro, Ben.Ingram
#[review] [at]Jason.Nadro, [at]Ben.Ingram
#preflight 6132ec79bf137d0001ae91ee
#jira UE-125101

#ushell-cherrypick of 17448576 by Arciel.Rekman

#ROBOMERGE-AUTHOR: arciel.rekman
#ROBOMERGE-SOURCE: CL 17448989 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17448996 by arciel rekman in ue5-release-engine-test branch]

2021-09-07 15:49:28 -04:00

..

ShaderCompileWorker.cpp

Shader compiler improvements.

2021-09-07 15:49:28 -04:00

XGEControlWorker.cpp

Do not downprioritize XGE tasks, also print more info.

2021-04-02 18:08:14 -04:00