Encoding improvements
* New hash table implementation
--- Move away from 16-bit hashes since we need more than 64k buckets to hold 2M entries efficiently
--- Change to CityHash64, which is faster and stronger
--- Remove hardcoded max limit
* NAME_INDEX changed from contigouos int to monotonically increasing int
--- Opens up for future deduplication schemes that are better than number suffix dedup.
--- Saves some memory since we don't need to maintain a contiguous array
--- Typed up to cause compile errors when used as integer directly
* Avoid touching data repeatedly, normal path only does single hash of data
* New constructor that allows supplying string length up front
* Avoid dynamic allocations in string conversions >= 128 characters
* Avoid extra copying when splitting numbers
* More efficient IsPureAnsi check
* Only do one global lazy initialization call instead of multiple
* Switch to faster RW locks
* Switch from single lock to sharded hash map with separate locks
* Memory optimizations that reduces per entry overhead: 4 bytes slots, 2 byte headers and don't store null terminator
Improved API & documentation
* Document that IsValid() rarely makes sense
* Hide global state such as GetNames()
* Reduce amount of implementation details visible in header
* NameTypes.h size down by ~1/3 while adding documentation, stronger type safety and new APIs
Future possibilities
* Memory savings: Removing public NAME_INDEX and global FName array allows using the 32-bit FNameEntryId for arbitrary deduplication schemes. This can save both actual stored strings memory by deduplication and half the size of FName instances from 8B to 4B in shipping / test configs by removing the number part.
* Implementation can be tweaked further, for instance could persist 32-bit slot index hash inside slot to increase encoding performance in development / debug at the cost of memory.
Perf & mem results for internal project:
* Editor startup wall time: 12% speedup, 22.0s -> 19.5s
--- Warm disk and asset registry cache
--- Some gains from nametable serialization improvements
--- Some of these gains might be from orthogonal asset discovery optimizations
* Win64 Test Client memory usage with ~400k names: 30.3Mb -> 22.2Mb
#rb steve.robb, pj.kack
#jira UE-59973
[CL 5774657 by Johan Torp in Dev-Core branch]
Now stores the linker environment block in the live coding manifest, and adds it to the cache before executing the linker.
#rb none
#jira UE-72324
#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: ben.marsh
#ROBOMERGE-SOURCE: CL 5740602 in //UE4/Release-4.22/... via CL 5740765
#ROBOMERGE-BOT: CORE (Main -> Dev-Core)
[CL 5743844 by ben marsh in Dev-Core branch]
- This feature allows different chunk layout on different platforms.
- Most mapping work is done in AssetRegistryGenerator.
- Terminology of ChunkID and Pakchunk Index are now separated and they are no longer the same if chunk id to pakchunk index mapping is defined. The ChunkIDs in FAssetData is not renamed to keep consistency, but it actually is pakchunk index. FAssetData.ChunkIDs is used for ChunkInstall.
- Chunks with encryption key guid or unique assetregistry name are not allowed to be mapped. There are code to filter out those chunks and log errors while initializing mapping
- CheckChunkAssetsAreNotInChild() and ResolveChunkDependencyGraph() is skipped when making build for XboxOne and PS4, since they are not needed. If for some reason, we need chunk dependency on console builds, we need to implement per-platform dependency graph, since now we support different chunk layout on different platforms.
- Disabled a few pak file overide rules for console builds, since they should only be applied to builds with Kairos chunk layout.
#test A preflight was kicked and tested on PS4 and XboxOne. No new issue was found.
#rb Daniel.Lamb
#ROBOMERGE-SOURCE: CL 5470414 via CL 5474050
[CL 5474308 by hongyi yu in Main branch]
#rb none
#jira
#ROBOMERGE-SOURCE: CL 5437980 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5437990 by ben marsh in Main branch]
* Removed code to invalidate makefiles when adding new source files. UBT should be reliable enough to make this determination itself nowadays, and ignored -invalidatemakefilesonly argument was causing modules to be recompiled.
* Fixed incorrect config section name when determining whether to allow hot reload from IDE. Now prevents hot reload from IDE when live coding is enabled.
* Added error message when trying to add a new class with Live Coding enabled.
* Added error messages when trying to start Live Coding after a hot reload has taken place.
* Added error messages when trying to hot reload with Live Coding enabled.
#jira UE-71253
#rb none
#ROBOMERGE-SOURCE: CL 5403464 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5403485 by ben marsh in Main branch]
#rb none
#jira UE-71399
#ROBOMERGE-SOURCE: CL 5365389 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5368509 by ben marsh in Main branch]
#rb none
#jira UE-71379
#ROBOMERGE-SOURCE: CL 5362145 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5368489 by ben marsh in Main branch]
#rb none
#jira UE-71269
#ROBOMERGE-SOURCE: CL 5361379 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5368488 by ben marsh in Main branch]
#rb none
#jira UE-71270
#ROBOMERGE-SOURCE: CL 5361279 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5368485 by ben marsh in Main branch]
#rb none
#jira UE-71359
#ROBOMERGE-SOURCE: CL 5361201 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5368484 by ben marsh in Main branch]
#rb none
#jira UE-71362
#ROBOMERGE-SOURCE: CL 5360814 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5368482 by ben marsh in Main branch]
#rb none
#jira UE-71274
#ROBOMERGE-SOURCE: CL 5353250 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5364960 by ben marsh in Main branch]
* Compile dropdown now just contains a checkbox for enabling live coding.
* Compile dropdown has a link to open the editor preferences window.
* Live coding can be enabled from the editor settings window.
* Option to show the console on startup is now folded into the startup mode.
* Options in the editor settings window are greyed out unless live coding is enabled.
#rb none
#jira
#ROBOMERGE-SOURCE: CL 5352692 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5364901 by ben marsh in Main branch]
#rb none
#jira
#ROBOMERGE-SOURCE: CL 5350764 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5364832 by ben marsh in Main branch]
#rb none
#jira UE-71257
#ROBOMERGE-SOURCE: CL 5350628 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5364828 by ben marsh in Main branch]
#rb none
#jira UE-71126
#ROBOMERGE-SOURCE: CL 5350227 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5364819 by ben marsh in Main branch]
#rb none
#jira UE-71077
#ROBOMERGE-SOURCE: CL 5343160 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5343162 by ben marsh in Main branch]
#rb none
#jira
#rnx
#ROBOMERGE-SOURCE: CL 5341380 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5341387 by ben marsh in Main branch]
#rb none
#jira
#ROBOMERGE-SOURCE: CL 5340479 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5340483 by ben marsh in Main branch]
* Console window is made visible and brought to the front when a compile is triggered.
* Patched application window is brought to the front once patching is complete.
#rb none
#jira UE-71128
#ROBOMERGE-SOURCE: CL 5337696 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5338036 by ben marsh in Main branch]
#rb none
#JIRA
#ROBOMERGE-OWNER: Juan.Canada
#ROBOMERGE-AUTHOR: juan.canada
#ROBOMERGE-SOURCE: CL 5317515 in //UE4/Release-4.22/...
#ROBOMERGE-BOT: RELEASE (Release-4.22 -> Main)
[CL 5335728 by Juan Canada in Main branch]
