#jira UE-80244
#rnx
#ROBOMERGE-SOURCE: CL 8747459 via CL 8747681 via CL 8782254
#ROBOMERGE-BOT: (v422-8689730)
[CL 8784623 by marc audy in Main branch]
#jira UE-79929
#rb robert.manuszewski
#ROBOMERGE-SOURCE: CL 8734585 via CL 8740979
#ROBOMERGE-BOT: (v422-8689730)
[CL 8741062 by johan torp in Main branch]
The change in behavior was introduced by an optimization in UE 4.23
#jira UE-79929
#rb robert.manuszewsk
(ushell-p4-cherrypick of 8633853 by Johan.Torp)
#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: johan.torp
#ROBOMERGE-SOURCE: CL 8686296 via CL 8689504
#ROBOMERGE-BOT: (v422-8689730)
[CL 8695250 by robert manuszewski in Main branch]
Most UE4 platforms use a 2-byte TCHAR, however some still use a 4-byte TCHAR. The platforms that use a 4-byte TCHAR expect their string data to be UTF-32, however there are parts of UE4 that serialize FString data as a series of UCS2CHAR, simply narrowing or widening each TCHAR in turn. This can result in invalid or corrupted UTF-32 strings (either UTF-32 strings containing UTF-16 surrogates, or UTF-32 code points that have been truncated to 2-bytes), which leads to either odd behavior or crashes.
This change updates the parts of UE4 that process FString data as a series of 2-byte values to do so on the correct UTF-16 interpretation of the data, converting to/from UTF-32 as required on platforms that use a 4-byte TCHAR. This conversion is a no-op on platforms that use a 2-byte TCHAR as the string is already assumed to be valid UTF-16 data. It should also be noted that while FString may contain UTF-16 code units on platforms using a 2-byte TCHAR, this change doesn't do anything to make FString represent a Unicode string on those platforms (ie, a string that understands and works on code points), but is rather just a bag of code units.
Two new variable-width string converters have be added to facilitate the conversion (modelled after the TCHAR<->UTF-8 converters), TUTF16ToUTF32_Convert and TUTF32ToUTF16_Convert. These are used for both TCHAR<->UTF16CHAR conversion when needed, but also for TCHAR<->wchar_t conversion on platforms that use char16_t for TCHAR along with having a 4-byte wchar_t (as defined by the new PLATFORM_WCHAR_IS_4_BYTES option).
These conversion routines are accessed either via the conversion macros (TCHAR_TO_UTF16, UTF16_TO_TCHAR, TCHAR_TO_WCHAR, and WCHAR_TO_TCHAR), or by using a conversion struct (FTCHARToUTF16, FUTF16ToTCHAR, FTCHARToWChar, and FWCharToTCHAR), which is the same pattern as the existing TCHAR<->UTF-8 conversion. Both the macros and the structs are defined as no-ops when the conversion isn't needed, but always exist so that code can be written in a portable way.
Very little code actually needed updating to use UTF-16, as the vast majority makes no assumptions about the size of TCHAR, nor how FString should be serialized. The main places were the FString archive serialization and the JSON reader/writer, along with some minor fixes to the UTF-8 conversion logic for platforms using a 4-byte TCHAR.
Tests have been added to verify that an FString representing a UTF-32 code point can be losslessly converted to/from UTF-8 and UTF-16, and serialized to/from an archive.
#jira
#rb Steve.Robb, Josh.Adams
#ROBOMERGE-SOURCE: CL 8676728 via CL 8687863
#ROBOMERGE-BOT: (v421-8677696)
[CL 8688048 by jamie dale in Main branch]
Get GIsRequestingExit now by IsEngineRequestingExit()
Set GIsRequestingExit now by RequestEngineExit(const TCHAR* Reason) or RequestEngineExit(const FString& Reason)
NOTE If Reason is 4 or less chars it will generate an ensure to force a reason to exit
The reason behind this change is right now setting GIsRequestingExit to true can cause many things to break mainly early on and with out any sort of log warning we have entered this state. We should wrap this behind a function to allow for proper handling
#rb Chris.Babcock, Michael.Trepka, Michael.Noland
#jira UE-79933
[FYI] Michael.Noland
#ROBOMERGE-SOURCE: CL 8649683 via CL 8653683
#ROBOMERGE-BOT: (v417-8656536)
[CL 8658680 by brandon schaefer in Main branch]
- reverted changes from CL 8231774
- added ShutdownAsyncThread in UObjectBaseShutdown
#jira UE-79704
#fyi Stefan.Boberg
#codereview pj.kack
[CL 8653832 by Per Larsson in Main branch]
#rb Steve.Robb, Michael.Noland
#jira
#ROBOMERGE-SOURCE: CL 8624616 via CL 8630200
#ROBOMERGE-BOT: (v409-8614070)
[CL 8631656 by marc audy in Main branch]
#rb Fred.Kimberley
[FYI] Jamie.Dale, Zak.Middleton
#rnx
#jira
#ROBOMERGE-SOURCE: CL 8624223 via CL 8630176
#ROBOMERGE-BOT: (v409-8614070)
[CL 8631650 by marc audy in Main branch]
#jira
#rb none
#rnx
#ROBOMERGE-SOURCE: CL 8376637 in //UE4/Release-4.23/...
#ROBOMERGE-BOT: RELEASE (Release-4.23 -> Main) (v406-8472469)
[CL 8498119 by ben marsh in Main branch]
#rb none
#rnx
#jira
#ROBOMERGE-SOURCE: CL 8374228 in //UE4/Release-4.23/...
#ROBOMERGE-BOT: RELEASE (Release-4.23 -> Main) (v406-8472469)
[CL 8497914 by ben marsh in Main branch]
#rb jack.porter
#ROBOMERGE-SOURCE: CL 8370381 via CL 8371517 via CL 8372527
#ROBOMERGE-BOT: (v402-8361577)
[CL 8372717 by dmitriy dyomin in Main branch]
- Up to CL8320930 from DevOnline and 8311605 Merge Down from Main
- skipped some Fortnite content/plugins/code where it tried to reintegrate files that had been moved pending investigation
#rb none
[CL 8321295 by Josh Markiewicz in Main branch]
