#rb Cory.Kolek
#ROBOMERGE-SOURCE: CL 12508127 via CL 12508133 via CL 12508137 via CL 12508141
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v673-12478461)
[CL 12508147 by federico oro in Main branch]
[FYI] [at]Rob.Cannaday
#ROBOMERGE-SOURCE: CL 11261731 via CL 11271810 via CL 11272686
#ROBOMERGE-BOT: (v647-11244347)
[CL 11272835 by michel champoux in Main branch]
[FYI] [at]Mic.Rooney, [at]Simon.Berube
#rb [at]Mic.Rooney, [at]Simon.Berube, [at]Rob.Cannaday
#ROBOMERGE-SOURCE: CL 11261262 via CL 11271792 via CL 11272683
#ROBOMERGE-BOT: (v647-11244347)
[CL 11272811 by michel champoux in Main branch]
JSON_SERIALIZE_DATETIME_MILLISECONDS_TIMESTAMP converts to and from a FDateTime to a value in milliseconds (based on the Unix time).
JSON_SERIALIZE_ENUM converts to and from an Enum using LexToString and LexFromString functions for that specific enum.
#ROBOMERGE-SOURCE: CL 11195132 via CL 11195186 via CL 11195197
#ROBOMERGE-BOT: (v640-11091645)
[CL 11195388 by fernando pereira in Main branch]
- Ptrdiff -> int32
- Float/int confusion and double/float
- size_t stuff; various changes to the algorithms to use a deduced IndexType template argument and/or decltype to use the appropriate size for indicies and counts
- Fixed GetNum(FString) incorrectly returning SIZE_T instead of int32, and GetNum(container) now returns whatever container.Num() does (so usually int32)
#jira UE-86949
#rb marc.audy, steve.robb
#ROBOMERGE-OWNER: michael.noland
#ROBOMERGE-AUTHOR: michael.noland
#ROBOMERGE-SOURCE: CL 11050799 via CL 11050828 via CL 11050837
#ROBOMERGE-BOT: (v637-11041722)
[CL 11051763 by michael noland in Main branch]
[at]jessica.hitchcock
[at]brad.johnston
[at]evan.kinney
[at]mitchell.fisher
#ROBOMERGE-OWNER: paul.moore
#ROBOMERGE-AUTHOR: paul.moore
#ROBOMERGE-SOURCE: CL 11034475 via CL 11035183 via CL 11035320 via CL 11035324
#ROBOMERGE-BOT: (v633-10983880)
[CL 11035327 by paul moore in Main branch]
#rnx
#rb none
#ROBOMERGE-OWNER: ryan.durand
#ROBOMERGE-AUTHOR: ryan.durand
#ROBOMERGE-SOURCE: CL 10869210 via CL 10869511 via CL 10869900
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870549 by ryan durand in Main branch]
Most UE4 platforms use a 2-byte TCHAR, however some still use a 4-byte TCHAR. The platforms that use a 4-byte TCHAR expect their string data to be UTF-32, however there are parts of UE4 that serialize FString data as a series of UCS2CHAR, simply narrowing or widening each TCHAR in turn. This can result in invalid or corrupted UTF-32 strings (either UTF-32 strings containing UTF-16 surrogates, or UTF-32 code points that have been truncated to 2-bytes), which leads to either odd behavior or crashes.
This change updates the parts of UE4 that process FString data as a series of 2-byte values to do so on the correct UTF-16 interpretation of the data, converting to/from UTF-32 as required on platforms that use a 4-byte TCHAR. This conversion is a no-op on platforms that use a 2-byte TCHAR as the string is already assumed to be valid UTF-16 data. It should also be noted that while FString may contain UTF-16 code units on platforms using a 2-byte TCHAR, this change doesn't do anything to make FString represent a Unicode string on those platforms (ie, a string that understands and works on code points), but is rather just a bag of code units.
Two new variable-width string converters have be added to facilitate the conversion (modelled after the TCHAR<->UTF-8 converters), TUTF16ToUTF32_Convert and TUTF32ToUTF16_Convert. These are used for both TCHAR<->UTF16CHAR conversion when needed, but also for TCHAR<->wchar_t conversion on platforms that use char16_t for TCHAR along with having a 4-byte wchar_t (as defined by the new PLATFORM_WCHAR_IS_4_BYTES option).
These conversion routines are accessed either via the conversion macros (TCHAR_TO_UTF16, UTF16_TO_TCHAR, TCHAR_TO_WCHAR, and WCHAR_TO_TCHAR), or by using a conversion struct (FTCHARToUTF16, FUTF16ToTCHAR, FTCHARToWChar, and FWCharToTCHAR), which is the same pattern as the existing TCHAR<->UTF-8 conversion. Both the macros and the structs are defined as no-ops when the conversion isn't needed, but always exist so that code can be written in a portable way.
Very little code actually needed updating to use UTF-16, as the vast majority makes no assumptions about the size of TCHAR, nor how FString should be serialized. The main places were the FString archive serialization and the JSON reader/writer, along with some minor fixes to the UTF-8 conversion logic for platforms using a 4-byte TCHAR.
Tests have been added to verify that an FString representing a UTF-32 code point can be losslessly converted to/from UTF-8 and UTF-16, and serialized to/from an archive.
#jira
#rb Steve.Robb, Josh.Adams
#ROBOMERGE-SOURCE: CL 8676728 via CL 8687863
#ROBOMERGE-BOT: (v421-8677696)
[CL 8688048 by jamie dale in Main branch]
