wojciech krywult 344371f68b Json: Reduced the probability of OOM crashes when running UE.Automation tests on some memory-constrained platforms. ...

The crashes were caused by us not being able to allocate large memory buffers. Plenty of physical memory was available but virtual memory was too fragmented. This fragmentation was caused by our tests repeatedly (after each rendering test) allocating very large chunks of memory i.e. ~70 MB when sending image data back to the connected PC coordinating the tests. This 70 MB was needed because of very inefficient serialization (FStructSerializer), noticeable in particular when handling raw binary buffers (TArray<int8>).

The core issue is that the serializer does the job in two passes. First, it creates descriptions of all collection items (FStructSerializerState), which are added to a common stack, and then iterating over all the stack elements. It means that each byte from a serialized binary buffer, we would allocate one FStructSerializerState (64 bytes). That's a 64x increase!

Reworking the whole FStructSerializer would be a pretty large change, so instead I'm implementing a workaround specifically for binary buffer. It implements the existing serialization logic in a slightly different way as a WritePODArray special case. We simply directly iterate over binary arrays and skip the state generation step completelly.

#rb wojciech.krywult
#jira UE-219991, UE-219992, UE-218514, UE-219986

[CL 35432011 by wojciech krywult in ue5-main branch]

2024-08-09 15:07:15 -04:00

..

Private

…

Public

…

Serialization.Build.cs

…