Commit Graph

1149 Commits

Author SHA1 Message Date
andrew scheidecker
6d87d814fb Fix for unversioned serialization of arrays of optionals (or other cases where an optional property might be nested in a container).
Also fix a memory leak when doing unversioned serialization of properties with destructors and default values: "loading as zero" doesn't call the destructor on the value being overwritten, and so isn't safe to use for properties that require destruction.
#rb Devin.Doucette, Robert.Manuszewski

[CL 26706988 by andrew scheidecker in ue5-main branch]
2023-07-31 11:22:52 -04:00
andrew scheidecker
37b316b300 Fix serialization of FP specials with the UE structured archive JSON formatter:
Like JS, JSON does not directly support FP specials in its numeric literal syntax. However, JS has Infinity and NaN global variables, and JSON does not have anything like that.
To work around the omission, this change makes the JSON formatter encode FP specials as a tagged string in the form:
    "Number:+inf"
    "Number:-inf"
    "Number:-nan:0x123"
    "Number:+nan:0xabc"
This change also gets rid of the special case that tried to print floats as integers as it invoked UB to cast an out-of-range float to an int.
#rb Francis.Hurteau

[CL 26704226 by andrew scheidecker in ue5-main branch]
2023-07-31 09:14:11 -04:00
neil henning
43a5d99013 Backout CL 26567699 because its causing an Android clang ICE.
[CL 26572214 by neil henning in ue5-main branch]
2023-07-25 10:05:36 -04:00
danny couture
8d027a4e3c Implement more granular progress tracking of async package loading in zenloader
- Allows callees of LoadPackageAsync to be notified when serialization is done so they can schedule the next load right from the async loading thread without having to pay for a round-trip to the game-thread
  - Add UnattendedAsyncLoading test as a sample of how this feature can be used
  - Add thread-safety tests for LoadPackageAsync
  - Can be exercised using a cooked client build with the following command-line
        -ExecCmds="Automation RunTest System.Engine.Loading.UnattendedAsyncLoadingTest+System.Engine.Loading.ThreadSafeAsyncLoadingTest;Quit"

#jira UE-188744
#rb Francis.Hurteau

[CL 26569151 by danny couture in ue5-main branch]
2023-07-25 07:36:40 -04:00
neil henning
55d6b8bfbb Add back the gnu::alloc_align change to explain to clang the alignment of allocated pointers.
#rb Devin.Doucette

[CL 26567709 by neil henning in ue5-main branch]
2023-07-25 03:45:51 -04:00
francis hurteau
bd3a51f5a8 Truncation warning fixes for CoreUObject
#jira UE-166274, UE-127233
#rb Devin.Doucette, Paul.Chipchase

[CL 26553935 by francis hurteau in ue5-main branch]
2023-07-24 16:13:27 -04:00
paul chipchase
ef7ef6393a Fix a bug where loading a bulkdata payload from a cooked package mounted in the IoStore could end up loading more data off disk than required.
#rb Per.Larsson

### Problem
- The IoStore does not support bulkdata payloads being compressed at the package level, instead compression will be applied to the .ucas file when it is created, this means that we don't really use the SizeOnDisk value for cooked bulkdata as it should always be the same as the in memory size.
- Some code paths in the editor however might support compression in the workspace domain and so use the SizeOnDisk value, such as calling ::GetCopy.
- In this cause the size being passed to the IoDispatcher would be INDEX_NONE, which the system converts to "please load the entire .ubulk section of the package"
- The buffer returned to the user would eventually be trimmed to the correct in memory size so this bug would be hard to notice, the only downside is wasting time loading more data off disk than required.

### Fix
- AsyncLoading2 now sets the SizeOnDisk to be a valid value (the size of the payload in memory) if we are not using runtime only bulkdata (USE_RUNTIME_BULKDATA) so when GetSizeOnDisk is called, it returns the correct value and we only load the payload we want.
- An assert has been added to make sure that the payload is not compressed (in which case the size would be wrong and we could end up overrunning valid memory) even though the flag is removed when cooking and it should not be possible to get to that point.

[CL 26404304 by paul chipchase in ue5-main branch]
2023-07-17 07:52:56 -04:00
matt peters
d0e927d0f0 Cooker: DiffWriterArchive: Fix a bug after refactor with FArchiveStackTraceIgnoreScope no longer being respected. Change DiffWriterArchive to report all messages to a message callback rather than logging them directly. Future users of DiffWriterArchive will store the messages rather than logging them.
#rb Zousar.Shaker
#rnx

[CL 26324673 by matt peters in ue5-main branch]
2023-06-29 15:40:21 -04:00
paul chipchase
80b3310281 Minor optimization - Early out when trying to load an empty bulkdata payload rather than allocating then freeing a zero byte buffer.
#rb Per.Larsson
#rnx

- Note that this should not change the outcome as ::TryLoadDataIntoMemory will return false if the length of the payload is zero s the previous behaviour was to allocate, fail to load, then delete. The new code path and the old code path both end up with the bulkdata containing no allocation.

[CL 26281649 by paul chipchase in ue5-main branch]
2023-06-28 07:36:01 -04:00
johan torp
b5c3964903 Add optional support to Unversioned Property Serialization
Previously small optionals worked, by accident rather than design, but large optionals didn't

#rb andew.scheidrecker,steve.robb

[CL 26281089 by johan torp in ue5-main branch]
2023-06-28 06:47:21 -04:00
matt peters
3c2682f374 FArchiveStackTrace: Deprecate the public interface and move it into a private helper class of DiffPackageWriter.
#rb Zousar.Shaker
#rnx

[CL 26269033 by matt peters in ue5-main branch]
2023-06-27 16:13:59 -04:00
Matt Peters
011a78ece9 LLM AssetTags: Fix the missing stats for AssetTags; they were accidentally disabled by CL 23599453.
#rnx
#rb Simon.Orr
#jira UE-188961

[CL 26185861 by Matt Peters in ue5-main branch]
2023-06-22 12:47:56 -04:00
andrew scheidecker
301cbe92f9 Move FOptionalProperty to CoreUObject.
Credit goes to Maxime.Mercier for a large part of the work on this.
#rb Maxime.Mercier, Robert.Manuszewski, Steve.Robb, Tim.Tillotson

[CL 26180447 by andrew scheidecker in ue5-main branch]
2023-06-22 09:14:07 -04:00
danny couture
054c92c777 Fix zenloader state progression when mixing cooked and uncooked assets
#rnx
#rb PJ.Kack

[CL 26178675 by danny couture in ue5-main branch]
2023-06-22 06:27:04 -04:00
danny couture
a14bf3dead Fix assert in zenloader steps now that we have multiple paths where states can come from
#rnx
#rb Francis.Hurteau

[CL 26121331 by danny couture in ue5-main branch]
2023-06-20 09:50:25 -04:00
danny couture
2e48f162b0 Make LoadPackageAsync callable from any thread for zenloader
#jira UE-188746
#rb Andriy.Tylychko

[CL 26120300 by danny couture in ue5-main branch]
2023-06-20 09:01:06 -04:00
robert millar
94b709b249 Propagate set of request ids for FlushAsyncLoading into inner loops.
#rb francis.hurteau,danny.couture,pj.kack

[CL 26106812 by robert millar in ue5-main branch]
2023-06-19 17:47:11 -04:00
ben zeigler
77666d67e5 CoreUObject deprecation cleanup pass to remove functions and classes deprecated before 5.0
This change does not remove things like PackageGuid that are still used in some cases
#rb matt.peters

[CL 26096036 by ben zeigler in ue5-main branch]
2023-06-19 13:29:53 -04:00
danny couture
5298c058f8 Fix crash during recursion by splitting nodes and waiting for dependencies instead of having recursive code
- Also prevents constructed objects from being assigned to the wrong package during highly recursive phase

#rnx
#rb Francis.Hurteau, PJ.Kack

[CL 26087239 by danny couture in ue5-main branch]
2023-06-19 08:03:05 -04:00
henrik karlsson
c90d3cb093 [CoreUObject]
* Moved dll export from types to methods/staticvar

#rb none

[CL 25910241 by henrik karlsson in ue5-main branch]
2023-06-10 01:58:50 -04:00
danny couture
ee463518cd Add missing PlayInEditorLoadingScope in zenloader
#rnx
#rb Francis.Hurteau

[CL 25870506 by danny couture in ue5-main branch]
2023-06-08 10:08:21 -04:00
danny couture
5311d0b7f6 Disable postload group in editor as it cause circular dependencies by trying to flush everything instead of only what's necessary
Only preload the current package instead of the whole hierarchy recursively

#rnx
#rb PJ.Kack, Francis.Hurteau
#preflight 64788f45947ff6973c0a1e7c

[CL 25788188 by danny couture in ue5-main branch]
2023-06-05 04:32:48 -04:00
Matt Peters
9337daf49c EditorBulkData: Remove accidentally-submitted debug code.
#fyi Paul.Chipchase
#rb None, trivial
#rnx
#preflight None, trivial

[CL 25716020 by Matt Peters in ue5-main branch]
2023-05-31 17:54:34 -04:00
Wojciech Krywult
d1baf24722 AsyncIO: Fixed incorrect handling of the STAT_AsyncFileMemory stat.
I've reviewed all code using async requests and we had multiple places incorrectly handling the stat: some placed failed to decrease the counter, while others would decrease it when they shouldn't. The main cause was the fact that when we create an async request and pass nullptr as the target memory, the request allocates the memory itself and increases the STAT_AsyncFileMemory. However, when we call GetReadResults() on such requests, the responsibility is on the caller to balance the calls and decrease the stat, which is not very obvious and leads to the mentioned bugs. On top of that, when we call GetReadResults() we may not have the knowledge as to whether the returned memory was allocated by the request or not (which may affect whether we should touch the stat at all).

Resolved by adding IAsyncReadRequest::ReleaseMemoryOwnershipImpl member, which may be used by request implementations to do the proper clean up (balance the stat). This way requests take the full responsibility for their changes and users of the requests don't need to know their implementation details.

#preflight none
#rb Patrick.Laflamme
#jira UE-185064

[CL 25664733 by Wojciech Krywult in ue5-main branch]
2023-05-29 12:11:37 -04:00
kirill zorin
b6ee3a6c64 Fix UE_LOG callsites that have format string-related UB
#rb trivial
#preflight 647107780a6634dbb236a0a6

[CL 25651638 by kirill zorin in ue5-main branch]
2023-05-26 15:52:39 -04:00