Commit Graph

450 Commits

Author SHA1 Message Date
per larsson
a71cafc0b6 BulkData - removed return value from issue batch
#rb Paul.Chipchase

[CL 31171929 by per larsson in ue5-main branch]
2024-02-05 09:24:10 -05:00
steve robb
da3a1210fc Made the JsonArchive structured formatters non-copyable so that their useless copy and move consturctors don't show up in IntelliSense.
#rb devin.doucette

[CL 30712630 by steve robb in ue5-main branch]
2024-01-19 05:41:27 -05:00
steve robb
66266c6a11 Fixed up DerivedDataCache, DesktopPlatform, ApplicationCore, AssetRegistry, Core, CoreUObject, Projects, Sockets code to use EAllowShrinking instead of bools.
[CL 30676428 by steve robb in ue5-main branch]
2024-01-17 19:51:06 -05:00
mattias hornlund
919c5e02e2 Take2: Changed ArchiveCrc32 operator<<(FArchive&) template generalization for structs to only apply for direct serialization to ArchiveCrc32 and if there is no better conversion available.
Details in Jira:

#rb Steve.Robb, Peter.Engstrom

[CL 30629867 by mattias hornlund in ue5-main branch]
2024-01-16 04:10:52 -05:00
dan thompson
a6b27c768f Track the channel min max on texture source, and optionally capture this information on texture save (disabled initially).
#rb charles.bloom
#jira UE-175149

[CL 30604649 by dan thompson in ue5-main branch]
2024-01-12 16:55:27 -05:00
mattias hornlund
b5664f8754 Reverted change.
Changed ArchiveCrc32 operator<<(FArchive&)  template generalization for structs to only apply for direct serialization to ArchiveCrc32 and if there is no better conversion available.

Details in Jira:

#rb Steve.Robb, Peter.Engstrom

[CL 30591200 by mattias hornlund in ue5-main branch]
2024-01-12 08:35:20 -05:00
mattias hornlund
44e6c8e7fa Changed ArchiveCrc32 operator<<(FArchive&) template generalization for structs to only apply for direct serialization to ArchiveCrc32 and if there is no better conversion available.
Details in Jira:

#rb Steve.Robb, Peter.Engstrom

[CL 30590970 by mattias hornlund in ue5-main branch]
2024-01-12 08:14:23 -05:00
per larsson
f1db8a6191 UnrealPak - dump bulk meta data to Json or Csv
Writes a package bulk data instances with offset, size and flags to the specified destination file.

Usage:
UnrealPak.exe -ListContainerBulkData=<Path/To/PakFolder/OrContainerFile> -cryptokeys=<Path> -Out=<Filename[.json|.csv]>

#rb Paul.Chipchase

[CL 30073803 by per larsson in ue5-main branch]
2023-12-04 04:27:29 -05:00
dave jones2
138d905209 Added enums for each of the supported loader types in the engine. GetLoaderType/LexToString now replace GetLoaderName.
Currently, loaders are identified via an FName returned by GetLoaderName, which is clumsy to use and a tad error prone. Instead, use an enum to identify the loader types via GetLoaderType.

This is also a prerequisite for UE-188423.

#rb Francis.Hurteau

[CL 29817770 by dave jones2 in ue5-main branch]
2023-11-17 15:04:14 -05:00
danny couture
98cf4f14e5 [ZenLoader] Fix recursive loading behavior in zenloader by following more closely how legacy loading works
- Add a new partial loading mode where flushing the requestID will exit after serialization is done (after exportdone but before postload)
  - Automatically opt-in partial loading when an explicit load is called recursively from another load (treated as an import)
  - During recursive loading, attach postload of the recursive load to the caller's postload
  - Add loading unit tests to validate the new behavior for zenloader
  - Add a way to detect which loader is currently active to run unit tests on zenloader only

#rnx
#jira UE-190649
#rb Francis.Hurteau

[CL 28227792 by danny couture in ue5-main branch]
2023-09-26 10:26:04 -04:00
bob tellez
c00095c628 [Backout] - CL28184258
[FYI] danny.couture
Original CL Desc
-----------------------------------------------------------------
[ZenLoader] Fix recursive loading behavior in zenloader by following more closely how legacy loading works
  - Add a new partial loading mode where flushing the requestID will exit after serialization is done (after exportdone but before postload)
  - Automatically opt-in partial loading when an explicit load is called recursively from another load (treated as an import)
  - During recursive loading, attach postload of the recursive load to the caller's postload
  - Add loading unit tests to validate the new behavior for zenloader
  - Add a way to detect which loader is currently active to run unit tests on zenloader only

#rnx
#jira UE-190649
#rb Francis.Hurteau

[CL 28214800 by bob tellez in ue5-main branch]
2023-09-25 22:04:05 -04:00
danny couture
f54ea28898 [ZenLoader] Fix recursive loading behavior in zenloader by following more closely how legacy loading works
- Add a new partial loading mode where flushing the requestID will exit after serialization is done (after exportdone but before postload)
  - Automatically opt-in partial loading when an explicit load is called recursively from another load (treated as an import)
  - During recursive loading, attach postload of the recursive load to the caller's postload
  - Add loading unit tests to validate the new behavior for zenloader
  - Add a way to detect which loader is currently active to run unit tests on zenloader only

#rnx
#jira UE-190649
#rb Francis.Hurteau

[CL 28184263 by danny couture in ue5-main branch]
2023-09-25 08:06:46 -04:00
per larsson
600eac785a IAS - allow reading from file cache when HTTP streaming is disabled
#rb Paul.Chipchase
#rnx

## Background
The on demand I/O backend doesn't store complete/hole I/O chunks in the local file cache, only the parts that has been requested by the game. There are currently
two ways for querying the I/O backend(s) whether a specific chunk exists or not. The primary one used by the texture streaming system is DoesChunkExist(ChunkId). This
method is called via FBulkData::DoesExist(). In order to be able to fufill this query for partially cached I/O chunks, two additional methods has been
added to the I/O dispatcher API. Currently private, only accessible by FBulkData.
 - DoesChunkExist(ChunkId, OffsetAndLength)
 - GetSizeForChunk(ChunkId, OffsetAndLength)

The method GetSizeForChunk is only ever used by the IAsyncFileHandle wrapper for bulk data and is a tricky compromise. The file handle
represents the hole chunk/file where the bulk data instance resides, i.e. .ubulk, .uptnl etc. Calling get file size on this file handle should return
the size for the entire chunk/file. It is technically possible to read outside of the bulk data range and this would result in the read request
coming back as failed, since the resolve function in the I/O backend wouldn't be found. The file handle wrapper for bulk data is currently only used
by virtual textures and should be deprecated in the long run.

[CL 27337787 by per larsson in ue5-main branch]
2023-08-24 07:52:54 -04:00
Matt Peters
7747519417 Cooker: DiffOnly: fix mismatched alloc/malloc calls on the previousbytes read from disk. They were being allocated with FMemory::Malloc but freed with delete. This mismatch is detected by ASAN and can cause crashes.
#rb Zousar.Shaker
#rnx

[CL 27250322 by Matt Peters in ue5-main branch]
2023-08-21 15:35:02 -04:00
paul chipchase
6e8473a9dd Delete the deprecated header BulkData2.h as it has not been in use for several releases.
#rb Per.Larsson
#rnx

[CL 26841133 by paul chipchase in ue5-main branch]
2023-08-04 04:03:08 -04:00
paul chipchase
fb7da6ad42 Improve the documentation for EBulkDataFlags to help people understand how to use them.
#rb Per.Larsson
#rnx

[CL 26840243 by paul chipchase in ue5-main branch]
2023-08-04 02:56:11 -04:00
neil henning
43a5d99013 Backout CL 26567699 because its causing an Android clang ICE.
[CL 26572214 by neil henning in ue5-main branch]
2023-07-25 10:05:36 -04:00
danny couture
8d027a4e3c Implement more granular progress tracking of async package loading in zenloader
- Allows callees of LoadPackageAsync to be notified when serialization is done so they can schedule the next load right from the async loading thread without having to pay for a round-trip to the game-thread
  - Add UnattendedAsyncLoading test as a sample of how this feature can be used
  - Add thread-safety tests for LoadPackageAsync
  - Can be exercised using a cooked client build with the following command-line
        -ExecCmds="Automation RunTest System.Engine.Loading.UnattendedAsyncLoadingTest+System.Engine.Loading.ThreadSafeAsyncLoadingTest;Quit"

#jira UE-188744
#rb Francis.Hurteau

[CL 26569151 by danny couture in ue5-main branch]
2023-07-25 07:36:40 -04:00
neil henning
55d6b8bfbb Add back the gnu::alloc_align change to explain to clang the alignment of allocated pointers.
#rb Devin.Doucette

[CL 26567709 by neil henning in ue5-main branch]
2023-07-25 03:45:51 -04:00
matt peters
d0e927d0f0 Cooker: DiffWriterArchive: Fix a bug after refactor with FArchiveStackTraceIgnoreScope no longer being respected. Change DiffWriterArchive to report all messages to a message callback rather than logging them directly. Future users of DiffWriterArchive will store the messages rather than logging them.
#rb Zousar.Shaker
#rnx

[CL 26324673 by matt peters in ue5-main branch]
2023-06-29 15:40:21 -04:00
matt peters
3c2682f374 FArchiveStackTrace: Deprecate the public interface and move it into a private helper class of DiffPackageWriter.
#rb Zousar.Shaker
#rnx

[CL 26269033 by matt peters in ue5-main branch]
2023-06-27 16:13:59 -04:00
robert millar
94b709b249 Propagate set of request ids for FlushAsyncLoading into inner loops.
#rb francis.hurteau,danny.couture,pj.kack

[CL 26106812 by robert millar in ue5-main branch]
2023-06-19 17:47:11 -04:00
ben zeigler
77666d67e5 CoreUObject deprecation cleanup pass to remove functions and classes deprecated before 5.0
This change does not remove things like PackageGuid that are still used in some cases
#rb matt.peters

[CL 26096036 by ben zeigler in ue5-main branch]
2023-06-19 13:29:53 -04:00
henrik karlsson
e7a889441d [Core]
* Moved dllexport from type to methods/staticvar. This improves compile times, memory and performance in dll builds

#rb

[CL 26081848 by henrik karlsson in ue5-main branch]
2023-06-17 15:59:58 -04:00
henrik karlsson
c90d3cb093 [CoreUObject]
* Moved dll export from types to methods/staticvar

#rb none

[CL 25910241 by henrik karlsson in ue5-main branch]
2023-06-10 01:58:50 -04:00