- Changed the recovery session activity stream to return 'I'm done' when it encounters an error to prevent the caller to ignore the error and continue asking new activities (which triggers the assert).
- Handled the error reported by the stream when searching for a recoverable session just after the Editor finished its initialization.
Fixed DisasterRecovery freezing the Editor if the new session database could not be created successfully.
- Reported the error to the client instead of silently ignoring it.
- Updated the verbose log to Warning to let the user know that a new session could not be created (and therefore disaster recovery is going to be off.
#jira UE-89128 - Re-Opening Game After Trying to Add a Cooked Material to a Level Results in a Crash
#rb Jamie.Dale
Details:
- The bug UE-89128 is possibly related to UE-89826 (at least on Mac) where the database file would fails to open successfully. Corrupted database/broken file API would fail to open the database file and will result in the assert below:
Assertion failed: (RequestFetchCount < 0 && LowestFetchedActivityId == 0) || (RequestFetchCount > 0 && LowestFetchedActivityId > 1) [File:/Users/build/Build/++UE4+Licensee/Sync/Engine/Plugins/Developer/Concert/ConcertSync/ConcertSyncClient/Source/ConcertSyncClient/Private/ConcertActivityStream.cpp] [Line: 66]
This CL fix can be tested by:
1. Launch the Editor, generate a crash with 'debug crash' command.
2. Comment 'SessionDatabase->Open(Session->GetSessionWorkingDirectory());' in Engine\Plugins\Developer\Concert\ConcertSync\ConcertSyncServer\Source\ConcertSyncServer\Private\ConcertSyncServerLiveSession.cpp to simulate SQLite failing to open the database.
3. Recompile CrashReportClientEditor(Windows) or UnrealRecoverySvc(Mac/Linux)
4. Relaunch the editor.
[CL 12393668 by Patrick Laflamme in 4.25 branch]
- Converted Concert API transferring package data in-memory only model to a streaming model to support packages bigger than 2 GB. (TNumberiLimit<int32>::max())
- Added the IConcertFileSharing interface to share large files between the client and the server. This is used as a side channel to the Concert request/response and event protocol.
- Fixed the ConcertClientPackageManager to prevent sending the package data for each the 'pre-save' when the 'live sync' is off. It only emits it once.
- Fixed UI to correctly report pre-save vs save vs auto-save for package as well as when a package is discarded.
#jira UE-85652 - Crash when importing large FBX with Morph Targets and Disaster Recovery enabled
#jira UE-78722 - Potential Memory Leak with Disaster Recovery Plugin
#rb Francis.Hurteau, Jamie.Dale
[CL 12113821 by Patrick Laflamme in 4.25 branch]
- Added the ability to copy and restore a live session, preventing the need to archive it in first place, making the server exist fast (releasing the session lock very quickly) before showing the crash UI and before the next Editor instance could starts.
Details:
This bug could manifest if various ways. An issue causing this bug was fixed in 11252374. This bug can also be observed if the crash reporting process doesn't release its lock on the crashed session quickly. Archiving a session may takes several minutes (depending on the session size) and while a session is archiving, its database is locked and cannot be restored until the archiving process complets. When the Editor reboots after a crash, it searches for a session to recover, but skip over any session that is mounted/locked assuming the session is concurrently used by a concurrent Editor process, potentially preventing it from restoring. The optimal way to work around this problem is to skip the archiving step. Instead, the live session is never archived (saving a copy), which allows the recovery service to shutdown and release the session lock very quickly ensuring that the session will be unlocked when the Editor restarts. On Editor start, it a crashed session is found and the user decides to restore it, the live session is copied into a new live session.
This changelist also affect those other jira in the following ways:
#jira UE-87899 - Disaster recovery prevents showing the crash reporting UI in a timely manner if the session is large
- This CL changes execution order to shut down the recovery service ASAP to release the lock, but the optimization above make it super fast, so the UI should always be shown in a timely manner.
#jira UE-87927 - Disaster Recovery doesn't restore a crash from a restored session
- This CL ensures the recovery service release the session lock faster than the next instance of the Editor can start.
#jira UE-87900 - Disaster Recovery stops recording transactions if the UDP transport layer restarts or auto-repair
#jira UE-88517 - Concert Log Spam - (ConcertKeepAlive) discarded
- This CL fixes an issues with endpoints timeout logic.
#jira UE-81049 - Clean up the DisasterRecovery Intermediate directory
- This CL added code to clean up the intermediate directory left over by crashed client.
#rb Francis.Hurteau
[CL 11632069 by Patrick Laflamme in 4.25 branch]
#jira UE-88070 - UnrealDisasterRecoveryService paths are too long
- Renamed UnrealDisasterRecoveryService as UnrealRecoverySvc
- Set a ConcertSyncServer ShortName as "CncrtSyncSvr" to ensure shorter build path.
The change saves 29 characters on the offending path. The path before vs the path after:
Engine\Plugins\Developer\Concert\ConcertSync\ConcertSyncServer\Intermediate\Build\Win64\UnrealDisasterRecoveryService\Development\ConcertSyncServer\UnrealDisasterRecoveryService-ConcertSyncServer.lib (Before, 199 chars)
Engine\Plugins\Developer\Concert\ConcertSync\ConcertSyncServer\Intermediate\Build\Win64\UnrealRecoverySvc\Development\CncrtSyncSvr\UnrealRecoverySvc-ConcertSyncServer.lib (After, 170 chars)
#rb Jamie.Dale
Edigrated 11281991 from Dev-VirtualProduction
[CL 11516806 by Patrick Laflamme in 4.25 branch]
- Fixed concert server to restore a session (by creating a new one) in the 'default repository' as expected rather than the creating it in the repository containing the archived session.
#rb Jamie.Dale
Edigrated 11252374 from Dev-VirtualProduction.
[CL 11515813 by Patrick Laflamme in 4.25 branch]
- Defaulted the Concert debug log level to 'Error' to avoid spamming the log when a remote endpoint disconnect.
#rb Jamie.Dale
Edigrated from 11445411.
[CL 11510926 by Patrick Laflamme in 4.25 branch]
Source Control validation now check if checked out files are actually modified instead of just being checked out, if validation mode is soft, Multi-User connection are allowed to proceed
Prompt on connection when dirty packages are found on soft validation.
Hot reload dirty packages on connection
#rb Jamie.Dale, Patrick.Laflamme
#jira UE-83300, UE-83302, UE-83303
[CL 11052914 by Francis Hurteau in Dev-VirtualProduction branch]
Fixed Concert server to unregister left over request handler on shutdown.
Fixed Concert server handler to report a repository as mounted if the requesting client already mounted it in a previous request.
Updated concert server to clean up its repository database when a repository doesn't exist anymore on disk.
#rb Francis.Hurteau
[CL 10882584 by Patrick Laflamme in Dev-VirtualProduction branch]
- Fixed verbosity of Multi-User/DisasterRecover endpoint discovery by adding a new log category "LogConcertDebug" defaulting to "Warning" except for the Multi-User server which default to "Log" level. The category verbosity can be adjusted from the command line as: -LogCmds="LogConcertDebug Verbose" or from a console command as: log LogConcertDebug Verbose.
- Prevented Disaster Recovery client from discovering (and logging) all recovery services running.
#rb Francis.Hurteau
[CL 10467794 by Patrick Laflamme in Dev-VirtualProduction branch]
#jira UE-83339
#rb Trivial.
#rnx
#ROBOMERGE-SOURCE: CL 10266187 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v591-10236483)
[CL 10266201 by patrick laflamme in Main branch]
- Fixed a disaster recovery bug preventing the Editor from recovering a session because another instance of the Editor on another project already locked all the sessions.
Problem:
On windows, the CrashReportClientEditor (hosting disaster recovery service) is started in the static initialization, before the engine is initialized, not allowing lot of command line configuration. The Editor project browser would start a first CrashReportClientEditor instance, which would load and lock all the available sessions (unless another CrashReportClientEditor was running). When the user selected a project, a new Editor and CrashReportClientEditor were launched before the first one was closed. The second instance could not access the existing sessions because they were still locked by the first instance.
Solution:
Because CrashReportClientEditor is launch before the engine is initialized, we don't have any context at the launch time. The best the was to delay the moment when the server reloads the existing sessions and enable each clients to store their sessions in different folders (repositories) mounted on demand by the server.
Implementation details:
- Implemented new RPC API to allow the client to list/create/load/drop specific repositories containing its own sessions on demand.
- Updated the Concert server to manage multiples directories where session can be stored/found (session repositories) rather than just one.
- Added a settings to allow the user to specify where the disaster recovery sessions should be stored on the disk. Now default in the current project folder.
- Added a settings to prevent the Concert server from scanning the sessions in the default location.
- Updated disaster recovery to start without any session repository and let the client decide if a new one needs to be created or an existing one be mounted to restore a previous session.
- Changed the code to let disaster recovery client manage its session history rather than letting the server rotate the old session. Defaulted the history to 0, user has no flow to visualize and pick from the history.
#rb Jamie.Dale
#ROBOMERGE-SOURCE: CL 10260823 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v591-10236483)
[CL 10260830 by patrick laflamme in Main branch]
- Cleared the concert server instance info on server shutdown.
- Shutdown the disaster recovery service when a crash is created. This enable the next server instance to grab the file lock and restore.
- Fixed archive rotation (delete oldest) that did not work when concurrent servers existed.
- Improved disaster recovery error messages.
- Fixed disaster recovery client not restoring a session that was crashed (server managed the crash), but for which the client process was still hanging around.
- Prevent showing the recovery UI if -unattended is specified on command line.
#rb Jamie.Dale
#ROBOMERGE-SOURCE: CL 9617188 in //UE4/Release-4.24/...
#ROBOMERGE-BOT: RELEASE (Release-4.24 -> Main) (v528-9595928)
[CL 9617199 by patrick laflamme in Main branch]
- Cleared the concert server instance info on server shutdown.
- Shutdown the disaster recovery service when a crash is created. This enable the next server instance to grab the file lock and restore.
- Fixed archive rotation (delete oldest) that did not work when concurrent servers existed.
- Improved disaster recovery error messages.
- Fixed disaster recovery client not restoring a session that was crashed (server managed the crash), but for which the client process was still hanging around.
- Prevent showing the recovery UI if -unattended is specified on command line.
#rb Jamie.Dale
[CL 9617188 by Patrick Laflamme in 4.24 branch]
