You've already forked UnrealEngineUWP
mirror of
https://github.com/izzy2lost/UnrealEngineUWP.git
synced 2026-03-26 18:15:20 -07:00
============================ MAJOR FEATURES & CHANGES ============================ Change 3873906 by Dan.Oconnor Revised fix for preventing delegate functions from appearing in context menu when using the compilation manager - needed to run after CallDelegateHandler changed statement type #jira UE-51726 Change 3873614 by Dan.Oconnor Prevent delegate functions from appearing in context menu when using the compilation manager and prevent crash if such a function is somehow compiled #jira UE-51726 Change 3873428 by Ben.Zeigler #jira UE-54753 Fix class pin on SpawnActorFromClass to correctly support browse. It was inherting from the wrong pin widget Copy of 3873408 Change 3873083 by Ethan.Geller [4.19]#jira UE-54865 fix seconds -> milliseconds conversion issue. #rb aaron.mcleran #lockdown cristina.riveron Change 3872714 by Dan.Oconnor Further revise fix for UE-53840, mistakenly reverted to old behavior when compiling synchronously #jira UE-53840 Change 3872648 by Ben.Zeigler #jira UE-54845 Fix crash in NextDebugTarget when there are no valid debug targets Copy of CL #3872636 Change 3872500 by Arne.Schober Back out changelist 3870283 #jira UE-54838 Change 3872412 by Mark.Satterthwaite Remove now unnecessary r.Metal.ManualVertexFetch from the device profiles. #jira UE-54853 Change 3872313 by Martin.Wilson Add UI to Live Link Client to warn live link users about background performance throttling #jira none Change 3872272 by Martin.Wilson Fix crash when clearing a skeletal mesh on a skeletal mesh component with an active Post Process Anim Instance #jira none Change 3872238 by Mark.Satterthwaite Duplicate CL #3871025 Make Manual Vertex Fetch a property of the shader platform for Metal - only the desktop platforms (METAL SM5/SM5_NoTess/MRT) will use manual vertex fetch. The mobile platforms use vertex descriptors. Prevents problems with cooked versions of games not working properly on Metal due to a mismatch between the runtime's Manual-Vertex-Fetch state versus the state used by the cooker when compiling shaders. #jira UE-54843 Change 3872087 by Yasiman.Ahsani Adding Python, libdisasm, musl, and LSS licenses. #JIRA n/a - adding licenses for new TPS Change 3872037 by Ben.Marsh BuildGraph: Add a task for compiling MSBuild projects. #jira Change 3871934 by Lina.Halper #jira: UE-54703 Change 3871595 by Michael.Trepka Fixed a problem with Mac editor not exiting with error code returned from GuardedMain function #jira UE-54830 Change 3870829 by Joe.Barnes Fix unintentional change to PhysX libs used in Debug builds. #jira ue-54817 Change 3870820 by Nick.Atamas Copying //Tasks/UE4/Dev-VR-4.19a@3870772 to Release-4.19 (//UE4/Release-4.19) #jira UE-54816 Change 3870755 by Chance.Ivey Min Android version set - should fix camera permissions issue found in #JIRA UE-54024 #rb none #fyi nick.atamas Change 3870547 by Krzysztof.Narkowicz Fixed FreezeRendering on non editor builds: ComputeAndMarkRelevanceForViewParallel was calling FrozenMatricesGuard on multiple threads, reading and writing view matrices state in parallel. #jira UE-53640 Change 3870546 by Krzysztof.Narkowicz Fixed tesselation shader tex/uniform initialization in OpenGL3/4 path #jira UE-54471 Change 3870284 by Ben.Zeigler #jira UE-54583 Fix issue where loading multiple already loaded assets via Async Load Asset node could return the wrong asset on the loaded pin. The assign variable node was happening at a slightly wrong time Copy of CL #3870279 Change 3870283 by Arne.Schober SafeRelease SRVs that might be hold by the Vertexfactories (maybe due to indirect use in GlobalResources) Note that the VFs are not owners of the data, e.g the underlying Buffers might be released before this and this reference counting should be uneccessary #jira none Change 3870098 by Ben.Marsh Fix OptimizeCode = CodeOptimization.Always causing compile errors in Linux debug builds. bUseInlining was not set correctly on the global compile environment used to build shared PCHs. Also fixed other settings not being propogated down from the target correctly. #jira UE-53855 Change 3870013 by Ben.Marsh UBT: Add an error if a user attempts to clean a target through hot-reload, rather than just failing to delete DLLs because they are locked. #jira UE-54179 Change 3870010 by Ben.Marsh UBT: Add an option to format output messages in a form that can be parsed by MSBuild. Prevents errors showing as "EXEC: Error:", and displays them correctly in the error list window. #jira Change 3869814 by Ben.Marsh UBT: Unify command line arguments to use -Name=Value syntax. -Module <Name>, -ModuleWithSuffix <Name> <Suffix>, -Plugin <Path> and -Receipt <Path> are no longer supported. Also remove the RemoteRoot option, which was not used anywhere. #jira Change 3869786 by Martin.Wilson Fix Live Link Remove Source button not working #Jira UE-54652 Change 3869660 by Martin.Wilson Fix missing message bus sources in the live link client (not repolling for new sources) #Jira UE-54712 Change 3869659 by Guillaume.Abadie Fixes SimpleComposure's BP_AdvancedCompositing keep allocating memory every frame. Credits for fixing the issue to Ron Radeztsk. #jira UE-54780 Change 3869401 by Lauren.Ridge Adding to the conditional in FindAllAncestorNamedSlotHostWidgetsForContent #jira UE-51470 Change 3869384 by Brandon.Schaefer Open the project in the explorer when NullSourceCodeAccess is done creating the project #jira UE-54630 Change 3869308 by Ben.Marsh PR #4452: Fixed FindFilesRecursively in IPlatformFilePak (Contributed by user37337) #jira UE-54568 Change 3869265 by Martin.Wilson Fix crash and subsequently found issues with unloading/reloading Maya Live Link plugin -Crash on reload due to core code getting reinitialize, made this a one time only thing (as Maya never gets rid of the module from memory) -Added manual ticking of FTicker, allows message bus objects to be cleaned up properly (cannot do this as part of normal flow as it is in engine code which we dont have) -rebuild binaries for Engine/Extras #Jira UE-54643 Change 3869206 by Benn.Gallagher Fixed crash updating clothing paint mode after tab spawners have been destroyed by the hosting application. #jira UE-54116 Change 3869064 by Benn.Gallagher Resolved skeletal mesh data changes and clothing section disable changes after collision. Re-added ability to disable sections at the mesh level and removed the ability to strip editor sections as these are required for a number of other features to correctly function. #jira UE-52557 Change 3869062 by Guillaume.Abadie Fixes "dynamic resolution is not supported on this platform" warning message being always visible. #jira UE-54655 Change 3868202 by Lauren.Ridge Fix for assert on expanding vector param in layered material #jira UE-54737 Change 3868161 by zak.parrish Replacing FaceARDebugUI with a blank UI until a new one can be created, due to a crash bug. #rb none #JIRA UE-54639 Change 3867750 by Ethan.Geller [4.19] #jira UE-54725 Fix for Envelope attack and release values not being properly set on Synth Components #rb aaron.mcleran #lockdown cristina.riveron Change 3867657 by Lauren.Ridge Adding if with editor wrapper to new function #jira cis fix Change 3867646 by Aaron.McLeran #jira UE-53867 Access violation on Switch when playing sound with specified time out of range Change 3867340 by Lauren.Ridge Fixes to Material Layers from 4.19 preview feedback -Need to pass through base attributes better/have better default nodes in layer (optional Example Layers and Blends checkbox now enables this in Experimental Settings) -Parent in function should be editable -Enforce only two layers in a blend -Mat layer should warn if it has incorrect output in the layer itself -Enforce not being able to delete outputs -Warn about creating a MAL node inside a function #jira UETOOL-1312 Change 3867317 by Aaron.McLeran #jira UE-53867 Access violation on Switch when playing sound with specified time out of range Change 3867000 by Lauren.Ridge Fix for folder favorites possibly becoming very large #jira UE-54704 Change 3866892 by Martin.Wilson Fix crash if clicking ok on message bus add source without having a source selected #jira UE-54572 Change 3866391 by Matt.Kuhlenschmidt Fix static analysis #jira UE-53379 Change 3866241 by Ryan.Vance #jira UE-54681 Fixed missing Vulkan texture GetNativeResource implementation that was lost in a bad merge. Change 3866071 by Nick.Shin UDN 412414 update HTMl5 readme file #jira none Change 3866005 by Max.Preussner Messaging: Preventing dangling references when removing message subscribers #jira UE-54680 Change 3865988 by Simon.Tourangeau Fix static analysis warnings #jira none Change 3865895 by zachary.wilson Renaming QA-PhysicalLightUnits to TM-PhysicalLightUnits in QAGame. #JIRA UE-29618 Change 3865469 by Simon.Tourangeau Support for DX11 quad buffer stereo rendering #jira UEENT-704 Change 3865461 by Chris.Babcock Add a wait for audio thread to pause audio on going to background #jira UE-54301 #ue4 #android Change 3865350 by Matt.Kuhlenschmidt Fix issue where cascade emitter UI would disapper #jira UE-53379 Change 3865336 by Arne.Schober REL - Fix UE-52356 Bone Weight #jira UE-52356 Change 3865257 by Ben.Marsh Fix editor failing to load content-only projects when compiled in DebugGame. #jira UE-54661 Change 3865238 by Simon.Tovey Pulling Olaf's fix from 3832595 over to release #tests EngineTests boots in vulkan #JIRA UE-54394 Change 3865191 by Sorin.Gradinaru UE-54317 DXT apps fail on first launch when 'For Distribution' enabled, Unsupported Texture Format #jira UE-54317 #Android #4.19 From //Dev-Mobile/3863322 Change 3865190 by Sorin.Gradinaru UE-54175 Selecting For Distribution no longer sets configuration to Shipping #UE4 #4.19 #jira UE-54175 From //Dev-Mobile/3863371 "For Distribution" enable-> reset "Build COnfiguration" to Shipping and force serialization to DefaultGame.ini (see UE-52845) Change 3865056 by Jamie.Dale Fixed culture being incorrect when added via the Localization Dashboard #jira none Change 3864826 by Max.Preussner WmfMedia: Added missing scope lock #jira UE-54365 Change 3864055 by Aaron.McLeran #jira UE-54213 Crash fix for GC'd plugin settings objects. Adding to AddReferencedObjects. Change 3863775 by Andrew.Porter MediaFrameworkTest: Removing Platform Media Source TVOS test content #jira UE-29618 Change 3863714 by Dan.Oconnor Make array, set, and map nodes no longer switch object type when a pin is disconnected. Also, you can no longer attach unrelated map types to the MakeMap node if only the value pin has been inferred #jira UE-54634 Change 3863696 by Max.Chen Sequencer: Fix crash open a new sequence while another is still active. #jira UE-54620 #jira UE-54624 Change 3863638 by Dan.Oconnor Make sure all blueprints that are skeleton compiled get a BroadcastChanged notification, also run BroadcastChanged after reinstancing, matching pre compilation manager behavior #jira UE-54193 Change 3863494 by Jamie.Dale Ensure Py_SetPythonHome is set correctly before the embedded Python interpreter is initialized #jira UE-54345 Change 3863433 by Max.Preussner MediaPlayerEditor: Removed tvOS from list of available platforms in Platform Media Source assets tvOS currently reports itself as iOS, so it cannot have its own setting. #jira UE-54559 Change 3863406 by Lauren.Ridge Fix for a crash when filtering a dropdown with no set value #jira UE-54631 Change 3863238 by Michael.Kirzinger #jira UE-52730: Fix mac voip crash Change 3862586 by Marc.Audy Clean up rendering thread heartbeat checks that got mangled in various integrations #jira Change 3862247 by Guillaume.Abadie Fixes editor lines that were using a simple element blending mode that was generating an incorrect alpha channel. #jira UE-53830 Change 3862120 by Guillaume.Abadie Fixes USceneCaptureComponent::HiddenActors always staying gray out in world editor. #jira UE-51458 Change 3861363 by Jamie.Dale Static analysis fixes #jira none Change 3861150 by Matt.Kuhlenschmidt Fix static mesh editor displaying 0 for all stats on LODs > 0 #jira UE-53776 Change 3860990 by Dan.Oconnor Clear subobjects not recreated on load, e.g. because we're running with -game and the subobject was created using CreateEditorOnlySubobject #jira UE-54551 Change 3860972 by Nick.Shin HTML5 - detect "SyntaxError: " and do a forced reload - an actual syntax error would be caught during compile time - this is usually error condition is usually seen when browser is running "old/partial" cached data and it's fairly safe to just reload the page #jira UE-54017 QAGame fails to launch properly on HTML5 Firefox 64 bit Change 3860951 by Arne.Schober Fix not releaseing SRV on render thread for FPositionVertexBuffer, FStaticMeshVertexBuffer, FColorVertexBuffer, FStaticMeshInstanceBuffer. #jira UE-54587 Change 3860950 by tim.gautier QAGame: Updating ML_Stone to include Texture Coordinate and Panner functions #jira UE-29618 Change 3860833 by Michael.Dupuis #jira UE-54181: Repopulate the foliage list when existing simulate mode Let the GC know that internal struct hold UObject ptr Change 3860762 by Jamie.Dale Ensure we invoke the correct version of Python from UBT #jira UE-54345 Change 3860676 by Simon.Tourangeau Remove DirectoryWatcher warning in output log #jira UEENT-846 Change 3860598 by Lauren.Ridge Fix for crash on opening new material instance #jira UE-54589 Change 3860338 by Michael.Lentine Integrate changes for fixing MorphTargets. #jira 54398 Change 3860215 by Ben.Marsh UAT: Fix exception consturcting target rules assembly inside UAT, now that there's an abstract TargetRules class. #jira UE-54578 Change 3860186 by Matt.Kuhlenschmidt Fix crash top 10 with the font editor shutting down and then a dpi event occuring #jira UE-54543 Change 3859854 by Graeme.Thornton PR #4124: VSCode: Changed VisualStudioCodeSourceCodeAccessor to work with non-default VSCode install paths on Linux (Contributed by christopherreed) #jira UE-51289 Change 3859848 by Graeme.Thornton Fix crypto.cs reading the wrong ini setting names for uasset encryption settings #jira UE-54566 Change 3859684 by Ben.Marsh PR #4436: Fix compile error when building BlankProgram because incorrect directory path (Contributed by windkey) #jira UE-54392 Change 3859657 by Ben.Marsh Fix FTransform being passed by value, causing alignment error when compiling for Win32. #jira Change 3859312 by nick.bullard Updating AEOverviewMan to remove reference to sub-level AEOverviewSWP which was deleted in 3859278 Still need to update menu to remove selection #jira UE-50784 Change 3859278 by Nick.Bullard Deleteing AEOverviewSWP.umap per request of Developer. Also updated TM-AnimPhys which refereneced this map as well. "This test map is a custom C++ implementation of a "procedural sound wave". This code is super janky and not worth fixing up. I'm sure there's a legit thread safety problem in there but the code is in QAGame only. We've since implemented "synth components" which do what this test does in a much safer/better way and maintaining/testing this old thing is *not* worth the time." #jira UE-50784 Change 3859124 by Dan.Oconnor Fix long standing crash when duplicating a blueprint that is missing its parent class. Entries in CrashReport go back to 4.16 #jira UE-54468 Change 3859086 by Ryan.Vance #jira UE-54470 We need to set the viewport in both cases. Change 3859006 by Dan.Oconnor Revise fix for archetype lookup when reinstancing. During compilation we do not want to force use of the up to date class #jira UE-54541 Change 3858990 by mason.seay Cleaned up blueprints to remove compile errors #jira UE-29618 Change 3858945 by Aaron.McLeran #jira UE-54265 PR #4428: Fixing Envelope Bug in AudioMixer (Contributed by Chrispykins) Change 3858719 by Aaron.McLeran #jira UE-54552 Fix for sample buffer reader Change 3858647 by Ethan.Geller PR #4439: Removes ambiX -> FuMa conversion (Contributed by mgorzel) #jira UE-54407 Change 3858364 by Michael.Dupuis #jira UE-52049: There was a case where adding and removing multiple time would lead to reordering the instances and this would cause the regeneration of the random stream for all the reorded instances. Change 3858268 by Dan.Oconnor Prevent postload logic running on REINST and SKEL classes #jira UE-54531 Change 3858205 by Mitchell.Wilson Removed particle looping from some emitters to resolve anim notify warnings. #jira UE-53823 Change 3858148 by Lauren.Ridge Changes based on Material Layer Feedback from previews - (Temporary) Disabled being able to create a layer or blend in the asset dropdown - Sections of the stack that have been disabled now inactivate that part of the UI - Create Function Instance now indicates if you are making a layer or a blend - Parent dropdown has been removed from layers and blends. Where relevant, a filter button has been added instead. #jira UETOOL-1328 Change 3857933 by Michael.Dupuis #jira UE-45854: Properly unregister callback when replacing foliage type with another one Change 3857898 by Michael.Dupuis #jira UE-54396: Remove the Ensure as it could be possible that the Landscape Info is invalid during an undo operation Change 3857878 by Max.Chen Sequencer: Assign the sequence id after the template is compiled. Copy from Dev-Sequencer #jira UE-54462 Change 3857808 by Michael.Dupuis #jira UE-54421: Prevent edition during Simulate when clicking on actor Change 3857786 by Rolando.Caloca UE4.19 - Fix recompute tangents and skin cache for OpenGL #jira UE-42108 PR #3271 Change 3857549 by Lina.Halper another jittering issue due to revision number change clear the motion vector after compile #jira: UE-53930 Change 3857439 by Lina.Halper Clear motion vector when end of sequencer when in editor world #jira:UE-54057 Change 3857384 by Graeme.Thornton Restore fix for visual studio source code accessor not correctly determining that a content project has no solution and opening a fresh instance of visual studio #jira UE-50020 Change 3856596 by Chris.Babcock Fix ResonanceAudioApi Android library architecture filtering #jira UE-54478 #ue4 #android Change 3856449 by Michael.Dupuis #jira UE-35097: Various cosmetic changes that were done in phase 2 that help improve clarity of the design. Change 3856415 by Dan.Oconnor Fix regression when loading instances that have had their class deleted. Issue detected by static analysis #jira UE-54467 Change 3856332 by Ben.Marsh Resaving TP_HandheldARBP assets to fix version again. #jira Change 3856319 by Ben.Marsh Back out changelist 3855588 Causes build failure in UFE because it introduces a dependency on the Engine module. UFE compiles with WITH_ENGINE = 0. #jira UE-54472 Change 3856292 by Ben.Marsh Fix app-local dependencies not being included in binary builds, and only working for code projects in source builds. #jira UE-54448 Change 3856190 by Martin.Wilson Fix bone access mismatch between raw mesh bones and final bones (that include virtual bones too) #jira UE-54266 Change 3856169 by Ben.Marsh Tag XGEControlWorker.exe as a build product, so it's included in the binary distro. #jira UE-54283 Change 3856123 by Chris.Babcock Fix missing ARCore file #jira UE-54453 #ue4 #android Change 3856005 by Richard.Wallis Clone of Dev-Rendering CL 3855993 - turn off FShaderCache. #jira UE-52928 Change 3855961 by Jian.Ru Copy 3855047 - fix DFAO Nan problem #jira UE-54403 Change 3855811 by Martin.Wilson Add build process for Maya Live Link plugin (standalone, manually triggered) + add built binaries to Engine/Extras (Maya 2016, Maya 2017, Maya 2018) #jira none Change 3855758 by Cosmin.Sulea UE-53569 - tvOS does not package or launch-on #jira UE-53569 Change 3855727 by Ben.Marsh Resaving assets with a versioned build in the editor, to fix warnings building DDC. #jira Change 3855626 by Adrian.Siminciuc https://jira.it.epicgames.net/browse/UE-50979 (MP4 doesn't resume playback when iOS 11 device wakes from sleep) #jira UE-50979 Change 3855588 by Graeme.Thornton Fix visual studio solution path being incorrect for content projects #jira UE-50020 Change 3855283 by Ben.Marsh Fix race condition where stdout/stderr write handles could be inherited by multiple processes, resulting in them not being closed (and the process exit being detected) until all processes that inherited them had been closed. Improves performance of ParallelExecutor. #jira Change 3855009 by Chance.Ivey Resaving with version number. #JIRA-54330 #rb none Change 3854943 by Dan.Oconnor Fix archetype lookup when searching hierarchy that has been partially reinstanced #jira UE-53840 Change 3854882 by Ryan.Vance #jira UE-54438 Removing vr related references to screen percentage. Removing previously removed gvr screen percentage code that came back in an integration from google. Change 3854806 by Mike.Beach Mirroring part of CL 3802176 to fix a crash that can occur when users try to use the default 'DisplayModel' on MotionController components. Also provided users a better error message to explain why a model might not be showing up. #jira UE-54214 Change 3854680 by Chance.Ivey Saving assets with version number#JIRA UE-54330 #rb none Change 3854652 by Uriel.Doyon Added a tooltip to the EV100 slider in the exposure menu. Using game settings now disables the slider. #jira UE-53945 Change 3854605 by Dan.Oconnor Make sure we don't create objects outered to a placeholder object, also make sure that archetypes that are reinstanced on load are relinked in to the linker table so that they are postload'd (and the old instance isn't) #jira UE-53954 Change 3854274 by Brandon.Schaefer Changes in CL 3842286 changed the function glslang::GlslangToSpv and requires a rebuild on Linux #jira UE-54302 #codeview Arciel.Rekman Change 3854255 by Phillip.Kavan Fix a scoping issue for local instanced subobject references in nativized Blueprint C++ code. Also, don't emit redundant assignment statements for instanced subobject reference properties. - Mirrored from //UE4/Dev-Framework (3853349). #jira UE-53960 Change 3854177 by Ethan.Geller #jira UE-54415 set EnabledByDefault to false for Resonance Audio Change 3854123 by Ethan.Geller #jira UE-54410 set AudioComponentID Change 3853775 by Lauren.Ridge Minor cleanup #jira UE-54054 Change 3853772 by Lauren.Ridge Don't create widgets when just testing if the selected widget is a replacement candidate #jira UE-54054 Change 3853715 by Rolando.Caloca UE4.19 - Fix for OpenGL overwriting texture units #jira UE-54401 Change 3853655 by Ben.Marsh Add a retry loop on creating the first directory before copying files. Attempt to work around problems copying to shared folders. #jira Change 3853535 by Ben.Marsh Expose the engine compatible changelist to Perforce. If EnginePatchVersion > 0, this will be the changelist of the original .0 release. #jira Change 3852583 by Nick.Atamas Resaved assets so they don't produce DDC warnings. #jira none Change3852552by Uriel.Doyon Fixed Pre-Exposure shader compilation and Temporal AA issue. #jira UE-54276 Change 3852354 by Nick.Atamas Hopefully fixes the static analysis warning from jira issue. #jira UE-54332 Change 3852281 by Nick.Atamas Merging CL 3851690 from //Tasks/UE4/Dev-VR-4.19a/... to //UE4/Telease-4.19/... #jira UE-54331 Change 3852274 by Simon.Tourangeau Back out changelist 3851041 until Win7 issue is resolved. #jira UE-54354 Change3852208by Jamie.Dale Merging CL# 3821754 from //UE4/Dev-Enterprise Class property conversion now goes through NativizeClass/PythonizeClass This allows it to coerce from Python wrapped object types #jira none Change 3852202 by Jamie.Dale More explicit handling of EngineDir for Python SDK #jira UE-54345 Change 3851982 by Brandon.Schaefer Workaround using a hardcoded path #jira UE-54136 Change 3851748 by Michael.Dupuis #jira UE-53904: Put the code back to what it was before, as we really only want to perform this code if bIsLayerThumbnail is true, it was changed to fix another issue complaining about a missing shader, that end up being added to the existing functions Change 3851545 by Marc.Audy Remove debugging code that slipped in #jira none Change 3851461 by Ben.Marsh Fix #includes with backslashes from crashing UBT. #jira UE-53996 Change 3851391 by Jamie.Dale Updated Python to prefer our TPS SDK #jira UE-54345 Change 3851372 by Jamie.Dale Added bat file to copy the Python SDK into TPS #jira UE-54345 Change 3851218 by Ben.Marsh Add missing template to installed engine build. #jira UE-54339 Change 3851117 by andrew.porter QAGame: Removing duplicate map #jira UE-29618 Change 3851041 by Simon.Tourangeau Support for DX11 quad buffer stereo rendering #jira UEENT-704 Change 3850548 by Ben.Marsh Add TP_HandheldARBP to installed engine build. #jira Change 3850424 by Ben.Zeigler Fix reported memory for asset registry to be correct, extracted from a larger change #jira none Change 3850324 by Ryan.Vance #jira UEVR-1050 Hook up the Rift dynamic res to the new IDynamicResolutionState framework Change 3849819 by JeanLuc.Corenthin Unhide Datasmith plugins in "regular" projects to allow easy project conversion #jira UEENT-795 Change 3849302 by Martin.Wilson Live Link is no longer experimental in 4.19 (moved from experimental folder to Animation) #jira none Change3849238by Max.Chen Sequencer: Remove bKeepStaleTracks so that stale tracks are now always purged. #jira UE-54248 Change 3849211 by Michael.Dupuis #jira UE-54181: Prevent foliage edition during PIE or simulate mode #coderevire jack.porter Change 3849123 by Benn.Gallagher Fixed crash in clothing actor creation when the clothing simulation mesh has no simulated vertices #jira UE-53741 Change 3849120 by Benn.Gallagher Fixed crash adding empty materials to destructible meshes in the destructible mesh editor #jira UE-53938 Change 3849047 by Jurre.deBaare Move material baking out of experimental #fix follow-up also remove the entry from experimental settings #jira UE-52685 Change 3848808 by Michael.Dupuis #jira UE-35097: Remove IncludeTessellationInShadowLOD & RestrictTessellationToShadowCascade from 4.19 as there is currently a bug regarding this in the dynamic rendering code path and would be too risky to fix for now. Fixed dynamic shadow code path issue vs static code path Change 3848659 by Lina.Halper Fix issue with animation resetting in the sequencer #jira: UE-54047 Change 3848635 by Rolando.Caloca UE4.19 - Fix static analysis #jira UE-50449 Change3848515by Sorin.Gradinaru Unshelved from pending changelist '3843541': WebBrowser Android crash on 4.4.3 #jira UE-53247 #Android #4.19 Crash on Android 4.x.x caused by a call to a Api Level 21 method Change 3848514 by Jurre.deBaare Moving over: CL 3832173 "Failed to import Alembic files #jira UE-53941 #fix Change Alembic thirdparty library setup - Removed old unused library files - Updated batch files for building HDF5 ILMBase and Alembic libraries to use new AlembicDeploy path - HDF5 is now build as a Dynamic library (which also enableds multi-threading) - Added DLLs for HDF5 to build.cs file CL 3838053 "Adding missing hdf5 dynamic libraries Change 3848245 by Ben.Marsh Downgrade warning about not using XGE shader compilation to a log message. Build machines do not have XGE. #jira UE-54237 Change 3847300 by Phillip.Kavan Fix shadowed local variable. #jira UE-54141 Change 3846922 by Max.Preussner ImgMedia: Fixed image media player never finished initialization if loading failed Copied from Dev-Sequencer CL# 3846902 #jira UE-54247 Change 3846831 by Arciel.Rekman Linux: only use lld for x86_64 (UE-54144). - lld support for other architectures seems to not ready for prime time. #jira UE-54144 Change 3846771 by Lauren.Ridge Material window now uses the background color set in preview scene. #jira UE-52215 Change 3846705 by Ben.Marsh Fix batch file paths not being quoted correctly when run through XGE. #jira Change 3846550 by Lauren.Ridge Toggling Show Background now updates the background as well #jira UE-52250 Change 3846417 by Matt.Kuhlenschmidt Fix crash resizing shootergame window #jira UE-53137 Change 3846295 by Rolando.Caloca UE4.19 - New Vulkan descriptor pooling mechanism (enabled on non-android) #jira UE-50449 Change 3846273 by tim.gautier QAGame: Updating Material Layer test assets to include more Params - Added temp assets, quicker repros for bugs #jira UE-54176, UE-54165 Change 3846255 by Lauren.Ridge Parameter tab is the primary tab for material instances #jira UE-54092 Change 3846086 by Chris.Babcock Add missing SecureHash.h include #jira UE-54026 #PR #4417 #ue4 #android Change 3846049 by Martin.Wilson Fix Set Root Motion Enabled Anim Data Modifier node (previously didn't set the enabled flag) #jira UE-54220 Change 3846033 by Martin.Wilson Fix root motion being repeatedly applied ( Clear() call only clears bHasRootMotion flag, not the transform itself ) #Jira UE-54219 Change 3845991 by andrew.porter QAGame: Updating bindings on activechannels sequencer test content #jira UE-29618 Change 3845933 by Lauren.Ridge Check for original material being valid #jira UE-54166 Change 3845920 by Martin.Wilson Optimized redundant key removal #jira UE-51303 Change 3845812 by Matt.Kuhlenschmidt Fix not being able to change BSP brush shape #jira UE-53738 Change 3845790 by Martin.Wilson Fix for assert failure when accumulating root motion in debug. #jira UE-53955 Change 3845730 by JeanLuc.Corenthin Fix build breakage warning: resaved disc.uasset with 4.19.0 preview #1 failure: set the correct default mesh for AreaLightStruct.uasset #jira none Change 3845693 by Lina.Halper Fix issue with previewing pose asset with curve data #jira: UE-53967 Change 3845533 by Andrew.Rodham Sequencer: Fixed sub sequences potentially being loaded during AddReferencedObjects #jira UE-54173 Change 3845472 by Thomas.Sarkanen Prevented debug object selection dropdown from displaying objects with pending kill outers #jira UE-54045 - Animation Blueprint Editor Crashes on Compile if the Debug Instance Selection is Other Than No Object or Preview Instance Change 3845401 by Yannick.Lange Reverting thumbnail capture from viewport. #jira UE-53775 #jira UE-53701 Change 3844693 by JeanMichel.Dignard Changed IES texture brightness to be the max candela value and set the texture multiplier to be 1. This fixes the IES lights intensity being too high. #jira UEENT-632 Change 3844689 by JeanLuc.Corenthin Update Datasmith content assets to latest Copying fix from Dev-Enterprise by Jean-Luc Corenthin CL 3809803 Updated assets with correct release version Cleanup some paths on static meshes and texture #jira UEENT-759 #jira UEENT-657 Change 3844571 by Martin.Wilson Fix motion controller motion source pin still showing when pin is connected to something #Jira UE-53236 Change 3844564 by Martin.Wilson Due to previously fixed bug some anim blueprint nodes could have duplicated guids, this fixes them #Jira UE-54174 Change 3844545 by Jamie.Dale No longer attempt to parse group separators for numeric inputs This avoids some ambiguity when parsing numbers for languages such as German #jira UE-54170 Change 3844221 by Nick.Shin HTML5 - filter out "windows/super" keys - these are not used in UE4 - but, keycode are not the expected "91 or 92" values, SDL keys are "227 & 231" instead... #jira UE-54056 HTML5 crashes inside browser upon pressing windows key Change 3843937 by JeanMichel.Dignard Fixed a crash when right clicking on a static mesh for which its AssetImportData class is currently unavailable (ie: in an unloaded plugin). #jira UEENT-764 Change 3843929 by Peter.Sauerbrei pull over fix for bad directory when copying launch images #jira UE-53177 Change 3843658 by Thomas.Sarkanen Text is red again in anim viewports #jira UE-53224 - Colouring removed from "Animation is being edited" warning messages Change 3843657 by Thomas.Sarkanen Enable picking via Enter for details panel asset pickers The previous (4.18) behavior was to only allow selection of the previous/next item in the list with arrow keys. A fix (CL 3783114) for pickers with many assets broke this. This change now allows for selection of any item by navigating with arrow keys and pressing Enter, as suggested in the Jira. #jira UE-53440 - Unable to select assets within modal Asset Selection dropdowns after navigating with keyboard entry Change 3843120 by Dan.Oconnor Avoid fixing component template games outside the editor, this logic does not work for games that have been nativized #jira UE-54009 Change 3842841 by Ben.Zeigler #jira UE-50020 Switch visual studio module back to using absolute paths so go to definition works, broken in CL #3796157 Change 3842582 by Lauren.Ridge Guards against the widget passed to scrollwidgetintoview being null #jira UE-54037 Change 3842575 by Max.Chen Sequence Recorder: Stop recording if the preview window is destroyed. #jira UE-49778 Change 3842551 by Michael.Dupuis #jira UE-35097: Minor bug fix, documentation, etc. to the landscape optim that was done in the phase 2 that have 0 risk. Change 3842371 by Max.Preussner Media: Merged 4.19 fixes from Dev-Sequencer CL 3807293 WmfMedia: Fixed YUY2 video format strides CL 3827988 MediaPlayerAssets: Fixed Media Texture is not linked to Media Player when created together CL 3805414 MediaAssets: Setting valid GUID when initializing media texture resource CL 3804183 MediaAssets: Added missing lock in media sound component CL 3831580 MfMedia: Media open events generated in same order as on other platforms CL 3807193 WmfMedia: Fixed incorrect buffer stride for RGB32 video tracks #jira UE-53532 #jira UE-53328 Change 3842356 by Max.Preussner ImgMedia: Fixed ensure when cooking project that uses ImgMediaSource #jira UE-51631 Change 3842335 by Aaron.McLeran #jira UE-54087 PR #4419: Fixes a crash due to nullptr dereference (Contributed by mgorzel) Change 3842286 by Rolando.Caloca UE4.19 - Fix for static analysis - Glslang 1.0.65.1 #jira UE-54128 Change 3842222 by andrew.porter QAGame: Updating framerate of EXR_Sequence #jira UE-29618 Change 3842211 by Ben.Marsh Fix determination of Windows version string. The manifest for UE4 applications declares compatibility with Windows 10 nowadays, so we get accurate version numbers returned from GetOSVersionInfo(). #jira UE-54035 Change 3842163 by Cosmin.Sulea UE-53303 - We do not check for remote connection before attempting remote shader compile, causing crashes when misconfigured #jira UE-53303 Change 3841770 by Max.Chen Sequencer: Fix to allow keying of an arbitrarily deep property path. #jira UE-54095 Change 3841758 by Max.Chen Sequencer: Fix unbound possessable components when pasting spawnables. #jira UE-54104 Change 3841415 by Lauren.Ridge Renaming a material layer or material layer blend will no longer cause the asset to appear removed from the stack #jira UE-53942 Change 3841327 by Arciel.Rekman Linux: fix Debug build (UE-53855) - A workaround. UBT should be using proper PCH files instead. #jira UE-53855 Change 3840975 by Rolando.Caloca UE4.19 - Updated VulkanRHI - Fixes for GPU frame time - Fixes for CPU performance #jira UE-50449 Change 3840838 by Michael.Dupuis #jira UE-53944: Make sure the LOD generated is in the valid range to prevent the crash Change 3840693 by Ben.Zeigler #jira UE-53923 Fix regression in 4.19 where PrintScriptCallstack is not always safe to call from the immediate window. I narrowed down the issue to some confusing optimized code so I turned off optimization Copy of CL #3840692 Change 3840680 by Aaron.McLeran Bringing fixes from Dev-AnimPhys to 4.19. #jira UE-53903 crash on load with oculus audio and old audio engine #jira UE-52786, UE-53910 Fix for broken spatialization on xaudio2, old audio engine. Change 3840663 by Rolando.Caloca UE4.19 - Fix for layout ensure on HMD projects on Vulkan #jira UE-50265 Change 3840577 by Rolando.Caloca UE4.19 - Fix for CPUs with more than 16 cores #jira UE-53434 Change 3840551 by andrew.porter QAGame: Setting Allow Bindings from Asset to false #jira UE-29618 Change 3840491 by Ben.Zeigler #jira UE-31662 Fix regression with renaming parent inherited function. It was not correctly searching the parent's skeleton class during the child's recompile so it was erroneously detecting the parent function as missing Copy of CL #3840489 Change 3840297 by Max.Chen Sequencer: Fix copy/paste crash for lights #jira UE-54084 Change 3840284 by Michael.Dupuis #jira UE-53053: Was having a mismatch between the remove reorder and the actual remove Change 3840215 by Sorin.Gradinaru Unshelved from pending changelist '3812852': UE-53550 Level doesn't render on Lenovo 939 UE-53592 Assertion right after rendering scene on Lenovo S939 #jira UE-53550 #jira UE-53592 #4.19 #Android UE-53550 Removed force disabling texture2DLodEXT and textureCubeLodEXT on Mali-400 devices The problem was that the shader compiler complains about code lines before the #extension directives. Placeholder // end extensions in the original shader code - to be replaced with round() functions UE-53592 Always use a new task for devices that have GIsThreadedRendering=false, even when the call is from the rendering thread Change 3840048 by Michael.Dupuis #jira UE-52975: Was always performing the equivalent of an Add, so now we use the Transform during the duplicate Change 3840005 by Richard.Wallis Clone of CL 3835252 Fix for shader library not working with iterative cooking, also fixes native Metal library not working with iterative cook. This works by saving a "backup" of the shader library file - this is reloaded only during iterative cook and adds back in shaders that are missing from the current cook. Fix for extracting/searching .metal files in different directories so debug tgz archiving wasn't working correctly and also support iterative cook. Includes first pass code review suggestions by Mark Sat and Dmitriy Dyomin. #jira UE-53815 Change 3839968 by Michael.Dupuis #jira UE-52289: When OnRegister is called on the component make sure our PerInstanceRenderData is up to date Prevent a possible crash if ClearInstanceSelection was called on a component with no PerInstanceRenderData existing ##codereview jack.porter Change 3839924 by Richard.Wallis Clone of CL 3838093 Fix for rewind / seek bugs in AvfMediaPlayer. - Don't initialise and send audio buffers that have a duration of Zero from the audio tap. This chokes the audio sink and adds overhead we don't need. - Don't faff around with current play rates during loop operation - normal seek doesn't do this so loop seek shouldn't either. - SetRate() should not required to be passed to media tracks - should be enough to do this on audio track select only. #jira UE-54019, UE-53027 Change 3839321 by andrew.porter QAGame: Adding missing bookmark to QA-Sequencer_Blending #jira UE-29618 Change 3839286 by Marcus.Wassmer Duplicate CL: 3823296 #jira UE-52784 Change 3839229 by Brandon.Schaefer Fix audio clean up crash when exiting PIE #jira UE-54050 #review-3839109 @Arciel.Rekman, @Aaron.McLeran Change 3839223 by andrew.porter QAGame: Rebinding pointlight actor to sequences #jira UE-29618 Change 3839098 by andrew.porter QAGame: Fixing missing sequencer blending test content #jira UE-29618 Change 3838919 by Mike.Erwin glTF: binary format's BIN chunk is not necessarily right after the JSON chunk. Discussed this with glTF spec authors. See https://github.com/KhronosGroup/glTF/issues/1177 Rearranged how we read "chunks" from the file since they are not fixed in number or order, besides JSON always being first. #jira UE-50695 Change 3838909 by Mike.Erwin glTF: base64 decoding of data buffers and images Data can be encoded inside the glTF JSON as a Base64 data URI. This CL addresses a known limitation of initial commit CL 3793018. I added this after because base64 is expected to be uncommon in the wild. Authoring software will typically use external BIN files (.gltf) or a BIN chunk (.glb) to store data. #jira UE-50695 Change 3838812 by Uriel.Doyon Integrated CL 3838576, 3838581 and 3831760 from DevRendering, fixing issues with texture streaming. #jira none Change 3838773 by Lauren.Ridge Fixing material layer filters #jira UE-54064 Change 3838748 by Michael.Trepka Fixed EngineTest runtime warning caused by CL 3838626 #jira UE-53893 Change 3838730 by Max.Chen Sequencer: Add notification when the blend type is changed. #jira UE-54046 Change 3838626 by Michael.Trepka Changed FMacMenu to store MultiBox and MenuEntryBox as weak pointers instead of shared pointers. This way we avoid a situation where FMacMenu would try to release them on the main Cocoa thread or where we'd try to execute a menu action for menu items that Slate considers released. #jira UE-53893 Change 3838392 by Arciel.Rekman Fix assert on a policy removal (UE-54042). - Applying Gil's safe fix which just sweeps the problem under the rug. #jira UE-54042 Change 3838162 by Arciel.Rekman Linux: fix crash due to lambda lifetime issues (UE-54040). - The lambda goes out of scope in FBufferVisualizationMenuCommands::CreateVisualizationCommands, crashing the editor if compiled with a recent clang (5.0+). #jira UE-54040 (Edigrating 3819174 to Release-4.19) Change 3838156 by Rolando.Caloca UE4.19 - Support for Vulkan devices that have no cached memory type #jira UE-54039 Change 3838096 by Brandon.Schaefer Set sound to unfocused volume multiplier if not focused #jira UE-51327 #review-3835736 arciel.rekman Change 3838087 by Brandon.Schaefer Fix arm server building. A fix from CL 3617084 remove inheriting from IHeadMountedDisplayVulkanExtensions. Remove overrides #jira UE-53901 #review-3838088 arciel.rekman Change 3837072 by Phillip.Kavan Fix for a build failure that could occur with Blueprint nativization enabled and EDL disabled. This was a regression introduced in 4.18. - Mirrored from //UE4/Dev-Framework (3836768). #jira UE-53908 Change 3837071 by Phillip.Kavan Emit proper syntax for set/map fields containing converted assets to generated C++ Blueprint class headers when Blueprint nativization is enabled. - Mirrored from //UE4/Dev-Framework (3835944, 3835965). #jira UE-42614 Change 3837070 by Phillip.Kavan #4202 - Blueprint nativization bug fixes (PR). - Mirrored from //UE4/Dev-Framework (3830562, 3832292). #jira UE-52188 Change 3836507 by Ryan.Vance #jira UE-53992 Due to hijacking the depth target directly from the scene context, we can't support depth compositing if it's being scaled by screen percentage since it wont match our color render target dimensions. Change 3836390 by Dan.Oconnor Fix failure to resolve archetype when using the compilation manager #jira UE-53840 Change 3836251 by Ryan.Vance #jira UE-53992 Change 3835852 by Mark.Satterthwaite Fix tessellation shaders in Metal with Manual Vertex Fetch enabled: - The control points idnex buffer shouldn't collide with anything else. - We can't use the optimisation of loading texture width & height from the buffer meta-table in tessellation shaders as the combined stages don't guarantee not to clobber unused buffer slots and screw it up when we use linear textures. #jira UE-53851 Change 3835802 by JeanMichel.Dignard UBT changes for Enterprise deployment - Allow building enterprise against an installed engine version - Added enterprise to the cleanup process if its not installed #jira UEENT-748 Change 3835625 by Bogdan.Vasilache UE-50257 --> (Skeletal meshes silently fail to render if they have more than 75 bones) --> changed logged error with a warning #jira UE-50257 Change 3833649 by Mike.Beach Enabling debug layer when multiview is enabled. #jira UE-49954 Change3833525by Ethan.Geller Fix copyright in ActiveSound.cpp #jira none #rb none Change 3642649 by Stewart.Lynch Renamed loctext key to stop it clashing with an existing entry #jira UE-49432 Change 3644762 by Stewart.Lynch LLM update: Added Total, Untracked and FMalloc Unused to LLMFULL stat page. Fixed occasionally missing allocs/frees. Removed platform csv. * removed CheckSize arg from OnLowLevelFree * show a warning in LLM Map when replacing existing value. This means that there has been an alloc/free mismatch. * minor optimization in LLMMap::GetMaxIndex - cache Mask value * added Total and Untracked stats to LLMFULL. LLMFULL now tracks almost all of the memory that LLMPlatform does, so there is no real need to use LLMPlatform. Removed the LLMPlatform csv. * added FMalloc Unused stat to LLMFULL to account for memory Binned2 has allocated internally. This can be used to track Binned2 fragmentation over time. * renamed Binned stats to FMemory to make it more general * added Default tracking to CustomVirtualAlloc and disable it where necessary. This catches the few VirtualAlloc calls that were missed. * added AllocType arg to all allocation tracking. This was needed in order to track the FMalloc total, and also to fix the pausing * fixed a bug in pausing where alloc/frees were being missed. Now only pauses a specific allocation type. * Trackers now maintain totals for each enum tag * tracking of Texture and mesh allocation on Windows D3D11 & D3D12 Change 3651334 by Joe.Barnes Fix misspelled function name. #jira 39441 #3016 Change 3653857 by Ben.Woodhouse Integrate from //UE4/Main/...@3653675 to //UE4/Dev-Console/... Change 3656553 by Joe.Barnes Add path for SCS_DeviceDepth. Change 3662703 by Ben.Woodhouse Merging CL 3659069 from //Fortnite/Main/... to to //ue4/dev-console/...: [FORTNITE] [CONSOLE] [+] Improved frame syncing mechanism - Improves input latency by allowing the game thread to sync to the swap chain flip of the previous frame. - Added "r.GTSyncType" CVar to control how the game thread syncs with the rest of the pipe. - r.GTSyncType 2 will sync the game thread with the flip of the swap chain, preventing the pipe from getting too long and causing excess input latency. Platforms are required to implement RHIWaitForFlip and RHISignalFlipEvent, and call RHIInitializeFlipTracking on RHI startup. A separate thread monitors the progress of frame flips and signals task graph events as they pass their corresponding frame index. In r.GTSyncType 2 mode, the game thread is signaled by this flip tracking thread. [~] Unified platform specific sync interval CVars into one: rhi.SyncInterval - 1 == 60Hz - 2 == 30Hz - 3 == 20Hz Change 3675239 by Keith.Judge Fixed thread priorities for Windows so that BelowNormal and SlightlyBelowNormal are the same. Bumped Lowest down a notch so they all fit. #jira UE-50626 Change 3676709 by Ben.Woodhouse Integrate from //UE4/Main/...@3675008 to //UE4/Dev-Console/... Change 3689712 by Ben.Woodhouse Integrate from //UE4/Main/...@3687781 to //UE4/Dev-Console/... Change 3701778 by Ben.Woodhouse Integrate from //UE4/Main/...@3699491 to //UE4/Dev-Console/... Change 3677043 by Ben.Woodhouse From StewartL: I also found that on Windows Fortnite is overflowing int32 values due to the number of allocations. I changed LLMArray and LLMMap to use uint32 and it seems to be Ok now. I didn't want to check this in at the last minute, so here's the shelf: 3645336 Change 3726532 by Luke.Thatcher [CONSOLE] [~] Enable XGE shader compilation by default. - Also set XGE mode to "force interception". With XGE on by default, we don't want people with an old Incredibuild version using the old system inadvertently. Change 3726554 by Luke.Thatcher [CONSOLE] [+] Improved Scoped Named Events - Added SCOPED_NAMED_EVENT(_FSTRING/_TEXT/_F) macros to allow use of string literals, FString and printf in scoped named event strings. - Replaced explicit use of FPlatformMisc::Begin/EndNamedEvent throughout the engine with macros, so the events can be compiled out. - Fixed performance issues on various platforms with named events. SCOPED_NAMED_EVENT macros were not considering whether the platform uses wide or ansi char strings. Change3751378by Ben.Woodhouse Integrate from //UE4/Main/...@3748735 to //UE4/Dev-Console/... Change 3751812 by Ben.Woodhouse Integrate from //UE4/Main/...@3750870 to //UE4/Dev-Console/... Change 3728571 by Luke.Thatcher [CONSOLE] [!] Drop another XGE controller warning to log. Change 3747150 by Joe.Barnes Add AuthoringToolHelper.bat file to installed build copy list. Change 3768585 by Ben.Woodhouse Integrate from //UE4/Main/...@3767531 to //UE4/Dev-Console/... Change 3772333 by Ben.Woodhouse Integrate from //UE4/Main/...@3771573 to //UE4/Dev-Console/... Change 3786872 by Ben.Woodhouse Integrate from //UE4/Main/...@3786785 to //UE4/Dev-Console/... Change 3787279 by Luke.Thatcher [CONSOLE] [~] Unified present threshold CVars. - CVars are now in RHIUtilities.cpp, and are renamed to rhi.PresentThreshold.Top and rhi.PresentThreshold.Bottom. - Platform implementations will be checked in shortly. Change 3787445 by Luke.Thatcher [CONSOLE] [^] Merging (as edit) improvements to low-latency frame syncing from //Fortnite/Main to //UE4/Dev-Console Original CLs - 3708949 - Added rhi.SyncSlackMS cvar to allow an offset of the game thread sync time by a number of milliseconds. - 3712693 - Fix for crash on startup in new frame syncing. - 3735765 - Fix r.GTSyncType logic when vsync is disabled (falls back to old behaviour when vsync is off). Change 3788417 by Ben.Woodhouse Duplicate from FN CL 3712515 CSV profiler GPU and pre-declared stat support - refactor the GPU profiler so it's no longer dependent on the stats system and can work in Test builds - add support for pre-declared CSV stats, using FNames (these are required for GPU stats) - add DECLARE_GPU_STAT macro which handles STATS and CsvProfiler declarations Note: still a few issues to resolve with GPU stats: these randomly go to 0 at times during a replay on XB1, the GPU total is lower than the stat unit number, and the unaccounted stat is too large due to missing stats Change 3807818 by Ben.Woodhouse Integrate from //UE4/Main/...@3803271 to //UE4/Dev-Console/... Change 3818577 by Ben.Woodhouse Integrate from //UE4/Main/...@3812936 to //UE4/Dev-Console/... Change 3821198 by Ben.Woodhouse Integrate from //UE4/Release-4.19/...@3820127 to //UE4/Dev-Console/... Change 3821519 by Ben.Woodhouse Integrate from //UE4/Release-4.19/...@3820753 to //UE4/Dev-Console/... #robomerge none Change 3813513 by Ben.Woodhouse CSV profiler refactor+ bug fix - Move the CSV profiler to core, so we can use it in modules other than engine - Profiler no longer enqueues RT commands - this is handled by a new core delegate type - Make begin/end requests more robust, enqueued via a command queue and processed in order - Defer end capture requests by a frame. This ensures stats on threads other than the gamethread (e.g. renderthread etc) are complete, so the last frame is not truncated - Fix long-standing bug with multiple captures where old/stale frames would appear in the first <128 frames of captures - Move unit stats and dynamic resolution out of the profiler itself and into the engine. Only frametime is recorded in the profiler now - Fix longstanding bug in first frame renderthread time in the engine Change 3814039 by Ben.Woodhouse More Csvprofiler improvements - Event support (via CSV_EVENT macros). These appear in the "EVENTS" column of the CSV and can be used to add context to the stat data - Reduced memory overhead for timestamps through bit-packing (now 16 bytes per marker instead of 24), and splitting stats into different types Change3814041by Ben.Woodhouse Integrate as edit CL 3796390 from Fortnite/Main: Fix CsvProfiler not compiling in shipping for now Change 3814229 by Ben.Woodhouse Integrate + refactor of CL 3792591 to reduce complexity and fix bugs Original changelist description: CSV profiler improvements: - The CSV profiler is now always compiled in on the server - The CSV profiler can now handle both int32 and float stats - In BeginCapture, the function can take additional arguments for some customization of filenames Fixes to the above: - Remove FCustomValue class - Fixes bug where all timestamp values from CSV_SCOPED_STATs were garbage (due to issues resulting from FCustomValue type ambiguity when adding values together) - FCsvCustomStat now just uses simple union + bitfield flag to reduce size and complexity (4 bytes instead of 8 per value) - FCsvColumn class modified to use doubles, which can represent both ints and floats without loss of precision - this class is not memory or time critical - Replace multiple overrides used by the server for filenameprefix/folder/postfix etc with a single FilenameOverride Change 3814242 by Ben.Woodhouse Disable CSV unit stats on the dedicated server Change 3817339 by Ben.Woodhouse Duplicate from 3816641: CSV profiler improvements - Added a low-pri processing thread to compress raw timing data into a much more efficient format - Reduces memory usage for 15 minute 30k frame capture with GPU stats enabled from 110MB to 3.4MB - Processing time : 0.1ms per frame - Improved name handling for char literal stats. We now use the string as the ID rather than the pointer so multiple stat uses of the same name string apply only to a single stat - Reworked thread data access to eliminate locking for anything time-critical. Frame boundaries now handled via a lock-free helper class - Fixed bug with queue implementation where 1 in 128 reads would duplicate the last block of stat data - Reduced #include dependencies for CSVProfiler.h - Removed AccumulateMax because it doesn't work, and implementing properly would add a lot of complexity - Added a simple test harness Change 3817582 by Ben.Woodhouse Fix android compile warning Change 3823242 by Ben.Woodhouse Integrate as edit from Fortnite/Main 3820067: Fix threading issue with D3D12 pipeline state caches for windows. This was caused by the usage of FRwScopeLock::RaiseLockToWrite. A pointer read before RaiseLockToWrite was called was invalidated because RaiseLockToWrite has to release the read lock before acquiring the write lock. Rename FRwScopeLock::RaiseLockToWrite to ReleaseReadOnlyLockAndAcquireWriteLock_USE_WITH_CAUTION to make it more explicit what's happening. As the comment says: // This function should be used with caution. // It releases the read lock _before_ acquiring a new write lock. This is not an atomic operation and the caller should // not treat it as such. // E.g. Pointers read from protected data structures prior to this call may be invalid after the function is called. Change 3823840 by Ben.Woodhouse Edigrate from 3823816 Fix an issue where the csvprofile console commands would sometimes result in an empty CSV. The root cause was GFrameNumber incrementing between the console command being read and FCSVProfiler::EndFrame(). We now use our own frame counter which is updated in EndFrame, so this can't happen Also fix an issue where calling csvprofile stop twice would cause all further commands to be ignored. Change 3827787 by Ben.Woodhouse Integrate-as-edit CL 3820678 from Fortnite/Main Allow the CSV Profiler to be compiled in to shipping dedicated server builds Change 3827842 by Ben.Woodhouse Integrate-as-edit CL 3827079 from Fortnite/Main CSV profiler category support Change 3827918 by Luke.Thatcher [CONSOLE] [!] Fix compile error in CSV custom stats. Change 3827964 by Luke.Thatcher [CONSOLE] [!] Fix inverted logic and spelling of boolean. - Functionally, the boolean did the correct thing, but the logic was backwards inside the build tool. Change3831661by Ben.Woodhouse Integrate-as-edit CL 3830630 from Fortnite/Main Fix CSVProfiler assert in dev builds on XB1 Change 3860300 by Joe.Barnes Use same method for src and dest rect calculation as other post processing passes so rects match between passes. Prevents read of unprocessed pixels. Change 3860347 by Joe.Barnes Delete existing SourceConfigFile before allocating a new one to prevent them leaking. Change 3860348 by Joe.Barnes Completely encapsulate GetLLMAlloc() in #ifdef. Change 3861772 by Ben.Woodhouse Integrate-as-edit CL 3861688 from Fortnite/Main: Forward lighting GPU crash fixes Change 3861774 by Ben.Woodhouse Integrate as edit CL 3833918 from dev-rendering (courtesy of DanielW): D3D12 RHI: only refcount uniform buffers if GRHINeedsExtraDeletionLatency is false, which is no longer the case for PC. The refcounting was heavy on performance as reported by a licensee because FRHIResource uses atomics for refcounting, which is only necessary when GRHINeedsExtraDeletionLatency is disabled. Change 3862214 by Ben.Woodhouse Integrate-as-edit CL 3859637 from Fortnite/Main Dynamic resolution console tweaks - Dynamic resolution high level switch driven by a cvar instead of code - Disable user settings dynamic resolution handling on non-desktop platforms. DynamicRes as a user setting does not make sense on consoles/mobile - we'll drive it from device profiles/scalability - Modify naming of GPUHeadRoom to make it clear that it's a percentage, not millseconds Change 3863919 by Ben.Woodhouse [Copy] Integrate console dynamic resolution interface changes from CL 3863354, 3862754, 3862639 Change 3864347 by Ben.Woodhouse Fix the editor build. This will need a proper fix before 4.19 ships. #lockdown Nick.Penwarden #rb none [CL 3913399 by Ben Marsh in Main branch]
2375 lines
87 KiB
Python
2375 lines
87 KiB
Python
# Author: Steven J. Bethard <steven.bethard@gmail.com>.
|
|
|
|
"""Command-line parsing library
|
|
|
|
This module is an optparse-inspired command-line parsing library that:
|
|
|
|
- handles both optional and positional arguments
|
|
- produces highly informative usage messages
|
|
- supports parsers that dispatch to sub-parsers
|
|
|
|
The following is a simple usage example that sums integers from the
|
|
command-line and writes the result to a file::
|
|
|
|
parser = argparse.ArgumentParser(
|
|
description='sum the integers at the command line')
|
|
parser.add_argument(
|
|
'integers', metavar='int', nargs='+', type=int,
|
|
help='an integer to be summed')
|
|
parser.add_argument(
|
|
'--log', default=sys.stdout, type=argparse.FileType('w'),
|
|
help='the file where the sum should be written')
|
|
args = parser.parse_args()
|
|
args.log.write('%s' % sum(args.integers))
|
|
args.log.close()
|
|
|
|
The module contains the following public classes:
|
|
|
|
- ArgumentParser -- The main entry point for command-line parsing. As the
|
|
example above shows, the add_argument() method is used to populate
|
|
the parser with actions for optional and positional arguments. Then
|
|
the parse_args() method is invoked to convert the args at the
|
|
command-line into an object with attributes.
|
|
|
|
- ArgumentError -- The exception raised by ArgumentParser objects when
|
|
there are errors with the parser's actions. Errors raised while
|
|
parsing the command-line are caught by ArgumentParser and emitted
|
|
as command-line messages.
|
|
|
|
- FileType -- A factory for defining types of files to be created. As the
|
|
example above shows, instances of FileType are typically passed as
|
|
the type= argument of add_argument() calls.
|
|
|
|
- Action -- The base class for parser actions. Typically actions are
|
|
selected by passing strings like 'store_true' or 'append_const' to
|
|
the action= argument of add_argument(). However, for greater
|
|
customization of ArgumentParser actions, subclasses of Action may
|
|
be defined and passed as the action= argument.
|
|
|
|
- HelpFormatter, RawDescriptionHelpFormatter, RawTextHelpFormatter,
|
|
ArgumentDefaultsHelpFormatter -- Formatter classes which
|
|
may be passed as the formatter_class= argument to the
|
|
ArgumentParser constructor. HelpFormatter is the default,
|
|
RawDescriptionHelpFormatter and RawTextHelpFormatter tell the parser
|
|
not to change the formatting for help text, and
|
|
ArgumentDefaultsHelpFormatter adds information about argument defaults
|
|
to the help.
|
|
|
|
All other classes in this module are considered implementation details.
|
|
(Also note that HelpFormatter and RawDescriptionHelpFormatter are only
|
|
considered public as object names -- the API of the formatter objects is
|
|
still considered an implementation detail.)
|
|
"""
|
|
|
|
__version__ = '1.1'
|
|
__all__ = [
|
|
'ArgumentParser',
|
|
'ArgumentError',
|
|
'ArgumentTypeError',
|
|
'FileType',
|
|
'HelpFormatter',
|
|
'ArgumentDefaultsHelpFormatter',
|
|
'RawDescriptionHelpFormatter',
|
|
'RawTextHelpFormatter',
|
|
'Namespace',
|
|
'Action',
|
|
'ONE_OR_MORE',
|
|
'OPTIONAL',
|
|
'PARSER',
|
|
'REMAINDER',
|
|
'SUPPRESS',
|
|
'ZERO_OR_MORE',
|
|
]
|
|
|
|
|
|
import collections as _collections
|
|
import copy as _copy
|
|
import os as _os
|
|
import re as _re
|
|
import sys as _sys
|
|
import textwrap as _textwrap
|
|
|
|
from gettext import gettext as _
|
|
|
|
|
|
def _callable(obj):
|
|
return hasattr(obj, '__call__') or hasattr(obj, '__bases__')
|
|
|
|
|
|
SUPPRESS = '==SUPPRESS=='
|
|
|
|
OPTIONAL = '?'
|
|
ZERO_OR_MORE = '*'
|
|
ONE_OR_MORE = '+'
|
|
PARSER = 'A...'
|
|
REMAINDER = '...'
|
|
_UNRECOGNIZED_ARGS_ATTR = '_unrecognized_args'
|
|
|
|
# =============================
|
|
# Utility functions and classes
|
|
# =============================
|
|
|
|
class _AttributeHolder(object):
|
|
"""Abstract base class that provides __repr__.
|
|
|
|
The __repr__ method returns a string in the format::
|
|
ClassName(attr=name, attr=name, ...)
|
|
The attributes are determined either by a class-level attribute,
|
|
'_kwarg_names', or by inspecting the instance __dict__.
|
|
"""
|
|
|
|
def __repr__(self):
|
|
type_name = type(self).__name__
|
|
arg_strings = []
|
|
for arg in self._get_args():
|
|
arg_strings.append(repr(arg))
|
|
for name, value in self._get_kwargs():
|
|
arg_strings.append('%s=%r' % (name, value))
|
|
return '%s(%s)' % (type_name, ', '.join(arg_strings))
|
|
|
|
def _get_kwargs(self):
|
|
return sorted(self.__dict__.items())
|
|
|
|
def _get_args(self):
|
|
return []
|
|
|
|
|
|
def _ensure_value(namespace, name, value):
|
|
if getattr(namespace, name, None) is None:
|
|
setattr(namespace, name, value)
|
|
return getattr(namespace, name)
|
|
|
|
|
|
# ===============
|
|
# Formatting Help
|
|
# ===============
|
|
|
|
class HelpFormatter(object):
|
|
"""Formatter for generating usage messages and argument help strings.
|
|
|
|
Only the name of this class is considered a public API. All the methods
|
|
provided by the class are considered an implementation detail.
|
|
"""
|
|
|
|
def __init__(self,
|
|
prog,
|
|
indent_increment=2,
|
|
max_help_position=24,
|
|
width=None):
|
|
|
|
# default setting for width
|
|
if width is None:
|
|
try:
|
|
width = int(_os.environ['COLUMNS'])
|
|
except (KeyError, ValueError):
|
|
width = 80
|
|
width -= 2
|
|
|
|
self._prog = prog
|
|
self._indent_increment = indent_increment
|
|
self._max_help_position = max_help_position
|
|
self._max_help_position = min(max_help_position,
|
|
max(width - 20, indent_increment * 2))
|
|
self._width = width
|
|
|
|
self._current_indent = 0
|
|
self._level = 0
|
|
self._action_max_length = 0
|
|
|
|
self._root_section = self._Section(self, None)
|
|
self._current_section = self._root_section
|
|
|
|
self._whitespace_matcher = _re.compile(r'\s+')
|
|
self._long_break_matcher = _re.compile(r'\n\n\n+')
|
|
|
|
# ===============================
|
|
# Section and indentation methods
|
|
# ===============================
|
|
def _indent(self):
|
|
self._current_indent += self._indent_increment
|
|
self._level += 1
|
|
|
|
def _dedent(self):
|
|
self._current_indent -= self._indent_increment
|
|
assert self._current_indent >= 0, 'Indent decreased below 0.'
|
|
self._level -= 1
|
|
|
|
class _Section(object):
|
|
|
|
def __init__(self, formatter, parent, heading=None):
|
|
self.formatter = formatter
|
|
self.parent = parent
|
|
self.heading = heading
|
|
self.items = []
|
|
|
|
def format_help(self):
|
|
# format the indented section
|
|
if self.parent is not None:
|
|
self.formatter._indent()
|
|
join = self.formatter._join_parts
|
|
for func, args in self.items:
|
|
func(*args)
|
|
item_help = join([func(*args) for func, args in self.items])
|
|
if self.parent is not None:
|
|
self.formatter._dedent()
|
|
|
|
# return nothing if the section was empty
|
|
if not item_help:
|
|
return ''
|
|
|
|
# add the heading if the section was non-empty
|
|
if self.heading is not SUPPRESS and self.heading is not None:
|
|
current_indent = self.formatter._current_indent
|
|
heading = '%*s%s:\n' % (current_indent, '', self.heading)
|
|
else:
|
|
heading = ''
|
|
|
|
# join the section-initial newline, the heading and the help
|
|
return join(['\n', heading, item_help, '\n'])
|
|
|
|
def _add_item(self, func, args):
|
|
self._current_section.items.append((func, args))
|
|
|
|
# ========================
|
|
# Message building methods
|
|
# ========================
|
|
def start_section(self, heading):
|
|
self._indent()
|
|
section = self._Section(self, self._current_section, heading)
|
|
self._add_item(section.format_help, [])
|
|
self._current_section = section
|
|
|
|
def end_section(self):
|
|
self._current_section = self._current_section.parent
|
|
self._dedent()
|
|
|
|
def add_text(self, text):
|
|
if text is not SUPPRESS and text is not None:
|
|
self._add_item(self._format_text, [text])
|
|
|
|
def add_usage(self, usage, actions, groups, prefix=None):
|
|
if usage is not SUPPRESS:
|
|
args = usage, actions, groups, prefix
|
|
self._add_item(self._format_usage, args)
|
|
|
|
def add_argument(self, action):
|
|
if action.help is not SUPPRESS:
|
|
|
|
# find all invocations
|
|
get_invocation = self._format_action_invocation
|
|
invocations = [get_invocation(action)]
|
|
for subaction in self._iter_indented_subactions(action):
|
|
invocations.append(get_invocation(subaction))
|
|
|
|
# update the maximum item length
|
|
invocation_length = max([len(s) for s in invocations])
|
|
action_length = invocation_length + self._current_indent
|
|
self._action_max_length = max(self._action_max_length,
|
|
action_length)
|
|
|
|
# add the item to the list
|
|
self._add_item(self._format_action, [action])
|
|
|
|
def add_arguments(self, actions):
|
|
for action in actions:
|
|
self.add_argument(action)
|
|
|
|
# =======================
|
|
# Help-formatting methods
|
|
# =======================
|
|
def format_help(self):
|
|
help = self._root_section.format_help()
|
|
if help:
|
|
help = self._long_break_matcher.sub('\n\n', help)
|
|
help = help.strip('\n') + '\n'
|
|
return help
|
|
|
|
def _join_parts(self, part_strings):
|
|
return ''.join([part
|
|
for part in part_strings
|
|
if part and part is not SUPPRESS])
|
|
|
|
def _format_usage(self, usage, actions, groups, prefix):
|
|
if prefix is None:
|
|
prefix = _('usage: ')
|
|
|
|
# if usage is specified, use that
|
|
if usage is not None:
|
|
usage = usage % dict(prog=self._prog)
|
|
|
|
# if no optionals or positionals are available, usage is just prog
|
|
elif usage is None and not actions:
|
|
usage = '%(prog)s' % dict(prog=self._prog)
|
|
|
|
# if optionals and positionals are available, calculate usage
|
|
elif usage is None:
|
|
prog = '%(prog)s' % dict(prog=self._prog)
|
|
|
|
# split optionals from positionals
|
|
optionals = []
|
|
positionals = []
|
|
for action in actions:
|
|
if action.option_strings:
|
|
optionals.append(action)
|
|
else:
|
|
positionals.append(action)
|
|
|
|
# build full usage string
|
|
format = self._format_actions_usage
|
|
action_usage = format(optionals + positionals, groups)
|
|
usage = ' '.join([s for s in [prog, action_usage] if s])
|
|
|
|
# wrap the usage parts if it's too long
|
|
text_width = self._width - self._current_indent
|
|
if len(prefix) + len(usage) > text_width:
|
|
|
|
# break usage into wrappable parts
|
|
part_regexp = r'\(.*?\)+|\[.*?\]+|\S+'
|
|
opt_usage = format(optionals, groups)
|
|
pos_usage = format(positionals, groups)
|
|
opt_parts = _re.findall(part_regexp, opt_usage)
|
|
pos_parts = _re.findall(part_regexp, pos_usage)
|
|
assert ' '.join(opt_parts) == opt_usage
|
|
assert ' '.join(pos_parts) == pos_usage
|
|
|
|
# helper for wrapping lines
|
|
def get_lines(parts, indent, prefix=None):
|
|
lines = []
|
|
line = []
|
|
if prefix is not None:
|
|
line_len = len(prefix) - 1
|
|
else:
|
|
line_len = len(indent) - 1
|
|
for part in parts:
|
|
if line_len + 1 + len(part) > text_width and line:
|
|
lines.append(indent + ' '.join(line))
|
|
line = []
|
|
line_len = len(indent) - 1
|
|
line.append(part)
|
|
line_len += len(part) + 1
|
|
if line:
|
|
lines.append(indent + ' '.join(line))
|
|
if prefix is not None:
|
|
lines[0] = lines[0][len(indent):]
|
|
return lines
|
|
|
|
# if prog is short, follow it with optionals or positionals
|
|
if len(prefix) + len(prog) <= 0.75 * text_width:
|
|
indent = ' ' * (len(prefix) + len(prog) + 1)
|
|
if opt_parts:
|
|
lines = get_lines([prog] + opt_parts, indent, prefix)
|
|
lines.extend(get_lines(pos_parts, indent))
|
|
elif pos_parts:
|
|
lines = get_lines([prog] + pos_parts, indent, prefix)
|
|
else:
|
|
lines = [prog]
|
|
|
|
# if prog is long, put it on its own line
|
|
else:
|
|
indent = ' ' * len(prefix)
|
|
parts = opt_parts + pos_parts
|
|
lines = get_lines(parts, indent)
|
|
if len(lines) > 1:
|
|
lines = []
|
|
lines.extend(get_lines(opt_parts, indent))
|
|
lines.extend(get_lines(pos_parts, indent))
|
|
lines = [prog] + lines
|
|
|
|
# join lines into usage
|
|
usage = '\n'.join(lines)
|
|
|
|
# prefix with 'usage:'
|
|
return '%s%s\n\n' % (prefix, usage)
|
|
|
|
def _format_actions_usage(self, actions, groups):
|
|
# find group indices and identify actions in groups
|
|
group_actions = set()
|
|
inserts = {}
|
|
for group in groups:
|
|
try:
|
|
start = actions.index(group._group_actions[0])
|
|
except ValueError:
|
|
continue
|
|
else:
|
|
end = start + len(group._group_actions)
|
|
if actions[start:end] == group._group_actions:
|
|
for action in group._group_actions:
|
|
group_actions.add(action)
|
|
if not group.required:
|
|
if start in inserts:
|
|
inserts[start] += ' ['
|
|
else:
|
|
inserts[start] = '['
|
|
inserts[end] = ']'
|
|
else:
|
|
if start in inserts:
|
|
inserts[start] += ' ('
|
|
else:
|
|
inserts[start] = '('
|
|
inserts[end] = ')'
|
|
for i in range(start + 1, end):
|
|
inserts[i] = '|'
|
|
|
|
# collect all actions format strings
|
|
parts = []
|
|
for i, action in enumerate(actions):
|
|
|
|
# suppressed arguments are marked with None
|
|
# remove | separators for suppressed arguments
|
|
if action.help is SUPPRESS:
|
|
parts.append(None)
|
|
if inserts.get(i) == '|':
|
|
inserts.pop(i)
|
|
elif inserts.get(i + 1) == '|':
|
|
inserts.pop(i + 1)
|
|
|
|
# produce all arg strings
|
|
elif not action.option_strings:
|
|
part = self._format_args(action, action.dest)
|
|
|
|
# if it's in a group, strip the outer []
|
|
if action in group_actions:
|
|
if part[0] == '[' and part[-1] == ']':
|
|
part = part[1:-1]
|
|
|
|
# add the action string to the list
|
|
parts.append(part)
|
|
|
|
# produce the first way to invoke the option in brackets
|
|
else:
|
|
option_string = action.option_strings[0]
|
|
|
|
# if the Optional doesn't take a value, format is:
|
|
# -s or --long
|
|
if action.nargs == 0:
|
|
part = '%s' % option_string
|
|
|
|
# if the Optional takes a value, format is:
|
|
# -s ARGS or --long ARGS
|
|
else:
|
|
default = action.dest.upper()
|
|
args_string = self._format_args(action, default)
|
|
part = '%s %s' % (option_string, args_string)
|
|
|
|
# make it look optional if it's not required or in a group
|
|
if not action.required and action not in group_actions:
|
|
part = '[%s]' % part
|
|
|
|
# add the action string to the list
|
|
parts.append(part)
|
|
|
|
# insert things at the necessary indices
|
|
for i in sorted(inserts, reverse=True):
|
|
parts[i:i] = [inserts[i]]
|
|
|
|
# join all the action items with spaces
|
|
text = ' '.join([item for item in parts if item is not None])
|
|
|
|
# clean up separators for mutually exclusive groups
|
|
open = r'[\[(]'
|
|
close = r'[\])]'
|
|
text = _re.sub(r'(%s) ' % open, r'\1', text)
|
|
text = _re.sub(r' (%s)' % close, r'\1', text)
|
|
text = _re.sub(r'%s *%s' % (open, close), r'', text)
|
|
text = _re.sub(r'\(([^|]*)\)', r'\1', text)
|
|
text = text.strip()
|
|
|
|
# return the text
|
|
return text
|
|
|
|
def _format_text(self, text):
|
|
if '%(prog)' in text:
|
|
text = text % dict(prog=self._prog)
|
|
text_width = max(self._width - self._current_indent, 11)
|
|
indent = ' ' * self._current_indent
|
|
return self._fill_text(text, text_width, indent) + '\n\n'
|
|
|
|
def _format_action(self, action):
|
|
# determine the required width and the entry label
|
|
help_position = min(self._action_max_length + 2,
|
|
self._max_help_position)
|
|
help_width = max(self._width - help_position, 11)
|
|
action_width = help_position - self._current_indent - 2
|
|
action_header = self._format_action_invocation(action)
|
|
|
|
# ho nelp; start on same line and add a final newline
|
|
if not action.help:
|
|
tup = self._current_indent, '', action_header
|
|
action_header = '%*s%s\n' % tup
|
|
|
|
# short action name; start on the same line and pad two spaces
|
|
elif len(action_header) <= action_width:
|
|
tup = self._current_indent, '', action_width, action_header
|
|
action_header = '%*s%-*s ' % tup
|
|
indent_first = 0
|
|
|
|
# long action name; start on the next line
|
|
else:
|
|
tup = self._current_indent, '', action_header
|
|
action_header = '%*s%s\n' % tup
|
|
indent_first = help_position
|
|
|
|
# collect the pieces of the action help
|
|
parts = [action_header]
|
|
|
|
# if there was help for the action, add lines of help text
|
|
if action.help:
|
|
help_text = self._expand_help(action)
|
|
help_lines = self._split_lines(help_text, help_width)
|
|
parts.append('%*s%s\n' % (indent_first, '', help_lines[0]))
|
|
for line in help_lines[1:]:
|
|
parts.append('%*s%s\n' % (help_position, '', line))
|
|
|
|
# or add a newline if the description doesn't end with one
|
|
elif not action_header.endswith('\n'):
|
|
parts.append('\n')
|
|
|
|
# if there are any sub-actions, add their help as well
|
|
for subaction in self._iter_indented_subactions(action):
|
|
parts.append(self._format_action(subaction))
|
|
|
|
# return a single string
|
|
return self._join_parts(parts)
|
|
|
|
def _format_action_invocation(self, action):
|
|
if not action.option_strings:
|
|
metavar, = self._metavar_formatter(action, action.dest)(1)
|
|
return metavar
|
|
|
|
else:
|
|
parts = []
|
|
|
|
# if the Optional doesn't take a value, format is:
|
|
# -s, --long
|
|
if action.nargs == 0:
|
|
parts.extend(action.option_strings)
|
|
|
|
# if the Optional takes a value, format is:
|
|
# -s ARGS, --long ARGS
|
|
else:
|
|
default = action.dest.upper()
|
|
args_string = self._format_args(action, default)
|
|
for option_string in action.option_strings:
|
|
parts.append('%s %s' % (option_string, args_string))
|
|
|
|
return ', '.join(parts)
|
|
|
|
def _metavar_formatter(self, action, default_metavar):
|
|
if action.metavar is not None:
|
|
result = action.metavar
|
|
elif action.choices is not None:
|
|
choice_strs = [str(choice) for choice in action.choices]
|
|
result = '{%s}' % ','.join(choice_strs)
|
|
else:
|
|
result = default_metavar
|
|
|
|
def format(tuple_size):
|
|
if isinstance(result, tuple):
|
|
return result
|
|
else:
|
|
return (result, ) * tuple_size
|
|
return format
|
|
|
|
def _format_args(self, action, default_metavar):
|
|
get_metavar = self._metavar_formatter(action, default_metavar)
|
|
if action.nargs is None:
|
|
result = '%s' % get_metavar(1)
|
|
elif action.nargs == OPTIONAL:
|
|
result = '[%s]' % get_metavar(1)
|
|
elif action.nargs == ZERO_OR_MORE:
|
|
result = '[%s [%s ...]]' % get_metavar(2)
|
|
elif action.nargs == ONE_OR_MORE:
|
|
result = '%s [%s ...]' % get_metavar(2)
|
|
elif action.nargs == REMAINDER:
|
|
result = '...'
|
|
elif action.nargs == PARSER:
|
|
result = '%s ...' % get_metavar(1)
|
|
else:
|
|
formats = ['%s' for _ in range(action.nargs)]
|
|
result = ' '.join(formats) % get_metavar(action.nargs)
|
|
return result
|
|
|
|
def _expand_help(self, action):
|
|
params = dict(vars(action), prog=self._prog)
|
|
for name in list(params):
|
|
if params[name] is SUPPRESS:
|
|
del params[name]
|
|
for name in list(params):
|
|
if hasattr(params[name], '__name__'):
|
|
params[name] = params[name].__name__
|
|
if params.get('choices') is not None:
|
|
choices_str = ', '.join([str(c) for c in params['choices']])
|
|
params['choices'] = choices_str
|
|
return self._get_help_string(action) % params
|
|
|
|
def _iter_indented_subactions(self, action):
|
|
try:
|
|
get_subactions = action._get_subactions
|
|
except AttributeError:
|
|
pass
|
|
else:
|
|
self._indent()
|
|
for subaction in get_subactions():
|
|
yield subaction
|
|
self._dedent()
|
|
|
|
def _split_lines(self, text, width):
|
|
text = self._whitespace_matcher.sub(' ', text).strip()
|
|
return _textwrap.wrap(text, width)
|
|
|
|
def _fill_text(self, text, width, indent):
|
|
text = self._whitespace_matcher.sub(' ', text).strip()
|
|
return _textwrap.fill(text, width, initial_indent=indent,
|
|
subsequent_indent=indent)
|
|
|
|
def _get_help_string(self, action):
|
|
return action.help
|
|
|
|
|
|
class RawDescriptionHelpFormatter(HelpFormatter):
|
|
"""Help message formatter which retains any formatting in descriptions.
|
|
|
|
Only the name of this class is considered a public API. All the methods
|
|
provided by the class are considered an implementation detail.
|
|
"""
|
|
|
|
def _fill_text(self, text, width, indent):
|
|
return ''.join([indent + line for line in text.splitlines(True)])
|
|
|
|
|
|
class RawTextHelpFormatter(RawDescriptionHelpFormatter):
|
|
"""Help message formatter which retains formatting of all help text.
|
|
|
|
Only the name of this class is considered a public API. All the methods
|
|
provided by the class are considered an implementation detail.
|
|
"""
|
|
|
|
def _split_lines(self, text, width):
|
|
return text.splitlines()
|
|
|
|
|
|
class ArgumentDefaultsHelpFormatter(HelpFormatter):
|
|
"""Help message formatter which adds default values to argument help.
|
|
|
|
Only the name of this class is considered a public API. All the methods
|
|
provided by the class are considered an implementation detail.
|
|
"""
|
|
|
|
def _get_help_string(self, action):
|
|
help = action.help
|
|
if '%(default)' not in action.help:
|
|
if action.default is not SUPPRESS:
|
|
defaulting_nargs = [OPTIONAL, ZERO_OR_MORE]
|
|
if action.option_strings or action.nargs in defaulting_nargs:
|
|
help += ' (default: %(default)s)'
|
|
return help
|
|
|
|
|
|
# =====================
|
|
# Options and Arguments
|
|
# =====================
|
|
|
|
def _get_action_name(argument):
|
|
if argument is None:
|
|
return None
|
|
elif argument.option_strings:
|
|
return '/'.join(argument.option_strings)
|
|
elif argument.metavar not in (None, SUPPRESS):
|
|
return argument.metavar
|
|
elif argument.dest not in (None, SUPPRESS):
|
|
return argument.dest
|
|
else:
|
|
return None
|
|
|
|
|
|
class ArgumentError(Exception):
|
|
"""An error from creating or using an argument (optional or positional).
|
|
|
|
The string value of this exception is the message, augmented with
|
|
information about the argument that caused it.
|
|
"""
|
|
|
|
def __init__(self, argument, message):
|
|
self.argument_name = _get_action_name(argument)
|
|
self.message = message
|
|
|
|
def __str__(self):
|
|
if self.argument_name is None:
|
|
format = '%(message)s'
|
|
else:
|
|
format = 'argument %(argument_name)s: %(message)s'
|
|
return format % dict(message=self.message,
|
|
argument_name=self.argument_name)
|
|
|
|
|
|
class ArgumentTypeError(Exception):
|
|
"""An error from trying to convert a command line string to a type."""
|
|
pass
|
|
|
|
|
|
# ==============
|
|
# Action classes
|
|
# ==============
|
|
|
|
class Action(_AttributeHolder):
|
|
"""Information about how to convert command line strings to Python objects.
|
|
|
|
Action objects are used by an ArgumentParser to represent the information
|
|
needed to parse a single argument from one or more strings from the
|
|
command line. The keyword arguments to the Action constructor are also
|
|
all attributes of Action instances.
|
|
|
|
Keyword Arguments:
|
|
|
|
- option_strings -- A list of command-line option strings which
|
|
should be associated with this action.
|
|
|
|
- dest -- The name of the attribute to hold the created object(s)
|
|
|
|
- nargs -- The number of command-line arguments that should be
|
|
consumed. By default, one argument will be consumed and a single
|
|
value will be produced. Other values include:
|
|
- N (an integer) consumes N arguments (and produces a list)
|
|
- '?' consumes zero or one arguments
|
|
- '*' consumes zero or more arguments (and produces a list)
|
|
- '+' consumes one or more arguments (and produces a list)
|
|
Note that the difference between the default and nargs=1 is that
|
|
with the default, a single value will be produced, while with
|
|
nargs=1, a list containing a single value will be produced.
|
|
|
|
- const -- The value to be produced if the option is specified and the
|
|
option uses an action that takes no values.
|
|
|
|
- default -- The value to be produced if the option is not specified.
|
|
|
|
- type -- A callable that accepts a single string argument, and
|
|
returns the converted value. The standard Python types str, int,
|
|
float, and complex are useful examples of such callables. If None,
|
|
str is used.
|
|
|
|
- choices -- A container of values that should be allowed. If not None,
|
|
after a command-line argument has been converted to the appropriate
|
|
type, an exception will be raised if it is not a member of this
|
|
collection.
|
|
|
|
- required -- True if the action must always be specified at the
|
|
command line. This is only meaningful for optional command-line
|
|
arguments.
|
|
|
|
- help -- The help string describing the argument.
|
|
|
|
- metavar -- The name to be used for the option's argument with the
|
|
help string. If None, the 'dest' value will be used as the name.
|
|
"""
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
dest,
|
|
nargs=None,
|
|
const=None,
|
|
default=None,
|
|
type=None,
|
|
choices=None,
|
|
required=False,
|
|
help=None,
|
|
metavar=None):
|
|
self.option_strings = option_strings
|
|
self.dest = dest
|
|
self.nargs = nargs
|
|
self.const = const
|
|
self.default = default
|
|
self.type = type
|
|
self.choices = choices
|
|
self.required = required
|
|
self.help = help
|
|
self.metavar = metavar
|
|
|
|
def _get_kwargs(self):
|
|
names = [
|
|
'option_strings',
|
|
'dest',
|
|
'nargs',
|
|
'const',
|
|
'default',
|
|
'type',
|
|
'choices',
|
|
'help',
|
|
'metavar',
|
|
]
|
|
return [(name, getattr(self, name)) for name in names]
|
|
|
|
def __call__(self, parser, namespace, values, option_string=None):
|
|
raise NotImplementedError(_('.__call__() not defined'))
|
|
|
|
|
|
class _StoreAction(Action):
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
dest,
|
|
nargs=None,
|
|
const=None,
|
|
default=None,
|
|
type=None,
|
|
choices=None,
|
|
required=False,
|
|
help=None,
|
|
metavar=None):
|
|
if nargs == 0:
|
|
raise ValueError('nargs for store actions must be > 0; if you '
|
|
'have nothing to store, actions such as store '
|
|
'true or store const may be more appropriate')
|
|
if const is not None and nargs != OPTIONAL:
|
|
raise ValueError('nargs must be %r to supply const' % OPTIONAL)
|
|
super(_StoreAction, self).__init__(
|
|
option_strings=option_strings,
|
|
dest=dest,
|
|
nargs=nargs,
|
|
const=const,
|
|
default=default,
|
|
type=type,
|
|
choices=choices,
|
|
required=required,
|
|
help=help,
|
|
metavar=metavar)
|
|
|
|
def __call__(self, parser, namespace, values, option_string=None):
|
|
setattr(namespace, self.dest, values)
|
|
|
|
|
|
class _StoreConstAction(Action):
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
dest,
|
|
const,
|
|
default=None,
|
|
required=False,
|
|
help=None,
|
|
metavar=None):
|
|
super(_StoreConstAction, self).__init__(
|
|
option_strings=option_strings,
|
|
dest=dest,
|
|
nargs=0,
|
|
const=const,
|
|
default=default,
|
|
required=required,
|
|
help=help)
|
|
|
|
def __call__(self, parser, namespace, values, option_string=None):
|
|
setattr(namespace, self.dest, self.const)
|
|
|
|
|
|
class _StoreTrueAction(_StoreConstAction):
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
dest,
|
|
default=False,
|
|
required=False,
|
|
help=None):
|
|
super(_StoreTrueAction, self).__init__(
|
|
option_strings=option_strings,
|
|
dest=dest,
|
|
const=True,
|
|
default=default,
|
|
required=required,
|
|
help=help)
|
|
|
|
|
|
class _StoreFalseAction(_StoreConstAction):
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
dest,
|
|
default=True,
|
|
required=False,
|
|
help=None):
|
|
super(_StoreFalseAction, self).__init__(
|
|
option_strings=option_strings,
|
|
dest=dest,
|
|
const=False,
|
|
default=default,
|
|
required=required,
|
|
help=help)
|
|
|
|
|
|
class _AppendAction(Action):
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
dest,
|
|
nargs=None,
|
|
const=None,
|
|
default=None,
|
|
type=None,
|
|
choices=None,
|
|
required=False,
|
|
help=None,
|
|
metavar=None):
|
|
if nargs == 0:
|
|
raise ValueError('nargs for append actions must be > 0; if arg '
|
|
'strings are not supplying the value to append, '
|
|
'the append const action may be more appropriate')
|
|
if const is not None and nargs != OPTIONAL:
|
|
raise ValueError('nargs must be %r to supply const' % OPTIONAL)
|
|
super(_AppendAction, self).__init__(
|
|
option_strings=option_strings,
|
|
dest=dest,
|
|
nargs=nargs,
|
|
const=const,
|
|
default=default,
|
|
type=type,
|
|
choices=choices,
|
|
required=required,
|
|
help=help,
|
|
metavar=metavar)
|
|
|
|
def __call__(self, parser, namespace, values, option_string=None):
|
|
items = _copy.copy(_ensure_value(namespace, self.dest, []))
|
|
items.append(values)
|
|
setattr(namespace, self.dest, items)
|
|
|
|
|
|
class _AppendConstAction(Action):
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
dest,
|
|
const,
|
|
default=None,
|
|
required=False,
|
|
help=None,
|
|
metavar=None):
|
|
super(_AppendConstAction, self).__init__(
|
|
option_strings=option_strings,
|
|
dest=dest,
|
|
nargs=0,
|
|
const=const,
|
|
default=default,
|
|
required=required,
|
|
help=help,
|
|
metavar=metavar)
|
|
|
|
def __call__(self, parser, namespace, values, option_string=None):
|
|
items = _copy.copy(_ensure_value(namespace, self.dest, []))
|
|
items.append(self.const)
|
|
setattr(namespace, self.dest, items)
|
|
|
|
|
|
class _CountAction(Action):
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
dest,
|
|
default=None,
|
|
required=False,
|
|
help=None):
|
|
super(_CountAction, self).__init__(
|
|
option_strings=option_strings,
|
|
dest=dest,
|
|
nargs=0,
|
|
default=default,
|
|
required=required,
|
|
help=help)
|
|
|
|
def __call__(self, parser, namespace, values, option_string=None):
|
|
new_count = _ensure_value(namespace, self.dest, 0) + 1
|
|
setattr(namespace, self.dest, new_count)
|
|
|
|
|
|
class _HelpAction(Action):
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
dest=SUPPRESS,
|
|
default=SUPPRESS,
|
|
help=None):
|
|
super(_HelpAction, self).__init__(
|
|
option_strings=option_strings,
|
|
dest=dest,
|
|
default=default,
|
|
nargs=0,
|
|
help=help)
|
|
|
|
def __call__(self, parser, namespace, values, option_string=None):
|
|
parser.print_help()
|
|
parser.exit()
|
|
|
|
|
|
class _VersionAction(Action):
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
version=None,
|
|
dest=SUPPRESS,
|
|
default=SUPPRESS,
|
|
help="show program's version number and exit"):
|
|
super(_VersionAction, self).__init__(
|
|
option_strings=option_strings,
|
|
dest=dest,
|
|
default=default,
|
|
nargs=0,
|
|
help=help)
|
|
self.version = version
|
|
|
|
def __call__(self, parser, namespace, values, option_string=None):
|
|
version = self.version
|
|
if version is None:
|
|
version = parser.version
|
|
formatter = parser._get_formatter()
|
|
formatter.add_text(version)
|
|
parser.exit(message=formatter.format_help())
|
|
|
|
|
|
class _SubParsersAction(Action):
|
|
|
|
class _ChoicesPseudoAction(Action):
|
|
|
|
def __init__(self, name, help):
|
|
sup = super(_SubParsersAction._ChoicesPseudoAction, self)
|
|
sup.__init__(option_strings=[], dest=name, help=help)
|
|
|
|
def __init__(self,
|
|
option_strings,
|
|
prog,
|
|
parser_class,
|
|
dest=SUPPRESS,
|
|
help=None,
|
|
metavar=None):
|
|
|
|
self._prog_prefix = prog
|
|
self._parser_class = parser_class
|
|
self._name_parser_map = _collections.OrderedDict()
|
|
self._choices_actions = []
|
|
|
|
super(_SubParsersAction, self).__init__(
|
|
option_strings=option_strings,
|
|
dest=dest,
|
|
nargs=PARSER,
|
|
choices=self._name_parser_map,
|
|
help=help,
|
|
metavar=metavar)
|
|
|
|
def add_parser(self, name, **kwargs):
|
|
# set prog from the existing prefix
|
|
if kwargs.get('prog') is None:
|
|
kwargs['prog'] = '%s %s' % (self._prog_prefix, name)
|
|
|
|
# create a pseudo-action to hold the choice help
|
|
if 'help' in kwargs:
|
|
help = kwargs.pop('help')
|
|
choice_action = self._ChoicesPseudoAction(name, help)
|
|
self._choices_actions.append(choice_action)
|
|
|
|
# create the parser and add it to the map
|
|
parser = self._parser_class(**kwargs)
|
|
self._name_parser_map[name] = parser
|
|
return parser
|
|
|
|
def _get_subactions(self):
|
|
return self._choices_actions
|
|
|
|
def __call__(self, parser, namespace, values, option_string=None):
|
|
parser_name = values[0]
|
|
arg_strings = values[1:]
|
|
|
|
# set the parser name if requested
|
|
if self.dest is not SUPPRESS:
|
|
setattr(namespace, self.dest, parser_name)
|
|
|
|
# select the parser
|
|
try:
|
|
parser = self._name_parser_map[parser_name]
|
|
except KeyError:
|
|
tup = parser_name, ', '.join(self._name_parser_map)
|
|
msg = _('unknown parser %r (choices: %s)') % tup
|
|
raise ArgumentError(self, msg)
|
|
|
|
# parse all the remaining options into the namespace
|
|
# store any unrecognized options on the object, so that the top
|
|
# level parser can decide what to do with them
|
|
|
|
# In case this subparser defines new defaults, we parse them
|
|
# in a new namespace object and then update the original
|
|
# namespace for the relevant parts.
|
|
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
|
|
for key, value in vars(subnamespace).items():
|
|
setattr(namespace, key, value)
|
|
|
|
if arg_strings:
|
|
vars(namespace).setdefault(_UNRECOGNIZED_ARGS_ATTR, [])
|
|
getattr(namespace, _UNRECOGNIZED_ARGS_ATTR).extend(arg_strings)
|
|
|
|
|
|
# ==============
|
|
# Type classes
|
|
# ==============
|
|
|
|
class FileType(object):
|
|
"""Factory for creating file object types
|
|
|
|
Instances of FileType are typically passed as type= arguments to the
|
|
ArgumentParser add_argument() method.
|
|
|
|
Keyword Arguments:
|
|
- mode -- A string indicating how the file is to be opened. Accepts the
|
|
same values as the builtin open() function.
|
|
- bufsize -- The file's desired buffer size. Accepts the same values as
|
|
the builtin open() function.
|
|
"""
|
|
|
|
def __init__(self, mode='r', bufsize=-1):
|
|
self._mode = mode
|
|
self._bufsize = bufsize
|
|
|
|
def __call__(self, string):
|
|
# the special argument "-" means sys.std{in,out}
|
|
if string == '-':
|
|
if 'r' in self._mode:
|
|
return _sys.stdin
|
|
elif 'w' in self._mode:
|
|
return _sys.stdout
|
|
else:
|
|
msg = _('argument "-" with mode %r') % self._mode
|
|
raise ValueError(msg)
|
|
|
|
# all other arguments are used as file names
|
|
try:
|
|
return open(string, self._mode, self._bufsize)
|
|
except IOError as e:
|
|
message = _("can't open '%s': %s")
|
|
raise ArgumentTypeError(message % (string, e))
|
|
|
|
def __repr__(self):
|
|
args = self._mode, self._bufsize
|
|
args_str = ', '.join(repr(arg) for arg in args if arg != -1)
|
|
return '%s(%s)' % (type(self).__name__, args_str)
|
|
|
|
# ===========================
|
|
# Optional and Positional Parsing
|
|
# ===========================
|
|
|
|
class Namespace(_AttributeHolder):
|
|
"""Simple object for storing attributes.
|
|
|
|
Implements equality by attribute names and values, and provides a simple
|
|
string representation.
|
|
"""
|
|
|
|
def __init__(self, **kwargs):
|
|
for name in kwargs:
|
|
setattr(self, name, kwargs[name])
|
|
|
|
__hash__ = None
|
|
|
|
def __eq__(self, other):
|
|
if not isinstance(other, Namespace):
|
|
return NotImplemented
|
|
return vars(self) == vars(other)
|
|
|
|
def __ne__(self, other):
|
|
if not isinstance(other, Namespace):
|
|
return NotImplemented
|
|
return not (self == other)
|
|
|
|
def __contains__(self, key):
|
|
return key in self.__dict__
|
|
|
|
|
|
class _ActionsContainer(object):
|
|
|
|
def __init__(self,
|
|
description,
|
|
prefix_chars,
|
|
argument_default,
|
|
conflict_handler):
|
|
super(_ActionsContainer, self).__init__()
|
|
|
|
self.description = description
|
|
self.argument_default = argument_default
|
|
self.prefix_chars = prefix_chars
|
|
self.conflict_handler = conflict_handler
|
|
|
|
# set up registries
|
|
self._registries = {}
|
|
|
|
# register actions
|
|
self.register('action', None, _StoreAction)
|
|
self.register('action', 'store', _StoreAction)
|
|
self.register('action', 'store_const', _StoreConstAction)
|
|
self.register('action', 'store_true', _StoreTrueAction)
|
|
self.register('action', 'store_false', _StoreFalseAction)
|
|
self.register('action', 'append', _AppendAction)
|
|
self.register('action', 'append_const', _AppendConstAction)
|
|
self.register('action', 'count', _CountAction)
|
|
self.register('action', 'help', _HelpAction)
|
|
self.register('action', 'version', _VersionAction)
|
|
self.register('action', 'parsers', _SubParsersAction)
|
|
|
|
# raise an exception if the conflict handler is invalid
|
|
self._get_handler()
|
|
|
|
# action storage
|
|
self._actions = []
|
|
self._option_string_actions = {}
|
|
|
|
# groups
|
|
self._action_groups = []
|
|
self._mutually_exclusive_groups = []
|
|
|
|
# defaults storage
|
|
self._defaults = {}
|
|
|
|
# determines whether an "option" looks like a negative number
|
|
self._negative_number_matcher = _re.compile(r'^-\d+$|^-\d*\.\d+$')
|
|
|
|
# whether or not there are any optionals that look like negative
|
|
# numbers -- uses a list so it can be shared and edited
|
|
self._has_negative_number_optionals = []
|
|
|
|
# ====================
|
|
# Registration methods
|
|
# ====================
|
|
def register(self, registry_name, value, object):
|
|
registry = self._registries.setdefault(registry_name, {})
|
|
registry[value] = object
|
|
|
|
def _registry_get(self, registry_name, value, default=None):
|
|
return self._registries[registry_name].get(value, default)
|
|
|
|
# ==================================
|
|
# Namespace default accessor methods
|
|
# ==================================
|
|
def set_defaults(self, **kwargs):
|
|
self._defaults.update(kwargs)
|
|
|
|
# if these defaults match any existing arguments, replace
|
|
# the previous default on the object with the new one
|
|
for action in self._actions:
|
|
if action.dest in kwargs:
|
|
action.default = kwargs[action.dest]
|
|
|
|
def get_default(self, dest):
|
|
for action in self._actions:
|
|
if action.dest == dest and action.default is not None:
|
|
return action.default
|
|
return self._defaults.get(dest, None)
|
|
|
|
|
|
# =======================
|
|
# Adding argument actions
|
|
# =======================
|
|
def add_argument(self, *args, **kwargs):
|
|
"""
|
|
add_argument(dest, ..., name=value, ...)
|
|
add_argument(option_string, option_string, ..., name=value, ...)
|
|
"""
|
|
|
|
# if no positional args are supplied or only one is supplied and
|
|
# it doesn't look like an option string, parse a positional
|
|
# argument
|
|
chars = self.prefix_chars
|
|
if not args or len(args) == 1 and args[0][0] not in chars:
|
|
if args and 'dest' in kwargs:
|
|
raise ValueError('dest supplied twice for positional argument')
|
|
kwargs = self._get_positional_kwargs(*args, **kwargs)
|
|
|
|
# otherwise, we're adding an optional argument
|
|
else:
|
|
kwargs = self._get_optional_kwargs(*args, **kwargs)
|
|
|
|
# if no default was supplied, use the parser-level default
|
|
if 'default' not in kwargs:
|
|
dest = kwargs['dest']
|
|
if dest in self._defaults:
|
|
kwargs['default'] = self._defaults[dest]
|
|
elif self.argument_default is not None:
|
|
kwargs['default'] = self.argument_default
|
|
|
|
# create the action object, and add it to the parser
|
|
action_class = self._pop_action_class(kwargs)
|
|
if not _callable(action_class):
|
|
raise ValueError('unknown action "%s"' % (action_class,))
|
|
action = action_class(**kwargs)
|
|
|
|
# raise an error if the action type is not callable
|
|
type_func = self._registry_get('type', action.type, action.type)
|
|
if not _callable(type_func):
|
|
raise ValueError('%r is not callable' % (type_func,))
|
|
|
|
# raise an error if the metavar does not match the type
|
|
if hasattr(self, "_get_formatter"):
|
|
try:
|
|
self._get_formatter()._format_args(action, None)
|
|
except TypeError:
|
|
raise ValueError("length of metavar tuple does not match nargs")
|
|
|
|
return self._add_action(action)
|
|
|
|
def add_argument_group(self, *args, **kwargs):
|
|
group = _ArgumentGroup(self, *args, **kwargs)
|
|
self._action_groups.append(group)
|
|
return group
|
|
|
|
def add_mutually_exclusive_group(self, **kwargs):
|
|
group = _MutuallyExclusiveGroup(self, **kwargs)
|
|
self._mutually_exclusive_groups.append(group)
|
|
return group
|
|
|
|
def _add_action(self, action):
|
|
# resolve any conflicts
|
|
self._check_conflict(action)
|
|
|
|
# add to actions list
|
|
self._actions.append(action)
|
|
action.container = self
|
|
|
|
# index the action by any option strings it has
|
|
for option_string in action.option_strings:
|
|
self._option_string_actions[option_string] = action
|
|
|
|
# set the flag if any option strings look like negative numbers
|
|
for option_string in action.option_strings:
|
|
if self._negative_number_matcher.match(option_string):
|
|
if not self._has_negative_number_optionals:
|
|
self._has_negative_number_optionals.append(True)
|
|
|
|
# return the created action
|
|
return action
|
|
|
|
def _remove_action(self, action):
|
|
self._actions.remove(action)
|
|
|
|
def _add_container_actions(self, container):
|
|
# collect groups by titles
|
|
title_group_map = {}
|
|
for group in self._action_groups:
|
|
if group.title in title_group_map:
|
|
msg = _('cannot merge actions - two groups are named %r')
|
|
raise ValueError(msg % (group.title))
|
|
title_group_map[group.title] = group
|
|
|
|
# map each action to its group
|
|
group_map = {}
|
|
for group in container._action_groups:
|
|
|
|
# if a group with the title exists, use that, otherwise
|
|
# create a new group matching the container's group
|
|
if group.title not in title_group_map:
|
|
title_group_map[group.title] = self.add_argument_group(
|
|
title=group.title,
|
|
description=group.description,
|
|
conflict_handler=group.conflict_handler)
|
|
|
|
# map the actions to their new group
|
|
for action in group._group_actions:
|
|
group_map[action] = title_group_map[group.title]
|
|
|
|
# add container's mutually exclusive groups
|
|
# NOTE: if add_mutually_exclusive_group ever gains title= and
|
|
# description= then this code will need to be expanded as above
|
|
for group in container._mutually_exclusive_groups:
|
|
mutex_group = self.add_mutually_exclusive_group(
|
|
required=group.required)
|
|
|
|
# map the actions to their new mutex group
|
|
for action in group._group_actions:
|
|
group_map[action] = mutex_group
|
|
|
|
# add all actions to this container or their group
|
|
for action in container._actions:
|
|
group_map.get(action, self)._add_action(action)
|
|
|
|
def _get_positional_kwargs(self, dest, **kwargs):
|
|
# make sure required is not specified
|
|
if 'required' in kwargs:
|
|
msg = _("'required' is an invalid argument for positionals")
|
|
raise TypeError(msg)
|
|
|
|
# mark positional arguments as required if at least one is
|
|
# always required
|
|
if kwargs.get('nargs') not in [OPTIONAL, ZERO_OR_MORE]:
|
|
kwargs['required'] = True
|
|
if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs:
|
|
kwargs['required'] = True
|
|
|
|
# return the keyword arguments with no option strings
|
|
return dict(kwargs, dest=dest, option_strings=[])
|
|
|
|
def _get_optional_kwargs(self, *args, **kwargs):
|
|
# determine short and long option strings
|
|
option_strings = []
|
|
long_option_strings = []
|
|
for option_string in args:
|
|
# error on strings that don't start with an appropriate prefix
|
|
if not option_string[0] in self.prefix_chars:
|
|
msg = _('invalid option string %r: '
|
|
'must start with a character %r')
|
|
tup = option_string, self.prefix_chars
|
|
raise ValueError(msg % tup)
|
|
|
|
# strings starting with two prefix characters are long options
|
|
option_strings.append(option_string)
|
|
if option_string[0] in self.prefix_chars:
|
|
if len(option_string) > 1:
|
|
if option_string[1] in self.prefix_chars:
|
|
long_option_strings.append(option_string)
|
|
|
|
# infer destination, '--foo-bar' -> 'foo_bar' and '-x' -> 'x'
|
|
dest = kwargs.pop('dest', None)
|
|
if dest is None:
|
|
if long_option_strings:
|
|
dest_option_string = long_option_strings[0]
|
|
else:
|
|
dest_option_string = option_strings[0]
|
|
dest = dest_option_string.lstrip(self.prefix_chars)
|
|
if not dest:
|
|
msg = _('dest= is required for options like %r')
|
|
raise ValueError(msg % option_string)
|
|
dest = dest.replace('-', '_')
|
|
|
|
# return the updated keyword arguments
|
|
return dict(kwargs, dest=dest, option_strings=option_strings)
|
|
|
|
def _pop_action_class(self, kwargs, default=None):
|
|
action = kwargs.pop('action', default)
|
|
return self._registry_get('action', action, action)
|
|
|
|
def _get_handler(self):
|
|
# determine function from conflict handler string
|
|
handler_func_name = '_handle_conflict_%s' % self.conflict_handler
|
|
try:
|
|
return getattr(self, handler_func_name)
|
|
except AttributeError:
|
|
msg = _('invalid conflict_resolution value: %r')
|
|
raise ValueError(msg % self.conflict_handler)
|
|
|
|
def _check_conflict(self, action):
|
|
|
|
# find all options that conflict with this option
|
|
confl_optionals = []
|
|
for option_string in action.option_strings:
|
|
if option_string in self._option_string_actions:
|
|
confl_optional = self._option_string_actions[option_string]
|
|
confl_optionals.append((option_string, confl_optional))
|
|
|
|
# resolve any conflicts
|
|
if confl_optionals:
|
|
conflict_handler = self._get_handler()
|
|
conflict_handler(action, confl_optionals)
|
|
|
|
def _handle_conflict_error(self, action, conflicting_actions):
|
|
message = _('conflicting option string(s): %s')
|
|
conflict_string = ', '.join([option_string
|
|
for option_string, action
|
|
in conflicting_actions])
|
|
raise ArgumentError(action, message % conflict_string)
|
|
|
|
def _handle_conflict_resolve(self, action, conflicting_actions):
|
|
|
|
# remove all conflicting options
|
|
for option_string, action in conflicting_actions:
|
|
|
|
# remove the conflicting option
|
|
action.option_strings.remove(option_string)
|
|
self._option_string_actions.pop(option_string, None)
|
|
|
|
# if the option now has no option string, remove it from the
|
|
# container holding it
|
|
if not action.option_strings:
|
|
action.container._remove_action(action)
|
|
|
|
|
|
class _ArgumentGroup(_ActionsContainer):
|
|
|
|
def __init__(self, container, title=None, description=None, **kwargs):
|
|
# add any missing keyword arguments by checking the container
|
|
update = kwargs.setdefault
|
|
update('conflict_handler', container.conflict_handler)
|
|
update('prefix_chars', container.prefix_chars)
|
|
update('argument_default', container.argument_default)
|
|
super_init = super(_ArgumentGroup, self).__init__
|
|
super_init(description=description, **kwargs)
|
|
|
|
# group attributes
|
|
self.title = title
|
|
self._group_actions = []
|
|
|
|
# share most attributes with the container
|
|
self._registries = container._registries
|
|
self._actions = container._actions
|
|
self._option_string_actions = container._option_string_actions
|
|
self._defaults = container._defaults
|
|
self._has_negative_number_optionals = \
|
|
container._has_negative_number_optionals
|
|
self._mutually_exclusive_groups = container._mutually_exclusive_groups
|
|
|
|
def _add_action(self, action):
|
|
action = super(_ArgumentGroup, self)._add_action(action)
|
|
self._group_actions.append(action)
|
|
return action
|
|
|
|
def _remove_action(self, action):
|
|
super(_ArgumentGroup, self)._remove_action(action)
|
|
self._group_actions.remove(action)
|
|
|
|
|
|
class _MutuallyExclusiveGroup(_ArgumentGroup):
|
|
|
|
def __init__(self, container, required=False):
|
|
super(_MutuallyExclusiveGroup, self).__init__(container)
|
|
self.required = required
|
|
self._container = container
|
|
|
|
def _add_action(self, action):
|
|
if action.required:
|
|
msg = _('mutually exclusive arguments must be optional')
|
|
raise ValueError(msg)
|
|
action = self._container._add_action(action)
|
|
self._group_actions.append(action)
|
|
return action
|
|
|
|
def _remove_action(self, action):
|
|
self._container._remove_action(action)
|
|
self._group_actions.remove(action)
|
|
|
|
|
|
class ArgumentParser(_AttributeHolder, _ActionsContainer):
|
|
"""Object for parsing command line strings into Python objects.
|
|
|
|
Keyword Arguments:
|
|
- prog -- The name of the program (default: sys.argv[0])
|
|
- usage -- A usage message (default: auto-generated from arguments)
|
|
- description -- A description of what the program does
|
|
- epilog -- Text following the argument descriptions
|
|
- parents -- Parsers whose arguments should be copied into this one
|
|
- formatter_class -- HelpFormatter class for printing help messages
|
|
- prefix_chars -- Characters that prefix optional arguments
|
|
- fromfile_prefix_chars -- Characters that prefix files containing
|
|
additional arguments
|
|
- argument_default -- The default value for all arguments
|
|
- conflict_handler -- String indicating how to handle conflicts
|
|
- add_help -- Add a -h/-help option
|
|
"""
|
|
|
|
def __init__(self,
|
|
prog=None,
|
|
usage=None,
|
|
description=None,
|
|
epilog=None,
|
|
version=None,
|
|
parents=[],
|
|
formatter_class=HelpFormatter,
|
|
prefix_chars='-',
|
|
fromfile_prefix_chars=None,
|
|
argument_default=None,
|
|
conflict_handler='error',
|
|
add_help=True):
|
|
|
|
if version is not None:
|
|
import warnings
|
|
warnings.warn(
|
|
"""The "version" argument to ArgumentParser is deprecated. """
|
|
"""Please use """
|
|
""""add_argument(..., action='version', version="N", ...)" """
|
|
"""instead""", DeprecationWarning)
|
|
|
|
superinit = super(ArgumentParser, self).__init__
|
|
superinit(description=description,
|
|
prefix_chars=prefix_chars,
|
|
argument_default=argument_default,
|
|
conflict_handler=conflict_handler)
|
|
|
|
# default setting for prog
|
|
if prog is None:
|
|
prog = _os.path.basename(_sys.argv[0])
|
|
|
|
self.prog = prog
|
|
self.usage = usage
|
|
self.epilog = epilog
|
|
self.version = version
|
|
self.formatter_class = formatter_class
|
|
self.fromfile_prefix_chars = fromfile_prefix_chars
|
|
self.add_help = add_help
|
|
|
|
add_group = self.add_argument_group
|
|
self._positionals = add_group(_('positional arguments'))
|
|
self._optionals = add_group(_('optional arguments'))
|
|
self._subparsers = None
|
|
|
|
# register types
|
|
def identity(string):
|
|
return string
|
|
self.register('type', None, identity)
|
|
|
|
# add help and version arguments if necessary
|
|
# (using explicit default to override global argument_default)
|
|
default_prefix = '-' if '-' in prefix_chars else prefix_chars[0]
|
|
if self.add_help:
|
|
self.add_argument(
|
|
default_prefix+'h', default_prefix*2+'help',
|
|
action='help', default=SUPPRESS,
|
|
help=_('show this help message and exit'))
|
|
if self.version:
|
|
self.add_argument(
|
|
default_prefix+'v', default_prefix*2+'version',
|
|
action='version', default=SUPPRESS,
|
|
version=self.version,
|
|
help=_("show program's version number and exit"))
|
|
|
|
# add parent arguments and defaults
|
|
for parent in parents:
|
|
self._add_container_actions(parent)
|
|
try:
|
|
defaults = parent._defaults
|
|
except AttributeError:
|
|
pass
|
|
else:
|
|
self._defaults.update(defaults)
|
|
|
|
# =======================
|
|
# Pretty __repr__ methods
|
|
# =======================
|
|
def _get_kwargs(self):
|
|
names = [
|
|
'prog',
|
|
'usage',
|
|
'description',
|
|
'version',
|
|
'formatter_class',
|
|
'conflict_handler',
|
|
'add_help',
|
|
]
|
|
return [(name, getattr(self, name)) for name in names]
|
|
|
|
# ==================================
|
|
# Optional/Positional adding methods
|
|
# ==================================
|
|
def add_subparsers(self, **kwargs):
|
|
if self._subparsers is not None:
|
|
self.error(_('cannot have multiple subparser arguments'))
|
|
|
|
# add the parser class to the arguments if it's not present
|
|
kwargs.setdefault('parser_class', type(self))
|
|
|
|
if 'title' in kwargs or 'description' in kwargs:
|
|
title = _(kwargs.pop('title', 'subcommands'))
|
|
description = _(kwargs.pop('description', None))
|
|
self._subparsers = self.add_argument_group(title, description)
|
|
else:
|
|
self._subparsers = self._positionals
|
|
|
|
# prog defaults to the usage message of this parser, skipping
|
|
# optional arguments and with no "usage:" prefix
|
|
if kwargs.get('prog') is None:
|
|
formatter = self._get_formatter()
|
|
positionals = self._get_positional_actions()
|
|
groups = self._mutually_exclusive_groups
|
|
formatter.add_usage(self.usage, positionals, groups, '')
|
|
kwargs['prog'] = formatter.format_help().strip()
|
|
|
|
# create the parsers action and add it to the positionals list
|
|
parsers_class = self._pop_action_class(kwargs, 'parsers')
|
|
action = parsers_class(option_strings=[], **kwargs)
|
|
self._subparsers._add_action(action)
|
|
|
|
# return the created parsers action
|
|
return action
|
|
|
|
def _add_action(self, action):
|
|
if action.option_strings:
|
|
self._optionals._add_action(action)
|
|
else:
|
|
self._positionals._add_action(action)
|
|
return action
|
|
|
|
def _get_optional_actions(self):
|
|
return [action
|
|
for action in self._actions
|
|
if action.option_strings]
|
|
|
|
def _get_positional_actions(self):
|
|
return [action
|
|
for action in self._actions
|
|
if not action.option_strings]
|
|
|
|
# =====================================
|
|
# Command line argument parsing methods
|
|
# =====================================
|
|
def parse_args(self, args=None, namespace=None):
|
|
args, argv = self.parse_known_args(args, namespace)
|
|
if argv:
|
|
msg = _('unrecognized arguments: %s')
|
|
self.error(msg % ' '.join(argv))
|
|
return args
|
|
|
|
def parse_known_args(self, args=None, namespace=None):
|
|
if args is None:
|
|
# args default to the system args
|
|
args = _sys.argv[1:]
|
|
else:
|
|
# make sure that args are mutable
|
|
args = list(args)
|
|
|
|
# default Namespace built from parser defaults
|
|
if namespace is None:
|
|
namespace = Namespace()
|
|
|
|
# add any action defaults that aren't present
|
|
for action in self._actions:
|
|
if action.dest is not SUPPRESS:
|
|
if not hasattr(namespace, action.dest):
|
|
if action.default is not SUPPRESS:
|
|
setattr(namespace, action.dest, action.default)
|
|
|
|
# add any parser defaults that aren't present
|
|
for dest in self._defaults:
|
|
if not hasattr(namespace, dest):
|
|
setattr(namespace, dest, self._defaults[dest])
|
|
|
|
# parse the arguments and exit if there are any errors
|
|
try:
|
|
namespace, args = self._parse_known_args(args, namespace)
|
|
if hasattr(namespace, _UNRECOGNIZED_ARGS_ATTR):
|
|
args.extend(getattr(namespace, _UNRECOGNIZED_ARGS_ATTR))
|
|
delattr(namespace, _UNRECOGNIZED_ARGS_ATTR)
|
|
return namespace, args
|
|
except ArgumentError:
|
|
err = _sys.exc_info()[1]
|
|
self.error(str(err))
|
|
|
|
def _parse_known_args(self, arg_strings, namespace):
|
|
# replace arg strings that are file references
|
|
if self.fromfile_prefix_chars is not None:
|
|
arg_strings = self._read_args_from_files(arg_strings)
|
|
|
|
# map all mutually exclusive arguments to the other arguments
|
|
# they can't occur with
|
|
action_conflicts = {}
|
|
for mutex_group in self._mutually_exclusive_groups:
|
|
group_actions = mutex_group._group_actions
|
|
for i, mutex_action in enumerate(mutex_group._group_actions):
|
|
conflicts = action_conflicts.setdefault(mutex_action, [])
|
|
conflicts.extend(group_actions[:i])
|
|
conflicts.extend(group_actions[i + 1:])
|
|
|
|
# find all option indices, and determine the arg_string_pattern
|
|
# which has an 'O' if there is an option at an index,
|
|
# an 'A' if there is an argument, or a '-' if there is a '--'
|
|
option_string_indices = {}
|
|
arg_string_pattern_parts = []
|
|
arg_strings_iter = iter(arg_strings)
|
|
for i, arg_string in enumerate(arg_strings_iter):
|
|
|
|
# all args after -- are non-options
|
|
if arg_string == '--':
|
|
arg_string_pattern_parts.append('-')
|
|
for arg_string in arg_strings_iter:
|
|
arg_string_pattern_parts.append('A')
|
|
|
|
# otherwise, add the arg to the arg strings
|
|
# and note the index if it was an option
|
|
else:
|
|
option_tuple = self._parse_optional(arg_string)
|
|
if option_tuple is None:
|
|
pattern = 'A'
|
|
else:
|
|
option_string_indices[i] = option_tuple
|
|
pattern = 'O'
|
|
arg_string_pattern_parts.append(pattern)
|
|
|
|
# join the pieces together to form the pattern
|
|
arg_strings_pattern = ''.join(arg_string_pattern_parts)
|
|
|
|
# converts arg strings to the appropriate and then takes the action
|
|
seen_actions = set()
|
|
seen_non_default_actions = set()
|
|
|
|
def take_action(action, argument_strings, option_string=None):
|
|
seen_actions.add(action)
|
|
argument_values = self._get_values(action, argument_strings)
|
|
|
|
# error if this argument is not allowed with other previously
|
|
# seen arguments, assuming that actions that use the default
|
|
# value don't really count as "present"
|
|
if argument_values is not action.default:
|
|
seen_non_default_actions.add(action)
|
|
for conflict_action in action_conflicts.get(action, []):
|
|
if conflict_action in seen_non_default_actions:
|
|
msg = _('not allowed with argument %s')
|
|
action_name = _get_action_name(conflict_action)
|
|
raise ArgumentError(action, msg % action_name)
|
|
|
|
# take the action if we didn't receive a SUPPRESS value
|
|
# (e.g. from a default)
|
|
if argument_values is not SUPPRESS:
|
|
action(self, namespace, argument_values, option_string)
|
|
|
|
# function to convert arg_strings into an optional action
|
|
def consume_optional(start_index):
|
|
|
|
# get the optional identified at this index
|
|
option_tuple = option_string_indices[start_index]
|
|
action, option_string, explicit_arg = option_tuple
|
|
|
|
# identify additional optionals in the same arg string
|
|
# (e.g. -xyz is the same as -x -y -z if no args are required)
|
|
match_argument = self._match_argument
|
|
action_tuples = []
|
|
while True:
|
|
|
|
# if we found no optional action, skip it
|
|
if action is None:
|
|
extras.append(arg_strings[start_index])
|
|
return start_index + 1
|
|
|
|
# if there is an explicit argument, try to match the
|
|
# optional's string arguments to only this
|
|
if explicit_arg is not None:
|
|
arg_count = match_argument(action, 'A')
|
|
|
|
# if the action is a single-dash option and takes no
|
|
# arguments, try to parse more single-dash options out
|
|
# of the tail of the option string
|
|
chars = self.prefix_chars
|
|
if arg_count == 0 and option_string[1] not in chars:
|
|
action_tuples.append((action, [], option_string))
|
|
char = option_string[0]
|
|
option_string = char + explicit_arg[0]
|
|
new_explicit_arg = explicit_arg[1:] or None
|
|
optionals_map = self._option_string_actions
|
|
if option_string in optionals_map:
|
|
action = optionals_map[option_string]
|
|
explicit_arg = new_explicit_arg
|
|
else:
|
|
msg = _('ignored explicit argument %r')
|
|
raise ArgumentError(action, msg % explicit_arg)
|
|
|
|
# if the action expect exactly one argument, we've
|
|
# successfully matched the option; exit the loop
|
|
elif arg_count == 1:
|
|
stop = start_index + 1
|
|
args = [explicit_arg]
|
|
action_tuples.append((action, args, option_string))
|
|
break
|
|
|
|
# error if a double-dash option did not use the
|
|
# explicit argument
|
|
else:
|
|
msg = _('ignored explicit argument %r')
|
|
raise ArgumentError(action, msg % explicit_arg)
|
|
|
|
# if there is no explicit argument, try to match the
|
|
# optional's string arguments with the following strings
|
|
# if successful, exit the loop
|
|
else:
|
|
start = start_index + 1
|
|
selected_patterns = arg_strings_pattern[start:]
|
|
arg_count = match_argument(action, selected_patterns)
|
|
stop = start + arg_count
|
|
args = arg_strings[start:stop]
|
|
action_tuples.append((action, args, option_string))
|
|
break
|
|
|
|
# add the Optional to the list and return the index at which
|
|
# the Optional's string args stopped
|
|
assert action_tuples
|
|
for action, args, option_string in action_tuples:
|
|
take_action(action, args, option_string)
|
|
return stop
|
|
|
|
# the list of Positionals left to be parsed; this is modified
|
|
# by consume_positionals()
|
|
positionals = self._get_positional_actions()
|
|
|
|
# function to convert arg_strings into positional actions
|
|
def consume_positionals(start_index):
|
|
# match as many Positionals as possible
|
|
match_partial = self._match_arguments_partial
|
|
selected_pattern = arg_strings_pattern[start_index:]
|
|
arg_counts = match_partial(positionals, selected_pattern)
|
|
|
|
# slice off the appropriate arg strings for each Positional
|
|
# and add the Positional and its args to the list
|
|
for action, arg_count in zip(positionals, arg_counts):
|
|
args = arg_strings[start_index: start_index + arg_count]
|
|
start_index += arg_count
|
|
take_action(action, args)
|
|
|
|
# slice off the Positionals that we just parsed and return the
|
|
# index at which the Positionals' string args stopped
|
|
positionals[:] = positionals[len(arg_counts):]
|
|
return start_index
|
|
|
|
# consume Positionals and Optionals alternately, until we have
|
|
# passed the last option string
|
|
extras = []
|
|
start_index = 0
|
|
if option_string_indices:
|
|
max_option_string_index = max(option_string_indices)
|
|
else:
|
|
max_option_string_index = -1
|
|
while start_index <= max_option_string_index:
|
|
|
|
# consume any Positionals preceding the next option
|
|
next_option_string_index = min([
|
|
index
|
|
for index in option_string_indices
|
|
if index >= start_index])
|
|
if start_index != next_option_string_index:
|
|
positionals_end_index = consume_positionals(start_index)
|
|
|
|
# only try to parse the next optional if we didn't consume
|
|
# the option string during the positionals parsing
|
|
if positionals_end_index > start_index:
|
|
start_index = positionals_end_index
|
|
continue
|
|
else:
|
|
start_index = positionals_end_index
|
|
|
|
# if we consumed all the positionals we could and we're not
|
|
# at the index of an option string, there were extra arguments
|
|
if start_index not in option_string_indices:
|
|
strings = arg_strings[start_index:next_option_string_index]
|
|
extras.extend(strings)
|
|
start_index = next_option_string_index
|
|
|
|
# consume the next optional and any arguments for it
|
|
start_index = consume_optional(start_index)
|
|
|
|
# consume any positionals following the last Optional
|
|
stop_index = consume_positionals(start_index)
|
|
|
|
# if we didn't consume all the argument strings, there were extras
|
|
extras.extend(arg_strings[stop_index:])
|
|
|
|
# if we didn't use all the Positional objects, there were too few
|
|
# arg strings supplied.
|
|
if positionals:
|
|
self.error(_('too few arguments'))
|
|
|
|
# make sure all required actions were present, and convert defaults.
|
|
for action in self._actions:
|
|
if action not in seen_actions:
|
|
if action.required:
|
|
name = _get_action_name(action)
|
|
self.error(_('argument %s is required') % name)
|
|
else:
|
|
# Convert action default now instead of doing it before
|
|
# parsing arguments to avoid calling convert functions
|
|
# twice (which may fail) if the argument was given, but
|
|
# only if it was defined already in the namespace
|
|
if (action.default is not None and
|
|
isinstance(action.default, basestring) and
|
|
hasattr(namespace, action.dest) and
|
|
action.default is getattr(namespace, action.dest)):
|
|
setattr(namespace, action.dest,
|
|
self._get_value(action, action.default))
|
|
|
|
# make sure all required groups had one option present
|
|
for group in self._mutually_exclusive_groups:
|
|
if group.required:
|
|
for action in group._group_actions:
|
|
if action in seen_non_default_actions:
|
|
break
|
|
|
|
# if no actions were used, report the error
|
|
else:
|
|
names = [_get_action_name(action)
|
|
for action in group._group_actions
|
|
if action.help is not SUPPRESS]
|
|
msg = _('one of the arguments %s is required')
|
|
self.error(msg % ' '.join(names))
|
|
|
|
# return the updated namespace and the extra arguments
|
|
return namespace, extras
|
|
|
|
def _read_args_from_files(self, arg_strings):
|
|
# expand arguments referencing files
|
|
new_arg_strings = []
|
|
for arg_string in arg_strings:
|
|
|
|
# for regular arguments, just add them back into the list
|
|
if not arg_string or arg_string[0] not in self.fromfile_prefix_chars:
|
|
new_arg_strings.append(arg_string)
|
|
|
|
# replace arguments referencing files with the file content
|
|
else:
|
|
try:
|
|
args_file = open(arg_string[1:])
|
|
try:
|
|
arg_strings = []
|
|
for arg_line in args_file.read().splitlines():
|
|
for arg in self.convert_arg_line_to_args(arg_line):
|
|
arg_strings.append(arg)
|
|
arg_strings = self._read_args_from_files(arg_strings)
|
|
new_arg_strings.extend(arg_strings)
|
|
finally:
|
|
args_file.close()
|
|
except IOError:
|
|
err = _sys.exc_info()[1]
|
|
self.error(str(err))
|
|
|
|
# return the modified argument list
|
|
return new_arg_strings
|
|
|
|
def convert_arg_line_to_args(self, arg_line):
|
|
return [arg_line]
|
|
|
|
def _match_argument(self, action, arg_strings_pattern):
|
|
# match the pattern for this action to the arg strings
|
|
nargs_pattern = self._get_nargs_pattern(action)
|
|
match = _re.match(nargs_pattern, arg_strings_pattern)
|
|
|
|
# raise an exception if we weren't able to find a match
|
|
if match is None:
|
|
nargs_errors = {
|
|
None: _('expected one argument'),
|
|
OPTIONAL: _('expected at most one argument'),
|
|
ONE_OR_MORE: _('expected at least one argument'),
|
|
}
|
|
default = _('expected %s argument(s)') % action.nargs
|
|
msg = nargs_errors.get(action.nargs, default)
|
|
raise ArgumentError(action, msg)
|
|
|
|
# return the number of arguments matched
|
|
return len(match.group(1))
|
|
|
|
def _match_arguments_partial(self, actions, arg_strings_pattern):
|
|
# progressively shorten the actions list by slicing off the
|
|
# final actions until we find a match
|
|
result = []
|
|
for i in range(len(actions), 0, -1):
|
|
actions_slice = actions[:i]
|
|
pattern = ''.join([self._get_nargs_pattern(action)
|
|
for action in actions_slice])
|
|
match = _re.match(pattern, arg_strings_pattern)
|
|
if match is not None:
|
|
result.extend([len(string) for string in match.groups()])
|
|
break
|
|
|
|
# return the list of arg string counts
|
|
return result
|
|
|
|
def _parse_optional(self, arg_string):
|
|
# if it's an empty string, it was meant to be a positional
|
|
if not arg_string:
|
|
return None
|
|
|
|
# if it doesn't start with a prefix, it was meant to be positional
|
|
if not arg_string[0] in self.prefix_chars:
|
|
return None
|
|
|
|
# if the option string is present in the parser, return the action
|
|
if arg_string in self._option_string_actions:
|
|
action = self._option_string_actions[arg_string]
|
|
return action, arg_string, None
|
|
|
|
# if it's just a single character, it was meant to be positional
|
|
if len(arg_string) == 1:
|
|
return None
|
|
|
|
# if the option string before the "=" is present, return the action
|
|
if '=' in arg_string:
|
|
option_string, explicit_arg = arg_string.split('=', 1)
|
|
if option_string in self._option_string_actions:
|
|
action = self._option_string_actions[option_string]
|
|
return action, option_string, explicit_arg
|
|
|
|
# search through all possible prefixes of the option string
|
|
# and all actions in the parser for possible interpretations
|
|
option_tuples = self._get_option_tuples(arg_string)
|
|
|
|
# if multiple actions match, the option string was ambiguous
|
|
if len(option_tuples) > 1:
|
|
options = ', '.join([option_string
|
|
for action, option_string, explicit_arg in option_tuples])
|
|
tup = arg_string, options
|
|
self.error(_('ambiguous option: %s could match %s') % tup)
|
|
|
|
# if exactly one action matched, this segmentation is good,
|
|
# so return the parsed action
|
|
elif len(option_tuples) == 1:
|
|
option_tuple, = option_tuples
|
|
return option_tuple
|
|
|
|
# if it was not found as an option, but it looks like a negative
|
|
# number, it was meant to be positional
|
|
# unless there are negative-number-like options
|
|
if self._negative_number_matcher.match(arg_string):
|
|
if not self._has_negative_number_optionals:
|
|
return None
|
|
|
|
# if it contains a space, it was meant to be a positional
|
|
if ' ' in arg_string:
|
|
return None
|
|
|
|
# it was meant to be an optional but there is no such option
|
|
# in this parser (though it might be a valid option in a subparser)
|
|
return None, arg_string, None
|
|
|
|
def _get_option_tuples(self, option_string):
|
|
result = []
|
|
|
|
# option strings starting with two prefix characters are only
|
|
# split at the '='
|
|
chars = self.prefix_chars
|
|
if option_string[0] in chars and option_string[1] in chars:
|
|
if '=' in option_string:
|
|
option_prefix, explicit_arg = option_string.split('=', 1)
|
|
else:
|
|
option_prefix = option_string
|
|
explicit_arg = None
|
|
for option_string in self._option_string_actions:
|
|
if option_string.startswith(option_prefix):
|
|
action = self._option_string_actions[option_string]
|
|
tup = action, option_string, explicit_arg
|
|
result.append(tup)
|
|
|
|
# single character options can be concatenated with their arguments
|
|
# but multiple character options always have to have their argument
|
|
# separate
|
|
elif option_string[0] in chars and option_string[1] not in chars:
|
|
option_prefix = option_string
|
|
explicit_arg = None
|
|
short_option_prefix = option_string[:2]
|
|
short_explicit_arg = option_string[2:]
|
|
|
|
for option_string in self._option_string_actions:
|
|
if option_string == short_option_prefix:
|
|
action = self._option_string_actions[option_string]
|
|
tup = action, option_string, short_explicit_arg
|
|
result.append(tup)
|
|
elif option_string.startswith(option_prefix):
|
|
action = self._option_string_actions[option_string]
|
|
tup = action, option_string, explicit_arg
|
|
result.append(tup)
|
|
|
|
# shouldn't ever get here
|
|
else:
|
|
self.error(_('unexpected option string: %s') % option_string)
|
|
|
|
# return the collected option tuples
|
|
return result
|
|
|
|
def _get_nargs_pattern(self, action):
|
|
# in all examples below, we have to allow for '--' args
|
|
# which are represented as '-' in the pattern
|
|
nargs = action.nargs
|
|
|
|
# the default (None) is assumed to be a single argument
|
|
if nargs is None:
|
|
nargs_pattern = '(-*A-*)'
|
|
|
|
# allow zero or one arguments
|
|
elif nargs == OPTIONAL:
|
|
nargs_pattern = '(-*A?-*)'
|
|
|
|
# allow zero or more arguments
|
|
elif nargs == ZERO_OR_MORE:
|
|
nargs_pattern = '(-*[A-]*)'
|
|
|
|
# allow one or more arguments
|
|
elif nargs == ONE_OR_MORE:
|
|
nargs_pattern = '(-*A[A-]*)'
|
|
|
|
# allow any number of options or arguments
|
|
elif nargs == REMAINDER:
|
|
nargs_pattern = '([-AO]*)'
|
|
|
|
# allow one argument followed by any number of options or arguments
|
|
elif nargs == PARSER:
|
|
nargs_pattern = '(-*A[-AO]*)'
|
|
|
|
# all others should be integers
|
|
else:
|
|
nargs_pattern = '(-*%s-*)' % '-*'.join('A' * nargs)
|
|
|
|
# if this is an optional action, -- is not allowed
|
|
if action.option_strings:
|
|
nargs_pattern = nargs_pattern.replace('-*', '')
|
|
nargs_pattern = nargs_pattern.replace('-', '')
|
|
|
|
# return the pattern
|
|
return nargs_pattern
|
|
|
|
# ========================
|
|
# Value conversion methods
|
|
# ========================
|
|
def _get_values(self, action, arg_strings):
|
|
# for everything but PARSER, REMAINDER args, strip out first '--'
|
|
if action.nargs not in [PARSER, REMAINDER]:
|
|
try:
|
|
arg_strings.remove('--')
|
|
except ValueError:
|
|
pass
|
|
|
|
# optional argument produces a default when not present
|
|
if not arg_strings and action.nargs == OPTIONAL:
|
|
if action.option_strings:
|
|
value = action.const
|
|
else:
|
|
value = action.default
|
|
if isinstance(value, basestring):
|
|
value = self._get_value(action, value)
|
|
self._check_value(action, value)
|
|
|
|
# when nargs='*' on a positional, if there were no command-line
|
|
# args, use the default if it is anything other than None
|
|
elif (not arg_strings and action.nargs == ZERO_OR_MORE and
|
|
not action.option_strings):
|
|
if action.default is not None:
|
|
value = action.default
|
|
else:
|
|
value = arg_strings
|
|
self._check_value(action, value)
|
|
|
|
# single argument or optional argument produces a single value
|
|
elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]:
|
|
arg_string, = arg_strings
|
|
value = self._get_value(action, arg_string)
|
|
self._check_value(action, value)
|
|
|
|
# REMAINDER arguments convert all values, checking none
|
|
elif action.nargs == REMAINDER:
|
|
value = [self._get_value(action, v) for v in arg_strings]
|
|
|
|
# PARSER arguments convert all values, but check only the first
|
|
elif action.nargs == PARSER:
|
|
value = [self._get_value(action, v) for v in arg_strings]
|
|
self._check_value(action, value[0])
|
|
|
|
# all other types of nargs produce a list
|
|
else:
|
|
value = [self._get_value(action, v) for v in arg_strings]
|
|
for v in value:
|
|
self._check_value(action, v)
|
|
|
|
# return the converted value
|
|
return value
|
|
|
|
def _get_value(self, action, arg_string):
|
|
type_func = self._registry_get('type', action.type, action.type)
|
|
if not _callable(type_func):
|
|
msg = _('%r is not callable')
|
|
raise ArgumentError(action, msg % type_func)
|
|
|
|
# convert the value to the appropriate type
|
|
try:
|
|
result = type_func(arg_string)
|
|
|
|
# ArgumentTypeErrors indicate errors
|
|
except ArgumentTypeError:
|
|
name = getattr(action.type, '__name__', repr(action.type))
|
|
msg = str(_sys.exc_info()[1])
|
|
raise ArgumentError(action, msg)
|
|
|
|
# TypeErrors or ValueErrors also indicate errors
|
|
except (TypeError, ValueError):
|
|
name = getattr(action.type, '__name__', repr(action.type))
|
|
msg = _('invalid %s value: %r')
|
|
raise ArgumentError(action, msg % (name, arg_string))
|
|
|
|
# return the converted value
|
|
return result
|
|
|
|
def _check_value(self, action, value):
|
|
# converted value must be one of the choices (if specified)
|
|
if action.choices is not None and value not in action.choices:
|
|
tup = value, ', '.join(map(repr, action.choices))
|
|
msg = _('invalid choice: %r (choose from %s)') % tup
|
|
raise ArgumentError(action, msg)
|
|
|
|
# =======================
|
|
# Help-formatting methods
|
|
# =======================
|
|
def format_usage(self):
|
|
formatter = self._get_formatter()
|
|
formatter.add_usage(self.usage, self._actions,
|
|
self._mutually_exclusive_groups)
|
|
return formatter.format_help()
|
|
|
|
def format_help(self):
|
|
formatter = self._get_formatter()
|
|
|
|
# usage
|
|
formatter.add_usage(self.usage, self._actions,
|
|
self._mutually_exclusive_groups)
|
|
|
|
# description
|
|
formatter.add_text(self.description)
|
|
|
|
# positionals, optionals and user-defined groups
|
|
for action_group in self._action_groups:
|
|
formatter.start_section(action_group.title)
|
|
formatter.add_text(action_group.description)
|
|
formatter.add_arguments(action_group._group_actions)
|
|
formatter.end_section()
|
|
|
|
# epilog
|
|
formatter.add_text(self.epilog)
|
|
|
|
# determine help from format above
|
|
return formatter.format_help()
|
|
|
|
def format_version(self):
|
|
import warnings
|
|
warnings.warn(
|
|
'The format_version method is deprecated -- the "version" '
|
|
'argument to ArgumentParser is no longer supported.',
|
|
DeprecationWarning)
|
|
formatter = self._get_formatter()
|
|
formatter.add_text(self.version)
|
|
return formatter.format_help()
|
|
|
|
def _get_formatter(self):
|
|
return self.formatter_class(prog=self.prog)
|
|
|
|
# =====================
|
|
# Help-printing methods
|
|
# =====================
|
|
def print_usage(self, file=None):
|
|
if file is None:
|
|
file = _sys.stdout
|
|
self._print_message(self.format_usage(), file)
|
|
|
|
def print_help(self, file=None):
|
|
if file is None:
|
|
file = _sys.stdout
|
|
self._print_message(self.format_help(), file)
|
|
|
|
def print_version(self, file=None):
|
|
import warnings
|
|
warnings.warn(
|
|
'The print_version method is deprecated -- the "version" '
|
|
'argument to ArgumentParser is no longer supported.',
|
|
DeprecationWarning)
|
|
self._print_message(self.format_version(), file)
|
|
|
|
def _print_message(self, message, file=None):
|
|
if message:
|
|
if file is None:
|
|
file = _sys.stderr
|
|
file.write(message)
|
|
|
|
# ===============
|
|
# Exiting methods
|
|
# ===============
|
|
def exit(self, status=0, message=None):
|
|
if message:
|
|
self._print_message(message, _sys.stderr)
|
|
_sys.exit(status)
|
|
|
|
def error(self, message):
|
|
"""error(message: string)
|
|
|
|
Prints a usage message incorporating the message to stderr and
|
|
exits.
|
|
|
|
If you override this in a subclass, it should not return -- it
|
|
should either exit or raise an exception.
|
|
"""
|
|
self.print_usage(_sys.stderr)
|
|
self.exit(2, _('%s: error: %s\n') % (self.prog, message))
|