ada/cpython
0
0
Fork 0
mirror of https://github.com/AdaCore/cpython.git synced 2026-02-12 12:57:15 -08:00
Code Issues Packages Projects Releases Wiki Activity

PEP 489: Multi-phase extension module initialization

Browse Source 
Known limitations of the current implementation:

- documentation changes are incomplete
- there's a reference leak I haven't tracked down yet

The leak is most visible by running:

  ./python -m test -R3:3 test_importlib

However, you can also see it by running:

  ./python -X showrefcount

Importing the array or _testmultiphase modules, and
then deleting them from both sys.modules and the local
namespace shows significant increases in the total
number of active references each cycle. By contrast,
with _testcapi (which continues to use single-phase
initialisation) the global refcounts stabilise after
a couple of cycles.
This commit is contained in:
Nick Coghlan
2015-05-23 22:24:10 +10:00
parent ec219ba1c0
commit d5cacbb1d9
34 changed files with 4462 additions and 3124 deletions
Download Patch File Download Diff File Expand all files Collapse all files

92
Doc/c-api/module.rst
View File

@@ -7,8 +7,6 @@ Module Objects
.. index:: object: module
There are only a few functions special to module objects.
.. c:var:: PyTypeObject PyModule_Type
@@ -109,6 +107,14 @@ There are only a few functions special to module objects.
unencodable filenames, use :c:func:`PyModule_GetFilenameObject` instead.
Per-interpreter module state
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Single-phase initialization creates singleton modules that can store additional
information as part of the interpreter, allow that state to be retrieved later
with only a reference to the module definition, rather than to the module
itself.
.. c:function:: void* PyModule_GetState(PyObject *module)
Return the "state" of the module, that is, a pointer to the block of memory
@@ -146,27 +152,6 @@ There are only a few functions special to module objects.
Initializing C modules
^^^^^^^^^^^^^^^^^^^^^^
These functions are usually used in the module initialization function.
.. c:function:: PyObject* PyModule_Create(PyModuleDef *module)
Create a new module object, given the definition in *module*. This behaves
like :c:func:`PyModule_Create2` with *module_api_version* set to
:const:`PYTHON_API_VERSION`.
.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version)
Create a new module object, given the definition in *module*, assuming the
API version *module_api_version*. If that version does not match the version
of the running interpreter, a :exc:`RuntimeWarning` is emitted.
.. note::
Most uses of this function should be using :c:func:`PyModule_Create`
instead; only use this if you are sure you need it.
.. c:type:: PyModuleDef
This struct holds all information that is needed to create a module object.
@@ -210,9 +195,10 @@ These functions are usually used in the module initialization function.
A pointer to a table of module-level functions, described by
:c:type:`PyMethodDef` values. Can be *NULL* if no functions are present.
.. c:member:: inquiry m_reload
.. c:member:: PyModuleDef_Slot* m_slots
Currently unused, should be *NULL*.
An array of slot definitions for multi-phase initialization, terminated by
a *NULL* entry.
.. c:member:: traverseproc m_traverse
@@ -229,6 +215,61 @@ These functions are usually used in the module initialization function.
A function to call during deallocation of the module object, or *NULL* if
not needed.
The module initialization function may create and return the module object
directly. This is referred to as "single-phase initialization", and uses one
of the following two module creation functions:
.. c:function:: PyObject* PyModule_Create(PyModuleDef *module)
Create a new module object, given the definition in *module*. This behaves
like :c:func:`PyModule_Create2` with *module_api_version* set to
:const:`PYTHON_API_VERSION`.
.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version)
Create a new module object, given the definition in *module*, assuming the
API version *module_api_version*. If that version does not match the version
of the running interpreter, a :exc:`RuntimeWarning` is emitted.
.. note::
Most uses of this function should be using :c:func:`PyModule_Create`
instead; only use this if you are sure you need it.
Alternatively, the module initialization function may instead return a
:c:type:`PyModuleDef` instance with a non-empty ``m_slots`` array. This is
referred to as "multi-phase initialization", and ``PyModuleDef`` instance
should be initialized with the following function:
.. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *module)
Ensures a module definition is a properly initialized Python object that
correctly reports its type and reference count.
.. XXX (ncoghlan): It's not clear if it makes sense to document PyModule_ExecDef
PyModule_FromDefAndSpec or PyModule_FromDefAndSpec2 here, as end user code
generally shouldn't be calling those.
The module initialization function (if using single phase initialization) or
a function called from a module execution slot (if using multiphase
initialization), can use the following functions to help initialize the module
state:
.. c:function:: int PyModule_SetDocString(PyObject *module, const char *docstring)
Set the docstring for *module* to *docstring*. Return ``-1`` on error, ``0``
on success.
.. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions)
Add the functions from the ``NULL`` terminated *functions* array to *module*.
Refer to the :c:type:`PyMethodDef` documentation for details on individual
entries (due to the lack of a shared module namespace, module level
"functions" implemented in C typically receive the module as their first
parameter, making them similar to instance methods on Python classes).
.. c:function:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value)
@@ -236,7 +277,6 @@ These functions are usually used in the module initialization function.
be used from the module's initialization function. This steals a reference to
*value*. Return ``-1`` on error, ``0`` on success.
.. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value)
Add an integer constant to *module* as *name*. This convenience function can be

28
Doc/library/importlib.rst
View File

@@ -55,6 +55,12 @@ generically as an :term:`importer`) to participate in the import process.
:pep:`451`
A ModuleSpec Type for the Import System
:pep:`488`
Elimination of PYO files
:pep:`489`
Multi-phase extension module initialization
:pep:`3120`
Using UTF-8 as the Default Source Encoding
@@ -756,9 +762,9 @@ find and load modules.
Only class methods are defined by this class to alleviate the need for
instantiation.
.. note::
Due to limitations in the extension module C-API, for now
BuiltinImporter does not implement :meth:`Loader.exec_module`.
.. versionchanged:: 3.5
As part of :pep:`489`, the builtin importer now implements
:meth:`Loader.create_module` and :meth:`Loader.exec_module`
.. class:: FrozenImporter
@@ -973,14 +979,18 @@ find and load modules.
Path to the extension module.
.. method:: load_module(name=None)
.. method:: create_module(spec)
Loads the extension module if and only if *fullname* is the same as
:attr:`name` or is ``None``.
Creates the module object from the given specification in accordance
with :pep:`489`.
.. note::
Due to limitations in the extension module C-API, for now
ExtensionFileLoader does not implement :meth:`Loader.exec_module`.
.. versionadded:: 3.5
.. method:: exec_module(module)
Initializes the given module object in accordance with :pep:`489`.
.. versionadded:: 3.5
.. method:: is_package(fullname)

19
Doc/whatsnew/3.5.rst
View File

@@ -93,6 +93,7 @@ Implementation improvements:
(:issue:`19977`).
* :pep:`488`, the elimination of ``.pyo`` files.
* :pep:`489`, multi-phase initialization of extension modules.
Significantly Improved Library Modules:
@@ -265,6 +266,21 @@ updated API to help with this change.
:pep:`488` -- Elimination of PYO files
PEP 489: Multi-phase extension module initialization
----------------------------------------------------
:pep:`489` updates extension module initialization to take advantage of the
two step module loading mechanism introduced by :pep:`451` in Python 3.4.
This change brings the import semantics of extension modules that opt-in to
using the new mechanism much closer to those of Python source and bytecode
modules, including the ability to any valid identifier as a module name,
rather than being restricted to ASCII.
.. seealso::
:pep:`488` -- Multi-phase extension module initialization
Other Language Changes
======================
@@ -667,7 +683,7 @@ time
tkinter
-------
* The :module:`tkinter._fix` module used for setting up the Tcl/Tk environment
* The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment
on Windows has been replaced by a private function in the :module:`_tkinter`
module which makes no permanent changes to environment variables.
(Contributed by Zachary Ware in :issue:`20035`.)
@@ -1012,7 +1028,6 @@ Changes in the Python API
program depends on patching the module level variable to capture the debug
output, you will need to update it to capture sys.stderr instead.
Changes in the C API
--------------------

62
Include/modsupport.h
View File

@@ -12,13 +12,13 @@ extern "C" {
/* If PY_SSIZE_T_CLEAN is defined, each functions treats #-specifier
to mean Py_ssize_t */
#ifdef PY_SSIZE_T_CLEAN
#define PyArg_Parse _PyArg_Parse_SizeT
#define PyArg_ParseTuple _PyArg_ParseTuple_SizeT
#define PyArg_ParseTupleAndKeywords _PyArg_ParseTupleAndKeywords_SizeT
#define PyArg_VaParse _PyArg_VaParse_SizeT
#define PyArg_VaParseTupleAndKeywords _PyArg_VaParseTupleAndKeywords_SizeT
#define Py_BuildValue _Py_BuildValue_SizeT
#define Py_VaBuildValue _Py_VaBuildValue_SizeT
#define PyArg_Parse _PyArg_Parse_SizeT
#define PyArg_ParseTuple _PyArg_ParseTuple_SizeT
#define PyArg_ParseTupleAndKeywords _PyArg_ParseTupleAndKeywords_SizeT
#define PyArg_VaParse _PyArg_VaParse_SizeT
#define PyArg_VaParseTupleAndKeywords _PyArg_VaParseTupleAndKeywords_SizeT
#define Py_BuildValue _Py_BuildValue_SizeT
#define Py_VaBuildValue _Py_VaBuildValue_SizeT
#else
PyAPI_FUNC(PyObject *) _Py_VaBuildValue_SizeT(const char *, va_list);
#endif
@@ -49,6 +49,9 @@ PyAPI_FUNC(int) PyModule_AddIntConstant(PyObject *, const char *, long);
PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char *);
#define PyModule_AddIntMacro(m, c) PyModule_AddIntConstant(m, #c, c)
#define PyModule_AddStringMacro(m, c) PyModule_AddStringConstant(m, #c, c)
PyAPI_FUNC(int) PyModule_SetDocString(PyObject *, const char *);
PyAPI_FUNC(int) PyModule_AddFunctions(PyObject *, PyMethodDef *);
PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def);
#define Py_CLEANUP_SUPPORTED 0x20000
@@ -67,35 +70,35 @@ PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char
Please add a line or two to the top of this log for each API
version change:
22-Feb-2006 MvL 1013 PEP 353 - long indices for sequence lengths
22-Feb-2006 MvL 1013 PEP 353 - long indices for sequence lengths
19-Aug-2002 GvR 1012 Changes to string object struct for
interning changes, saving 3 bytes.
19-Aug-2002 GvR 1012 Changes to string object struct for
interning changes, saving 3 bytes.
17-Jul-2001 GvR 1011 Descr-branch, just to be on the safe side
17-Jul-2001 GvR 1011 Descr-branch, just to be on the safe side
25-Jan-2001 FLD 1010 Parameters added to PyCode_New() and
PyFrame_New(); Python 2.1a2
14-Mar-2000 GvR 1009 Unicode API added
3-Jan-1999 GvR 1007 Decided to change back! (Don't reuse 1008!)
3-Jan-1999 GvR 1007 Decided to change back! (Don't reuse 1008!)
3-Dec-1998 GvR 1008 Python 1.5.2b1
3-Dec-1998 GvR 1008 Python 1.5.2b1
18-Jan-1997 GvR 1007 string interning and other speedups
18-Jan-1997 GvR 1007 string interning and other speedups
11-Oct-1996 GvR renamed Py_Ellipses to Py_Ellipsis :-(
11-Oct-1996 GvR renamed Py_Ellipses to Py_Ellipsis :-(
30-Jul-1996 GvR Slice and ellipses syntax added
30-Jul-1996 GvR Slice and ellipses syntax added
23-Jul-1996 GvR For 1.4 -- better safe than sorry this time :-)
23-Jul-1996 GvR For 1.4 -- better safe than sorry this time :-)
7-Nov-1995 GvR Keyword arguments (should've been done at 1.3 :-( )
7-Nov-1995 GvR Keyword arguments (should've been done at 1.3 :-( )
10-Jan-1995 GvR Renamed globals to new naming scheme
10-Jan-1995 GvR Renamed globals to new naming scheme
9-Jan-1995 GvR Initial version (incompatible with older API)
9-Jan-1995 GvR Initial version (incompatible with older API)
*/
/* The PYTHON_ABI_VERSION is introduced in PEP 384. For the lifetime of
@@ -105,10 +108,11 @@ PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char
#define PYTHON_ABI_STRING "3"
#ifdef Py_TRACE_REFS
/* When we are tracing reference counts, rename PyModule_Create2 so
/* When we are tracing reference counts, rename module creation functions so
modules compiled with incompatible settings will generate a
link-time error. */
#define PyModule_Create2 PyModule_Create2TraceRefs
#define PyModule_FromDefAndSpec2 PyModule_FromDefAndSpec2TraceRefs
#endif
PyAPI_FUNC(PyObject *) PyModule_Create2(struct PyModuleDef*,
@@ -116,10 +120,22 @@ PyAPI_FUNC(PyObject *) PyModule_Create2(struct PyModuleDef*,
#ifdef Py_LIMITED_API
#define PyModule_Create(module) \
PyModule_Create2(module, PYTHON_ABI_VERSION)
PyModule_Create2(module, PYTHON_ABI_VERSION)
#else
#define PyModule_Create(module) \
PyModule_Create2(module, PYTHON_API_VERSION)
PyModule_Create2(module, PYTHON_API_VERSION)
#endif
PyAPI_FUNC(PyObject *) PyModule_FromDefAndSpec2(PyModuleDef *def,
PyObject *spec,
int module_api_version);
#ifdef Py_LIMITED_API
#define PyModule_FromDefAndSpec(module, spec) \
PyModule_FromDefAndSpec2(module, spec, PYTHON_ABI_VERSION)
#else
#define PyModule_FromDefAndSpec(module, spec) \
PyModule_FromDefAndSpec2(module, spec, PYTHON_API_VERSION)
#endif
#ifndef Py_LIMITED_API

16
Include/moduleobject.h
View File

@@ -30,6 +30,9 @@ PyAPI_FUNC(void) _PyModule_ClearDict(PyObject *);
PyAPI_FUNC(struct PyModuleDef*) PyModule_GetDef(PyObject*);
PyAPI_FUNC(void*) PyModule_GetState(PyObject*);
PyAPI_FUNC(PyObject *) PyModuleDef_Init(struct PyModuleDef*);
PyTypeObject PyModuleDef_Type;
typedef struct PyModuleDef_Base {
PyObject_HEAD
PyObject* (*m_init)(void);
@@ -44,18 +47,29 @@ typedef struct PyModuleDef_Base {
NULL, /* m_copy */ \
}
typedef struct PyModuleDef_Slot{
int slot;
void *value;
} PyModuleDef_Slot;
typedef struct PyModuleDef{
PyModuleDef_Base m_base;
const char* m_name;
const char* m_doc;
Py_ssize_t m_size;
PyMethodDef *m_methods;
inquiry m_reload;
PyModuleDef_Slot* m_slots;
traverseproc m_traverse;
inquiry m_clear;
freefunc m_free;
}PyModuleDef;
#define Py_mod_create 1
#define Py_mod_exec 2
#ifndef Py_LIMITED_API
#define _Py_mod_LAST_SLOT 2
#endif
#ifdef __cplusplus
}

33
Lib/imp.py
View File

@@ -8,15 +8,15 @@ functionality over this module.
# (Probably) need to stay in _imp
from _imp import (lock_held, acquire_lock, release_lock,
get_frozen_object, is_frozen_package,
init_builtin, init_frozen, is_builtin, is_frozen,
init_frozen, is_builtin, is_frozen,
_fix_co_filename)
try:
from _imp import load_dynamic
from _imp import create_dynamic
except ImportError:
# Platform doesn't support dynamic loading.
load_dynamic = None
create_dynamic = None
from importlib._bootstrap import _ERR_MSG, _exec, _load
from importlib._bootstrap import _ERR_MSG, _exec, _load, _builtin_from_name
from importlib._bootstrap_external import SourcelessFileLoader
from importlib import machinery
@@ -312,3 +312,28 @@ def reload(module):
"""
return importlib.reload(module)
def init_builtin(name):
"""**DEPRECATED**
Load and return a built-in module by name, or None is such module doesn't
exist
"""
try:
return _builtin_from_name(name)
except ImportError:
return None
if create_dynamic:
def load_dynamic(name, path, file=None):
"""**DEPRECATED**
Load an extension module.
"""
import importlib.machinery
loader = importlib.machinery.ExtensionFileLoader(name, path)
return loader.load_module()
else:
load_dynamic = None

23
Lib/importlib/_bootstrap.py
View File

@@ -735,16 +735,17 @@ class BuiltinImporter:
return spec.loader if spec is not None else None
@classmethod
@_requires_builtin
def load_module(cls, fullname):
"""Load a built-in module."""
# Once an exec_module() implementation is added we can also
# add a deprecation warning here.
with _ManageReload(fullname):
module = _call_with_frames_removed(_imp.init_builtin, fullname)
module.__loader__ = cls
module.__package__ = ''
return module
def create_module(self, spec):
"""Create a built-in module"""
if spec.name not in sys.builtin_module_names:
raise ImportError('{!r} is not a built-in module'.format(spec.name),
name=spec.name)
return _call_with_frames_removed(_imp.create_builtin, spec)
@classmethod
def exec_module(self, module):
"""Exec a built-in module"""
_call_with_frames_removed(_imp.exec_dynamic, module)
@classmethod
@_requires_builtin
@@ -764,6 +765,8 @@ class BuiltinImporter:
"""Return False as built-in modules are never packages."""
return False
load_module = classmethod(_load_module_shim)
class FrozenImporter:

33
Lib/importlib/_bootstrap_external.py
View File

@@ -378,7 +378,8 @@ def _check_name(method):
if name is None:
name = self.name
elif self.name != name:
raise ImportError('loader cannot handle %s' % name, name=name)
raise ImportError('loader for %s cannot handle %s' %
(self.name, name), name=name)
return method(self, name, *args, **kwargs)
try:
_wrap = _bootstrap._wrap
@@ -875,7 +876,7 @@ class SourcelessFileLoader(FileLoader, _LoaderBasics):
EXTENSION_SUFFIXES = []
class ExtensionFileLoader:
class ExtensionFileLoader(FileLoader, _LoaderBasics):
"""Loader for extension modules.
@@ -894,24 +895,20 @@ class ExtensionFileLoader:
def __hash__(self):
return hash(self.name) ^ hash(self.path)
@_check_name
def load_module(self, fullname):
"""Load an extension module."""
# Once an exec_module() implementation is added we can also
# add a deprecation warning here.
with _bootstrap._ManageReload(fullname):
module = _bootstrap._call_with_frames_removed(_imp.load_dynamic,
fullname, self.path)
_verbose_message('extension module loaded from {!r}', self.path)
is_package = self.is_package(fullname)
if is_package and not hasattr(module, '__path__'):
module.__path__ = [_path_split(self.path)[0]]
module.__loader__ = self
module.__package__ = module.__name__
if not is_package:
module.__package__ = module.__package__.rpartition('.')[0]
def create_module(self, spec):
"""Create an unitialized extension module"""
module = _bootstrap._call_with_frames_removed(
_imp.create_dynamic, spec)
_verbose_message('extension module {!r} loaded from {!r}',
spec.name, self.path)
return module
def exec_module(self, module):
"""Initialize an extension module"""
_bootstrap._call_with_frames_removed(_imp.exec_dynamic, module)
_verbose_message('extension module {!r} executed from {!r}',
self.name, self.path)
def is_package(self, fullname):
"""Return True if the extension module is a package."""
file_name = _path_split(self.path)[1]

167
Lib/test/test_importlib/extension/test_loader.py
View File

@@ -7,6 +7,8 @@ import os.path
import sys
import types
import unittest
import importlib.util
import importlib
class LoaderTests(abc.LoaderTests):
@@ -80,6 +82,171 @@ class LoaderTests(abc.LoaderTests):
Source_LoaderTests
) = util.test_both(LoaderTests, machinery=machinery)
class MultiPhaseExtensionModuleTests(abc.LoaderTests):
"""Test loading extension modules with multi-phase initialization (PEP 489)
"""
def setUp(self):
self.name = '_testmultiphase'
finder = self.machinery.FileFinder(None)
self.spec = importlib.util.find_spec(self.name)
assert self.spec
self.loader = self.machinery.ExtensionFileLoader(
self.name, self.spec.origin)
# No extension module as __init__ available for testing.
test_package = None
# No extension module in a package available for testing.
test_lacking_parent = None
# Handling failure on reload is the up to the module.
test_state_after_failure = None
def test_module(self):
'''Test loading an extension module'''
with util.uncache(self.name):
module = self.load_module()
for attr, value in [('__name__', self.name),
('__file__', self.spec.origin),
('__package__', '')]:
self.assertEqual(getattr(module, attr), value)
with self.assertRaises(AttributeError):
module.__path__
self.assertIs(module, sys.modules[self.name])
self.assertIsInstance(module.__loader__,
self.machinery.ExtensionFileLoader)
def test_functionality(self):
'''Test basic functionality of stuff defined in an extension module'''
with util.uncache(self.name):
module = self.load_module()
self.assertIsInstance(module, types.ModuleType)
ex = module.Example()
self.assertEqual(ex.demo('abcd'), 'abcd')
self.assertEqual(ex.demo(), None)
with self.assertRaises(AttributeError):
ex.abc
ex.abc = 0
self.assertEqual(ex.abc, 0)
self.assertEqual(module.foo(9, 9), 18)
self.assertIsInstance(module.Str(), str)
self.assertEqual(module.Str(1) + '23', '123')
with self.assertRaises(module.error):
raise module.error()
self.assertEqual(module.int_const, 1969)
self.assertEqual(module.str_const, 'something different')
def test_reload(self):
'''Test that reload didn't re-set the module's attributes'''
with util.uncache(self.name):
module = self.load_module()
ex_class = module.Example
importlib.reload(module)
self.assertIs(ex_class, module.Example)
def test_try_registration(self):
'''Assert that the PyState_{Find,Add,Remove}Module C API doesn't work'''
module = self.load_module()
with self.subTest('PyState_FindModule'):
self.assertEqual(module.call_state_registration_func(0), None)
with self.subTest('PyState_AddModule'):
with self.assertRaises(SystemError):
module.call_state_registration_func(1)
with self.subTest('PyState_RemoveModule'):
with self.assertRaises(SystemError):
module.call_state_registration_func(2)
def load_module(self):
'''Load the module from the test extension'''
return self.loader.load_module(self.name)
def load_module_by_name(self, fullname):
'''Load a module from the test extension by name'''
origin = self.spec.origin
loader = self.machinery.ExtensionFileLoader(fullname, origin)
spec = importlib.util.spec_from_loader(fullname, loader)
module = importlib.util.module_from_spec(spec)
loader.exec_module(module)
return module
def test_load_twice(self):
'''Test that 2 loads result in 2 module objects'''
module1 = self.load_module_by_name(self.name)
module2 = self.load_module_by_name(self.name)
self.assertIsNot(module1, module2)
def test_unloadable(self):
'''Test nonexistent module'''
name = 'asdfjkl;'
with self.assertRaises(ImportError) as cm:
self.load_module_by_name(name)
self.assertEqual(cm.exception.name, name)
def test_unloadable_nonascii(self):
'''Test behavior with nonexistent module with non-ASCII name'''
name = 'fo\xf3'
with self.assertRaises(ImportError) as cm:
self.load_module_by_name(name)
self.assertEqual(cm.exception.name, name)
def test_nonmodule(self):
'''Test returning a non-module object from create works'''
name = self.name + '_nonmodule'
mod = self.load_module_by_name(name)
self.assertNotEqual(type(mod), type(unittest))
self.assertEqual(mod.three, 3)
def test_null_slots(self):
'''Test that NULL slots aren't a problem'''
name = self.name + '_null_slots'
module = self.load_module_by_name(name)
self.assertIsInstance(module, types.ModuleType)
assert module.__name__ == name
def test_bad_modules(self):
'''Test SystemError is raised for misbehaving extensions'''
for name_base in [
'bad_slot_large',
'bad_slot_negative',
'create_int_with_state',
'negative_size',
'export_null',
'export_uninitialized',
'export_raise',
'export_unreported_exception',
'create_null',
'create_raise',
'create_unreported_exception',
'nonmodule_with_exec_slots',
'exec_err',
'exec_raise',
'exec_unreported_exception',
]:
with self.subTest(name_base):
name = self.name + '_' + name_base
with self.assertRaises(SystemError):
self.load_module_by_name(name)
def test_nonascii(self):
'''Test that modules with non-ASCII names can be loaded'''
# punycode behaves slightly differently in some-ASCII and no-ASCII
# cases, so test both
cases = [
(self.name + '_zkou\u0161ka_na\u010dten\xed', 'Czech'),
('\uff3f\u30a4\u30f3\u30dd\u30fc\u30c8\u30c6\u30b9\u30c8',
'Japanese'),
]
for name, lang in cases:
with self.subTest(name):
module = self.load_module_by_name(name)
self.assertEqual(module.__name__, name)
self.assertEqual(module.__doc__, "Module named in %s" % lang)
(Frozen_MultiPhaseExtensionModuleTests,
Source_MultiPhaseExtensionModuleTests
) = util.test_both(MultiPhaseExtensionModuleTests, machinery=machinery)
if __name__ == '__main__':

1
Misc/ACKS
View File

@@ -1458,6 +1458,7 @@ Mike Verdone
Jaap Vermeulen
Nikita Vetoshkin
Al Vezza
Petr Victorin
Jacques A. Vidrine
John Viega
Dino Viehland

2
Misc/NEWS
View File

@@ -10,6 +10,8 @@ Release date: 2015-05-24
Core and Builtins
-----------------
- Issue #24268: PEP 489: Multi-phase extension module initialization
- Issue #23955: Add pyvenv.cfg option to suppress registry/environment
lookup for generating sys.path on Windows.

3
Modules/_testcapimodule.c
View File

@@ -4048,6 +4048,9 @@ static struct PyModuleDef _testcapimodule = {
NULL
};
/* Per PEP 489, this module will not be converted to multi-phase initialization
*/
PyMODINIT_FUNC
PyInit__testcapi(void)
{

567
Modules/_testmultiphase.c Normal file
View File

File diff suppressed because it is too large Load Diff

50
Modules/arraymodule.c
View File

@@ -2981,34 +2981,17 @@ static PyMethodDef a_methods[] = {
{NULL, NULL, 0, NULL} /* Sentinel */
};
static struct PyModuleDef arraymodule = {
PyModuleDef_HEAD_INIT,
"array",
module_doc,
-1,
a_methods,
NULL,
NULL,
NULL,
NULL
};
PyMODINIT_FUNC
PyInit_array(void)
static int
array_modexec(PyObject *m)
{
PyObject *m;
char buffer[Py_ARRAY_LENGTH(descriptors)], *p;
PyObject *typecodes;
Py_ssize_t size = 0;
struct arraydescr *descr;
if (PyType_Ready(&Arraytype) < 0)
return NULL;
return -1;
Py_TYPE(&PyArrayIter_Type) = &PyType_Type;
m = PyModule_Create(&arraymodule);
if (m == NULL)
return NULL;
Py_INCREF((PyObject *)&Arraytype);
PyModule_AddObject(m, "ArrayType", (PyObject *)&Arraytype);
@@ -3031,5 +3014,30 @@ PyInit_array(void)
Py_DECREF(m);
m = NULL;
}
return m;
return 0;
}
static PyModuleDef_Slot arrayslots[] = {
{Py_mod_exec, array_modexec},
{0, NULL}
};
static struct PyModuleDef arraymodule = {
PyModuleDef_HEAD_INIT,
"array",
module_doc,
0,
a_methods,
arrayslots,
NULL,
NULL,
NULL
};
PyMODINIT_FUNC
PyInit_array(void)
{
return PyModuleDef_Init(&arraymodule);
}

2
Modules/config.c.in
View File

@@ -13,7 +13,7 @@ redistribution of this file, and for a DISCLAIMER OF ALL WARRANTIES.
/* !!! !!! !!! This file is edited by the makesetup script !!! !!! !!! */
/* This file contains the table of built-in modules.
See init_builtin() in import.c. */
See create_builtin() in import.c. */
#include "Python.h"

55
Modules/xxlimited.c
View File

@@ -222,25 +222,9 @@ static PyMethodDef xx_methods[] = {
PyDoc_STRVAR(module_doc,
"This is a template module just for instruction.");
/* Initialization function for the module (*must* be called PyInit_xx) */
static struct PyModuleDef xxmodule = {
PyModuleDef_HEAD_INIT,
"xxlimited",
module_doc,
-1,
xx_methods,
NULL,
NULL,
NULL,
NULL
};
PyMODINIT_FUNC
PyInit_xxlimited(void)
static int
xx_modexec(PyObject *m)
{
PyObject *m = NULL;
PyObject *o;
/* Due to cross platform compiler issues the slots must be filled
@@ -254,11 +238,6 @@ PyInit_xxlimited(void)
if (Xxo_Type == NULL)
goto fail;
/* Create the module and add the functions */
m = PyModule_Create(&xxmodule);
if (m == NULL)
goto fail;
/* Add some symbolic constants to the module */
if (ErrorObject == NULL) {
ErrorObject = PyErr_NewException("xxlimited.error", NULL, NULL);
@@ -279,8 +258,34 @@ PyInit_xxlimited(void)
if (o == NULL)
goto fail;
PyModule_AddObject(m, "Null", o);
return m;
return 0;
fail:
Py_XDECREF(m);
return NULL;
return -1;
}
static PyModuleDef_Slot xx_slots[] = {
{Py_mod_exec, xx_modexec},
{0, NULL}
};
static struct PyModuleDef xxmodule = {
PyModuleDef_HEAD_INIT,
"xxlimited",
module_doc,
0,
xx_methods,
xx_slots,
NULL,
NULL,
NULL
};
/* Export function for the module (*must* be called PyInit_xx) */
PyMODINIT_FUNC
PyInit_xxlimited(void)
{
return PyModuleDef_Init(&xxmodule);
}

54
Modules/xxmodule.c
View File

@@ -334,26 +334,10 @@ static PyMethodDef xx_methods[] = {
PyDoc_STRVAR(module_doc,
"This is a template module just for instruction.");
/* Initialization function for the module (*must* be called PyInit_xx) */
static struct PyModuleDef xxmodule = {
PyModuleDef_HEAD_INIT,
"xx",
module_doc,
-1,
xx_methods,
NULL,
NULL,
NULL,
NULL
};
PyMODINIT_FUNC
PyInit_xx(void)
static int
xx_exec(PyObject *m)
{
PyObject *m = NULL;
/* Due to cross platform compiler issues the slots must be filled
* here. It's required for portability to Windows without requiring
* C++. */
@@ -366,11 +350,6 @@ PyInit_xx(void)
if (PyType_Ready(&Xxo_Type) < 0)
goto fail;
/* Create the module and add the functions */
m = PyModule_Create(&xxmodule);
if (m == NULL)
goto fail;
/* Add some symbolic constants to the module */
if (ErrorObject == NULL) {
ErrorObject = PyErr_NewException("xx.error", NULL, NULL);
@@ -389,8 +368,33 @@ PyInit_xx(void)
if (PyType_Ready(&Null_Type) < 0)
goto fail;
PyModule_AddObject(m, "Null", (PyObject *)&Null_Type);
return m;
return 0;
fail:
Py_XDECREF(m);
return NULL;
return -1;
}
static struct PyModuleDef_Slot xx_slots[] = {
{Py_mod_exec, xx_exec},
{0, NULL},
};
static struct PyModuleDef xxmodule = {
PyModuleDef_HEAD_INIT,
"xx",
module_doc,
0,
xx_methods,
xx_slots,
NULL,
NULL,
NULL
};
/* Export function for the module (*must* be called PyInit_xx) */
PyMODINIT_FUNC
PyInit_xx(void)
{
return PyModuleDef_Init(&xxmodule);
}

75
Modules/xxsubtype.c
View File

@@ -257,13 +257,50 @@ static PyMethodDef xxsubtype_functions[] = {
{NULL, NULL} /* sentinel */
};
static int
xxsubtype_exec(PyObject* m)
{
/* Fill in deferred data addresses. This must be done before
PyType_Ready() is called. Note that PyType_Ready() automatically
initializes the ob.ob_type field to &PyType_Type if it's NULL,
so it's not necessary to fill in ob_type first. */
spamdict_type.tp_base = &PyDict_Type;
if (PyType_Ready(&spamdict_type) < 0)
return -1;
spamlist_type.tp_base = &PyList_Type;
if (PyType_Ready(&spamlist_type) < 0)
return -1;
if (PyType_Ready(&spamlist_type) < 0)
return -1;
if (PyType_Ready(&spamdict_type) < 0)
return -1;
Py_INCREF(&spamlist_type);
if (PyModule_AddObject(m, "spamlist",
(PyObject *) &spamlist_type) < 0)
return -1;
Py_INCREF(&spamdict_type);
if (PyModule_AddObject(m, "spamdict",
(PyObject *) &spamdict_type) < 0)
return -1;
return 0;
}
static struct PyModuleDef_Slot xxsubtype_slots[] = {
{Py_mod_exec, xxsubtype_exec},
{0, NULL},
};
static struct PyModuleDef xxsubtypemodule = {
PyModuleDef_HEAD_INIT,
"xxsubtype",
xxsubtype__doc__,
-1,
0,
xxsubtype_functions,
NULL,
xxsubtype_slots,
NULL,
NULL,
NULL
@@ -273,37 +310,5 @@ static struct PyModuleDef xxsubtypemodule = {
PyMODINIT_FUNC
PyInit_xxsubtype(void)
{
PyObject *m;
/* Fill in deferred data addresses. This must be done before
PyType_Ready() is called. Note that PyType_Ready() automatically
initializes the ob.ob_type field to &PyType_Type if it's NULL,
so it's not necessary to fill in ob_type first. */
spamdict_type.tp_base = &PyDict_Type;
if (PyType_Ready(&spamdict_type) < 0)
return NULL;
spamlist_type.tp_base = &PyList_Type;
if (PyType_Ready(&spamlist_type) < 0)
return NULL;
m = PyModule_Create(&xxsubtypemodule);
if (m == NULL)
return NULL;
if (PyType_Ready(&spamlist_type) < 0)
return NULL;
if (PyType_Ready(&spamdict_type) < 0)
return NULL;
Py_INCREF(&spamlist_type);
if (PyModule_AddObject(m, "spamlist",
(PyObject *) &spamlist_type) < 0)
return NULL;
Py_INCREF(&spamdict_type);
if (PyModule_AddObject(m, "spamdict",
(PyObject *) &spamdict_type) < 0)
return NULL;
return m;
return PyModuleDef_Init(&xxsubtypemodule);
}

339
Objects/moduleobject.c
View File

@@ -20,7 +20,7 @@ static PyMemberDef module_members[] = {
{0}
};
static PyTypeObject moduledef_type = {
PyTypeObject PyModuleDef_Type = {
PyVarObject_HEAD_INIT(&PyType_Type, 0)
"moduledef", /* tp_name */
sizeof(struct PyModuleDef), /* tp_size */
@@ -28,6 +28,20 @@ static PyTypeObject moduledef_type = {
};
PyObject*
PyModuleDef_Init(struct PyModuleDef* def)
{
if (PyType_Ready(&PyModuleDef_Type) < 0)
return NULL;
if (def->m_base.m_index == 0) {
max_module_number++;
Py_REFCNT(def) = 1;
Py_TYPE(def) = &PyModuleDef_Type;
def->m_base.m_index = max_module_number;
}
return (PyObject*)def;
}
static int
module_init_dict(PyModuleObject *mod, PyObject *md_dict,
PyObject *name, PyObject *doc)
@@ -97,26 +111,13 @@ PyModule_New(const char *name)
return module;
}
PyObject *
PyModule_Create2(struct PyModuleDef* module, int module_api_version)
/* Check API/ABI version
* Issues a warning on mismatch, which is usually not fatal.
* Returns 0 if an exception is raised.
*/
static int
check_api_version(const char *name, int module_api_version)
{
PyObject *d, *v, *n;
PyMethodDef *ml;
const char* name;
PyModuleObject *m;
PyInterpreterState *interp = PyThreadState_Get()->interp;
if (interp->modules == NULL)
Py_FatalError("Python import machinery not initialized");
if (PyType_Ready(&moduledef_type) < 0)
return NULL;
if (module->m_base.m_index == 0) {
max_module_number++;
Py_REFCNT(module) = 1;
Py_TYPE(module) = &moduledef_type;
module->m_base.m_index = max_module_number;
}
name = module->m_name;
if (module_api_version != PYTHON_API_VERSION && module_api_version != PYTHON_ABI_VERSION) {
int err;
err = PyErr_WarnFormat(PyExc_RuntimeWarning, 1,
@@ -125,7 +126,30 @@ PyModule_Create2(struct PyModuleDef* module, int module_api_version)
name,
PYTHON_API_VERSION, name, module_api_version);
if (err)
return NULL;
return 0;
}
return 1;
}
PyObject *
PyModule_Create2(struct PyModuleDef* module, int module_api_version)
{
const char* name;
PyModuleObject *m;
PyInterpreterState *interp = PyThreadState_Get()->interp;
if (interp->modules == NULL)
Py_FatalError("Python import machinery not initialized");
if (!PyModuleDef_Init(module))
return NULL;
name = module->m_name;
if (!check_api_version(name, module_api_version)) {
return NULL;
}
if (module->m_slots) {
PyErr_Format(
PyExc_SystemError,
"module %s: PyModule_Create is incompatible with m_slots", name);
return NULL;
}
/* Make sure name is fully qualified.
@@ -156,53 +180,260 @@ PyModule_Create2(struct PyModuleDef* module, int module_api_version)
memset(m->md_state, 0, module->m_size);
}
d = PyModule_GetDict((PyObject*)m);
if (module->m_methods != NULL) {
n = PyUnicode_FromString(name);
if (n == NULL) {
if (PyModule_AddFunctions((PyObject *) m, module->m_methods) != 0) {
Py_DECREF(m);
return NULL;
}
for (ml = module->m_methods; ml->ml_name != NULL; ml++) {
if ((ml->ml_flags & METH_CLASS) ||
(ml->ml_flags & METH_STATIC)) {
PyErr_SetString(PyExc_ValueError,
"module functions cannot set"
" METH_CLASS or METH_STATIC");
Py_DECREF(n);
Py_DECREF(m);
return NULL;
}
v = PyCFunction_NewEx(ml, (PyObject*)m, n);
if (v == NULL) {
Py_DECREF(n);
Py_DECREF(m);
return NULL;
}
if (PyDict_SetItemString(d, ml->ml_name, v) != 0) {
Py_DECREF(v);
Py_DECREF(n);
Py_DECREF(m);
return NULL;
}
Py_DECREF(v);
}
Py_DECREF(n);
}
if (module->m_doc != NULL) {
_Py_IDENTIFIER(__doc__);
v = PyUnicode_FromString(module->m_doc);
if (v == NULL || _PyDict_SetItemId(d, &PyId___doc__, v) != 0) {
Py_XDECREF(v);
if (PyModule_SetDocString((PyObject *) m, module->m_doc) != 0) {
Py_DECREF(m);
return NULL;
}
Py_DECREF(v);
}
m->md_def = module;
return (PyObject*)m;
}
PyObject *
PyModule_FromDefAndSpec2(struct PyModuleDef* def, PyObject *spec, int module_api_version)
{
PyModuleDef_Slot* cur_slot;
PyObject *(*create)(PyObject *, PyModuleDef*) = NULL;
PyObject *nameobj;
PyObject *m = NULL;
int has_execution_slots = 0;
char *name;
int ret;
PyModuleDef_Init(def);
nameobj = PyObject_GetAttrString(spec, "name");
if (nameobj == NULL) {
return NULL;
}
name = PyUnicode_AsUTF8(nameobj);
if (name == NULL) {
goto error;
}
if (!check_api_version(name, module_api_version)) {
goto error;
}
if (def->m_size < 0) {
PyErr_Format(
PyExc_SystemError,
"module %s: m_size may not be negative for multi-phase initialization",
name);
goto error;
}
for (cur_slot = def->m_slots; cur_slot && cur_slot->slot; cur_slot++) {
if (cur_slot->slot == Py_mod_create) {
if (create) {
PyErr_Format(
PyExc_SystemError,
"module %s has multiple create slots",
name);
goto error;
}
create = cur_slot->value;
} else if (cur_slot->slot < 0 || cur_slot->slot > _Py_mod_LAST_SLOT) {
PyErr_Format(
PyExc_SystemError,
"module %s uses unknown slot ID %i",
name, cur_slot->slot);
goto error;
} else {
has_execution_slots = 1;
}
}
if (create) {
m = create(spec, def);
if (m == NULL) {
if (!PyErr_Occurred()) {
PyErr_Format(
PyExc_SystemError,
"creation of module %s failed without setting an exception",
name);
}
goto error;
} else {
if (PyErr_Occurred()) {
PyErr_Format(PyExc_SystemError,
"creation of module %s raised unreported exception",
name);
goto error;
}
}
} else {
m = PyModule_New(name);
if (m == NULL) {
goto error;
}
}
if (PyModule_Check(m)) {
((PyModuleObject*)m)->md_state = NULL;
((PyModuleObject*)m)->md_def = def;
} else {
if (def->m_size > 0 || def->m_traverse || def->m_clear || def->m_free) {
PyErr_Format(
PyExc_SystemError,
"module %s is not a module object, but requests module state",
name);
goto error;
}
if (has_execution_slots) {
PyErr_Format(
PyExc_SystemError,
"module %s specifies execution slots, but did not create "
"a ModuleType instance",
name);
goto error;
}
}
if (def->m_methods != NULL) {
ret = PyModule_AddFunctions(m, def->m_methods);
if (ret != 0) {
goto error;
}
}
if (def->m_doc != NULL) {
ret = PyModule_SetDocString(m, def->m_doc);
if (ret != 0) {
goto error;
}
}
return m;
error:
Py_DECREF(nameobj);
Py_XDECREF(m);
return NULL;
}
int
PyModule_ExecDef(PyObject *module, PyModuleDef *def)
{
PyModuleDef_Slot *cur_slot;
const char *name;
int ret;
name = PyModule_GetName(module);
if (name == NULL) {
return -1;
}
if (PyModule_Check(module) && def->m_size >= 0) {
PyModuleObject *md = (PyModuleObject*)module;
if (md->md_state == NULL) {
/* Always set a state pointer; this serves as a marker to skip
* multiple initialization (importlib.reload() is no-op) */
md->md_state = PyMem_MALLOC(def->m_size);
if (!md->md_state) {
PyErr_NoMemory();
return -1;
}
memset(md->md_state, 0, def->m_size);
}
}
if (def->m_slots == NULL) {
return 0;
}
for (cur_slot = def->m_slots; cur_slot && cur_slot->slot; cur_slot++) {
switch (cur_slot->slot) {
case Py_mod_create:
/* handled in PyModule_CreateFromSlots */
break;
case Py_mod_exec:
ret = ((int (*)(PyObject *))cur_slot->value)(module);
if (ret != 0) {
if (!PyErr_Occurred()) {
PyErr_Format(
PyExc_SystemError,
"execution of module %s failed without setting an exception",
name);
}
return -1;
}
if (PyErr_Occurred()) {
PyErr_Format(
PyExc_SystemError,
"execution of module %s raised unreported exception",
name);
return -1;
}
break;
default:
PyErr_Format(
PyExc_SystemError,
"module %s initialized with unknown slot %i",
name, cur_slot->slot);
return -1;
}
}
return 0;
}
int
PyModule_AddFunctions(PyObject *m, PyMethodDef *functions)
{
PyObject *name, *func;
PyMethodDef *fdef;
name = PyModule_GetNameObject(m);
if (name == NULL) {
return -1;
}
for (fdef = functions; fdef->ml_name != NULL; fdef++) {
if ((fdef->ml_flags & METH_CLASS) ||
(fdef->ml_flags & METH_STATIC)) {
PyErr_SetString(PyExc_ValueError,
"module functions cannot set"
" METH_CLASS or METH_STATIC");
Py_DECREF(name);
return -1;
}
func = PyCFunction_NewEx(fdef, (PyObject*)m, name);
if (func == NULL) {
Py_DECREF(name);
return -1;
}
if (PyObject_SetAttrString(m, fdef->ml_name, func) != 0) {
Py_DECREF(func);
Py_DECREF(name);
return -1;
}
Py_DECREF(func);
}
Py_DECREF(name);
return 0;
}
int
PyModule_SetDocString(PyObject *m, const char *doc)
{
PyObject *v;
_Py_IDENTIFIER(__doc__);
v = PyUnicode_FromString(doc);
if (v == NULL || _PyObject_SetAttrId(m, &PyId___doc__, v) != 0) {
Py_XDECREF(v);
return -1;
}
Py_DECREF(v);
return 0;
}
PyObject *
PyModule_GetDict(PyObject *m)

2
PC/config.c
View File

@@ -1,7 +1,7 @@
/* Module configuration */
/* This file contains the table of built-in modules.
See init_builtin() in import.c. */
See create_builtin() in import.c. */
#include "Python.h"

Some files were not shown because too many files have changed in this diff Show More