Merge in all documentation changes since branching 3.4.0rc1.

Doc/c-api/arg.rst

@@ -45,6 +45,7 @@ in any early abort case).
 Unless otherwise stated, buffers are not NUL-terminated.
 
 .. note::
+
    For all ``#`` variants of formats (``s#``, ``y#``, etc.), the type of
    the length argument (int or :c:type:`Py_ssize_t`) is controlled by
    defining the macro :c:macro:`PY_SSIZE_T_CLEAN` before including

Doc/c-api/exceptions.rst

@@ -525,11 +525,11 @@ Exception Objects
    reference, as accessible from Python through :attr:`__cause__`.
 
 
-.. c:function:: void PyException_SetCause(PyObject *ex, PyObject *ctx)
+.. c:function:: void PyException_SetCause(PyObject *ex, PyObject *cause)
 
-   Set the cause associated with the exception to *ctx*.  Use *NULL* to clear
-   it.  There is no type check to make sure that *ctx* is either an exception
-   instance or :const:`None`.  This steals a reference to *ctx*.
+   Set the cause associated with the exception to *cause*.  Use *NULL* to clear
+   it.  There is no type check to make sure that *cause* is either an exception
+   instance or :const:`None`.  This steals a reference to *cause*.
 
    :attr:`__suppress_context__` is implicitly set to ``True`` by this function.
 

Doc/c-api/init.rst

@@ -582,6 +582,7 @@ code, or when embedding the Python interpreter:
    .. index:: module: _thread
 
    .. note::
+
       When only the main thread exists, no GIL operations are needed. This is a
       common situation (most Python programs do not use threads), and the lock
       operations slow the interpreter down a bit. Therefore, the lock is not

Doc/c-api/module.rst

@@ -120,7 +120,7 @@ There are only a few functions special to module objects.
 
    Return a pointer to the :c:type:`PyModuleDef` struct from which the module was
    created, or *NULL* if the module wasn't created with
-   :c:func:`PyModule_Create`.i
+   :c:func:`PyModule_Create`.
 
 .. c:function:: PyObject* PyState_FindModule(PyModuleDef *def)
 

Doc/c-api/object.rst

@@ -357,9 +357,9 @@ attribute is considered sufficient for this determination.
 
 .. c:function:: Py_ssize_t PyObject_LengthHint(PyObject *o, Py_ssize_t default)
 
-   Return an estimated length for the object *o*. First trying to return its
-   actual length, then an estimate using ``__length_hint__``, and finally
-   returning the default value. On error ``-1`` is returned. This is the
+   Return an estimated length for the object *o*. First try to return its
+   actual length, then an estimate using :meth:`~object.__length_hint__`, and
+   finally return the default value. On error return ``-1``. This is the
    equivalent to the Python expression ``operator.length_hint(o, default)``.
 
    .. versionadded:: 3.4

Doc/c-api/tuple.rst

@@ -129,6 +129,14 @@ type.
    Initializes a struct sequence type *type* from *desc* in place.
 
 
+.. c:function:: int PyStructSequence_InitType2(PyTypeObject *type, PyStructSequence_Desc *desc)
+
+   The same as ``PyStructSequence_InitType``, but returns ``0`` on success and ``-1`` on
+   failure.
+
+   .. versionadded:: 3.4
+
+
 .. c:type:: PyStructSequence_Desc
 
    Contains the meta information of a struct sequence type to create.

Doc/c-api/typeobj.rst

@@ -205,9 +205,8 @@ type objects) *must* have the :attr:`ob_size` field.
    bit currently defined is :const:`Py_PRINT_RAW`. When the :const:`Py_PRINT_RAW`
    flag bit is set, the instance should be printed the same way as :c:member:`~PyTypeObject.tp_str`
    would format it; when the :const:`Py_PRINT_RAW` flag bit is clear, the instance
-   should be printed the same was as :c:member:`~PyTypeObject.tp_repr` would format it. It should
-   return ``-1`` and set an exception condition when an error occurred during the
-   comparison.
+   should be printed the same way as :c:member:`~PyTypeObject.tp_repr` would format it. It should
+   return ``-1`` and set an exception condition when an error occurs.
 
    It is possible that the :c:member:`~PyTypeObject.tp_print` field will be deprecated. In any case,
    it is recommended not to define :c:member:`~PyTypeObject.tp_print`, but instead to rely on

Doc/conf.py

@@ -61,6 +61,8 @@ add_module_names = True
 # By default, highlight as Python 3.
 highlight_language = 'python3'
 
+needs_sphinx = '1.1'
+
 
 # Options for HTML output
 # -----------------------
@@ -118,11 +120,11 @@ _stdauthor = r'Guido van Rossum\\Fred L. Drake, Jr., editor'
 latex_documents = [
     ('c-api/index', 'c-api.tex',
      'The Python/C API', _stdauthor, 'manual'),
-    ('distutils/index', 'distutils.tex',
+    ('distributing/index', 'distributing.tex',
      'Distributing Python Modules', _stdauthor, 'manual'),
     ('extending/index', 'extending.tex',
      'Extending and Embedding Python', _stdauthor, 'manual'),
-    ('install/index', 'install.tex',
+    ('installing/index', 'installing.tex',
      'Installing Python Modules', _stdauthor, 'manual'),
     ('library/index', 'library.tex',
      'The Python Library Reference', _stdauthor, 'manual'),

Doc/contents.rst

@@ -11,8 +11,8 @@
    library/index.rst
    extending/index.rst
    c-api/index.rst
-   distutils/index.rst
-   install/index.rst
+   distributing/index.rst
+   installing/index.rst
    howto/index.rst
    faq/index.rst
    glossary.rst
@@ -21,3 +21,11 @@
    bugs.rst
    copyright.rst
    license.rst
+
+.. include legacy packaging docs in build
+
+.. toctree::
+   :hidden:
+
+   distutils/index.rst
+   install/index.rst

Doc/distributing/index.rst

@@ -0,0 +1,143 @@
+.. _distributing-index:
+
+###############################
+  Distributing Python Modules
+###############################
+
+:Email: distutils-sig@python.org
+
+
+As a popular open source development project, Python has an active
+supporting community of contributors and users that also make their software
+available for other Python developers to use under open source license terms.
+
+This allows Python users to share and collaborate effectively, benefiting
+from the solutions others have already created to common (and sometimes
+even rare!) problems, as well as potentially contributing their own
+solutions to the common pool.
+
+This guide covers the distribution part of the process. For a guide to
+installing other Python projects, refer to the
+:ref:`installation guide <installing-index>`.
+
+.. note::
+
+   For corporate and other institutional users, be aware that many
+   organisations have their own policies around using and contributing to
+   open source software. Please take such policies into account when making
+   use of the distribution and installation tools provided with Python.
+
+
+Key terms
+=========
+
+* the `Python Package Index <https://pypi.python.org/pypi>`__ is a public
+  repository of open source licensed packages made available for use by
+  other Python users
+* the `Python Packaging Authority
+  <http://packaging.python.org/en/latest/future.html>`__ are the group of
+  developers and documentation authors responsible for the maintenance and
+  evolution of the standard packaging tools and the associated metadata and
+  file format standards. They maintain a variety of tools, documentation
+  and issue trackers on both `GitHub <https://github.com/pypa>`__ and
+  `BitBucket <https://bitbucket.org/pypa/>`__.
+* ``distutils`` is the original build and distribution system first added to
+  the Python standard library in 1998. While direct use of ``distutils`` is
+  being phased out, it still laid the foundation for the current packaging
+  and distribution infrastructure, and it not only remains part of the
+  standard library, but its name lives on in other ways (such as the name
+  of the mailing list used to coordinate Python packaging standards
+  development).
+
+
+Open source licensing and collaboration
+=======================================
+
+In most parts of the world, software is automatically covered by copyright.
+This means that other developers require explicit permission to copy, use,
+modify and redistribute the software.
+
+Open source licensing is a way of explicitly granting such permission in a
+relatively consistent way, allowing developers to share and collaborate
+efficiently by making common solutions to various problems freely available.
+This leaves many developers free to spend more time focusing on the problems
+that are relatively unique to their specific situation.
+
+The distribution tools provided with Python are designed to make it
+reasonably straightforward for developers to make their own contributions
+back to that common pool of software if they choose to do so.
+
+The same distribution tools can also be used to distribute software within
+an organisation, regardless of whether that software is published as open
+source software or not.
+
+
+Installing the tools
+====================
+
+The standard library does not include build tools that support modern
+Python packaging standards, as the core development team has found that it
+is important to have standard tools that work consistently, even on older
+versions of Python.
+
+The currently recommended build and distribution tools can be installed
+using ``pip``::
+
+    pip install setuptools wheel twine
+
+
+Reading the guide
+=================
+
+The Python Packaging User Guide covers the various key steps and elements
+involved in creating a project
+
+* `Project structure`_
+* `Building and packaging the project`_
+* `Uploading the project to the Python Package Index`_
+
+.. _Project structure: \
+   http://packaging.python.org/en/latest/tutorial.html#creating-your-own-project
+.. _Building and packaging the project: \
+   http://packaging.python.org/en/latest/tutorial.html#building-packaging-your-project
+.. _Uploading the project to the Python Package Index: \
+   http://packaging.python.org/en/latest/tutorial.html#uploading-your-project-to-pypi
+
+
+How do I...?
+============
+
+These are quick answers or links for some common tasks.
+
+... choose a name for my project?
+---------------------------------
+
+This isn't an easy topic, but here are a few tips:
+
+* check the Python Package Index to see if the name is already in use
+* check popular hosting sites like GitHub, BitBucket, etc to see if there
+  is already a project with that name
+* check what comes up in a web search for the name you're considering
+* avoid particularly common words, especially ones with multiple meanings,
+  as they can make it difficult for users to find your software when
+  searching for it
+
+
+... create and distribute binary extensions?
+--------------------------------------------
+
+This is actually quite a complex topic, with a variety of alternatives
+available depending on exactly what you're aiming to achieve. See the
+Python Packaging User Guide for more information and recommendations.
+
+.. seealso::
+
+   `Python Packaging User Guide: Binary Extensions
+   <http://packaging.python.org/en/latest/extensions.html>`__
+
+.. other topics:
+
+   Once the Development & Deployment part of PPUG is fleshed out, some of
+   those sections should be linked from new questions here (most notably,
+   we should have a question about avoiding depending on PyPI that links to
+   http://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches)

Doc/distutils/apiref.rst

@@ -853,17 +853,6 @@ Windows.  It also contains the Mingw32CCompiler class which handles the mingw32
 port of GCC (same as cygwin in no-cygwin mode).
 
 
-:mod:`distutils.emxccompiler` --- OS/2 EMX Compiler
-===================================================
-
-.. module:: distutils.emxccompiler
-   :synopsis: OS/2 EMX Compiler support
-
-
-This module provides the EMXCCompiler class, a subclass of
-:class:`UnixCCompiler` that handles the EMX port of the GNU C compiler to OS/2.
-
-
 :mod:`distutils.archive_util` ---  Archiving utilities
 ======================================================
 
@@ -1171,15 +1160,6 @@ other utility module.
    underscore. No { } or ( ) style quoting is available.
 
 
-.. function:: grok_environment_error(exc[, prefix='error: '])
-
-   Generate a useful error message from an :exc:`OSError` exception object.
-   Handles Python 1.5.1 and later styles, and does what it can to deal with
-   exception objects that don't have a filename (which happens when the error
-   is due to a two-file operation, such as :func:`~os.rename` or :func:`~os.link`).
-   Returns the error message as a string prefixed with *prefix*.
-
-
 .. function:: split_quoted(s)
 
    Split a string up according to Unix shell-like rules for quotes and backslashes.
@@ -1943,8 +1923,12 @@ Subclasses of :class:`Command` must define the following methods.
 .. module:: distutils.command.clean
    :synopsis: Clean a package build area
 
+This command removes the temporary files created by :command:`build`
+and its subcommands, like intermediary compiled object files.  With
+the ``--all`` option, the complete build directory will be removed.
 
-.. % todo
+Extension modules built :ref:`in place <distutils-build-ext-inplace>`
+will not be cleaned, as they are not in the build directory.
 
 
 :mod:`distutils.command.config` --- Perform package configuration

Doc/distutils/configfile.rst

@@ -69,6 +69,8 @@ universal :option:`--help` option, e.g. ::
 Note that an option spelled :option:`--foo-bar` on the command-line  is spelled
 :option:`foo_bar` in configuration files.
 
+.. _distutils-build-ext-inplace:
+
 For example, say you want your extensions to be built "in-place"---that is, you
 have an extension :mod:`pkg.ext`, and you want the compiled extension file
 (:file:`ext.so` on Unix, say) to be put in the same source directory as your

Doc/distutils/index.rst

@@ -1,8 +1,8 @@
 .. _distutils-index:
 
-###############################
-  Distributing Python Modules
-###############################
+##############################################
+  Distributing Python Modules (Legacy version)
+##############################################
 
 :Authors: Greg Ward, Anthony Baxter
 :Email: distutils-sig@python.org

Doc/distutils/setupscript.rst

@@ -685,6 +685,8 @@ include the following code fragment in your :file:`setup.py` before the
         DistributionMetadata.download_url = None
 
 
+.. _debug-setup-script:
+
 Debugging the setup script
 ==========================
 
@@ -700,7 +702,8 @@ installation is broken because they don't read all the way down to the bottom
 and see that it's a permission problem.
 
 On the other hand, this doesn't help the developer to find the cause of the
-failure. For this purpose, the DISTUTILS_DEBUG environment variable can be set
+failure. For this purpose, the :envvar:`DISTUTILS_DEBUG` environment variable can be set
 to anything except an empty string, and distutils will now print detailed
-information what it is doing, and prints the full traceback in case an exception
-occurs.
+information about what it is doing, dump the full traceback when an exception
+occurs, and print the whole command line when an external program (like a C
+compiler) fails.

Doc/extending/index.rst

@@ -21,14 +21,31 @@ Python) that give the language its wide application range.
 For a detailed description of the whole Python/C API, see the separate
 :ref:`c-api-index`.
 
-.. note::
 
-   This guide only covers the basic tools for creating extensions provided
-   as part of this version of CPython. Third party tools may offer simpler
-   alternatives. Refer to the `binary extensions section
-   <https://python-packaging-user-guide.readthedocs.org/en/latest/extensions.html>`__
-   in the Python Packaging User Guide for more information.
+Recommended third party tools
+=============================
 
+This guide only covers the basic tools for creating extensions provided
+as part of this version of CPython. Third party tools like Cython,
+``cffi``, SWIG and Numba offer both simpler and more sophisticated
+approaches to creating C and C++ extensions for Python.
+
+.. seealso::
+
+   `Python Packaging User Guide: Binary Extensions <https://packaging.python.org/en/latest/extensions.html>`_
+      The Python Packaging User Guide not only covers several available
+      tools that simplify the creation of binary extensions, but also
+      discusses the various reasons why creating an extension module may be
+      desirable in the first place.
+
+
+Creating extensions without third party tools
+=============================================
+
+This section of the guide covers creating C and C++ extensions without
+assistance from third party tools. It is intended primarily for creators
+of those tools, rather than being a recommended way to create your own
+C extensions.
 
 .. toctree::
    :maxdepth: 2
@@ -38,4 +55,17 @@ For a detailed description of the whole Python/C API, see the separate
    newtypes.rst
    building.rst
    windows.rst
+
+Embedding the CPython runtime in a larger application
+=====================================================
+
+Sometimes, rather than creating an extension that runs inside the Python
+interpreter as the main application, it is desirable to instead embed
+the CPython runtime inside a larger application. This section covers
+some of the details involved in doing that successfully.
+
+.. toctree::
+   :maxdepth: 2
+   :numbered:
+
    embedding.rst

Doc/faq/general.rst

@@ -471,7 +471,8 @@ that is written in Python using Tkinter.  PythonWin is a Windows-specific IDE.
 Emacs users will be happy to know that there is a very good Python mode for
 Emacs.  All of these programming environments provide syntax highlighting,
 auto-indenting, and access to the interactive interpreter while coding.  Consult
-http://www.python.org/editors/ for a full list of Python editing environments.
+`the Python wiki <https://wiki.python.org/moin/PythonEditors>`_ for a full list
+of Python editing environments.
 
 If you want to discuss Python's use in education, you may be interested in
 joining `the edu-sig mailing list

Doc/faq/library.rst

@@ -513,6 +513,7 @@ For data that is more regular (e.g. a homogeneous list of ints or floats),
 you can also use the :mod:`array` module.
 
 .. note::
+
    To read and write binary data, it is mandatory to open the file in
    binary mode (here, passing ``"rb"`` to :func:`open`).  If you use
    ``"r"`` instead (the default), the file will be open in text mode

Doc/faq/programming.rst

@@ -1103,6 +1103,7 @@ Use a list comprehension::
 
    result = [obj.method() for obj in mylist]
 
+.. _faq-augmented-assignment-tuple-error:
 
 Why does a_tuple[i] += ['item'] raise an exception when the addition works?
 ---------------------------------------------------------------------------

Doc/glossary.rst

@@ -783,6 +783,14 @@ Glossary
       mapping rather than a sequence because the lookups use arbitrary
       :term:`immutable` keys rather than integers.
 
+      The :class:`collections.abc.Sequence` abstract base class
+      defines a much richer interface that goes beyond just
+      :meth:`__getitem__` and :meth:`__len__`, adding :meth:`count`,
+      :meth:`index`, :meth:`__contains__`, and
+      :meth:`__reversed__`. Types that implement this expanded
+      interface can be registered explicitly using
+      :func:`~abc.register`.
+
    single dispatch
       A form of :term:`generic function` dispatch where the implementation is
       chosen based on the type of a single argument.

Doc/howto/pyporting.rst

@@ -26,6 +26,32 @@ Porting Python 2 Code to Python 3
    For help with porting, you can email the python-porting_ mailing list with
    questions.
 
+The Short Version
+=================
+
+* Decide what's the oldest version of Python 2 you want to support (if at all)
+* Make sure you have a thorough test suite and use continuous integration
+  testing to make sure you stay compatible with the versions of Python you care
+  about
+* If you have dependencies, check their Python 3 status using caniusepython3
+  (`command-line tool <https://pypi.python.org/pypi/caniusepython3>`__,
+  `web app <https://caniusepython3.com/>`__)
+
+With that done, your options are:
+
+* If you are dropping Python 2 support, use 2to3_ to port to Python 3
+* If you are keeping Python 2 support, then start writing Python 2/3-compatible
+  code starting **TODAY**
+
+  + If you have dependencies that have not been ported, reach out to them to port
+    their project while working to make your code compatible with Python 3 so
+    you're ready when your dependencies are all ported
+  + If all your dependencies have been ported (or you have none), go ahead and
+    port to Python 3
+
+* If you are creating a new project that wants to have 2/3 compatibility,
+  code in Python 3 and then backport to Python 2
+
 
 Before You Begin
 ================
@@ -85,7 +111,7 @@ between Python 2 and 3 easier.
 Projects to Consider
 --------------------
 
-The lowest level library for suppoting Python 2 & 3 simultaneously is six_.
+The lowest level library for supporting Python 2 & 3 simultaneously is six_.
 Reading through its documentation will give you an idea of where exactly the
 Python language changed between versions 2 & 3 and thus what you will want the
 library to help you continue to support.
@@ -548,7 +574,10 @@ Backporting Python 3 code to Python 2
 
 If you have Python 3 code and have little interest in supporting Python 2 you
 can use 3to2_ to translate from Python 3 code to Python 2 code. This is only
-recommended if you don't plan to heavily support Python 2 users.
+recommended if you don't plan to heavily support Python 2 users. Otherwise
+write your code for Python 3 and then backport as far back as you want. This
+is typically easier than going from Python 2 to 3 as you will have worked out
+any difficulties with e.g. bytes/strings, etc.
 
 
 Other Resources