github.com/Technologicat/setup-template-cython @v0.1.5

setup-template-cython

setuptools-based setup.py template for Cython projects, plus notes on packaging.

TL;DR

Copy setup.py, customize, enjoy.

Introduction

Setuptools[[1]][setuptools] has become the tool of choice[[2]][packaging] for packaging Python projects, yet not much documentation is available on how to use setuptools in Cython projects. As of this writing (Cython 0.25), Cython's official packaging instructions are based on distutils.

For packages to be distributed (especially through PyPI), setuptools is preferable, since the documentation on distributing packages[[3]][distributing] assumes that is what the developer uses. Also, setuptools adds dependency resolution (over distutils), which is an essential feature of pip.

This very minimal project documents the author's best guess at current best practices for the packaging and distribution of Cython projects, by piecing together information from various sources. Possible corrections, if any, are welcome.

General notes

A (very short) terminology can be found in the Python documentation on distributing Python modules. Probably the best practical documentation on actually distributing your own Python projects, though, is the packaging guide.

There is a timeline of the history of Python packaging (as of August 2017, up to 2015) on the PyPA website.

This blog post, dated 2012 (before the introduction of the wheel format), explains many interesting technical details, such as the differences between distutils, setuptools and early pip, the install directory structure used by setuptools, .pth files, and the two kinds of .eggs, namely directories and zipfiles.

If you are familiar with distutils, but new to setuptools, see the list of new and changed keywords in the setuptools documentation.

Content and aim

The included setup.py is a setuptools-based packaging and install script template for new Cython projects.

This is similar to simple-cython-example, but our focus is on numerical scientific projects, where a custom Cython extension (containing all-new code) can bring a large speedup. The aim is to help open-sourcing such extensions in a manner that lets others effortlessly compile them, thus advancing the openness and repeatability of science.

For completeness, a minimal Cython-based example library is included, containing examples of things such as absolute cimports, subpackages, NumPyDoc style docstrings, and using memoryviews for passing arrays (for the last two, see compute.pyx). The example in the test/ subdirectory demonstrates usage of the example library after it is installed.

A pruned-down version of setup.py for pure Python projects, called setup-purepython.py, is also provided for comparison.

Features

Our setup.py features the following:

• The most important fields of setup()
• If this is all you need, simple-cython-example is much cleaner.
• If this is all you need, and you somehow ended up here even though your project is pure Python, PyPA's sampleproject (as mentioned in [[4]][setup-py]) has more detail on this.
• How to get absolute_import working in a Cython project
• For compatibility with both Python 3 and Python 2.7 (with from __future__ import absolute_import)
• For scientific developers used to Python 2.7, this is perhaps the only tricky part in getting custom Cython code to play nicely with Python 3. (As noted elsewhere[a][b], it is time to move to Python 3.)
• How to automatically grab __version__ from mylibrary/__init__.py (using AST; no import or regexes), so that you DontRepeatYourself declaring your package version (based on [[5]][getversion])
• Hopefully appropriate compiler and linker flags for math and non-math modules on x86_64, in production and debug configurations.
• Also compiler and linker flags for OpenMP, to support cython.parallel.prange.
• How to make setup.py pick up non-package data files, such as your documentation and usage examples (based on [[6]][datafolder]). However, see the section on Packaging data files below.
• How to make setup.py pick up data files inside your Python packages.
• How to enforce that setup.py is running under a given minimum Python version (considered harmful, but if duck-checking for individual features is not an option for a reason or another) (based on [[7]][enforcing]).
• Disabling zip_safe. Having zip_safe enabled (which will in practice happen by default) is a bad idea for Cython projects, because:
• Cython (as of this writing, version 0.25) will not see .pxd headers inside installed .egg zipfiles. Thus other libraries cannot cimport modules from yours if it has zip_safe set.
• At (Python-level) import time, the OS's dynamic library loader usually needs to have the .so unzipped (from the .egg zipfile) to a temporary directory anyway.

Usage

See the setuptools manual. Perhaps the most useful commands are:

python setup.py build_ext  # compile binary (Cython) extensions
python setup.py build      # copy .py files in the Python packages into the build directory
python setup.py install    # will automatically "build" and "bdist" first
python setup.py sdist      # create source distribution

Substitute python2 or python3 for python if needed.

For build_ext, the switch --inplace may be useful for one-file throwaway projects, but packages to be installed are generally much better off by letting setuptools create a build/ subdirectory.

For install, the switch --user may be useful. As can, alternatively, running the command through sudo, depending on your installation.

Uninstalling your installed package

Sometimes it is useful to uninstall the installed copy of your package, such as during the development and testing of the install step for a new version.

Whereas setuptools does not know how to uninstall packages, pip does. This applies also to setuptools-based packages not installed by pip itself.

(In contrast, legacy distutils-based packages contain no metadata and cannot be automatically uninstalled.)

As an example, to uninstall this template project (if you have installed it):

pip uninstall setup-template-cython

The package name is the name argument provided to setup() in setup.py.

Note that, when you invoke the command, if the current working directory of your terminal has a subdirectory with the same name as the package to be uninstalled (here setup-template-cython), its presence will mask the package, which is probably not what you want.

If you have installed several versions of the package manually, the above command will uninstall only the most recent version. In this case, invoke the command several times until it reports that setup-template-cython is not installed.

Note that pip will automatically check also the user directory of the current user (packages installed with python setup.py install --user) for the package to uninstall; there is no need to specify any options for that.

Substitute pip2 or pip3 for pip as needed; run through sudo if needed.

To check whether your default pip manages your Python 2 or Python 3 packages, use pip --version.

Distributing binaries

Windows and Mac OS users may be interested in python setup.py bdist_wheel to create platform wheels (platform-specific binary distributions).

As for Linux, as noted in the Python packaging guide, PyPI accepts platform wheels for Linux only if they conform to the manylinux1 ABI, so in this case running python setup.py bdist_wheel on an arbitrary development machine is generally not very useful for the purposes of distribution.

For the adventurous, PyPA provides instructions along with a Docker image.

For the less adventurous, just make an sdist and upload that; scientific Linux users are likely not scared by an automatic compilation step, and will have the development tools already installed anyway.

Packaging your data files

This is rather intricate. From the viewpoint of Python packaging, data files in your project come in two varieties:

• Non-package data files are files in the project that are to be distributed, but do not belong to any Python package. Typically, this means README.md et al., end-user documentation, and usage examples.
• Package data are data files inside Python packages. Typically, this means data files needed by your library (to be loaded at runtime via the pkg_resources API, TL;DR [1] [2]), or developer documentation specific to a particular package that you want to install into the same location as the package itself.

Non-package data files arguably have no natural install location, unless they are specified with an absolute target path. Thus it is almost always better to package them only into the source distribution (sdist).

On what gets included into the sdist by default, refer to the documentation. GitHub users specifically note that README.txt gets included, but README.md does not.

Mechanisms

There are three main mechanisms to make setuptools pick up data files: data_files, package_data, and the manifest template MANIFEST.in.

The data_files option of setup() is meant for non-package data files. However, any data_files specified with a relative path will install directly under sys.prefix. Importantly, Python environments in different operating systems may behave differently.

For example, on Linux Mint, when setuptools installs packages, each .egg directory effectively (if not strictly speaking) becomes the prefix for that particular package (as far as the install procedure is concerned).

However, on Mac OS, setuptools will use the system prefix, typically /usr/local. If setup.py specifies (for example) test/* to be included as data_files, then these files will try to install into /usr/local/test/*, which will fail (for good reason).

Thus, although setup.py provides an example of this, using data_files is usually not recommended.

For a long time, the package_data option of setup() was used only for binary distributions and installation, and was ignored for the sdist [1] [2]. The manifest template MANIFEST.in was the way to customize the sdist.

However, both of these features have since been partially extended to cover some tasks of the other, perhaps in an attempt to simplify packaging in simple cases.

The documentation says that in Python 3.1+ (and also in 2.7), all files specified as package_data will be included also into the sdist, but only if no manifest template is provided.

The manifest template MANIFEST.in is an optional, separate configuration file for setuptools, to be placed in the same directory as setup.py. It can be used to include additional files (both package and non-package data) into the sdist, and to exclude any undesired files that would be included in the sdist by default.

Nowadays, files listed in MANIFEST.in can also be included in binary distributions and installation, by setting include_package_data=True in the call to setup(). The option has no effect on the contents of the sdist. Also, as the name suggests, it only affects files that reside inside Python packages; any non-package data files included by MANIFEST.in will still be packaged only into the sdist.

Format of MANIFEST.in

For an overview, see this quick explanation. For available commands, see the (very short) documentation.

Simple example MANIFEST.in:

``` include .md include doc/.txt