There's a lot of misunderstanding between setup.py and requirements.txt and their roles. A lot of people have felt they are duplicated information and have even created tools to handle this "duplication".

Python Libraries

A Python library in this context is something that has been developed and released for others to use. You can find a number of them on PyPI that others have made available. A library has a number of pieces of metadata that need to be provided in order to successfully distribute it. These are things such as the Name, Version, Dependencies, etc. The setup.py file gives you the ability to specify this metadata like:

from setuptools import setup

setup(
    name="MyLibrary",
    version="1.0",
    install_requires=[
        "requests",
        "bcrypt",
    ],
    # ...
)

This is simple enough, you have the required pieces of metadata declared. However something you don't see is a specification as to where you'll be getting those dependencies from. There's no url or filesystem where you can fetch these dependencies from, it's just "requests" and "bcrypt". This is important and for lack of a better term I call these "abstract dependencies". They are dependencies which exist only as a name and an optional version specifier. Think of it like duck typing your dependencies, you don't care what specific "requests" you get as long as it looks like "requests".

Python Applications

Here when I speak of a Python application I'm going to typically be speaking about something that you specifically deploy. It may or may not exist on PyPI but it's something that likely does not have much in the way of reusability. An application that does exist on PyPI typically requires a deploy specific configuration file and this section deals with the "deploy specific" side of a Python application.

An application typically has a set of dependencies, often times even a very complex set of dependencies, that it has been tested against. Being a specific instance that has been deployed, it typically does not have a name, nor any of the other packaging related metadata. This is reflected in the abilities of a piprequirements file. A typical requirements file might look something like:

# This is an implicit value, here for clarity
--index https://pypi.python.org/simple/

MyPackage==1.0
requests==1.2.0
bcrypt==1.0.2

Here you have each dependency shown along with an exact version specifier. While a library tends to want to have wide open ended version specifiers an application wants very specific dependencies. It may not have mattered up front what version of requests was installed but you want the same version to install in production as you developed and tested with locally.

At the top of this file you'll also notice a --index https://pypi.python.org/simple/. Your typical requirements.txt won't have this listed explicitly like this unless they are not using PyPI, it is however an important part of a requirements.txt. This single line is what turns the abstract dependency of requests==1.2.0 into a "concrete" dependency of "requests 1.2.0 from https://pypi.python.org/simple/". This is not like duck typing, this is the packaging equivalent of an isinstance() check.

So Why Does Abstract and Concrete Matter?

You've read this far and maybe you've said, ok I know that setup.py is designed for redistributable things and that requirements.txt is designed for non-redistributable things but I already have something that reads a requirements.txt and fills out my install_requires=[...] so why should I care?

This split between abstract and concrete is an important one. It was what allows the PyPI mirroring infrastructure to work. It is what allows a company to host their own private package index. It is even what enables you to fork a library to fix a bug or add a feature and use your own fork. Because an abstract dependency is a name and an optional version specifier you can install it from PyPI or from Crate.io, or from your own filesystem. You can fork a library, change the code, and as long as it has the right name and version specifier that library will happily go on using it.

A more extreme version of what can happen when you use a concrete requirement where an abstract requirement should be used can be found in the Go language. In the go language the default package manager (go get) allows you to specify your imports via an url inside the code which the package manager collects and downloads. This would look something like:

import (
        "github.com/foo/bar"
)

Here you can see that an exact url to a dependency has been specified. Now if I used a library that specified its dependencies this way and I wanted to change the "bar" library because of a bug that was affecting me or a feature I needed, I would not only need to fork the bar library, but I would also need to fork the library that depended on the bar library to update it. Even worse, if the bar library was say, 5 levels deep, then that's a potential of 5 different packages that I would need to fork and modify only to point it at a slightly different "bar".

A Setuptools Misfeature

Setuptools has a feature similar to the Go example. It's calleddependency links and it looks like this:

from setuptools import setup

setup(
    # ...
    dependency_links = [
        "http://packages.example.com/snapshots/",
        "http://example2.com/p/bar-1.0.tar.gz",
    ],
)

This "feature" of setuptools removes the abstractness of its dependencies and hardcodes an exact url from which you can fetch the dependency from. Now very similarly to Go if we want to modify packages, or simply fetch them from a different server we'll need to go in and edit each package in the dependency chain in order to update the dependency_links.

Developing Reusable Things or How Not to Repeat Yourself

The "Library" and "Application" distinction is all well and good, but whenever you're developing a Library, in a way it becomes your application. You want a specific set of dependencies that you want to fetch from a specific location and you know that you should have abstract dependencies in your setup.py and concrete dependencies in your requirements.txt but you don't want to need to maintain two separate lists which will inevitably go out of sync. As it turns out pip requirements file have a construct to handle just such a case. Given a directory with a setup.py inside of it you can write a requirements file that looks like:

--index https://pypi.python.org/simple/

-e .

Now your pip install -r requirements.txt will work just as before. It will first install the library located at the file path . and then move on to its abstract dependencies, combining them with its --index option and turning them into concrete dependencies and installing them.

This method grants another powerful ability. Let's say you have two or more libraries that you develop as a unit but release separately, or maybe you've just split out part of a library into its own piece and haven't officially released it yet. If your top level library still depends on just the name then you can install the development version when using the requirements.txt and the release version when not, using a file like:

--index https://pypi.python.org/simple/

-e https://github.com/foo/bar.git#egg=bar
-e .

This will first install the bar library from https://github.com/foo/bar.git, making it equal to the name "bar", and then will install the local package, again combining its dependencies with the --index option and installing but this time since the "bar" dependency has already been satisfied it will skip it and continue to use the in development version.

Recognition: This post was inspired by Yehuda Katz's blog post on a similar issue in Ruby with Gemfile and gemspec.

文章选自：setup-vs-requirements

Python模块发布文件setup.py vs requirements.txt，首发于润物无声。

A completely incomplete guide to packaging a Python module and sharing it with the world on PyPI.

Abstract

Few things give me caremads like Python modules I want to use that aren’t on PyPI (pronounced “pie pee eye”, or “cheese shop”, not “pie pie”!). On the other hand – aspydanny points out – the current situation on packaging is rather confusing.

Therefore I want to help everyone who has some great code but feels lost with getting it on PyPI. I will be using my latest opus “pem” as a realistic yet simple example of how to get a pure-Python 2 and 3 module packaged up, tested and uploaded to PyPI. Including the new and shiny binary wheel format that’s faster and allows for binary extensions (read the wheel story if you want to know more)!

I’ll keep it super simple to get everyone started. At the end, I’ll link more complete documentation to show you the way ahead. Sorry, no Windows.

Tools Used

This is not a history lesson, therefore we will use:

• pip 1.4+ (if you have no pip yet, bootstrapping is easy),
• setuptools 0.9+,
• and wheel 0.21+.

$ pip install -U "pip>=1.4" "setuptools>=0.9" "wheel>=0.21"

Please make sure all installs succeed, ancient installations may need some extra manual labor.

A Minimal Glimpse Into The Past

Forget that there ever was distribute (cordially merged into setuptools), easy_install (part of setuptools, supplanted by pip), or distutils2 aka packaging (was supposed to be the official thing from Python 3.3 on, didn’t get done in time due to lack of helping hands, got ripped out by a heart-broken Éric and abandoned now).

Be just vaguely aware that there are distutils and distlibsomewhere underneath but ideally it shouldn't matter to you at all for now.

setup.py

Nowadays, every project that you want to package needs asetup.py file. Let’s have a look at what pem’s could look like:

from setuptools import setup

setup(
    name='pem',
    version='0.1.0',
    description='Parse and split PEM files painlessly.',
    long_description=(open('README.rst').read() + '\n\n' +
                      open('HISTORY.rst').read() + '\n\n' +
                      open('AUTHORS.rst').read()),
    url='http://github.com/hynek/pem/',
    license='MIT',
    author='Hynek Schlawack',
    author_email='hs@ox.cx',
    py_modules=['pem'],
    include_package_data=True,
    classifiers=[
        'Development Status :: 5 - Production/Stable',
        'Intended Audience :: Developers',
        'Natural Language :: English',
        'License :: OSI Approved :: MIT License',
        'Operating System :: OS Independent',
        'Programming Language :: Python',
        'Programming Language :: Python :: 2',
        'Programming Language :: Python :: 2.6',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.3',
        'Topic :: Software Development :: Libraries :: Python Modules',
    ],
)

In the simplest case, you just import setup from setuptoolsand run it with some keyword arguments.

Before I go into these arguments, I want to point out one of the most neglected and yet most important one: license.Always set a license! Otherwise nobody can use your module, which would be a pity, right?

I take a few shortcuts here which you may want to copy too:

• I love DRY, therefore my long description is just a concatenation of my README, change log and author credits.
• I use bumpversion for manipulating my version strings.

An essential field is py_modules which you'll use to denote the relevant code for packaging, if your package is actually just a single module like in pem’s case. If you have a full-blown package (i.e. a directory), you’ll go for:

from setuptools import setup, find_packages

setup(
...
    packages=find_packages(exclude=['tests*']),
...
)

The exclude makes sure that a top-level tests package doesn’t get installed (it’s still part of the source distribution) since that would wreak havoc.

You can also specify them by hand like I used to fordoc2dash’s setup.py. However, unless you have good reasons, just use find_packages() and be done.

The classifiers field’s usefulness is openly disputed, nevertheless pick them from here. PyPI will refuse to accept packages with unknown classifiers, hence I like to use "Private :: Do Not Upload" for private packages to protect myself from my own stupidity.

One icky thing are dependencies. Unless you really know what you’re doing, don’t pin them (specifying minimal version your package requires to work is fine of course) or your users won’t be able to install security updates of your dependencies:

setup(
...
    install_requires=['Django', 'django-annoying'],
...
)

Rule of thumb: requirements.txt should contain only ==,setup.py the rest (>=, !=, <=, …).

Non-Code Files

Every Python project has a “Fix MANIFEST.in” commit. Look it up, it’s true.

But it’s not really that hard: you add all files and directories that are not packaged due to py_modules or packages in your setup.py.

For “pem”, it’s just

include *.rst LICENSE

For more commands, have a look at the MANIFEST.in docs – especially recursive-include is used widely. There’s also a handy tool to validate your MANIFEST.in: check-manifest.

Important: If you want the files and directories fromMANIFEST.in to also be installed (e.g. if it’s runtime-relevant data), you will have to set include_package_data=True in yoursetup() call as in the example above!

Configuration

For our minimal Python-only project, we’ll only need two lines in setup.cfg:

[wheel]
universal = 1

These will make wheel build a universal wheel file (e.g.pem-0.1.0-py2.py3-none-any.whl) and you won’t have to circle through virtual environments of all supported Python versions to build them separately.

Documentation

As I’ve hopefully established, every open source project needs a license. No excuses.

Additionally, even the simplest package needs a README that tells potential users what they’re looking at. Make itreStructuredText (reST) so PyPI can properly render it on your project page. As a courtesy to your users, also keep a change log so they know what to expect from your releases. And finally it’s good style to credit your contributors in a third file. I like to call them README.rst,HISTORY.rst, and AUTHORS.rst respectively, but there’s no hard rule.

My long project description and thus PyPI text is the concatenation of those three (and I’ve shamelessly stolen it from Kenneth Reitz).

If you host on GitHub, you may want to add aCONTRIBUTING.md that gets displayed when someone wants to open a pull request. Unfortunately it has to be a Markdown file because GitHub doesn’t support reST for this. Have a look at pem’s if you need inspiration.

Let’s Build Finally!

Building a source distribution (which is what is usually used for pure-Python project) is just a matter of

$ python setup.py sdist

Because it’s 2013 and wheels are awesome, we’ll build one too! Fortunately, it’s just as easy (please note that you’ll need to install wheel as per above for this to work!):

$ python setup.py bdist_wheel

Of course, the wheel package is optional, but your users will thank you for much faster installation times.

Now you should have a new directory called dist containing a source distribution file and a wheel file. For pem, it currently looks like this:

dist
├── pem-0.1.0-py2.py3-none-any.whl
└── pem-0.1.0.tar.gz

You can test whether both install properly before we move on to uploading:

$ rm -rf 27-sdist  # ensure clean state if ran repeatedly
$ virtualenv 27-sdist
...
$ 27-sdist/bin/pip install --no-index dist/pem-0.1.0.tar.gz
Ignoring indexes: https://pypi.python.org/simple/
Unpacking ./dist/pem-0.1.0.tar.gz
  Running setup.py egg_info for package from file:///Users/hynek/Projects/pem/dist/pem-0.1.0.tar.gz

Installing collected packages: pem
  Running setup.py install for pem

Successfully installed pem
Cleaning up...
$ 27-sdist/bin/python
Python 2.7.4 (default, Apr 21 2013, 09:35:41)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pem
>>> pem.__version__
'0.1.0'

and

$ rm -rf 27-wheel  # ensure clean state if ran repeatedly
$ virtualenv 27-wheel
...
$ 27-wheel/bin/pip install --use-wheel --no-index --find-links dist pem
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking pem
Installing collected packages: pem
Successfully installed pem
Cleaning up...
$ 27-wheel/bin/python
Python 2.7.4 (default, Apr 21 2013, 09:35:41)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pem
>>> pem.__version__
'0.1.0'

Please note the lack of “Running setup.py install for pem” in the second test. Yes, you can finally install packages without executing arbitrary code!

So you’re confident that your package is perfect? Let’s use the Test PyPI server to find out!

The PyPI Staging Server

Again, I’ll be using “pem” as the example project name to avoid <your project name> everywhere.

First, sign up on the test server, you will receive a user name and a password. Please note that this is independent from the live servers. Thus you’ll have to re-register both yourself and your packages. It also gets cleaned from time to time so don’t be surprised if it suddenly doesn’t know about you or your projects anymore. Just re-register.

Next, create a ~/.pypirc consisting of:

[distutils]
index-servers=
    test

[test]
repository = https://testpypi.python.org/pypi
username = <your user name goes here>
password = <your password goes here>

Then use

$ python setup.py register -r test

to register your project with the PyPI test server.

Finally, upload your distributions:

$ python setup.py sdist upload -r test
$ python setup.py bdist_wheel upload -r test

Please note that calling upload without building something in the same command line will give you a:

error: No dist file created in earlier command

Now test your packages again:

$ pip install -i https://testpypi.python.org/pypi pem

Everything dandy? Then lets tackle the last step: putting it on the real PyPI!

The Final Step

First, register at PyPI, then complete your ~/.pypirc:

[distutils]
index-servers=
    pypi
    test

[test]
repository = https://testpypi.python.org/pypi
username = <your test user name goes here>
password = <your test password goes here>

[pypi]
repository = http://pypi.python.org/pypi
username = <your production user name goes here>
password = <your production password goes here>

One last deep breath and let’s rock:

$ python setup.py sdist upload -r pypi
$ python setup.py bdist_wheel upload -r pypi

And thus, your package is only a pip install away for everyone! Congratulations, do more of that!

Next Steps

The information herein will probably get you pretty far but if you get stuck, the current canonical truths for Python packaging are:

• Python Packaging User Guide
• setuptools
• wheel
• distutils
• and specifically the PyPI chapter in the latter one.

Please don’t let your software rot on GitHub or launchpad! Share!

Thanks

This article has been kindly proof-read by Lynn Root,Donald Stufft, Alex Gaynor, Thomas Heinrichsdobler, andJannis Leidel. All mistakes are still mine though.

I’d be happy if you have any feedback to make this article even more straight-forward for people new to Python packaging!

文章出自：sharing-your-labor-of-love-pypi-quick-and-dirty

分享Python模块到PyPI，首发于润物无声。