setup.py: which "Twisted" distribution name should we depend on?

[warner]

We've learned a bit today about the overzealous downloading/installation of Twisted by our setup.py on platforms that supposedly already have it installed.

The entry in our setup.py (really _auto_deps.py) stanza that declares which other packages we depend upon refers to a setuptools "Distribution" (which is equivalent to a debian package). These distribution names are not case sensitive, and one can use either hyphens or underscores to separate the words. For example, "twisted-core", "Twisted-Core", and "Twisted_Core" are all equivalent.

When Twisted-8.0.1 is built by running 'python setup.py install' on the top-level setup.py, it produces a single .egg directory (i.e. a single "distribution", named "Twisted".

When Twisted-8.0.1 is built by running 'setup.py install' on each individual subproject's setup.py, it produces a separate .egg directory for each subproject: Twisted-Core, Twisted-Web, etc. This is how debian/sid current does it. In sid, the 'python-twisted' debian package is a metapackage that depends upon 'python-twisted-core', 'python-twisted-web', etc. It does not contain any .egg directories. So if you have debian sid, you have the "Twisted-Core" distribution, but not the "Twisted" distribution. The most recent debian release ("etch") has twisted-2.4.0, but no .egg files.

Ubuntu feisty and gutsy have Twisted-2.5.0, and have just the single "Twisted" distribution. Dapper has Twisted-2.2.0, but does not have any .egg files.

Hardy has Twisted-2.5.0, and its "python-twisted-core" Deb package provides the "Twisted" distribution name.

The extra download/installs we've been seeing are because the tahoe setup.py declares a dependency upon "Twisted", whereas the debian packages installed on the system provide "Twisted-Core". The setuptools dependency language doesn't provide for boolean-or, so we can't declare a dependency upon one or the other.

So we need to decide what to do in the Tahoe setup.py:

• depend upon "Twisted": builds on Feisty, Gutsy, and Hardy will not download anything extra, and will use the system Twisted instead. Builds on debian etch and sid will download Twisted-8.0.1 and install it into support/lib .
• depend upon "Twisted-Core": builds on debian sid will not download anything extra. Builds on debian etch, and on ubuntu feisty/gutsy/hardy will download and install Twisted-8.0.1 .
• modify our dependencies based upon whether pkg_resources.require("Twisted") succeeds or not. We could choose one of the two options above as our default, but then say that a system that provides "Twisted" will depend upon "Twisted" rather than "Twisted-Core", or vice-versa. This would avoid extra downloads on debian sid and ubuntu feisty/gutsy/hardy (i.e. all platforms on which the system Twisted has .egg_info data). However it would raise the question of what dependencies should wind up in the record on pypi.

I think Foolscap will need to make the same decision.

[warner]

Note: distutils has a 'provides' keyword argument, but it refers to packages (not distributions), and doesn't cause any additional information to get placed in the EGG-INFO data. We were hoping that we could convince the twisted folks to make their top-level setup.py declare that the "Twisted" distribution provides "Twisted-Core" and friends, but it looks like the machinery for that just isn't there.

We were also hoping that the setuptools dependency language was rich enough to allow OR clauses, so we could declare a dependency upon "Twisted | Twisted-Core", but it is not.

[zooko]

Ubuntu Hardy's "python-twisted-core" Deb package provides the "Twisted" distribution name.

[warner]

FYI, here is the debian bug filed about this one: ​http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=477384

[warner]

This is most probably fixed by the recent upload of python-twisted-8.1.0-1 into debian/sid. Note that the "Twisted.egg-info" file comes from the python-twisted package, not the python-twisted-core package (the latter still provides a Twisted-Core.egg-info file, but that doesn't matter as long as there is a Twisted.egg-info file present).

I need to double-check that this solves the problem on my work computer, but I'm 90% sure it's fixed now. If so, I'll close this ticket on monday.

[warner]

This is indeed fixed. (I was missing the python-twisted package on my work machine). Thanks doko!