use setuptools's --multi-version mode

[zooko]

​http://peak.telecommunity.com/DevCenter/setuptools#develop-deploy-the-project-source-in-development-mode

This would make it so that if there is already an older version of a dependency, e.g. pyutil, present on the system and we require a newer version, that it would go ahead and install the newer version, and Tahoe would use the newer version, without requiring the user to delete the older version from her system.

[zooko]

This was implemented in 0de6e616e0d0890d but it caused a mysterious problem on most buildbots. I think the problem might be due to an interaction with our "fake setuptools" code in our "Trial" class in setup.py​ and that the problem might be fixed by Chris Galvan's solution to #505 (wanted: a setuptools plugin to make unit tests be executed with trial instead of with pyunit). Anyway, I undid this change in 1d377cc2d9871e39 in order to make the buildbots green again. By the way, this issue seems to be interfering with Francois's attempt to fix #534 ("tahoe cp" command encoding issue) since an older version of simplejson is installed system-wide on the buildslave that he is experimenting with.

[warner]

Zooko mentioned, in response to #555 ("tahoe .deb cannot be installed on hardy because of simplejson dependency"), that this #530 ticket "would have meant that simplejson upgrade happened automatically".

I'm not sure I see how --multi-version would help this. We know that the tahoe .deb is not supposed to contain anything but tahoe code. What --multi-version might do is to make it easier to build tahoe from source on a host that has an older version of e.g. simplejson installed from a deb. But that seems to work anyways right now.

[zooko]

Oh I guess I confused #555 with the deployment issue that arose this weekend. That issue involved building from source (or, equivalently, running easy_install allmydata-tahoe. This ticket -- #530 -- would have made it so that deployment attempt would have automatically succeeded by automatically downloading and installing a newer simplejson in egg or source form. Which isn't what #555 is requesting.

[zooko]

Sigh, 8148366d93c34dbe unapplies this patch, again, because setuptools doesn't create the easy-install.pth, setuptools.pth, and site.py files when we add --multi-version.

​http://bugs.python.org/setuptools/issue57 # develop doesn't create .pth files and site.py if --multi-version

[davidsarah]

Replying to zooko:

​http://bugs.python.org/setuptools/issue57 # develop doesn't create .pth files and site.py if --multi-version

which got closed with the following comment:

Behavior is as intended. More precisely, it is an intended feature of

--multi-version that it does not require .pth or site.py support in the target directory.

Can someone translate this to English for those unfamiliar with the vagaries of Python package management? Is this just more of the same "setuptools is broken-by-design" that seems to be causing so many other problems?

[zooko]

See ​http://tahoe-lafs.org/pipermail/tahoe-dev/2010-October/005324.html

[zooko@…]

In 98ffbfb31faccdaf:

setup: add --multi-version to the "setup.py develop" command-line
fixes #530. I earlier tried this twice (see #530 for history) and then twice rolled it back due to some problems that arose. However, I didn't write down what the problems were in enough detail on the ticket that I can tell today whether those problems are still issues, so here goes the third attempt. (I did write down on the ticket that it would not create site.py or .pth files in the target directory with --multi-version mode, but I didn't explain why *that* was a problem.)

[warner]

After this change, a "python setup.py build" that shouldn't do anything (i.e. do it twice and capture the output from the second run) emits 353 lines, including 16 copies of a warning about how to use versioned dependencies in the importing code:

Using /home/warner/stuff/tahoe/darcs-trunk/support/lib/python2.6/site-packages/zbase32-1.1.2-py2.6.egg

Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:

    pkg_resources.require("zbase32")  # latest installed version
    pkg_resources.require("zbase32==1.1.2")  # this exact version
    pkg_resources.require("zbase32>=1.1.2")  # this version or higher


Note also that the installation directory must be on sys.path at runtime for
this to work.  (e.g. by being the application's script directory, by being on
PYTHONPATH, or by being added to sys.path by your code.)

I suppose that this is a benign warning, but there's too much output there for me to tell: I first assumed that it was a fatal error.

After building it, trying to run setup.py test on my debian box failed (after emitting the same 353 lines of warnings):

pkg_resources.VersionConflict: (pycryptopp 0.5.17 (/usr/lib/pymodules/python2.6), Requirement.parse('pycryptopp>=0.5.20'))

That 0.5.17 is from the current debian/sid python-pycryptopp .deb package, which includes .egg-info data (and pkg_resources.require("pycryptopp") returns 0.5.17).

Does --multi-version break if there is a version on PYTHONPATH that was installed without --multi-version?

[warner]

Running python -c 'import pkg_resources; print pkg_resources.require("pycryptopp")', either from inside or outside my tahoe tree, reports:

[pycryptopp 0.5.17 (/usr/lib/pymodules/python2.6)]

Amending that to use "pycryptopp>=0.5.20" results in an exception in both places:

64:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("pycryptopp>=0.5.20")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 654, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 556, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (pycryptopp 0.5.17 (/usr/lib/pymodules/python2.6), Requirement.parse('pycryptopp>=0.5.20'))

Adding support/lib/python2.6/site-packages to my PYTHONPATH (which is what I vaguely remember we used to do inside bin/tahoe) doesn't change its behavior. This is the same error I'm seeing when I do setup.py test.

I started with a setup.py build, so my support/lib/python2.6/site-packages directory contains:

55:warner@fluxx% ls support/lib/python2.6/site-packages/
allmydata-tahoe.egg-link  foolscap-0.5.1-py2.6.egg/                  setuptools-0.6c16dev2.egg/
argparse-1.1-py2.6.egg/   pycryptopp-0.5.25-py2.6-linux-x86_64.egg/  zbase32-1.1.2-py2.6.egg/
darcsver-1.6.3.egg/       pyutil-1.7.12-py2.6.egg/                   zfec-1.4.7-py2.6-linux-x86_64.egg/

I wonder if that -linux-x86_64 in there is causing problems.

[davidsarah]

Reopening until the warnings in comment:9 are fixed.

[zooko]

Maybe we don't need --multi-version? Now that we have a reliable test of this behavior (see the step "test-with-fake-pkg" on buildbot, or run python misc/build_helpers/test-with-fake-pkg.py), maybe we should try removing --multi-version and see if that test stays green.

[davidsarah]

Replying to zooko:

Maybe we don't need --multi-version? Now that we have a reliable test of this behavior (see the step "test-with-fake-pkg" on buildbot, or run python misc/build_helpers/test-with-fake-pkg.py), maybe we should try removing --multi-version and see if that test stays green.

+1

[zooko]

I created a ticket530 branch to test patches on the buildbot. Then I forced a build on all buildbots with the current ticket530 (which was unchanged from the current trunk), and now I will push a patch to remove --multi-version (rolling back that part of 98ffbfb31faccdaf) to see if it fares better or worse than trunk.

[zooko]

Okay I committed 5f61bad92d8766e7 to the source:ticket530 branch.

[zooko]

Removing --multi-version didn't induce any regressions on buildbot. I guess I would like to add a unit test making sure that build doesn't emit certain scary warnings, which test will go from red to green when I commit 5f61bad92d8766e7 to trunk...

[zooko]

Fixed in 5f61bad92d8766e7. I decided not to add a unit tests as I couldn't think of a way to check whether the output of build was too scary without "overfitting" and just checking for this particular message text.