#717 closed defect (fixed)

unnecessary rebuild of dependencies when tahoe-deps/ is present

Reported by: warner Owned by: cgalvan
Priority: major Milestone: 1.11.0
Component: packaging Version: 1.4.1
Keywords: setuptools install Cc: nicolas.pouillard@…
Launchpad Bug:

Description (last modified by warner)

Using a trunk 9a02f4a1043d6288 tree, on my debian/sid box, performing "python setup.py build" with the tahoe-deps tarball unpacked in the parent directory causes several dependency libraries to be built and installed in support/lib/python2.5/site-packages even though those libraries are already present, usable, and visible to pkg_resources.requires in the standard PYTHONPATH /usr/lib/python2.5/site-packages directory.

The libraries that are unnecessarily rebuilt are:

  • foolscap-0.4.1
  • twisted-8.2.0
  • simplejson-2.0.9

Removing ../tahoe-deps, erasing support/lib, and re-running "python setup.py build" correctly skips the build of these libraries, resulting in a support/lib that only contains setuptools-0.6c12dev and the allmydata-tahoe.egg-link file (well, and the usual setuptools noise: setup.py, setuptools.pth, and easy-install.pth).

Something in setuptools is buggy: having ../tahoe-deps available should not influence the decision to install a library or not.

This needs to be added to the setuptools bugtracker at http://bugs.python.org/setuptools/ .

Change History (13)

comment:1 Changed at 2009-05-29T01:40:26Z by zooko

Thank you for the bug report. As I'm sure you know, a good way to report this bug to setuptools devs would be to create a minimal test case. Here's a crack at one:

Make a setup.py file in a new empty directory with the following contents:

from setuptools import setup
setup(name="minbug", install_requires="simplejson>=1.4", find_links="../tahoe-deps")

Then run python ./setup.py build, with and without a simplejson-2.0.9.tar.gz located in ../tahoe-deps.

comment:2 Changed at 2009-06-04T16:23:08Z by zooko

This may be related to http://bugs.python.org/setuptools/issue17 (easy_install will install a package that is already there). There is a work-around documented in that ticket.

comment:3 Changed at 2009-06-06T01:07:01Z by warner

I'm not sure it is related.. as I understand it, setuptools-17 is about setuptools building packages that are present in an unusual directory that the platform has added to sys.path (like the /var/lib/python-support/python2.5 on debian), probably because it isn't honoring the platform's changes to sys.path when it scans for those packages.

In contrast, what I observed was that all of the unecessarily-rebuilt libraries were present in /usr/lib/python2.5/site-packages . None were in /var/lib/python-support/python2.5 .

comment:4 Changed at 2009-08-06T18:30:42Z by zooko

This may be related to #657 ("python ./setup.py test" rebuilds packages).

comment:5 Changed at 2009-09-24T19:59:14Z by cgalvan

Note that once #668 is resolved, switching to Distribute, this issue may or may not be resolved as a result.

comment:6 Changed at 2009-10-14T15:08:20Z by zooko

It looks like this may be related to http://bugs.python.org/setuptools/issue65 (setuptools bug,inconsistent replacement of eggs when installing from file:// urls.) which is fixed in setuptools-0.6c10 prerelease. setuptools 65 is related to http://bugs.python.org/setuptools/issue17 (easy_install will install a package that is already there) and http://bugs.python.org/setuptools/issue20 (package required at build time seems to be not fully present at install time?).

comment:7 Changed at 2009-10-14T15:09:51Z by zooko

  • Owner changed from somebody to cgalvan

Maybe someone (cgalvan?) could test this issue with the setuptools v0.6c10.

comment:8 Changed at 2010-06-12T22:26:01Z by davidsarah

  • Keywords setuptools install added

comment:9 Changed at 2010-09-18T17:40:10Z by zooko

There was a different bug which caused a similar behavior in #229.

comment:10 follow-up: Changed at 2010-09-18T18:21:50Z by zooko

In comment:7:ticket:1168 and comment:10:ticket:1168 we commented on this issue. David-Sarah wrote:

I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running python setup.py trial. foolscap 0.5.1 is downloaded again on every run of trial, even though python setup.py build copied it into support/site-packages and the build completed successfully.

and I wrote:

Looks like it might be #717. Isn't the buildbot already testing this by running the "Desert Island build"? e.g: http://tahoe-lafs.org/buildbot/builders/clean/builds/2522

In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario (comment:8)?

Makefile

comment:11 in reply to: ↑ 10 Changed at 2010-09-18T20:35:42Z by davidsarah

Replying to zooko:

[...] Isn't the buildbot already testing this by running the "Desert Island build"? e.g: http://tahoe-lafs.org/buildbot/builders/clean/builds/2522

In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario (comment:8)?

In my scenario, I download a non-SUMO tarball, run python setup.py trial python setup.py build followed by python setup.py trial (expecting the dependencies to be downloaded and built before the test), and then run python setup.py trial again (expecting the previously downloaded dependencies to be used and not rebuilt). Are these expectations correct?

In any case, this is not the same thing as downloading tahoe-deps.tar.gz, untarring it in the distribution directory and then running python setup.py build, which is what test-desert-island does.

Last edited at 2010-09-18T20:38:49Z by davidsarah (previous) (diff)

comment:12 Changed at 2010-10-30T12:59:14Z by np

  • Cc nicolas.pouillard@… added

comment:13 Changed at 2016-03-26T21:22:37Z by warner

  • Description modified (diff)
  • Milestone changed from undecided to 1.11.0
  • Resolution set to fixed
  • Status changed from new to closed

We no longer provide or support the tahoe-deps tarball/directory, and the pip http+wheel cache works correctly to avoid rebuilds. Closing this out.

Note: See TracTickets for help on using tickets.