#437 new defect

automatically schedule tests of large files

Reported by: zooko Owned by: somebody
Priority: major Milestone: eventually
Component: dev-infrastructure Version: 1.0.0
Keywords: test large Cc:
Launchpad Bug:


#435 (automate testing of large files) is to create an automated test which, once launched, can determine if a given version of Tahoe correctly uploads and downloads large files. For starters, we need to test files > 4 GiB, and also we need to test files > 12 GiB and probably also even larger in the future.

This ticket is to automatically schedule those tests to get run under certain conditions. Obviously running such a test can be expensive in terms of disk, CPU, and network, so we don't necessarily want to run such a test on the normal schedule of all of our unit tests, which are done on every darcs commit on all test builders.

Also running such a test takes a long time, so we don't want to habitually block other processes (such as running other unit tests or building packages) on such a large-file test completing.

Brian has already expressed a desire that large-file tests are not automatically launched at all, and instead just get launched by the specific decision of a human user. I disagree and would like for us to have an automated policy for running large-file tests. If we go with Brian's preference then we'll just close this ticket as "invalid" or "wontfix".

Here are some policies which are easily implemented in buildbot which can reduce the problems of an expensive test like this:

  1. We can limit the large-file test to only certain buildslaves.
  1. We can make it so that other processes go ahead without waiting to see the result of the large-file test (such as other unit tests, buildbot results e-mails, building of packages, etc.)
  1. Buildbot normally always makes it so that a new test won't launch if one is already in progress, and so that if multiple triggers for a test accumulate then only one test is needed to satisfy all those triggers. That can help with this.
  1. We could make it so that large-file tests are not triggered by darcs commit activity, but instead are triggered once per day, or once per week, or whatever.

We could implement more detailed policies, such as "do it every day at midnight in UTC-7 if there was any darcs commit that day" or "if it takes X hours to run the test, then wait at least 3 * X hours before starting the next test", but I don't know off the top of my head how easy it is to program buildbot for such a policy.

Change History (5)

comment:1 Changed at 2008-06-02T19:30:12Z by warner

yeah, running it in an automated fashion like once a week seems like it could be reasonable. How about we create a target in the Makefile called "make expensive-test", and then have a buildslave that runs it once a week or something? of course, first we need to write the test and measure how long it actually takes, and then decide on what hardware we want to use for this purpose (after seeing just how slammed the machine gets)

comment:2 Changed at 2008-06-02T21:16:18Z by zooko

Sounds good.

comment:3 Changed at 2008-06-04T01:17:52Z by zooko

  • Milestone changed from 1.1.0 to 1.1.1

comment:4 Changed at 2009-03-28T19:41:51Z by zooko

  • Milestone changed from 1.3.1 to eventually

Moving this out of the 1.3.1 Milestone.

comment:5 Changed at 2010-02-11T03:33:47Z by davidsarah

  • Keywords test large added
Note: See TracTickets for help on using tickets.