rusty dusty server fails on tests

[arch_o_median]

I'm installing on a:

Hardware: Pentium III; 64 Mb RAM; 731 Mhz OS: Debian 5.0 Lenny 2.6.26-1-686

The attached file includes:

1) dpkg_l_Log: the output of a dpkg -l

"included because I installed non-base packages including python-devel gcc and darcs 2.1.2"

2) buildlog and testlog[_2]: outputs of setup.py build and test respectively

Attachments have brief descriptive headers echoing the commandlines that invoked them.

[arch_o_median]

build test logs

[zooko]

The test log includes failures, starting with this:

allmydata.test.test_immutable
  Test
    test_download ...                                                      [OK]
    test_download_abort_if_too_many_corrupted_shares ...                   [OK]
    test_download_abort_if_too_many_missing_shares ...                  [ERROR]
                 [ERROR]
                 [ERROR]
    test_download_from_only_3_remaining_shares ...                      [ERROR]
                     [ERROR]
                     [ERROR]
    test_download_from_only_3_shares_with_good_crypttext_hash ...       [ERROR]
      [ERROR]
    test_test_code ...                                                  [ERROR]
                                                 [ERROR]

Would you please re-run just these tests (so that it won't take all day), with python ./setup.py test -s allmydata.test.test_immutable, let it run to completion, and attach the output as well as the contents of _trial_temp/test.log ?

[arch_o_median]

Replying to zooko:

Here they come!

[arch_o_median]

Replying to arch_o_median:

Replying to zooko:

test.log is _trial_temp/test.log

[zooko]

What? Tests passed this time! Darn! Nondeterministic computing!

Could you please try to figure out what makes the tests pass or fail? Perhaps it is that when you run the full suite then the test_immutable part fails? Perhaps it has to do with whether the machine is loaded with other processes at the same time? Or maybe you should have to run test_immutable several times in a row to see it fail?

[zooko]

Handy dandy shell script:

/bin/true # to set $? to 0
while [ $? = 0 ] ; do
    python ./setup.py test -s allmydata.test.test_immutable
done

That will execute the test_immutable tests over and over, stopping if python exits with a non-zero exit code, which it will if any test fails.

[arch_o_median]

OK, running overnight or until test failure... using my very first from scratch BASH script of more that a couple of lines.

[arch_o_median]

I'm running "python ./setup.py test" repeatedly. I plan to let it run for 72 hours.

[zooko]

Wait a minute! You mean that when you reran the exact same test which earlier produced "test.log" which you attached in ​http://allmydata.org/trac/tahoe/ticket/659#comment:1 , it didn't produce the same output? (The test_immutable and some other ones didn't fail?)

[arch_o_median]

Script used to run test on a loop.

[arch_o_median]

I ran it using the attached script. Maybe it doesn't do what I think it does? I assume a failed test produces a non-zero exit code.

The script ran 95 times, before I killed it.

I have changed state on the system. In particular I darcs "got" the latest version of tahoe and put it in a directory parallel to the one containing the version obtained by walking through the install. I ran setup.py in the darcs version. I then rm -rf'd the darcs-version directory, and reacquired the "walk-through" 1.3.0 version, and built it. The tests I mention were run on this reinstalled version of 1.3.0, so if tahoe changed state outside of its install directory (I also rm -rf'd ~/.tahoe) then I've made things harder on myself.

I've also apt-get install vim. I _may_ have apt-get installed other packages but I don't remember for sure and I haven't convinced dpkg to tell me what I installed most recently, yet.

I notice that the ERROR's in the first and second logs here: ​http://allmydata.org/trac/tahoe/attachment/ticket/659/old_sys_inst_setup_test_logs.tar.bz2 are not in the same tests. The second log passes test_immutable, but produces errors in test_repairer.

[zooko]

Maybe it doesn't do what I think it does? I assume a failed test produces a non-zero exit code.

Did you capture the stdout/stderr so you can grep it and see if any tests failed?

[zooko]

if tahoe changed state outside of its install directory (I also rm -rf'd ~/.tahoe)

It shouldn't have changed anything outside of those two directories. In fact, just running the unit tests shouldn't have changed anything outside the install directory.

[zooko]

I have changed state on the system. In particular I darcs "got" the latest version of tahoe and put it in a directory parallel to the one containing the version obtained by walking through the install. I ran setup.py in the darcs version. I then rm -rf'd the darcs-version directory, and reacquired the "walk-through" 1.3.0 version, and built it. The tests I mention were run on this reinstalled version of 1.3.0,

Which tests? Maybe you could identify which tests you mean using a URL to their log file or something.

I still don't understand why it failed one time but didn't fail another time. I think, but am not entirely certain, that you actually ran the same tests on the same source code with the same system both times, which implies that there is a non-determinism problem in the tests (probably a timing problem). But can you confirm that this is what you did? I really want a report of a controlled experiment in which the exact same test produced two different results.

By the way, I like the command_repeater.sh script!

[arch_o_median]

Replying to zooko:

Wait a minute! You mean that when you reran the exact same test which earlier produced "test.log" which you attached in ​http://allmydata.org/trac/tahoe/ticket/659#comment:1 , it didn't produce the same output? (The test_immutable and some other ones didn't fail?)

Not quite:

When I ran "python ./setup.py test" the first time I got this result: ​http://allmydata.org/trac/tahoe/attachment/ticket/659/test_all_one

When I ran "python ./setup.py test" the second time I got this results: ​http://allmydata.org/trac/tahoe/attachment/ticket/659/test_all_two

When I ran "python ./setup.py test -s allmydata.test.test_immutable" 96 times I generated 0 [ERRORS].

The [ERROR] sets generated by "setup.py test" are not the same.

Given the above observations I thought it would be interesting to look at patterns in [ERROR] messages generated across a large number of "setup.py test" runs. I had resolved to do this when a new development arose. The Rusty Dusty server, died.

That is it generated some terrifying error message and became totally non-responsive. Unfortunately this occurred several days ago and I cannot remember the specifics of the error. I eventually powered the machine down. Today when I pressed the power button to bring it back up, it started emitting a regular BEEEP BEEEP noise, and gave no other signs of life (e.g. no activity on monitor). I did this twice.

My new plan is to run the above tests on a different old desktop, while debugging Rusty.

[zooko]

Heh heh heh. Closing this ticket as "invalid". You should try running memtest86 and other such diagnostics.