#173 new enhancement

How does tahoe filesystem layer (lafs) handle lots of file-upload tasks?

Reported by: zooko Owned by: somebody
Priority: major Milestone: eventually
Component: dev-infrastructure Version: 0.6.1
Keywords: scalability upload performance memory Cc:
Launchpad Bug:

Description

Rob, Peter, Justin and I just fixed up the commercial, proprietary cousin of Tahoe -- the http://allmydata.com product -- to behave better when the user asks it to upload 100,000 files ASAP. How would Tahoe handle a similar situation? To close this ticket, implement an automated measurement (akin to the memory and upload/download rate measurements) which measures how much RAM, wall-clock time, CPU load, etc. a Tahoe node requires to simultaneously upload 1000, 10,000, or 100,000 files.

Change History (9)

comment:1 Changed at 2007-10-19T23:11:38Z by zooko

  • Version changed from 0.6.0 to 0.6.1

comment:2 Changed at 2007-12-18T00:08:53Z by zooko

  • Keywords scaling added

comment:3 Changed at 2008-02-28T01:58:33Z by zooko

  • Milestone changed from undecided to 0.9.0 (Allmydata 3.0 final)

Fabrice, the entrepeneur behind Allmydata, says that our reference test should be 25,000 files and 100 GB -- "VERY WELL and RELIABLY!". He says this is necessary to be competitive with Carbonite, Mozy, etc.. He promises to raise the bar to 100,000 files and 500 GB as soon as we reach 25,000 files and 100 GB. ;-)

Putting this into the 0.9.0 (Allmydata 3.0 final) Milestone.

comment:4 Changed at 2008-03-10T19:23:28Z by zooko

  • Milestone changed from 0.9.0 (Allmydata 3.0 final) to 0.10.0

comment:5 Changed at 2008-03-10T19:57:45Z by zooko

  • Summary changed from How does tahoe handle lots of simultaneous file-upload tasks? to How does tahoe filesystem layer (laughfs) handle lots of file-upload tasks?

This ticket is just for the tahoe filesystem layer ("laughfs"). See also #337 (how does the whole system including the Windows native client handle lots of file-upload tasks?).

This ticket is hereby vagueified to be unclear about whether we're talking about lots of parallel upload tasks or a lot of sequential upload tasks or some combination thereof.

comment:6 Changed at 2008-03-10T20:04:18Z by warner

incidentally, I'm -1 on the "laughfs" term. I have no idea what it refers to.

comment:7 Changed at 2008-03-21T22:31:09Z by zooko

  • Summary changed from How does tahoe filesystem layer (laughfs) handle lots of file-upload tasks? to How does tahoe filesystem layer (lafs) handle lots of file-upload tasks?

comment:8 Changed at 2008-03-21T22:31:29Z by zooko

  • Milestone changed from 0.10.0 to undecided

comment:9 Changed at 2009-12-13T03:01:59Z by davidsarah

  • Keywords upload performance memory added; scaling removed
Note: See TracTickets for help on using tickets.