[tahoe-dev] [tahoe-lafs] #598: add 'tahoe backup' command: fast versioned readonly backups
tahoe-lafs
trac at allmydata.org
Fri Feb 20 15:46:42 PST 2009
#598: add 'tahoe backup' command: fast versioned readonly backups
-----------------------------------+----------------------------------------
Reporter: warner | Owner:
Type: enhancement | Status: closed
Priority: major | Milestone: 1.3.0
Component: code-frontend-cli | Version: 1.2.0
Resolution: fixed | Keywords:
Launchpad_bug: |
-----------------------------------+----------------------------------------
Comment(by azazel):
I've done some little benchmarks of uploading one of my darcs repos to the
production grid. I've uploaded it first using "tahoe cp -r -v" and then i
uploaded a tar (not zipped) of the same data. The repo is composed of 67
dirs and 4098 files, the tar size is 27 MB. The "cp -r -v" took roughly
3.5 hours, the "cp repo.tar" took 760 seconds.
The client is configured to use an helper.
Here are the stats for one of the files involved in the first upload:
{{{
* Timings:
o File Size: 3424 bytes
o Total: 3.88s (882Bps)
+ Storage Index: 194us (17.64MBps)
+ [Contacting Helper]: 723ms
# [Helper Already-In-Grid Check]: 228ms
+ [Upload Ciphertext To Helper]: 352ms (9.7kBps)
+ Peer Selection: 879ms
+ Encode And Push: 1.05s (69.9kBps)
# Cumulative Encoding: 705us (4.86MBps)
# Cumulative Pushing: 48ms (71.0kBps)
# Send Hashes And Close: 881ms
+ [Helper Total]: 3.37s
}}}
Next, the stats for the tar upload:
{{{
* Timings:
o File Size: 27176960 bytes
o Total: 760.13s (35.8kBps)
+ Storage Index: 261us (104099.02MBps)
+ [Contacting Helper]: 702ms
# [Helper Already-In-Grid Check]: 454ms
+ [Upload Ciphertext To Helper]: 723.25s (37.6kBps)
+ Peer Selection: 461ms
+ Encode And Push: 35.01s (807.5kBps)
# Cumulative Encoding: 1.50s (18.13MBps)
# Cumulative Pushing: 32.16s (845.1kBps)
# Send Hashes And Close: 996ms
+ [Helper Total]: 759.47s
}}}
This small test demostrated an overhead of 1.5 ~ 2 seconds for every
upload operation.
Lastly i post the results of a "du --si $repo; find $repo -type f |wc -l;
find $repo -type d |wc-l" command:
{{{
33k wip/cute/lib/python2.5/site-packages/fixture-1.1.1-py2.5.egg/EGG-
INFO
136k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture/command/generate
144k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture/command
25k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture/examples/db
29k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture/examples
115k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture/loadable
29k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture/setup_cmd
78k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture/test/test_command/test_generate
82k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture/test/test_command
213k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture/test/test_loadable
426k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture/test
922k wip/cute/lib/python2.5/site-
packages/fixture-1.1.1-py2.5.egg/fixture
955k wip/cute/lib/python2.5/site-packages/fixture-1.1.1-py2.5.egg
353k wip/cute/lib/python2.5/site-
packages/zope.schema-3.5.0a2-py2.5.egg/zope/schema/tests
627k wip/cute/lib/python2.5/site-
packages/zope.schema-3.5.0a2-py2.5.egg/zope/schema
635k wip/cute/lib/python2.5/site-
packages/zope.schema-3.5.0a2-py2.5.egg/zope
54k wip/cute/lib/python2.5/site-packages/zope.schema-3.5.0a2-py2.5.egg
/EGG-INFO
689k wip/cute/lib/python2.5/site-packages/zope.schema-3.5.0a2-py2.5.egg
46k wip/cute/lib/python2.5/site-
packages/zope.interface-3.5.0-py2.5-linux-i686.egg/zope/interface/common/tests
168k wip/cute/lib/python2.5/site-
packages/zope.interface-3.5.0-py2.5-linux-i686.egg/zope/interface/common
267k wip/cute/lib/python2.5/site-
packages/zope.interface-3.5.0-py2.5-linux-i686.egg/zope/interface/tests
1,0M wip/cute/lib/python2.5/site-
packages/zope.interface-3.5.0-py2.5-linux-i686.egg/zope/interface
1,1M wip/cute/lib/python2.5/site-
packages/zope.interface-3.5.0-py2.5-linux-i686.egg/zope
87k wip/cute/lib/python2.5/site-
packages/zope.interface-3.5.0-py2.5-linux-i686.egg/EGG-INFO
1,1M wip/cute/lib/python2.5/site-
packages/zope.interface-3.5.0-py2.5-linux-i686.egg
21k wip/cute/lib/python2.5/site-
packages/zope.event-3.4.0-py2.5.egg/zope/event
29k wip/cute/lib/python2.5/site-
packages/zope.event-3.4.0-py2.5.egg/zope
29k wip/cute/lib/python2.5/site-packages/zope.event-3.4.0-py2.5.egg
/EGG-INFO
58k wip/cute/lib/python2.5/site-packages/zope.event-3.4.0-py2.5.egg
41k wip/cute/lib/python2.5/site-
packages/zope.component-3.5.1-py2.5.egg/zope/component/bbb
29k wip/cute/lib/python2.5/site-
packages/zope.component-3.5.1-py2.5.egg/zope/component/testfiles
672k wip/cute/lib/python2.5/site-
packages/zope.component-3.5.1-py2.5.egg/zope/component
680k wip/cute/lib/python2.5/site-
packages/zope.component-3.5.1-py2.5.egg/zope
95k wip/cute/lib/python2.5/site-
packages/zope.component-3.5.1-py2.5.egg/EGG-INFO
775k wip/cute/lib/python2.5/site-
packages/zope.component-3.5.1-py2.5.egg
4,0M wip/cute/lib/python2.5/site-packages
13k wip/cute/lib/python2.5/distutils
4,0M wip/cute/lib/python2.5
4,0M wip/cute/lib
0 wip/cute/include
1,2M wip/cute/bin
5,3M wip/cute/cute/_darcs/pristine.hashed
13M wip/cute/cute/_darcs/patches
21k wip/cute/cute/_darcs/prefs
435k wip/cute/cute/_darcs/inventories
19M wip/cute/cute/_darcs
463k wip/cute/cute/docs/tutorial/images
517k wip/cute/cute/docs/tutorial
0 wip/cute/cute/docs/experiments
517k wip/cute/cute/docs
29k wip/cute/cute/lib/cute/app
25k wip/cute/cute/lib/cute/ui/widgets
91k wip/cute/cute/lib/cute/ui/resources
13k wip/cute/cute/lib/cute/ui/designer_plugins
13k wip/cute/cute/lib/cute/ui/ui
8,2k wip/cute/cute/lib/cute/ui/test
304k wip/cute/cute/lib/cute/ui
8,2k wip/cute/cute/lib/cute/db/search
17k wip/cute/cute/lib/cute/db/source
91k wip/cute/cute/lib/cute/db
3,6M wip/cute/cute/lib/cute/tests/sample_data/birt/images/logos
263k wip/cute/cute/lib/cute/tests/sample_data/birt/images/productlines
4,1M wip/cute/cute/lib/cute/tests/sample_data/birt/images
4,3M wip/cute/cute/lib/cute/tests/sample_data/birt
4,3M wip/cute/cute/lib/cute/tests/sample_data
4,3M wip/cute/cute/lib/cute/tests
4,8M wip/cute/cute/lib/cute
4,8M wip/cute/cute/lib
24M wip/cute/cute
33M wip/cute
3647
70
}}}
--
Ticket URL: <http://allmydata.org/trac/tahoe/ticket/598#comment:9>
tahoe-lafs <http://allmydata.org>
secure decentralized file storage grid
More information about the tahoe-dev
mailing list