[tahoe-dev] N.B. Garbage Collection; TestGrid and ProdGrid changes

[Rob Kinninmont]

IMPORTANT: please read the first few paragraphs of this message, to
ensure the integrity of your data.  Thank you.


I will this week be commencing a Garbage Collection pass over the
allmydata.com production grid. Additionally we will be repurposing
storage from the test grid into the production grid.


If your files are all stored in directories rooted at an allmydata.com
account's "root dir", and you have no important data stored in the test
grid, you do not need to take any action.


If you are one of the small number of allmydata.com users who are
storing files in directories whose root caps you have kept private,
then you will need to act promptly to add leases to those files.
More details on that below.

If you have data stored in the test grid which you are not comfortable
losing, then you should act promptly to download and backup that data
via some other means.  The test grid is considered 'scratch' space for
development and testing, and should not be relied upon for storage of
important data, but I'm providing this 'heads up' in case anyone is
relying upon that storage.


If you have any questions or concerns, please don't hesitate to raise
them.  Thanks for your attention.

cheers,
rob



More detailed information:
--------------------------


Test Grid:

We're going to be reducing the capacity of the test grid in a few days,
and this will result in a loss of some of the data contained therein.
The test grid should remain functional for ongoing testing and
development throughout, but you should not rely on any data currently
within the test grid remaining accessible.  There's also a chance that
it might case false negatives in tests which are running at the time
that the capacity is reduced.

If you have any data not otherwise backed up, you should download it
as soon as possible.  If you have any questions or concerns about this,
please let me know.


Production Grid Garbage Collection:

Storage on the grid is technically _leased_ from the storage servers,
and barring failures, the storage servers agree to keep data available
until the expiration of any leases upon it.  After that point the data
is a candidate for storage reclamation, and in principle might be
deleted at any time.

The garbage collection process is a standard 'mark and sweep'.  All
data which should be retained is 'marked' by adding a fresh lease to
it.  Once the mark phase is complete, then the 'sweep' phase reclaims
space by deleting any data which has no unexpired leases upon it.

All data stored through the allmydata.com web service and backup client
are stored within a directory structure the root of which is recorded
in the user's allmdata.com account.  I will be updating leases on all
user data so reachable for current paid-up accounts before commencing
the sweep.  Hence if you only use the allmydata.com service via the
website and backup client you do not need to act further.

A small number of you, as tahoe developers, will have perhaps used
other features of tahoe to store data which is not reachable from such
an account; e.g. you have created a private directory, for example
accessed through the alias mechanism of the 'tahoe' command, and that
directory is not reachable from an account's root directory.

If that applies to you, you will need to add leases to that data
yourself before the sweep process runs.  It will probably take a week
or two to 'mark' all the allmydata.com accounts, but you should act as
soon as possible to ensure your leases are updated.  For each private
rootcap you have, you will need to run

     tahoe deep-check --add-lease $ROOTCAP

This might take a while to run, dependant on how many files and
directories are involved.  The deep-check runs at about 1-2 checks
per second (dependant on round-trip latency to the production grid
storage servers).  You can obtain summary statistics by adding --raw,
i.e. 'tahoe deep-check --raw --add-lease $ROOTCAP' which will print out
statistics in JSON format.

If you have any questions, concerns, or encounter problems with the
process of adding leases to your data, please let me know.

It would be helpful and instructive for us to get an idea of how many
people are using private rootcaps to maintain data on the production
storage grid.  While it's not necessary (unlike the add-lease which
_is_ necessary) it would be appreciated if you could drop me a line if
this applies to you, perhaps including an estimate of how much data is
involved.

Thanks for reading.

cheers,
rob