IF the mutable file update problem is due to memory use problems with downloading the file, keeping it in memory then re-uploading it, then (for this use case) it is worth the overhead (to me) for now.<div><br></div><div>IF there are other issues that I need to be aware of, please let me know.</div>
<div><br></div><div>Question: Why was the decision made to download the file before uploading the new version?</div><div><br></div><div>thanks,</div><div>jody<br clear="all">----<br>- Think carefully.<br>- Contra mundum - "Against the world" (St. Athanasius)<br>
- Credo ut intelliga - "I believe that I may know" (St. Augustin of Hippo)<br>
<br><br><div class="gmail_quote">On Mon, Jan 4, 2010 at 11:33 AM, Peter Secor <span dir="ltr"><<a href="mailto:secorp@allmydata.com">secorp@allmydata.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
You are correct Kevan, the hard limit has been removed, sorry I didn't<br>
realize that had been fixed already!<br>
<br>
So, the limit is removed, but Jody you may want to read the ticket #694<br>
notes, there is a good discussion of why we originally had a hard-limit.<br>
<br>
Ps<br>
<div class="im"><br>
On 1/4/10 10:24 AM, Kevan Carstensen wrote:<br>
> I'm sure someone will correct me if I'm wrong, but I think #694 removed<br>
> the hard limit on mutable files. So your data should be backed up.<br>
><br>
> (apologies for formatting - I'm on a phone)<br>
> --<br>
</div>> Kevan Carstensen | <<a href="mailto:kevan@isnotajoke.com">kevan@isnotajoke.com</a> <mailto:<a href="mailto:kevan@isnotajoke.com">kevan@isnotajoke.com</a>>><br>
<div class="im">><br>
> On Jan 4, 2010, at 10:10 AM, Jody Harris <<a href="mailto:imhavoc@gmail.com">imhavoc@gmail.com</a><br>
</div><div class="im">> <mailto:<a href="mailto:imhavoc@gmail.com">imhavoc@gmail.com</a>>> wrote:<br>
><br>
</div><div class="im">>> Ack!<br>
>><br>
>> So, I'm NOT backing up my data!? (I'm just backing up the first 2MB of<br>
>> each compressed table?)<br>
>><br>
>> Okay, if immutable files are my only choice, how do I do garbage<br>
>> collection to insure that a minimum of dead space is taken up by<br>
>> "yesterday's" files?<br>
>><br>
>> jody<br>
>> ----<br>
>> - Think carefully.<br>
>> - Contra mundum - "Against the world" (St. Athanasius)<br>
>> - Credo ut intelliga - "I believe that I may know" (St. Augustin of Hippo)<br>
>><br>
>><br>
>> On Mon, Jan 4, 2010 at 11:05 AM, Peter Secor <<br>
</div>>> <mailto:<a href="mailto:secorp@allmydata.com">secorp@allmydata.com</a>><a href="mailto:secorp@allmydata.com">secorp@allmydata.com</a><br>
<div class="im">>> <mailto:<a href="mailto:secorp@allmydata.com">secorp@allmydata.com</a>>> wrote:<br>
>><br>
>> Mutable files currently have a small (2MB?) limit and cannot be<br>
>> used for<br>
>> larger files. Immutable files (the default) have a much higher limit<br>
>> (12GB now?) and are much better suited for the purpose you<br>
>> described of<br>
>> large snapshots of data that won't change later.<br>
>><br>
>> There are ideas behind larger mutable files (#393) but they are not<br>
>> implemented yet.<br>
>><br>
>> Ps<br>
>><br>
>> On 1/4/10 7:37 AM, Jody Harris wrote:<br>
>> > Oh, I forgot to answer the "how?"<br>
>> ><br>
>> > ... put --mutable ...<br>
>> ><br>
>> > jody<br>
>> > ----<br>
>> > - Think carefully.<br>
>> > - Contra mundum - "Against the world" (St. Athanasius)<br>
>> > - Credo ut intelliga - "I believe that I may know" (St. Augustin<br>
>> of Hippo)<br>
>> ><br>
>> ><br>
>> > On Sun, Jan 3, 2010 at 10:59 PM, Jody Harris <<br>
</div>>> <mailto:<a href="mailto:imhavoc@gmail.com">imhavoc@gmail.com</a>><a href="mailto:imhavoc@gmail.com">imhavoc@gmail.com</a> <mailto:<a href="mailto:imhavoc@gmail.com">imhavoc@gmail.com</a>><br>
>> > <mailto: <mailto:<a href="mailto:imhavoc@gmail.com">imhavoc@gmail.com</a>><a href="mailto:imhavoc@gmail.com">imhavoc@gmail.com</a><br>
<div class="im">>> <mailto:<a href="mailto:imhavoc@gmail.com">imhavoc@gmail.com</a>>>> wrote:<br>
>> ><br>
>> > I have a script that:<br>
>> > - iterates though my MySQL tables<br>
>> > - dumps each table<br>
>> > - pipes the dump through gzip<br>
>> > - write the file to a temp file<br>
>> > - copies the file to [remote storage] (now Tahoe grid)<br>
>> > - deletes temp file<br>
>> ><br>
>> > This gives me a daily snapshot of all of the active databases on my<br>
>> > server every night.<br>
>> ><br>
>> > The current backup is ~ 300 MB/night, which isn't bad, but after one<br>
>> > year, that's 110+GB x 3.3 and counting of consumed tahoe space.<br>
>> > Unless tahoe has a better garbage collection system than I have come<br>
>> > to believe, that's going to be a problem on a 500 GB grid.<br>
>> ><br>
>> > In my reading of the tahoe docs, I have not come across any alarms<br>
>> > concerning mutable files. Is there something I should be made<br>
>> aware of?<br>
>> ><br>
>> > jody<br>
>> > ----<br>
>> > - Think carefully.<br>
>> > - Contra mundum - "Against the world" (St. Athanasius)<br>
>> > - Credo ut intelliga - "I believe that I may know" (St. Augustin of<br>
>> > Hippo)<br>
>> ><br>
>> ><br>
>> > On Sun, Jan 3, 2010 at 10:50 PM, Zooko O'Whielacronx<br>
</div>>> > < <mailto:<a href="mailto:zookog@gmail.com">zookog@gmail.com</a>><a href="mailto:zookog@gmail.com">zookog@gmail.com</a><br>
>> <mailto:<a href="mailto:zookog@gmail.com">zookog@gmail.com</a>> <mailto:<br>
>> <mailto:<a href="mailto:zookog@gmail.com">zookog@gmail.com</a>><a href="mailto:zookog@gmail.com">zookog@gmail.com</a><br>
<div class="im">>> <mailto:<a href="mailto:zookog@gmail.com">zookog@gmail.com</a>>>> wrote:<br>
>> ><br>
>> > On Sun, Jan 3, 2010 at 10:33 PM, Jody Harris<br>
</div>>> > < <mailto:<a href="mailto:havoc@harrisdev.com">havoc@harrisdev.com</a>><a href="mailto:havoc@harrisdev.com">havoc@harrisdev.com</a><br>
>> <mailto:<a href="mailto:havoc@harrisdev.com">havoc@harrisdev.com</a>> <mailto:<br>
>> <mailto:<a href="mailto:havoc@harrisdev.com">havoc@harrisdev.com</a>><a href="mailto:havoc@harrisdev.com">havoc@harrisdev.com</a><br>
<div><div></div><div class="h5">>> <mailto:<a href="mailto:havoc@harrisdev.com">havoc@harrisdev.com</a>>>> wrote:<br>
>> > > I have set up my web server to store backup files on my tahoe<br>
>> > grid, shifting<br>
>> > > from the use of a server at my house. One thing that<br>
>> > initially alarmed me<br>
>> > > was that the timestamps of the backed up files is not updated<br>
>> > in the<br>
>> > > tahoe-lafs interface. I did confirm that the shares<br>
>> > timestamps are updated.<br>
>> ><br>
>> > Tahoe-LAFS doesn't store timestamps or other metadata on files, only<br>
>> > on links to files, which links are stored in directories. So if you<br>
>> > have a directory which contains a link named "README.txt" to<br>
>> > version 1<br>
>> > of your file, and then you upload version 2 of your file and you<br>
>> > change that directory to link to version 2, then Tahoe-LAFS will<br>
>> > update the timestamp on that link named "README.txt" when it updates<br>
>> > the link to point to the new file.<br>
>> ><br>
>> > Did you say "mutable files"? You should probably not be using<br>
>> > mutable<br>
>> > files for anything. Immutable files (plus directories) probably fit<br>
>> > all your needs for backups, and mutable files have significant<br>
>> > performance problems.<br>
>> ><br>
>> > How do you trigger these backups -- are you using "tahoe cp" or<br>
>> > "tahoe<br>
>> > backup" or sending HTTP requests to the WAPI or something else?<br>
>> ><br>
>> > Regards,<br>
>> ><br>
>> > Zooko<br>
>> > _______________________________________________<br>
>> > tahoe-dev mailing list<br>
</div></div>>> > <mailto:<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a>><a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a><br>
>> <mailto:<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a>> <mailto:<br>
>> <mailto:<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a>><a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a><br>
<div class="im">>> <mailto:<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a>>><br>
>> ><br>
</div>>> <<a href="http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev" target="_blank">http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev</a>><a href="http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev" target="_blank">http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev</a><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > tahoe-dev mailing list<br>
>> > <mailto:<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a>><a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a><br>
<div class="im">>> <mailto:<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a>><br>
>> ><br>
</div>>> <<a href="http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev" target="_blank">http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev</a>><a href="http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev" target="_blank">http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev</a><br>
>> _______________________________________________<br>
>> tahoe-dev mailing list<br>
>> <mailto:<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a>><a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a><br>
<div class="im">>> <mailto:<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a>><br>
</div>>> <<a href="http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev" target="_blank">http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev</a>><a href="http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev" target="_blank">http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev</a><br>
<div><div></div><div class="h5">>><br>
>><br>
>> _______________________________________________<br>
>> tahoe-dev mailing list<br>
>> <a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a> <mailto:<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a>><br>
>> <a href="http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev" target="_blank">http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev</a><br>
><br>
><br>
> _______________________________________________<br>
> tahoe-dev mailing list<br>
> <a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a><br>
> <a href="http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev" target="_blank">http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev</a><br>
_______________________________________________<br>
tahoe-dev mailing list<br>
<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a><br>
<a href="http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev" target="_blank">http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev</a><br>
</div></div></blockquote></div><br></div>