v1.4.1 storage servers sending a negative number for maximum-immutable-share-size?

[zooko]

I'm looking at the logs of allmydata.com user gar5 and I see this. I believe the storage servers in question are tahoe-lafs v1.4.1 storage servers operated by allmydata.com:

05:16:27.571 [3121]: connected to pfavfmv3, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5412811776L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
05:16:27.572 [3122]: connected to 5ws3m43h, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5369373696L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
05:16:27.573 [3123]: connected to hndn3zdv, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5419058176L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
05:16:27.573 [3124]: connected to gsulyyiv, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5401027584L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
05:16:27.574 [3125]: connected to 7yun2nc3, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5405422592L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}

[zooko]

Hm, but in the same flog incident file, other 1.4.1 servers report positive numbers:

local#118354 11:38:18.121: connected to f7342k6g, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 283901230080L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
local#118355 11:38:18.122: connected to 5n3rnbx3, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 267428404224L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
local#118356 11:38:18.123: connected to p3677neu, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 274421722112L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
local#118357 11:38:18.124: connected to ipxyurtk, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 270720273408L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
local#118358 11:38:18.125: connected to yeelbkr2, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 276353047552L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
local#118359 11:38:18.126: connected to jtobcyqt, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 270300511232L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}

[warner]

I'd suspect a problem with the how-much-space-is-left calculation, since I think the maximum-immutable-share-size value that each server reports is actually min(format_support, unreserved_space_left). If the server is overfull and there's not a clamp at zero, you could get this sort of thing.

I also vaguely remember fixing this problem at some point, maybe after 1.4.1?

[zooko]

Hm, it looks like this bug is still in place in trunk: storage/server.py#L209​. How to test it? Maybe monkeypatch in a fake os.statvfs()?

And how do clients currently behave when storage servers tell them that they have negative "maximum-immutable-share-size"? immutable/upload.py#L187​ They just compare whether it is >= the space that they need, so they work fine when it reports a negative number. However, it seems better if they would report 0 instead of a negative number.

[zooko]

If you like this ticket you might also like #637 (support "keep this much disk space free" on Windows as well as other platforms).

[davidsarah]

The current code for get_available_space in storage/server.py subtracts the 'reserved_space' value from the free space available to non-root users, without clamping to zero. I had already spotted that and changed it to clamp to zero in the (unfinished) patch for #637.

It is also possible for the maximum-immutable-share-size to be 264 if os.statvfs is not supported (or if neither os.statvfs nor win32api.GetDiskFreeSpaceEx? is supported after #637 is fixed). Will 264 be JSON-encoded properly?

[davidsarah]

Just to confirm that we have the right explanation, what was the reserved_space set to on gar5?

[davidsarah]

Replying to davidsarah:

Just to confirm that we have the right explanation, what was the reserved_space set to on gar5?

I mean, on the servers listed in the log. (I misread gar5 as a server name.)

[zooko]

Well for example this one: 05:16:27.571 [3121]: connected to pfavfmv3, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5412811776L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'} Has reserved_space = 10G in its tahoe.cfg file.

[davidsarah]

That makes sense (if reserved_space had been less than 5412811776 bytes, i.e. ~5.4 GB, it would have excluded this explanation).

[davidsarah]

This should be fixed by the latest patch attached to #637.

[davidsarah]

This was reviewed with #637.

[warner]

This has been subsumed by #637, which was just committed, so this can be closed.