#167 closed enhancement (fixed)

try out Jerasure

Reported by: zooko Owned by: zooko
Priority: minor Milestone: undecided
Component: code-encoding Version: 0.6.0
Keywords: Cc:
Launchpad Bug:

Description

Professor James Plank has published a new library that has implementations of various erasure coding algorithms, Jerasure:

http://www.cs.utk.edu/~plank/plank/papers/CS-07-603.html

It would be fun to benchmark these against the current zfec

http://pypi.python.org/pypi/zfec

and if some algorithm in there is faster, switch the internals of zfec to use it instead.

Change History (3)

comment:1 Changed at 2007-10-04T19:19:34Z by zooko

  • Status changed from new to assigned

comment:2 Changed at 2008-06-01T20:58:19Z by warner

  • Milestone changed from eventually to undecided

comment:3 Changed at 2009-03-29T05:02:42Z by zooko

  • Resolution set to fixed
  • Status changed from assigned to closed

Prof. James Plank and co-authors (including me) did some good investigation about comparative performance:

http://www.cs.utk.edu/~plank/plank/papers/FAST-2009.html

The upshot is that zfec is definitely fast enough, and is actually competitive with the others in the sense of being consistently fast on different machines and different K and M combinations. The paper doesn't show graphs of numbers of check blocks (a.k.a. secondary shares) larger than 4, but I think the alternatives get slower with larger numbers of secondary shares faster than zfec gets slower. Also note that Jack Lloyd has contributed some code for zfec to use SIMD when appropriate for even better performance. Closing this as "fixed".

Note: See TracTickets for help on using tickets.