Obsolete:Media server/2011 Media Storage plans/Conference call 2010-12-21

From Wikitech

Present: CT Woo, NeilK, Russell, Ariel, Mark (joined halfway through)

This meeting is a followup from previous call on 2010-12-14, to decide our direction on media storage technologies for Wikimedia.

Following the meeting last time, which added some new criteria to investigate, over the past week Russell and Neil have fleshed out the documentation on our technology choices.

Reducing the number of choices

Three choices is too much, so the sense was that we should try to reduce it to a binary choice. However, nobody (even Russell) could see one that was immediately worth removing, or one that was so much better it definitely was not to be removed. So we discussed various criteria.

Authentication

We anticipate using authentication in Wikimedia projects in the near future. MogileFS does not do any authentication out of the box. OpenStack and Gluster do.

However, there are many apps built using MogileFS which do have privacy -- it's just that Mogile handles managing replicas and uploading them to various servers, and authentication is handled elsewhere. This did not seem to be a clear win or lose for Mogile.

Dependencies

  • CT, Russell do not like Mogile's MySQL dependency. Another thing that can fail. We don't have a DBA here.
  • Ariel, Neil okay with MySQL, don't see it as a reason to reject it. MySQL is well known, used for everything at Wikimedia anyway.

No clear conclusion.

Difficulty of implementation

Russell proposed that, since Gluster and OpenStack expose POSIX APIs, that caches should speak directly to them via this API.

Ariel does not like that -- the caches should be agnostic as to what the backend is doing. Should just know HTTP

No obvious winner on difficulty of implementation. Mogile is simpler, but requires building more of it ourselves. The others did not obviously seem harder or easier.

Stability?

CT asked which was most stable.

Russell: differing ideas of what stable is

NeilK says Mogile is simple & dumb, around since 2005, others are iterating rapidly

Russell likes that OpenStack has releases. A lot of people excited about Swift

Gluster is up to version 3.1 -- stable.

CT: let's drop OpenStack since it is the newest entry -- very early in its history.

Ariel: it's early in its release to the public, it's been used for a while at RackSpace

No clear conclusion.

Dropping one of Gluster and OpenStack just because they're similar

CT suggests Mogile + one of (Gluster / OpenStack).

Russell not happy about choosing just for the sake of an easier choice; what if Gluster and OpenStack are BOTH better?

Did not seem like a good idea, no conclusion.

Sucks/wins ratio

There is a row in the table for number of search results of "X sucks" versus "X wins" on the web. Generally agreed this is not a great criterion. NeilK is surprised that MogileFS has fewer mentions, since it clearly has to be the winner in number of implemented sites.

Languages and extensions

Discussions about ease of extending. The three choices require different languages -- C, Perl, and OpenStack can be extended (?) via the web services API and is itself written in Python. None of these are unacceptable to us.

MogileFS is written in Perl. NeilK has read the source, finds it easy to understand. Mark was not impressed with MogileFS code the last time he checked.

OpenStack is all python, based on Twisted. Mark & NeilK praise Twisted as a good framework.

Gluster is C. There are fewer people here who can write C but still substantial expertise.

Russell suggests this is more of a trivial point, no disagreement.

Adding nodes

Russell points out that Gluster requires restart of daemon to add nodes, which is a big difference.

Everybody investigates of "remounting" concept in Gluster. Looks not so awesome since requires file mount.

First time we have a clear loser.

Support Model

CT asks about commercial support. Likes that there is gluster.com.

  • probably no support for openstack, since it's so new
  • mogile -- you're on your own

Russell: I will add this to the matrix.

Anecdotal

Mark - mentioned friend who tested Gluster at ISP, found it to be unstable

Ariel : what did they go with?

Mark: doesn't know -- was a system we already considered and rejected because it focused on encryption

Conclusion

GlusterFS seems to be the loser, but not by much, and we could revisit this.

Main factors people are uncomfortable with:

  • can't add nodes on the fly
  • Mark's anecdote that it was unstable when tested
  • apparent requirement to be mounted as a filesystem, which may replicate the issues we've had with NFS

Next steps

For now, eliminate Gluster from the systems we are examining.

Let's try these on real nodes, 1 or 2 nodes. It would be nice if multiple people could play around with it so please let's not to do it on our own boxes.

Provisioning? Wait? Get nodes on Amazon or something?

Ariel: what's the time frame for getting real servers or provisioning new servers

Ryan is getting new hardware in a week or two for virtualization tests, maybe we can reuse them

CT if you had spare servers for such miscellaneous use it would be better

Bottom line - We should have servers available for this in a couple of weeks, 1st of January.


  1. get ready to provision servers & test OpenStack and MogileFS early in 2011
  2. Russell to figure out commercial support options for all 3 (Mogile, OpenStack, Gluster), add info to matrix
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Obsolete:Media_server/2011_Media_Storage_plans/Conference_call_2010-12-21&oldid=75480"