Swift/Deploy Plan - Shard Smaller Wikis

From Wikitech


This page describes the plan to introduce sharding containers for thumbnails to a number of wikis. Currently (2012-05-08) only commons and enwiki are sharded. We would like to shard all wikis with over 25 thousand objects to make sure there's plenty of headroom for growth.

List of wikis to shard: 'commonswiki', 'dewiki', 'enwiki', 'fiwiki', 'frwiki', 'hewiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'rowiki', 'ruwiki', 'thwiki', 'trwiki', 'ukwiki', 'zhwiki'

List of containers to shard: wikipedia-commons-local-thumb,wikipedia-de-local-thumb,wikipedia-en-local-thumb,wikipedia-fi-local-thumb,wikipedia-fr-local-thumb,wikipedia-he-local-thumb,wikipedia-hu-local-thumb,wikipedia-id-local-thumb,wikipedia-it-local-thumb,wikipedia-ja-local-thumb,wikipedia-ro-local-thumb,wikipedia-ru-local-thumb,wikipedia-th-local-thumb,wikipedia-tr-local-thumb,wikipedia-uk-local-thumb,wikipedia-zh-local-thumb

Order of operations:

  • create sharded containers in swift (following instructions) [DONE]
 pass='aoeu'
 for cont in wikipedia-de-local-thumb wikipedia-fi-local-thumb wikipedia-fr-local-thumb wikipedia-he-local-thumb wikipedia-hu-local-thumb wikipedia-id-local-thumb wikipedia-it-local-thumb wikipedia-ja-local-thumb wikipedia-ro-local-thumb wikipedia-ru-local-thumb wikipedia-th-local-thumb wikipedia-tr-local-thumb wikipedia-uk-local-thumb wikipedia-zh-local-thumb ; do
 for i in {0..9} {a..f} ; do for j in {0..9} {a..f} ; do 
   echo "creating container $cont.$i$j"; 
   swift -A http://127.0.0.1/auth/v1.0 -U mw:thumbnail -K $pass post -r '.r:*' ${cont}.$i$j
 done; done
 done
  • change mediawiki configuration to look at the sharded containers for purging [DONE]
  • change rewrite.py in swift to use the sharded containers [DONE]
  • verify it's all working and happy [DONE]
    • checked new objects get created and verified that purging still works [DONE]
  • wait a day or two [DONE]
  • delete all the content in the old unsharded containers and drop them [in progress]

We are not prepopulating the sharded containers; image latency for the newly sharded containers (all in the list except commons and enwiki) will degrade for a period while (a few days to a week is my guess)

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Swift/Deploy_Plan_-_Shard_Smaller_Wikis&oldid=59901"