Obsolete:Media server/Distributed File Storage choices
Please examine this evaluation of the existing distributed file storage choices. I only considered Open Source solutions. I'm eliminating certain columns:
- goes because the IRC community only had three people when I checked it in November 2010, plus their metadata server is only a hot backup, not a replicated / redundant server.
- Eliminated because it is not production code yet (although it looks good).
- Eliminated because its metadata server is a single point of failure with no obvious path to replication.
- Eliminated because nobody but me likes it.
Please look at the remaining DFS choices, Mogile, OpenStack, and GlusterFS to see if you can veto (or praise) any of them. I'd like to have a binary choice when we decide to go. RussNelson 02:27, 24 November 2010 (UTC)
- Was an experiment by Domas IIRC Hashar 12:29, 7 December 2010 (UTC)
- Ganglia might help : http://ganglia.wikimedia.org/
- Hrm. It only reports CPU and memory, not network load. RussNelson 21:51, 15 December 2010 (UTC)
- Do not forget to take in account the strategic plan up to 2015. Those statistics should sky rocket in the next couple years. Hashar 12:29, 7 December 2010 (UTC)
- I didn't find anything detailed enough to affect these plans in any specific way RussNelson 21:51, 15 December 2010 (UTC)
- Ganglia might help : http://ganglia.wikimedia.org/
Right now, the original and thumb server are NFS servers. The thumbnailer is reading files from the originals server, and writing them back to the thumb server. The apache servers have the original and thumb servers mounted via NFS. So, if we are to use a distributed filesystem, the API to them needs to either appear in the filesystem "just like" NFS, or else we need to modify all the code which touches the filesystem so that it goes through the DFS API instead. This section documents, for the top three candidates, what will be required to adapt our code to their interface.
Mogile has a PHP interface, amusingly derived from the experimental MediaWiki code added by Domas. Probably better to bring Domas' code up to date with the current MediaWiki code. Mogile does not support FUSE, but Danga has experimental code for it.
OpenStack's REST interface apparently doesn't support deep levels of directories. Plus accesses need an authorization token. This bodes ill for the idea of directly exposing it to the caches (unless the caches can add a header) and letting it look up our existing files via REST. Nor does it support filesystem access via FUSE. It's not really a "file" system ... but more like an object store. No searching of metadata (which is stored in the Linux filesystem's xattrs).
Design has a hard limit of 5GB/file. Containers get slow if they have more than a million files. We currently have 10M originals. Thus, we will have to apportion files across containers.
There are two recommended interfaces to GlusterFS: NFS (which we're pretty sure we don't want to use, BUT it would be a drop-in solution), and their native client. It exposes files via a POSIX interface to the Linux FUSE client in user space. I don't have any experience with it, but I am dubious of the idea of running a file system in user space. First, it seems like a kludge, second it seems like it would be slow, with multiple context switches into the kernel, and third it seems like it would be unreliable. But those a fears; not necessarily rational.
- Using fuse in the thumbnailers wouldn't need to slow things. The actual network fetching would be orders of magnitude slower. Platonides 23:03, 14 December 2010 (UTC)
We decided to eliminate GlusterFS at least for now, and try running Mogile and OpenStackSwift. I'm going to document my experiences installing both, so that we have an idea of how difficult it is.
Following these instructions: http://code.google.com/p/mogilefs/wiki/InstallHowTo
Mogile consists of three packages: Server, Client, and Utils. They're written in Perl (and some C), but install straightforwardly. I used the 'cpan' program on Ubuntu 10.4 to install the dependencies. The Server requires the Client to be installed first. Attempts to build the Makefile.PL cause it to check for missing dependencies. It was sufficient to run "sudo cpan" and name the missing Perl module on the command line. The modules install into system libraries, which is why they require root. Here are the commands I use after fetching the packages:
for i in MogileFS-*; do tar xpfz $i; done sudo apt-get install mysql-server sudo mysql -p # and then issue the commands in the Moguile docs to create the database sudo cpan Danga::Socket sudo cpan IO::AIO sudo cpan Net::Netmask sudo cpan Perlbal cd MogileFS-Client-1.13/ sudo cpan IO::WrapTie perl Makefile.PL make make test sudo make install cd ../MogileFS-Utils-2.18/ perl Makefile.PL make make test sudo make install cd ../MogileFS-Server-2.44/ perl Makefile.PL make make test sudo make install
Note that the "make test" fails unless you give it the database password per the documentation. Even then it's still throwing various errors. I'm going to track them down; errors in published tests make me worry.
Following the instructions given here: http://swift.openstack.org/howto_installmultinode.html
The installation went fairly well. I ran into a few documentation problems, and spent some time rewriting / reordering things. It's better now. It should be fairly easy to write a Puppet script which will turn a machine into a storage node, and register with the object ring.
Documentation for Swift in our environment is now on its own page.
I have five laptops configured to serve as both a mogile and a swift cluster. They have 8GB SD cards for storage, and the Ubuntu and cluster software is stored on 4GB USB drivers. They are borrowed machines, (and I can borrow more), but I also can't use the hard drives. They're probably reasonable for comparing mogile against swift, although of course for eventual performance of the cluster, this will not be a valid test.
I have a 5MB file produced using:
dd if=/dev/zero of=bigfile1.tgz count=10240
I uploaded that file to both clusters. The current machine is a.ra-tes.org. It is running the database for Mogile and the sole tracker. It is also running the auth server and the only proxy server for Swift.
root@a:~# time st -A https://a.ra-tes.org:11000/v1.0 -U system:root -K testpass upload myfiles bigfile1.tgz bigfile2.tgz real 0m5.300s user 0m1.260s sys 0m0.368s root@a:~# time mogtool inject --trackers=127.0.0.1:7001 --domain=testdomain bigfile1.tgz BigFile1 file BigFile2: 5f363e0e58a95f06cbe9bbc662c5dfb6, len = 5242880 Spawned child 14343 to deal with chunk number 1. chunk 1 saved in 1.03 seconds. Child 14343 successfully finished with chunk 1. Replicating: 1 Replicating: 1 Replicating: 1 Replicating: 1 real 0m7.266s user 0m0.692s sys 0m0.188s
Mogile is generally slower than Swift at writing, by about 1/2 second, except that for one trial, it was faster by 1/2 second. Repeating the same upload ten times in a row took the same time for bot Mogile and Swift: 60.8 seconds. Replicating it was 57.3s for Mogile and 59.8 for Swift. Running two Mogile uploads simultaneously took 66s and 64.8s. Interestingly, running two Swift uploads simultaneously took 92.4 seconds and 96.2 seconds. Using the Swift's client (st) native threading takes 30.7 seconds to upload ten files. Running two of them at a time takes 62.5 and 62.9 seconds.
Made a set of small files as so:
for i in 1 2 3 4 5 6 7 8 9 10; do dd if=/dev/zero of=smallfile$i.png count=$i bs=5120; done
Swift uploaded ten files in ten seconds. Mogile took 39 seconds. Swift uploaded them in 1.5s using its native threading.
Ariel suggested that I do a test using files more representative of our files. So, I fetched 73 thumbnails (including two originals), and stored them in my cluster. 2/3rds of the files were under 1K in size. Biggest was 81K. According to du, 704K. According to "cat * | wc -c" 456K. Remember that this is a very wimpy cluster, so times should be compared against DFSes not against your expectations. Took 2m45s to store them into Swift. Disk space increase was 76636-74820 on b, 75668 - 73712 on c, 68592 - 67644 on d. (didn't check the size on a). Took 4m41s to store them into Mogile. Disk space increase was 76876 - 76636 on b, 75988 - 75668 on c, 68956 - 68592 on d. One machine (e) was unavailable while this operation took place. It seems to have a dodgy Ethernet .. but that seems like a fair test, because if either one can't handle a down host, then we don't want it!
root@a:~# time st-null -A https://a.ra-tes.org:11000/v1.0 -U system:root -K testpass download myfiles bigfile2.tgz bigfile2.tgz real 0m1.431s user 0m0.892s sys 0m0.112s root@a:~# time mogtool extract --overwrite --trackers=127.0.0.1:7001 --domain=testdomain BigFile2 /dev/null Fetching piece 1... Trying http://10.0.0.75:7500/dev4/0/000/000/0000000007.fid... Done. real 0m1.970s user 0m1.192s sys 0m0.236s
Similarly, Mogile is slower than Swift at reading, again by about 1/2 second (but a larger percentage of time). To repeat the same fetch ten times, with Mogile took 19.6 seconds, but with Swift took 15.0 seconds. In both cases, it's writing the file to /dev/null. Reading two sets of ten concurrently using Swift took 22.5 and 22.7 seconds. The same test with Mogile took 20 and 20.4 seconds. Using Swift's client (st) native threading takes 7 seconds to download ten files. Running two of them at a time takes 9.1 and 13.4 seconds.
Pulling down the small file set using Swift took 5.8s. Using Mogile took 7.3 seconds. Using Swift's native threading took 0.77 seconds.
We've decided to do live testing of Swift, just to make sure sure that we can make this work for us. To do this, we'll set up a cluster containing 1/256th of the current media store. Then we'll use the squid's ability to rewrite URLs. We'll turn the project and language into a container name, and then use the rest of the filename as the object name. So, for example, we will look for
and turn it into
and look for
and turn it into
Cluster.wikimedia.org will be a real cluster, but it will only have 1/256th of the current media store.
Live Test Configuration
Installed swift according to the multinode instructions (which I improved and have contributed back to OpenStack). After much angst with Squid (which hadn't been compiled with SSL enabled), and Swift's proxy (which turns out not to require SSL if you remove the certificate from the configuration), I discovered that you can load custom code into Swift's proxy. We're using this code for two ends. First, to reformat the path into that required by Swift's account/container/object path. We could insert a header containing the authorization token, but it's just as easy to mark the container as public so anyone may access it. Second, to recognize a missing file, fetch it from upload.wikimedia.org, and write it into the cluster.
To set a container's Read Access Control List so that anybody can read a container, you need to change the Read ACL (Access Control List). Use the 'st' command, like this:
st -A https://127.0.0.1:11000/v1.0 -U system:media -K password -r '.r:*' post project%2Flanguage
This can also be used to create a container which didn't previously exist.
Anyway, this is useful: http://swift.openstack.org/admin_guide.html and this http://swift.openstack.org/development_saio.html and of course the root of their documentation: http://swift.openstack.org/ and WSGI (a python web service gateway interface): http://pythonweb.org/projects/webmodules/doc/0.5.3/html_multipage/lib/wsgi.html and the WSGI PEP: http://www.python.org/dev/peps/pep-0333/
http://pythonpaste.org/deploy/ explains the contents of these files:
Once you understand that, you'll be able to use http://eventlet.net to understand this code:
After much painful documentation reading and source browsing, I've finally figured out the relationship between the various bits and pieces from which Swift is comprised. We need middleware to adjust our incoming URLs to the swift schema, and we need middleware to act as a 404 handler so that we can create thumbnails as we do now. In the short term, we'll just fall back onto the Media Store to fetch thumbnails that don't exist. In the longer term, we'll have to locate the original image, fetch it, scale it, and write it back to the user / store it back into the cluster. It's possible that we can fold the scaler machines into the cluster and do scaling on the proxies (we'll need to have a lot of them anyway), or it's possible that we'll write a Swift Repo for the scaler. We'll certainly have to have Swift Repo code for the uploader, but that's a longer-term issue.
Let's take a tour of the middleware.
- First place to look is /etc/swift/proxy-server.conf . It's a standard Python Paste Deploy file with our 'rewrite' added to it. The Deploy system is designed to make it easy to create web services just by creating a config file that tells how everything is pasted together. The "egg" reference is to the distribution method for Python packages (Pythons are reptiles and lay eggs). Swift is the name of the egg, and the # portion tells what module out of the egg is being referenced. By default, a $THING_factory gets called, depending on whether it's an application or a filter (there are other types, but they're not in this config file).
- Look for our rewrite portions. They create a filter and put it into the proxy pipeline. Our Python code is on the Python path, and so we need only name it for Paste Deploy to be able to find it. It's stored in /usr/local/lib/python2.6/site-packages/wmf/rewrite.py on alsted.
- Rewrite imports webob. WebOb is part of the PythonPaste project: http://pythonpaste.org/webob/ . This is an object which keeps everything about a web server request and web server response in one place. It parses the headers and makes them easy to change.
- Rewrite uses a regexp to modify the incoming path to put it into the form that Swift needs (account/container/object). Note that the container cannot include internal slashes, and that the object may contain multiple slashes. We use our project names / languages / sections as the container name, and the name of the media file as the object name. We retain the hashed "subdirectories" so that we can locate the original file.
- Once we have the final Swift name, we continue with the pipeline. To prevent the pipeline from starting the HTTP response, we point that callable to a subroutine of our own which just saves the parameters.
- If we get a good response, we call the response with the appropriate header and body to return the file to them.
- If we get a bad response,
- we construct a URL pointing to the media store.
- We open up that URL to fetch the image out of the media store.
- We modify the request so that it becomes a 'PUT' to store the image into the cluster.
- As we read from the Media Store, we write to the cluster and return the media file to the user.
- if anything goes wrong, we delete the object off the cluster.
Mediawiki has never virtualized its filesystem accesses. So you'll find filesystems calls sprinkled here and there throughout the code. Here are the files which I already know contain filesystem calls which modify the repository (unlink, rename, opendir/readdir/closedir, link, fopen).
Ariel points to /media, DjVuImage.php, MimeMagic.php, looks like the upload stash stuff (/upload/*php etc)... Math.php (as you already discovered)
- Upload a file. Visit http://ersch.wikimedia.org/index.php/Special:Upload and upload the image of your choice. Call it Test1.jpg
- Observe that all the thumbnail links work.
- Move the file. http://ersch.wikimedia.org/index.php/Special:MovePage/Test1.jpg . Move it to Test1a.jpg
- Delete the file. http://ersch.wikimedia.org/index.php?title=File:Test1a.jpg&action=delete
- Observe that you can't view the image anymore.
- Restore (undelete) the file by clicking on http://ersch.wikimedia.org/index.php/Special:Undelete/File:Test1a.jpg
- Observe that you can now view the file again.
- Upload a new version by clicking on http://ersch.wikimedia.org/index.php?title=Special:Upload&wpDestFile=Test1a.jpg&wpForReUpload=1
- Observe on the results page that both thumbnails are available. Click on the thumbs to view the full-size images.
- Revert an archived image so that it becomes the new images.
- Observe that this generates an error: "SwiftRepo::getZonePath: not implemented". As far as I know, this is the only missing part.
The Evaluation Matrix
Note http://leemoonsoo.blogspot.com/2009/04/simple-comparison-open-source.html for an older comparison. If a column contains the title of the column, it needs to be filled in with information.
|Our Req't||Mogile||OpenStack's swift||GlusterFS|
|Endorsements||Last.fm, Digg, Veoh, Uh, Boobs&Kittens, LiveJournal, Six Apart, Friendster, Wikispaces, blip.tv manages 1.4 petabytes in MogileFS, Support for pear derived from Mediawiki support, Mediawiki???||Rackspace Cloud Files, Cloudscaling (planned), SMEStorage (planned)||Listing of users, My list of testimonials|
|Mailing list activity in the last month||19||14, but IRC channel is active||21|
|Documentation available||Sketchy, incomplete wiki. Most knowledge seems to be accumulated from IRC, mailing lists or reading source.||http://swift.openstack.org/||http://www.gluster.com/community/documentation/index.php (using MediaWiki)|
|Commercial support available?||apparently none||Although they claim commercial support exists, they don't make it easy to find!||Yes, from Gluster.com|
|Single point of failure?||MySQL, use replication||No||No, uses hash to find files|
|Runs on Commodity Hardware?||Yes||Yes||Yes|
|Direct HTTP or are
web front-ends needed?
|Seekable?||Yes, if the underlying HTTP server supports Range: header||Yes, using the Range: header, per IRC.||Claims POSIX compatibility, so it MUST, right?|
|Load balanced for the same data?||Returns URLs in load balanced order, with first one known good||Issues a GET on all copies and returns the first one that responds, per IRC.||Yes|
|What kind of API?||Library||REST||Library|
|Authentication?||No||It's a plugin. Comes with a sample "devauth", but soon will have a production "swauth".||Yes; IP or password|
|Supports unpublished files
for new upload?
|Yes||Yes; they have authen/author included.||Yes|
|Yes, see MogileFS and Multiple Networks/Datacenters. Can configure policies for cross-colo replication, # of replicas per colo, ZoneLocal will help to preferentially serve from local. IPs of each network must be in a range.||Yes||Not obviously|
|Free of Proprietary Software?||Yes||Yes||Yes|
|Maximum file size?||Limited only by toolchain (web server, underlying FS). How to stream uploads.||Unlimited read length; write 5GB chunks then paste them into an object, per IRC||2^64|
|Maximum number of files?||Depends on MySQL config. 2^64 with --with-big-tables.||No limit, but write performance drops as you put more files into a container. We could do our own hashing.||2^64|
|Replicated?||Yes, across hosts||Yes, via rsync||Yes|
|Easy to add new boxes?||mogadm add host||Yes||Must restart|
|Node failures detected and
|Yes, often-failing node is taken out of rotation for writes/reads, will continue to be tested by monitoring job, in case it comes back. Files that were on that node are not re-replicated until node marked dead manually.||Yes, but "Replication is an area of active development, and likely rife with potential improvements to speed and correctness."||Yes, and updated with changes when restored to operation.|
|What happens when a failed node comes back up?||Monitoring job should notice it is back and will eventually route writes/reads there||Uses a "push" model, where data holders make sure that their partners have the right data.||Any changes while it was down are re-synched.|
|How are node failures reported?||syslog||swift-stats-report will tell you.||They have a management console with logs|
|Is Nagios already supported?||Yes||As of this writing, no, per IRC.||3rd party|
|How are replication policy failures reported?||syslog||swift-stats-report will tell you.||They have a management console with logs|
|Can we set # of copies?||Yes, configurable by app-defined "class" of file. Could store multiple copies of originals, fewer for thumbnails.||Yes||Yes; just add more volumes|
|How long before a new file is replicated?||< 1 second if queue is empty||PUTs are done in parallel and success is reported when two have succeeded, per IRC.||Can't find it in the documentation, and #glusterfs isn't really good at answering questions.|
|How are files replicated?||Desired replications queued in DB table. Then worker processes pick them off, use HTTP GET to fetch, pipe to an HTTP PUT request.||Each partition owner compares a hash of its partition against other partitions, and pushes the files over using rsync.||Uses a changelog to determine which copies need updating.|
|Do updates appear everywhere simultaneously?||No, async||No, but deletes and new uploads do.||Yes, unless there are conflicting updates.|
|Want: Files stored individually on filesystem, with name like MediaWiki title?||No, uses automatically assigned IDs.||No||Yes, but you can't predict which server will hold a file.|
|Want: free of esoteric kernel features?||Yes||Yesbut requires xattrs on||Yesbut requires xattrs on|
|Want: Consistency Audit?||Yes||Yes||Maybe not needed??|
|Want: Extensible?||Yes, hook-based system similar to MediaWiki. (Perl)||Yes, using wsgi (web services standard, many langs)||Yes, via plugins (C)|
Discarded distributed FS
Not production code
|Mailing list activity in the last month||64, but IRC channel is dead||167||67||0|
|Single point of failure?||Yes, metadata server can have
a hot backup.
|No||DIR and MRC||No|
|Runs on Commodity Hardware?||Yes||Yes||Yes||Yes|
|File access via HTTP?||No||Ceph||No||Yes|
|Replicated?||Yes||Both file and metadata||No||Yes|
|Easy to add new boxes?||Yes||Yes||Yes||Yes|
|Can preserve FileRepo layer?||POSIX via FUSE||POSIX with kernel module
Or direct access w/ library
|POSIX via FUSE||Yes|
|Supports unpublished files
for new upload?
|Not obviously||Not obviously||Yes||Yes|
|Free of Proprietary Software?||Yes||Yes||Yes - GPL||Yes|
|Want: Files stored as files with names?||No||Ceph||No||Yes|
|Want: free of esoteric kernel features?||Yes||Ceph||Yes||Yes|
|Want: Consistency Audit?||Yes||Ceph||Not obviously||Yes|
I solicited advice from some people on the list of GlusterFS users. Tobias Wilken says "We didn't test the 3.1 version yet. With 3.0.5 we are quite happy, except of some special inotify issues." Ken Bigelow of Pytec Design is using 33 servers with 6 1TB drives interconnected using InfiniBand @ 40GB/sec. That's many fewer drives than most are using, but he's also getting very good performance out of his cluster where others who try to put petabyte arrays on single servers are having trouble. Sabuj Pattanayek of Vanderbilt University is having a bit of trouble with FUSE (user-space filesystems), and his writes tend to be slow, even across a striped system, but he's otherwise happy with GlusterFS.
Image test methodology
On the call today, people were surprised that 2/3rds of the media files were <1K. Here's how I came by that conclusion. I selected the set of pages by clicking on "Random Page" off the main page. I pulled the media files out of the HTML source. I expect that some of these files will be commonly used by multiple pages (e.g. Commons_logo) and so will live in the caches full-time, so that image usage in the page will not get translated into cluster accesses.
wget http://upload.wikimedia.org/wikipedia/en/thumb/7/7e/Replace_this_image_male.svg/150px-Replace_this_image_male.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Flag_of_Sweden.svg/22px-Flag_of_Sweden.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/e/eb/Swimming_pictogram.svg/50px-Swimming_pictogram.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Flag_of_Sweden.svg/25px-Flag_of_Sweden.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/1/18/Pictgram_swimming.svg/50px-Pictgram_swimming.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Portrait_of_George_Washington.jpeg/27px-Portrait_of_George_Washington.jpeg wget http://upload.wikimedia.org/wikipedia/commons/9/9c/Dramalis.jpg wget http://upload.wikimedia.org/wikipedia/en/thumb/d/d2/Hells_Kitchen_title.png/245px-Hells_Kitchen_title.png wget http://upload.wikimedia.org/wikipedia/en/thumb/1/13/Padlock-olive-arrow.svg/20px-Padlock-olive-arrow.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/2/2b/Flag_of_the_Arab_League.svg/22px-Flag_of_the_Arab_League.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/b/b9/Flag_of_Australia.svg/22px-Flag_of_Australia.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/9/92/Flag_of_Belgium_%28civil%29.svg/22px-Flag_of_Belgium_%28civil%29.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/0/05/Flag_of_Brazil.svg/22px-Flag_of_Brazil.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/9/9a/Flag_of_Bulgaria.svg/22px-Flag_of_Bulgaria.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/c/cf/Flag_of_Canada.svg/22px-Flag_of_Canada.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/b/bc/Flag_of_Finland.svg/22px-Flag_of_Finland.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/Flag_of_France.svg/22px-Flag_of_France.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Flag_of_Germany.svg/22px-Flag_of_Germany.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/c/c1/Flag_of_Hungary.svg/22px-Flag_of_Hungary.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/4/41/Flag_of_India.svg/22px-Flag_of_India.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/4/45/Flag_of_Ireland.svg/22px-Flag_of_Ireland.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/d/d4/Flag_of_Israel.svg/22px-Flag_of_Israel.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/0/03/Flag_of_Italy.svg/22px-Flag_of_Italy.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Flag_of_Japan.svg/22px-Flag_of_Japan.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/6/66/Flag_of_Malaysia.svg/22px-Flag_of_Malaysia.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/f/fc/Flag_of_Mexico.svg/22px-Flag_of_Mexico.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/2/20/Flag_of_the_Netherlands.svg/22px-Flag_of_the_Netherlands.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/3/3e/Flag_of_New_Zealand.svg/22px-Flag_of_New_Zealand.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Flag_of_Norway.svg/22px-Flag_of_Norway.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/1/12/Flag_of_Poland.svg/22px-Flag_of_Poland.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/5/5c/Flag_of_Portugal.svg/22px-Flag_of_Portugal.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/f/f0/Flag_of_Slovenia.svg/22px-Flag_of_Slovenia.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Flag_of_Sweden.svg/22px-Flag_of_Sweden.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/4/49/Flag_of_Ukraine.svg/22px-Flag_of_Ukraine.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Flag_of_the_United_Kingdom.svg/22px-Flag_of_the_United_Kingdom.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/a/a4/Flag_of_the_United_States.svg/22px-Flag_of_the_United_States.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/1/1c/Wiki_letter_w_cropped.svg/20px-Wiki_letter_w_cropped.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/d/d6/Wikiquote-logo-en.svg/40px-Wikiquote-logo-en.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/9/99/Question_book-new.svg/50px-Question_book-new.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/f/f2/Prince_Edward_Road_East_and_West.jpg/220px-Prince_Edward_Road_East_and_West.jpg wget http://upload.wikimedia.org/wikipedia/en/thumb/2/26/Perw.jpg/220px-Perw.jpg wget http://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/P1070289.JPG/220px-P1070289.JPG wget http://upload.wikimedia.org/wikipedia/commons/thumb/a/ab/HK_Kln_Prince_Edward_Road_East_590.jpg/220px-HK_Kln_Prince_Edward_Road_East_590.jpg wget http://upload.wikimedia.org/wikipedia/commons/thumb/4/4a/Commons-logo.svg/30px-Commons-logo.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/5/5f/Disambig_gray.svg/30px-Disambig_gray.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/2/28/Bandeira_da_Bahia.svg/23px-Bandeira_da_Bahia.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/f/fb/Brazil_State_Bahia.svg/130px-Brazil_State_Bahia.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/9/9a/Erioll_world.svg/18px-Erioll_world.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/f/ff/Lajosmizse.jpg/250px-Lajosmizse.jpg wget http://upload.wikimedia.org/wikipedia/commons/thumb/c/c4/HUN_Lajosmizse_COA.jpg/100px-HUN_Lajosmizse_COA.jpg wget http://upload.wikimedia.org/wikipedia/commons/thumb/a/ac/Hungary_location_map.svg/250px-Hungary_location_map.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/c/c1/Flag_of_Hungary.svg/22px-Flag_of_Hungary.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/c/c5/HU_county_Bacs-Kiskun.svg/100px-HU_county_Bacs-Kiskun.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/9/9f/Philadelphia_colours.svg/16px-Philadelphia_colours.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/1/1e/New_Haven_colours.svg/16px-New_Haven_colours.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/7/76/New_York_colors.svg/16px-New_York_colors.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/8/88/Northern_Raiders_colors.svg/16px-Northern_Raiders_colors.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/8/8e/Aston_DSC-Glen_Mills_colours.svg/16px-Aston_DSC-Glen_Mills_colours.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/9/94/Connecticut_colors.svg/16px-Connecticut_colors.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/9/9e/Jacksonville_colors.svg/16px-Jacksonville_colors.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/c/c5/New_Jersey_colours.svg/16px-New_Jersey_colours.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/f/f5/Washington_DC_colours.svg/16px-Washington_DC_colours.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/f/fc/Fairfax_Eagles_colors.svg/16px-Fairfax_Eagles_colors.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/1/17/Rio_Osun.jpg/250px-Rio_Osun.jpg wget http://upload.wikimedia.org/wikipedia/commons/thumb/9/99/Nigeria_Osun_State_map.png/250px-Nigeria_Osun_State_map.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/7/79/Flag_of_Nigeria.svg/22px-Flag_of_Nigeria.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/2/25/Templo_Osun3.jpg/220px-Templo_Osun3.jpg wget http://upload.wikimedia.org/wikipedia/commons/thumb/7/79/Flag_of_Nigeria.svg/23px-Flag_of_Nigeria.svg.png wget http://upload.wikimedia.org/wikipedia/en/2/25/Machinesmith.png wget http://upload.wikimedia.org/wikipedia/en/thumb/1/18/Machinesmith.jpg/220px-Machinesmith.jpg wget http://upload.wikimedia.org/wikipedia/commons/thumb/2/24/UK_Chequered_flag.svg/25px-UK_Chequered_flag.svg.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/6/62/Salic_Law.png/280px-Salic_Law.png wget http://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Wikisource-logo.svg/38px-Wikisource-logo.svg.png wget http://upload.wikimedia.org/wikipedia/en/thumb/a/ae/Devil_Face%2C_Angel_Heart_DVD.jpg/220px-Devil_Face%2C_Angel_Heart_DVD.jpg wget http://upload.wikimedia.org/wikipedia/commons/thumb/9/93/Hong_Kong_film.svg/29px-Hong_Kong_film.svg.png