Obsolete:Media server/Performance requirements
Much of this data is very low resolution, but no better stats are yet available. These servers apparently do not keep logs so the only way to know, for instance, peak bytes uploaded per second, was to reconstruct it from associated database records. However, those are created in an instant after the upload is completed, so they do not tell us everything we want to know.
| Characteristic | Sustained | Peak | Anticipated | Comments |
|---|---|---|---|---|
| Uploaded files / time | 6000/day or 200,000/month | 3/second or 120,000/day or 350,000/month | ?? | From examination of Bryan Tong Minh's commons growth graph, associated data, also Ariel's Uploads per day in Nov 2010, Data gathered from toolserver queries below (see wiki source for SQL queries) |
| Uploaded bytes / time | 208K/second or 750M/hour or 9GB/day | 1.6M/second or 100M/minute or 15GB/day | ?? | MB uploaded per day, Nov 2010, also see data gathered below from toolserver queries (view source for SQL). However, note that there are sometimes *much* bigger spikes due to mass imports of entire collections. In the past these imports were handled by the users RoanKattouw/Catrope or Multichil. |
| Storage growth GB/day | ?? | ?? | ?? | Not quite the same as uploaded bytes, as there are deletions, thumbnails, etc. Should be a constant multiple of uploaded bytes, though. |
| Requested files/s | 355 | 632 | ?? | In our setup, this is pre-caching - it's files requested by Squid. Some data available in torrus, i.e. these graphs circa Jan 2011 |
| Requested bytes/s | ?? | ?? | ?? | In our setup, this is pre-caching - it's files requested by Squid. |
| Transformation files/s | ?? | ?? | ?? | |
| Transformation bytes/s | ~2MB/s | ~15MB/s | ?? | Image scalers ganglia retrieved Jan 2011. Very high peak seems to be related to some network outage; normally peaks more like 5MB/s. |
| Deletions/time | 25000/month | 50000/month | ?? | See Commons deletion ratio - if this graph is accurate, up to 40% of files uploaded per month will be removed by admins. Uncertain if this means purged or just removed from view; probably the latter. |
Statistics
This section contains some ad hoc stats from 2010-11-01T00:00:00 through 2010-11-07T00:00:00. View source for SQL.
Average upload size: 1.62MB
Median upload size: 702K
Maximum uploads completing within one second: 3
n.b. This will definitely change when UploadWizard is more common; it does simultaneous uploads.
Maximum upload throughput (spike for 1 minute): ~100M/minute or 1.6M/second
Maximum upload throughput (sustained over an hour): ~750M/hour or 208K/second
Typical sizes of upload
| size (bytes) | count |
|---|---|
| less than 1K | 220 |
| 1K - 10K | 789 |
| 10K - 100K | 5185 |
| 100K - 1M | 11032 |
| 1M - 10M | 12671 |
| 10M - 100M | 339 |
| 100M - 1GB | 1 |
Typical bytes uploaded per minute
| bytes uploaded | minutes |
|---|---|
| less than 1K | 61 |
| 1K-10K | 346 |
| 10K-100K | 1071 |
| 100K-1MB | 4736 |
| 1MB-10MB | 1645 |
| 10MB-100MB | 4 |
Other info
Some of this is helpful, but mostly does not have the high resolution we would want.
- WikiStats for Commons
- Ganglia on image scalers
- Meta - Statistics
- Bryan Tong Minh's various stats & graphs - more useful than most
- Some Wikimedia-wide data from November 2010:
