Jump to content

Obsolete:Media server/Performance requirements

From Wikitech

Much of this data is very low resolution, but no better stats are yet available. These servers apparently do not keep logs so the only way to know, for instance, peak bytes uploaded per second, was to reconstruct it from associated database records. However, those are created in an instant after the upload is completed, so they do not tell us everything we want to know.

Characteristic Sustained Peak Anticipated Comments
Uploaded files / time 6000/day or 200,000/month 3/second or 120,000/day or 350,000/month ?? From examination of Bryan Tong Minh's commons growth graph, associated data, also Ariel's Uploads per day in Nov 2010, Data gathered from toolserver queries below (see wiki source for SQL queries)
Uploaded bytes / time 208K/second or 750M/hour or 9GB/day 1.6M/second or 100M/minute or 15GB/day ?? MB uploaded per day, Nov 2010, also see data gathered below from toolserver queries (view source for SQL). However, note that there are sometimes *much* bigger spikes due to mass imports of entire collections. In the past these imports were handled by the users RoanKattouw/Catrope or Multichil.
Storage growth GB/day ?? ?? ?? Not quite the same as uploaded bytes, as there are deletions, thumbnails, etc. Should be a constant multiple of uploaded bytes, though.
Requested files/s 355 632 ?? In our setup, this is pre-caching - it's files requested by Squid. Some data available in torrus, i.e. these graphs circa Jan 2011
Requested bytes/s ?? ?? ?? In our setup, this is pre-caching - it's files requested by Squid.
Transformation files/s ?? ?? ??
Transformation bytes/s ~2MB/s ~15MB/s ?? Image scalers ganglia retrieved Jan 2011. Very high peak seems to be related to some network outage; normally peaks more like 5MB/s.
Deletions/time 25000/month 50000/month ?? See Commons deletion ratio - if this graph is accurate, up to 40% of files uploaded per month will be removed by admins. Uncertain if this means purged or just removed from view; probably the latter.


Statistics

This section contains some ad hoc stats from 2010-11-01T00:00:00 through 2010-11-07T00:00:00. View source for SQL.

Average upload size: 1.62MB

Median upload size: 702K

Maximum uploads completing within one second: 3
n.b. This will definitely change when UploadWizard is more common; it does simultaneous uploads.

Maximum upload throughput (spike for 1 minute): ~100M/minute or 1.6M/second

Maximum upload throughput (sustained over an hour): ~750M/hour or 208K/second

Typical sizes of upload

size (bytes) count
less than 1K 220
1K - 10K 789
10K - 100K 5185
100K - 1M 11032
1M - 10M 12671
10M - 100M 339
100M - 1GB 1

Typical bytes uploaded per minute

bytes uploaded minutes
less than 1K 61
1K-10K 346
10K-100K 1071
100K-1MB 4736
1MB-10MB 1645
10MB-100MB 4

Other info

Some of this is helpful, but mostly does not have the high resolution we would want.