Obsolete:Backup procedures
Jump to navigation
Jump to search
![]() | This page contains historical information. It is probably no longer true. |
THIS PAGE IS OLD AND OUTDATED. It is kept for historical reasons. Most of the info inside will either be or already is wrong/irrelevant. Backup infrastructure has been replaced by Bacula
This page is an inventory of our backups for both clusters data and office data.
Some definitions:
- On-site
- here means the backup is in the same physical location as the master, but on a different machine.
- Off-site
- means the backup is in a physically separate location. In most cases it's nice to have both!
- Red cells
- denote a lacking area which we want/need.
Service backup state
Service | Master | On-site | Off-site | Notes | Approx. Space Needed |
---|---|---|---|---|---|
Wiki primary data | |||||
Wiki main databases | EQIAD db* | EQIAD slave replication, LVM snapshots every 8 hours (have we tested restoring from one?) | PMTPA db slaves & snapshots / ESAMS toolserver replication (how complete is the toolserver repl?) | http://noc.wikimedia.org/dbtree/ | 4 TB - check |
External storage | EQIAD db* | EQIAD slave replication | PMTPA - ? /ESAMS toolserver replication - how complete are these? | 5 TB - check | |
Wiki data dumps | PMTPA Obsolete:dataset2 | Gluster storage (labstore1 under /publicdata-project directory) Last 5 good only. | EQIAD: via rolling rsync Obsolete:dataset1001; via daily rsync, all public data: dumps.wikimedia.your.org, last 5 good dumps: wikipedia.c3sl.ufpr.br | See Dumps/Mirror status for more info. | 6T (last 5 good)/ 29T (all) - update these |
Images and media | |||||
Uploads | PMTPA Swift | 3 copies per file | EQIAD Ceph | 23T? | |
Thumbs | PMTPA Swift | 3 copies per file | EQIAD Ceph | 12T? | |
Software development and configuration | |||||
MediaWiki config | PMTPA NFS | tridge | Subversion | ||
Apache config | PMTPA NFS | tridge | Automatic | rsync | |
Subversion | Formey | tridge | 3 GB | ||
Gerrit | Manganese | Formey | Databases on db9/10 | 3 GB | |
Bugzilla data | PMTPA db9 | db10 replication | EQIAD db1008/db1025 & snapshots | 1 GB | |
Bugzilla config | PMTPA db9 | db10 replication | EQIAD db1008/db1025 & snapshots | db10.pmtpa.w /a/sqldata 0 66532680 9020197 13.6 31:49 4724.1 31:49 4724.1 | 1.5 GB |
Bugzilla frontend | PMTPA isidore | daily rsync to Tridge | |||
Wikitech wiki | linode | PMTPA tridge; ESAMS ? | |||
Communications | |||||
OTRS data | PMTPA db9 | db10 replication | EQIAD db1008/db1025 & snapshots | 57 GB | |
OTRS config | PMTPA bart | 19 GB | |||
blogs frontend data | PMTPA singer | Daily backups to Tridge | |||
bugzilla | PMTPA isidore | Daily backups to Tridge | |||
IMAP mail | PMTPA sanger | PMTPA mchenry, Daily backups to tridge | see Mail#Backups | 32 GB | |
Mailing lists | ESAMS lily - decommissioned | EQIAD sodium | PMTPA mchenry | See Mailing lists#Backups | 50 GB |
DNS | EQIAD ns0, PMTPA ns1 | in gerrit | ESAMS ns2 | In gerrit | |
Google docs | Google :P | ad-hoc downloads | |||
Soft data | |||||
Search databases | PMTPA various | EQIAD is primary now | SDTPA | ||
HTTP logs | PMTPA locke | emery | /a/squid | oxygen (eqiad) | |
MediaWiki logs | PMTPA NFS | tridge | fluorine | /home/wikipedia/logs | |
Fundraising | |||||
Fundraising Front-End | EQIAD aluminium | software:svn logs:storage3->tridge |
puppet:files/misc/scripts/offhost_backups | 5GB | |
Fundraising Databases | EQIAD db1008 | db1025 replication + snapshots | PMTPA storage3 replication, snapshots, dumps tridge daily rsync of dumps |
puppet:files/misc/scripts/dump_fundraisingdb | 1TB (15GB/day) |
Payment Processing | software:svn db+logs:PMTPA |
db:mysql replication db+logs:encrypt+copy daily to tridge |
db+logs:replicate to silicon | payments cluster | 100GB(?) |
Impression Logs | PMTPA nas1 | nas1001 | 1.5TB year | ||
Misc data | |||||
Server home directories | PMTPA nfs1 / nfs2 | tridge | Daily backups to Tridge | 30 GB | |
stat1 /a and /home | - | Amanda to tridge | gerrit change | 400 GB |