Dumps/Snapshot hosts

From Wikitech
Jump to navigation Jump to search

Snapshot (XML dumps generation) cluster information

Hardware

These hosts generate the XML dumps. For information about the hosts that serve them, see Dumps/Dump servers.

We have one mini snapshot cluster. Eventually another will be set up in codfw.

In D.C.:

  • snapshot1005: primary, HP Proliant DL360, Debian 9 (stretch), 64GB RAM, 2 Xeon E5-2698 16-core CPUs, 2 480GB SDs in hw raid 1
  • snapshot1006: primary, HP Proliant DL360, Debian 9 (stretch), 64GB RAM, 2 Xeon E5-2698 16-core CPUs, 2 480GB SDs in hw raid 1
  • snapshot1007: primary, HP Proliant DL360, Debian 9 (stretch), 64GB RAM, 2 Xeon E5-2698 16-core CPUs, 2 480GB SDs in hw raid 1
  • snapshot1008: primary, Dell PowerEdge R440, Debian 9 (stretch), 64GB RAM, 2 Intel Xeon Gold 6130 16-core CPUs, 2 480GB SSDs in software raid 1
  • snapshot1009: primary, Dell PowerEdge R440, Debian 9 (stretch), 64GB RAM, 2 Intel Xeon Gold 6130 16-core CPUs, 2 480GB SSDs in software raid 1

Current setup

The dumps monitor which cleans up stale locks and generates the index.html file, currently runs on snapshot1007.

Worker nodes:

En wikipedia dumps run on snapshot1009 via the dump scheduler. Wikidata dumps run on snapshot1006 via the scheduler. All other xml/sql dumps run on snapshot1005,6,7, via the dump scheduler. Dumps other than the xml/sql dumps run on snapshot1008.

For info on running the dumps, see Dumps#Starting_dump_runs.

Other tasks

All misc cron jobs apart from the sql/xml dump runs take place on snapshot1008. From here adds-changes dumps are run, wikidata weekly entity dumps are generated, and so on.