Dumps/Snapshot hosts

From Wikitech
Jump to: navigation, search

Snapshot (XML dumps generation) cluster information

Hardware

These hosts generate the XML dumps. For information about the hosts that serve them, see Dumps/Dump servers.

We have one mini snapshot cluster. Eventually another will be set up in codfw.

In D.C.:

  • snapshot1001: canary, PowerEdge R815, Ubuntu 12.04.2, 64GB RAM, 4 8-core Opterons, 2 80GB HDs
  • snapshot1005: primary, HP Proliant DL360, Ubuntu 14.04, 64GB RAM, 2 Xeon E5-2698 16-core CPUs, 2 480GB SDs in hw raid 1
  • snapshot1006: primary, HP Proliant DL360, Ubuntu 14.04, 64GB RAM, 2 Xeon E5-2698 16-core CPUs, 2 480GB SDs in hw raid 1
  • snapshot1007: primary, HP Proliant DL360, Ubuntu 14.04, 64GB RAM, 2 Xeon E5-2698 16-core CPUs, 2 480GB SDs in hw raid 1

Current setup

The dumps monitor which cleans up stale locks and generates the index.html file, currently runs on snapshot1004.

Worker nodes:

En wikipedia dumps run on snapshot1005 via the dump scheduler. All other dumps run on snapshot1006-7 via the dump scheduler.

For info on running the dumps, see Dumps#Starting_dump_runs.

Other tasks

All misc cron jobs apart from the main dump runs take place on snapshot1003. From here central auth dumps are run, wikidata json dumps are generated, and so on.