Data Platform/Systems/Hadoop/Test
The Analytics team runs a Hadoop test cluster in production. This seems controversial since we could use Cloud virtual machines instead, but due to the complexity of the Hadoop set up (Kerberos auth, TLS encryption, Kafka topics to sample to generate data, etc..) it would be very resource demanding to run a test cluster in there.
The purpose of the cluster is to test any kind of infrastructure change before production, from a small config to big and invasive changes like upgrading the Hadoop distribution or changing the security auth schema.
Hosts and their roles
an-test-master1003.eqiad.wmnet - Hadoop master (HDFS Namenode, Yarn ResourceManager, Mapreduce History server).
an-test-master1004.eqiad.wmnet - Hadoop master standby ((HDFS Namenode, Yarn ResourceManager).
an-test-coord1002.eqiad.wmnet - Multi-purpose node (Hive Server/Metastore, Oozie, Presto, systemd timers for refine and druid load).
an-test-worker100[1,2,3].eqiad.wmnet - Hadoop worker nodes (HDFS Datanode, Yarn Nodemanagerm HDFS Journalnode).
an-test-druid1001.eqiad.wmnet - Single node Druid test cluster (all Druid daemons running on it).
an-test-ui1001.eqiad.wmnet - Yarn UI.
an-test-client1002.eqiad.wmnet - analytics client
an-test-presto1001.eqiad.wmnet - Presto server.
an-test-druid1001.eqiad.wmnet - Druid server.
UI quicklinks
Yarn: ssh -L 8080:an-test-ui1001.eqiad.wmnet:80 an-test-ui1001.eqiad.wmnet and then set Host: yarn.wikimedia.org in the browser's custom header settings
Druid Coordinator/Overlord: ssh -L 8081:localhost:8081 an-test-druid1001.eqiad.wmnet
Hadoop upgrade - jobs to test
Before rolling a new hadoop version to the production server we should have tested the following jobs on the upgrade test cluster:
- Gobblin data import (subset of webrequest and events)
- Webrequest Load (on smaller data)
- Events Refine (on smaller data)
- Druid load (webrequest_sampled_128 for instance)
- Wikitext-current (single small XML dump imported manually)