Nova Resource:Math/Flink

This page document setup of Flink on the math labs cluster.

Install Apache Flink and run job via YARN

1. A working Hadoop cluster needs to be installed first (follow these instructions).
2. Sign in to master node (e.g. hadoop000)
3. Set up Apache Flink
   • Download Flink files: wget http://www-eu.apache.org/dist/flink/flink-1.1.3/flink-1.1.3-bin-hadoop26-scala_2.10.tgz
   • Extract files tar zxvf flink-1.1.3-bin-hadoop26-scala_2.10.tgz
   • Let's Flink discover the Hadoop configuration directory export HADOOP_CONF_DIR=/etc/hadoop/conf
4. Run Flink job on YARN
   • WordCount.jar prints output to console flink/bin/flink run -m yarn-cluster -yn 1 -yjm 1024 -ytm 4096 flink/examples/batch/WordCount.jar
   • See official Flink YARN documentation for more details.