Instructions for use of LCSR's hadoop cluster

Setup

• Download the hadoop software from http://apache.ziply.com/hadoop/common/hadoop-0.20.203.0/.
• Unpack it in your home directory.
• Add the following to the <configuration> definition in hadoop-0.20.203.0/conf/core-site.xml:

      <property>
         <name>fs.default.name</name>
         <value>hdfs://hadoop41.rutgers.edu:7000/</value>
      </property>
      <property>
         <name>hadoop.tmp.dir</name>
         <value>/tmp/hadoop-NetID</value>
      </property>
  

  changing "NetID" to your own user name.
• In hadoop-0.20.203.0/conf/hadoop-env.sh, set up JAVA_HOME and HADOOP_HOME.
  (HADOOP_HOME should point to the directory where you unpacked the software, presumably /your/home/dir/hadoop-0.20.203.0).
• Add the following to the <configuration> definition in hadoop-0.20.203.0/conf/mapred-site.xml:

      <property>
          <name>mapred.job.tracker</name>
          <value>hadoop42.rutgers.edu:7001</value>
      </property>
  

• In hadoop-0.20.203.0/conf/masters, change localhost to hadoop42.

Usage

[Note: to be able to submit jobs to our hadoop cluster, a directory has to be created for you. Ask the CS professor sponsoring your work to request one.]

To submit (a sample) job:

    cd hadoop-0.20.203.0
    bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'

One user reported an error trying to submit the example saying "Input path does not exist." He was able to fix this with the command

    bin/hadoop dfs -mkdir input

    bin/hadoop fs -get output output
    cat output/*