Hadoop 2.4.1 설치 (Ubuntu 14.04 server)

System

• Ubuntu server 14.04
• 64-bit

Pre-requistive

• SSH 설치
• • sudo apt-get install openssh-server rsync
• JDK 설치
• • sudo apt-get install openjdk-7-jdk
  • sudo ln -s /usr/lib/jvm/java-7-openjdk-amd64 /usr/lib/jvm/jdk
  • • /usr/lib/jvm/jdk/bin/java -version
    • /usr/lib/jvm/jdk/bin/javac -version

Add Hadoop Group and User

• 사용자 추가
• • sudo adduser hadoop
• 사용자를 sudo 그룹에 추가
• • sudo adduser hadoop sudo

Setup SSH Certificate

• ssh-keygen -t rsa -P ''
• cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Download Hadoop 2.4.1

• wget https://archive.apache.org/dist/hadoop/core/hadoop-2.4.1/hadoop-2.4.1.tar.gz
• sudo tar zxvf hadoop-2.4.1.tar.gz
• sudo mv hadoop-2.4.1 /usr/local/hadoop
• sudo chown -R hadoop:hadoop /usr/local/hadoop

Setup Hadoop Environment Variables

• vi ~/.bashrc
• • export JAVA_HOME=/usr/lib/jvm/jdk/
  • export HADOOP_PREFIX=/usr/local/hadoop
  • export HADOOP_HOME=$HADOOP_PREFIX
  • export HADOOP_COMMON_HOME=$HADOOP_PREFIX
  • export HADOOP_MAPRED_HOME=$HADOOP_PREFIX
  • export HADOOP_HDFS_HOME=$HADOOP_PREFIX
  • export HADOOP_YARN_HOME=$HADOOP_PREFIX
  • export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin
• 
  
• # 64bit Java를 사용하기 위해선 아래 코드 추가
• • export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
  • export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
• 
  
• vi /usr/local/hadoop/etc/hadoop/hadoop-env.sh
• • export JAVA_HOME=/usr/lib/jvm/jdk/
• source ~/.bashrc
• hadoop version

Configure Hadoop (/usr/local/hadoop/etc/hadoop)

• vi core-site.xml

<property>

  <name>fs.default.name</name>

  <value>hdfs://localhost:9000</value>

</property>

• mkdir -p ~/hdfs/namenode ~/hdfs/datanode

• vi hdfs-site.xml

<property>

  <name>dfs.namenode.name.dir</name>

  <value>file:/home/hadoop/hdfs/namenode</value>

</property>

<property>

  <name>dfs.datanode.data.dir</name>

  <value>file:/home/hadoop/hdfs/datanode</value>

</property>

<property>

  <name>dfs.replication</name>

  <value>1</value>

</property>

• vi yarn-site.xml

<property>

  <name>yarn.nodemanager.aux-services</name>

  <value>mapreduce_shuffle</value>

</property>

<property>

  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

  <value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

• cp mapred-site.xml.template mapred-site.xml

• vi mapred-site.xml

<property>

  <name>mapreduce.framework.name</name>

  <value>yarn</value>

</property>

Format Namenode

• hdfs namenode -format

Start Hadoop Service

• start-dfs.sh
• start-yarn.sh
• jps

Run Hadoop Example

• cd /usr/local/hadoop
• hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar pi 2 5
• hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar pi 16 100000
• http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-latest/bk_using-apache-hadoop/content/running_mapreduce_examples_on_yarn.html

P.S.

• 다음 경고문을 해결하려면
• • WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
• $HADOOP_HOME/lib/native/libhadoop.so.1.0.0 를 64bit에서 재컴파일 한다음에 교체해야 한다.
• Hadoop을 운영하는데는 하등의 영향이 없다.