Server - Web and Hadoop Development Environment.

■ Web service Development Environment.

Host Name	Ip address	Installed software
centos01	192.168.11.20	mysql(sudo /etc/init.d/mysql.server start|stop) - njoonk tomcat(/etc/init.d/tomcat start|stop) - njoonk
centos02	192.168.11.21	nginx(/etc/init.d/nginx start|stop) - root redis(/etc/init.d/redis_6379 start|stop) - root gitlab(/etc/init.d/gitlab start|stop) - root - https://192.168.11.21
centos03	192.168.11.22	artifactory(/etc/init.d/artifactory start|stop) - root - http://192.168.11.22:8081 tomcat(/etc/init.d/tomcat start|stop) - njoonk - http://192.168.11.22:8080

■ This is my Hadoop Development Environment.

Host Name	Ip address	Installed software
hadoop01	192.168.11.23	hadoop(master name node), hbase
hadoop02	192.168.11.24	hadoop(secondary name node, data node1), zookeeper1
hadoop03	192.168.11.25	hadoop(data node2), zookeeper2, Flume1
hadoop04	192.168.11.26	hadoop(data node3), zookeeper3, Flume2
hadoop05	192.168.11.27	zookeeper, kafka

Hbase - How to install Hbase

@ Download
$ wget http://ftp.riken.jp/net/apache/hbase/stable/hbase-1.0.0-bin.tar.gz

@ uncompress on the [/usr/local/src]
$ tar xvf ./hbase-1.0.0-bin.tar.gz
$ mv ./hbase-1.0.0 ../hbase
$ chown -R hadoop.hadoop ./hbase/
$ cd ./hbase/conf

@ Set configuration
$ vim ./hbase-env.sh

export JAVA_HOME=/usr/local/java
export HBASE_MANAGES_ZK=false

@ Set environment variables 
$ vim ./.bash_profile

export HBASE_HOME=/usr/local/hbase
export PATH=$HBASE_HOME/bin:$PATH

@Make a public key (on hadoop user)
$ ssh-keygen -t rsa

@Copy a public key into authorized_keys in other server (on hadoop user)

@ Make a folder hbase( on Name Server)
$ cd /usr/local/hadoop/bin
$ ./hadoop fs -mkdir /hbase

@ If there are 3 server on the zookeeper.
$ vim ./hbase-site.xml
# Execute start-hbase.sh on the master Hbase <----
# You have to execute the zookeeper first

@Check URL
http://192.168.11.23:60010/master-status?filter=handler

Zookeeper - How to install zookeeper

@ DownLoad zookeeper.
@ http://ftp.riken.jp/net/apache/zookeeper/
$ cd /usr/local/src/
$ wget http://ftp.riken.jp/net/apache/zookeeper/stable/zookeeper-3.4.6.tar.gz

@ Decompress zookeeper-3.4.5.tar.gz on the [/usr/local/src]
$ tar xvf ./zookeeper-3.4.6.tar.gz
$ mv ./zookeeper-3.4.6 ../zookeeper
$ chown -R hadoop.hadoop /usr/local/src/zookeeper
$ cd /usr/local/zookeeper/conf/
$ mv ./zoo_sample.cfg ./zoo.cfg

@ Set configuration
$ vim ./zoo.cfg

@ If you want Running Replicated ZooKeeper.
@ Add  the following settings into zoo.cfg
server.1=192.168.11.24:2888:3888
server.2=192.168.11.25:2888:3888
server.3=192.168.11.26:2888:3888

@ Make a tar file with zookeeper
@ Copy the zookeeper.tar to each server

@ start
$ bin/zkServer.sh start
@ If this error occur, set myId to every server.

2015-04-03 01:22:15,509 [myid:] - ERROR [main:QuorumPeerMain@85] - Invalid config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /usr/local/zookeeper/bin/../conf/zoo.cfg
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.lang.IllegalArgumentException: /tmp/zookeeper/myid file is missing
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:350)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)

@ First server
$ echo "1" > /tmp/zookeeper/myid
@ Second server
$ echo "2" > /tmp/zookeeper/myid
@ Third server
$ echo "3" > /tmp/zookeeper/myid

http://youngdeok-k.com/entry/7-%EC%A3%BC%ED%82%A4%ED%8D%BC-zookeeper-%ED%99%9C%EC%9A%A9-ndash-%EB%B6%84%EC%82%B0%EC%84%9C%EB%B2%84-%EA%B5%AC%ED%98%84-2%ED%8E%B8

Hadoop - How to Install Hadoop-1.2.1

@How to Install Hadoop@

@ Makes the hadoop user (in root).
@ If you don't have a hadoop group, commend like the following this.
$ /usr/sbin/groupadd hadoop
$ /usr/sbin/useradd -d /home/hadoop -m hadoop -g hadoop

@ You don't need to set the Hadoop's password
@ But you have to set Nonpass between the host server and node server on Linux.

@ Add hosts
$ vim /etc/hosts

192.168.11.23   server01 # (nameNode)

192.168.11.24   server02 # (secondaryNameNode01, dataNode01)

192.168.11.25   server03 # (dataNode02)
192.168.11.26   server04 # (dataNode03)

@Make a public key (in hadoop)
$ ssh-keygen -t rsa

@ Contribute the public key to master and slave server on Hadoop's user
@ Copy a public key in id_rsa.pub into authorized_keys.
$ vim /home/hadoop/.ssh/authorized_keys
@ Changes the permission
$ chmod 644 /home/hadoop/.ssh/authorized_keys

@ Make the name directory (in hadoop, server is nameNode)

$ mkdir -p /home/hadoop/data/name

@ Make the secondary name directory (in hadoop, server is secondaryNameNode01)

$ mkdir -p /home/hadoop/data/checkpoint
$ chown -R hadoop.hadoop /home/hadoop/data/checkpoint

@ Make the name directory (in hadoop, server is dataNode01~03)

$ mkdir -p /home/hadoop/data01 
$ mkdir -p /home/hadoop/data02
$ chown -R hadoop.hadoop /home/hadoop/data01
$ chown -R hadoop.hadoop /home/hadoop/data02

@@ Install Haddop @@

@ There is a hadoop
@ http://ftp.kddilabs.jp/infosystems/apache/hadoop/common/
$ cd /usr/local/src/
@ user is root
$ wget http://ftp.kddilabs.jp/infosystems/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
@ Install Hadoop (master and slave both)
$ tar xvf ./hadoop-1.2.1.tar.gz
$ mv ./hadoop-1.2.1 ./hadoop
$ chown -R hadoop.hadoop /usr/local/hadoop

@Configure the Hadoop(master)

## conf/hadoop-env.sh ##
#=================================
# Not to show the warning deprecated
HADOOP_HOME_WARN_SUPPRESS=TRUE
export HADOOP_HOME_WARN_SUPPRESS
# The java implementation to use.  Required.
export JAVA_HOME=/usr/local/java
#================================

@## core-site.xml ##@
@## hdfs-site.xml ##@

@ Set master server
(This is secondary namenode)
$ vi master
@ Add below this

## conf/master ##
#=================================
server02

@Set slave server
$ vi slave
@ Add below this
server02
server03
server04

## conf/slave ##
#=================================
server02
server03
server04

@ Set the hadoop home directory path
$ vi /etc/profile
@ Add below this at end of file. (master, slave both)

# /etc/profile
# =====================================
export HADOOP_HOME=/usr/local/hadoop
# =====================================

@Apply the value
$ source /etc/profile

@ Install slaves as the master.

@ Make the directory as DataNode (on the nameNode server)
@ You may need to do connection first for SSH
$ ./bin/slaves.sh mkdir -p /home/hadoop/data01/hdfs
$ ./bin/slaves.sh mkdir -p /home/hadoop/data02/hdfs

@ Format the Namenode (on the master)
$ ./bin/hadoop namenode -format

[hadoop@centos04 hadoop]$ ./bin/hadoop namenode -format
15/03/22 01:41:36 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = server01/192.168.11.23
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152;
compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_75
************************************************************/
Re-format filesystem in /home/hadoop/data/name ? (Y or N) Y

@ If this error occur, you'd better check hosts on file.(Delete 127.0.0.1 server01)
#org.apache.hadoop.ipc.RPC: Server at server01/127.0.0.1:9000 not available yet, Zzzzz...


@ Add the follow information to open ports and access each other in server.
$ vim /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 50070 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 50090 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 50100 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 50105 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9000 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9001 -j ACCEPT

@Start the hadoop
$ ./bin/start-dfs.sh

@ If the following error occur, Just Iptables off lisk this ($ chkconfig iptables off)

error: java.io.IOException: File /home/hadoop/data/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1

@ If you want to do reformat ($ ./bin/hadoop namenode -format), you'd better do after delete all of old directorys(name, data, data01, data02)

@Stop the hadoop

$ ./bin/stop-dfs.sh

@You can see the hadoop on Firefox

http://192.168.11.23:50070

@You can see the hadoop on Console

$ ./bin/hadoop dfsadmin -report

@@@Truble Shouting@@@

@ If a dataNode doesn't run, Check It's permission
$ ./bin/slaves.sh chmod 755 /home/hadoop/data01/hdfs
$ ./bin/slaves.sh chmod 755 /home/hadoop/data02/hdfs 

2013-05-17 13:42:37,457 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: Incorrect permission for /home/hadoop/data01/hdfs, expected: rwxr-xr-x, while actual: rwxrwxr-x
2013-05-17 13:42:37,462 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: Incorrect permission for /home/hadoop/data02/hdfs, expected: rwxr-xr-x, while actual: rwxrwxr-x
2013-05-17 13:42:37,462 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in dfs.data.dir are invalid.

@In Addition, This [mapred-site.xml]
#This [/.../mapred/system]directory on Hadoop.
#This [/.../mapred/local]directory on Local System.
$ ./bin/slaves.sh chmod 755 /home/hadoop/data
#==========================================================#

#==========================================================#

I refer to this post 
http://blog.beany.co.kr/archives/412
http://blog.beany.co.kr/archives/1373
http://www.slideshare.net/TaeYoungLee1/20141029-25-hive
Thank you

Troubleshooting - When a data is putted into the Hive.

@  The permission is cause that the following is error.
@ You'd better change date on the log file.
--------------------------------------------------------------------
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
(88 bytes are truncated)

Hive - Install Parquet into Hive

@ 1. Get the source download
$ git clone https://github.com/Parquet/parquet-mr.git
@ 2. Change the tag
$ git checkout parquet-1.0.0
@ 3. You only need to command the following like this.
$ mvn install -rf :parquet-hive -e -DskipTests=true
$ mv ./parquet-hive-1.0.0.jar /usr/local/hive/lib/
@ 4. You need the library under the lib
$ cd /usr/local/hive/lib
$ for f in parquet-avro parquet-cascading parquet-column parquet-common parquet-encoding parquet-generator parquet-hadoop parquet-hive parquet-pig parquet-scrooge parquet-test-hadoop2 parquet-thrift
> do
> curl -O https://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/${f}/1.2.5/${f}-1.2.5.jar
> done
> curl -O https://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/parquet-format/1.0.0/parquet-format-1.0.0.jar
 
Thank you
http://cmenguy.github.io/blog/2013/10/30/using-hive-with-parquet-format-in-cdh-4-dot-3/

Hive - Install on CentOs

@1. Download the hive(user:root, place:/usr/local/src)
$ wget http://ftp.riken.jp/net/apache/hive/stable/hive-0.11.0.tar.gz

@ 2.Next you need to unpack the tarball(user:root, place:/usr/local/src)
$ tar xvf ./hive-0.11.0.tar.gz

@ 3. Move the hive unpacked(user:root, place:/usr/local/src)
$ mv ./hive-0.11.0 ../hive

@ 4. Change own(user:root, place:/usr/local)
$ chown -R hadoop.hadoop ./hive/

@5. Set the environment variable(user:root, place:/usr/local)
$ vim /etc/profile
export HIVE_HOME=/usr/local/hive
@OR (user:hadoop, place:/home/hadoop)
export HIVE_HOME=/usr/local/hive
export PATH=$HIVE_HOME/bin:$PATH

@6. You must create /tmp and /hive
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /hive
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /hive

@7. You must change the path on Hive(user:hadoop, place:/usr/local/hive/conf)
$ cp ./hive-default.xml.template ./hive-default.xml
# /user/hive/warehouse -> /hive
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/hive</value>
  <description>location of default database for the warehouse</description>
</property>

@8. Execute hive
$ $HIVE_HOME/bin/hive

Hive - How to stop a job in Hive.

$ hadoop job -kill <job_id>.

DataArtists - What is the Data Artists?

Let's watch the Ted.

Hadoop - exclude a node on ruuning server

■dfs.hosts.exclude:
Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded.

# Add below this to hdfs-site.xml
       <property>
              <name>dfs.hosts.exclude</name>
              <value>/home/hadoop/hadoop/conf/excludes</value>
      </property>

■mapred.hosts.exclude
Names a file that contains the list of hosts that should be excluded by the jobtracker. If the value is empty, no hosts are excluded. # Add below this to mapred-site.xml
    <property>
        <name>mapred.hosts.exclude</name>
        <value>/home/hadoop/hadoop/conf/excludes</value>
    </property>

# Excute
$ bin/hadoop dfsadmin -refreshNodes

# Excute Banlancer to banlanc for data
bin/hadoop balancer

Hadoop - Searching for something

Hadoop - Remove node

@1. Get the IP or Hosts list by running "report" command
$ $HADOOP_HOME/hadoop dfsadmin -report | grep Name

@2. You include IP:Port The following file.
@$HADOOP_HOME/conf/excludes
00.xx.xxx.001:50010

@3. invoke command:
$ $HADOOP_HOME/bin/hadoop dfsadmin -refreshNodes

@4. Verification
$ $HADOOP_HOME/bin/hadoop dfsadmin -report | grep -Eiw ‘Name|Decommission’

@4. This time is MapReduce
@If It has the exclude file, You can command this
$ $HADOOP_HOME/bin/hadoop mradmin -refreshNodes

http://pearlin.info/2012/04/best-way-to-blacklist-node-from-live-hadoop-cluster/

Hadoop - commands

@Find files you want to see.
hadoop dfs -lsr /hadoop/flume/ | grep [search_term].

@hadoop error(release safe mode)
$./bin/hadoop dfsadmin -safemode leave