Hadoop - How to Install Hadoop-1.2.1

@How to Install Hadoop@

@ Makes the hadoop user (in root).
@ If you don't have a hadoop group, commend like the following this.
$ /usr/sbin/groupadd hadoop
$ /usr/sbin/useradd -d /home/hadoop -m hadoop -g hadoop

@ You don't need to set the Hadoop's password
@ But you have to set Nonpass between the host server and node server on Linux.

@ Add hosts
$ vim /etc/hosts

192.168.11.23   server01 # (nameNode)

192.168.11.24   server02 # (secondaryNameNode01, dataNode01)

192.168.11.25   server03 # (dataNode02)
192.168.11.26   server04 # (dataNode03)

@Make a public key (in hadoop)
$ ssh-keygen -t rsa

@ Contribute the public key to master and slave server on Hadoop's user
@ Copy a public key in id_rsa.pub into authorized_keys.
$ vim /home/hadoop/.ssh/authorized_keys
@ Changes the permission
$ chmod 644 /home/hadoop/.ssh/authorized_keys

@ Make the name directory (in hadoop, server is nameNode)

$ mkdir -p /home/hadoop/data/name

@ Make the secondary name directory (in hadoop, server is secondaryNameNode01)

$ mkdir -p /home/hadoop/data/checkpoint
$ chown -R hadoop.hadoop /home/hadoop/data/checkpoint

@ Make the name directory (in hadoop, server is dataNode01~03)

$ mkdir -p /home/hadoop/data01 
$ mkdir -p /home/hadoop/data02
$ chown -R hadoop.hadoop /home/hadoop/data01
$ chown -R hadoop.hadoop /home/hadoop/data02

@@ Install Haddop @@

@ There is a hadoop
@ http://ftp.kddilabs.jp/infosystems/apache/hadoop/common/
$ cd /usr/local/src/
@ user is root
$ wget http://ftp.kddilabs.jp/infosystems/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
@ Install Hadoop (master and slave both)
$ tar xvf ./hadoop-1.2.1.tar.gz
$ mv ./hadoop-1.2.1 ./hadoop
$ chown -R hadoop.hadoop /usr/local/hadoop

@Configure the Hadoop(master)

## conf/hadoop-env.sh ##
#=================================
# Not to show the warning deprecated
HADOOP_HOME_WARN_SUPPRESS=TRUE
export HADOOP_HOME_WARN_SUPPRESS
# The java implementation to use.  Required.
export JAVA_HOME=/usr/local/java
#================================

@## core-site.xml ##@

	<configuration>
	<property>
	<name>fs.default.name</name>
	<value>hdfs://server01:9000</value>
	</property>
	<property>
	<name>hadoop.tmp.dir</name>
	<value>/tmp/hadoop-${user.name}</value>
	</property>
	</configuration>

view raw core-site.xml hosted with ❤ by GitHub

@## hdfs-site.xml ##@

	<configuration>
	<property>
	<name>dfs.name.dir</name>
	<value>/home/hadoop/data/name</value>
	</property>
	<property>
	<name>dfs.data.dir</name>
	<value>/home/hadoop/data01/hdfs,/home/hadoop/data02/hdfs</value>
	</property>
	<property>
	<name>dfs.replication</name>
	<value>2</value>
	</property>
	<property>
	<name>dfs.permissions</name>
	<value>false</value>
	</property>
	<property>
	<name>fs.trash.interval</name>
	<value>30</value>
	</property>
	<property>
	<name>dfs.http.address</name>
	<value>server01:50070</value>
	</property>
	<property>
	<name>dfs.secondary.http.address</name>
	<value>server02:50090</value>
	</property>
	<property>
	<name>fs.checkpoint.dir</name>
	<value>/home/hadoop/data/checkpoint</value>
	</property>
	<property>
	<name>fs.checkpoint.period</name>
	<value>360</value>
	</property>
	<property>
	<name>dfs.support.append</name>
	<value>true</value>
	</property>
	<property>
	<name>dfs.datanode.max.xcievers</name>
	<value>4096</value>
	</property>
	<property>
	<name>dfs.webhdfs.enabled</name>
	<value>true</value>
	</property>
	<property>
	<name>dfs.backup.address</name>
	<value>server02:50100</value>
	</property>
	<property>
	<name>dfs.backup.http.address</name>
	<value>server02:50105</value>
	</property>
	<property>
	<name>dfs.hosts.exclude</name>
	<value>/usr/local/hadoop/conf/excludes</value>
	</property>
	</configuration>

view raw hdfs-site.xml hosted with ❤ by GitHub

@ Set master server
(This is secondary namenode)
$ vi master
@ Add below this

## conf/master ##
#=================================
server02

@Set slave server
$ vi slave
@ Add below this
server02
server03
server04

## conf/slave ##
#=================================
server02
server03
server04

@ Set the hadoop home directory path
$ vi /etc/profile
@ Add below this at end of file. (master, slave both)

# /etc/profile
# =====================================
export HADOOP_HOME=/usr/local/hadoop
# =====================================

@Apply the value
$ source /etc/profile

@ Install slaves as the master.

@ Make the directory as DataNode (on the nameNode server)
@ You may need to do connection first for SSH
$ ./bin/slaves.sh mkdir -p /home/hadoop/data01/hdfs
$ ./bin/slaves.sh mkdir -p /home/hadoop/data02/hdfs

@ Format the Namenode (on the master)
$ ./bin/hadoop namenode -format

[hadoop@centos04 hadoop]$ ./bin/hadoop namenode -format
15/03/22 01:41:36 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = server01/192.168.11.23
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152;
compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_75
************************************************************/
Re-format filesystem in /home/hadoop/data/name ? (Y or N) Y

@ If this error occur, you'd better check hosts on file.(Delete 127.0.0.1 server01)
#org.apache.hadoop.ipc.RPC: Server at server01/127.0.0.1:9000 not available yet, Zzzzz...


@ Add the follow information to open ports and access each other in server.
$ vim /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 50070 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 50090 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 50100 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 50105 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9000 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9001 -j ACCEPT

@Start the hadoop
$ ./bin/start-dfs.sh

@ If the following error occur, Just Iptables off lisk this ($ chkconfig iptables off)

error: java.io.IOException: File /home/hadoop/data/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1

@ If you want to do reformat ($ ./bin/hadoop namenode -format), you'd better do after delete all of old directorys(name, data, data01, data02)

@Stop the hadoop

$ ./bin/stop-dfs.sh

@You can see the hadoop on Firefox

http://192.168.11.23:50070

@You can see the hadoop on Console

$ ./bin/hadoop dfsadmin -report

@@@Truble Shouting@@@

@ If a dataNode doesn't run, Check It's permission
$ ./bin/slaves.sh chmod 755 /home/hadoop/data01/hdfs
$ ./bin/slaves.sh chmod 755 /home/hadoop/data02/hdfs 

2013-05-17 13:42:37,457 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: Incorrect permission for /home/hadoop/data01/hdfs, expected: rwxr-xr-x, while actual: rwxrwxr-x
2013-05-17 13:42:37,462 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: Incorrect permission for /home/hadoop/data02/hdfs, expected: rwxr-xr-x, while actual: rwxrwxr-x
2013-05-17 13:42:37,462 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in dfs.data.dir are invalid.

@In Addition, This [mapred-site.xml]
#This [/.../mapred/system]directory on Hadoop.
#This [/.../mapred/local]directory on Local System.
$ ./bin/slaves.sh chmod 755 /home/hadoop/data
#==========================================================#

	<configuration>
	<property>
	<name>mapred.job.tracker </name>
	<value>master:9001</value>
	</property>
	<property>
	<name>mapred.system.dir</name>
	<value>/home/hadoop/data/mapred/system</value>
	</property>
	<property>
	<name>mapred.local.dir</name>
	<value>/home/hadoop/data/mapred/local</value>
	</property>
	<property>
	<name>mapred.hosts.exclude</name>
	<value>/home/hadoop/hadoop/conf/excludes</value>
	</property>
	</configuration>

view raw mapred-site.xml hosted with ❤ by GitHub

#==========================================================#

I refer to this post 
http://blog.beany.co.kr/archives/412
http://blog.beany.co.kr/archives/1373
http://www.slideshare.net/TaeYoungLee1/20141029-25-hive
Thank you

The solutions for communicating on SSH Commend

@ If you faced the following error, the solution would help you

@ Error 1
>Pseudo-terminal will not be allocated because stdin is not a terminal.
@ the solution 1
@ [-oStrictHostKeyChecking=no] Option make a fingerprint to write in a known_hosts
@ (maybe -T -oStrictHostKeyChecking=no)
$ ssh -oStrictHostKeyChecking=no user@somehost sudo /home/tomcat/sh/sample.sh

@ Error 2
>sudo: sorry, you must have a tty to run sudo
@ the solution 1
$ ssh -t -oStrictHostKeyChecking=no user@somehost sudo /home/tomcat/sh/sample.sh
or
@ Comment the requiretty in /etc/sudoers like the following
#Defaults requiretty 

@ Extra explanation.
@ -T Disable pseudo-tty allocation.
@ -t Force pseudo-tty allocation.

http://aero.sarang.net/blog/2009/01/ssh-1.html

Mysql - useful Tips

@ Check the character set on mysql.
mysql> status

mysql> show variables like 'character%';

@ How to copy 1 column from one DB to another DB.
mysql> INSERT INTO `target_databases`.`target_table` (`column21`, `column22`, `column23`) 
SELECT `column11`, `column12`, 1 FROM `source_databases`.`source_table` WHERE column15='test';

@ When these are different DB, -- This is option (--skip-column-names, -N)
$ ./mysql -e "SELECT column1, column2, 1 FROM database_name.table_name WHERE column3 in ('test1', 'test2')" -u root -p xxxx database_name > ~/dumpFile.txt
mysql>LOAD DATA LOCAL INFILE '~/dumpFile.txt' INTO TABLE target_table_name;

@ Backup only the schema
$ mysqldump -u root -p --no-data database_name > ~/onlySchema.sql

@ Ignores foreign keys constraints (0 : ignore, 1:value)
mysql> SET FOREIGN_KEY_CHECKS=0

@ Backup databases
$ ./mysqldump -u root -p Database_Name > DB_backup.sql
$ ./mysql -u root -p Database_Name < DB_backup.sql

@ Create a databases
@ Check character set
mysql> show variables like 'character%';
mysql> CREATE DATABASE databases DEFAULT CHARACTER SET utf8 or utf8mb4;

@SELECT VARIABLES values
mysql>show VARIABLES where Variable_name like '%log%';
@ Start to print logs into a file in mysql
mysql>SET GLOBAL general_log='on';
@If you want to change a path of a log file, change FILE for a path you want
mysql>SET GLOBAL log_output = ‘FILE’;

@Modify a column property
mysql> ALTER TABLE apns_book MODIFY book_badge VARCHAR(2) NULL;

@Remove [on update CURRENT_TIMESTAMP]
# Set the same column name
mysql> ALTER TABLE table_name CHANGE column_name column_name TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;

@ How to load data into table from CSV in Mac
mysql> LOAD DATA LOCAL INFILE '~/test.csv' INTO TABLE table_name FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r';

@ How to control data encrypted.
# select
mysql>SELECT AES_DECRYPT(column_name, 'password') as name FROM table_name;
# update
mysql>UPDATE table_name SET name=AES_ENCRYPT('Update Data', 'password') WHERE id=1;

@ Select data encrypted
mysql>SELECT distinct AES_DECRYPT(column_name, 'password') as test FROM table_name WHERE AES_DECRYPT(column_name, 'password') LIKE '%test%';

@ Change character set
mysql> SET character_set_client = utf8;
mysql> SET character_set_results = utf8;
mysql> SET character_set_connection = utf8;
mysql> ALTER DATABASE DB명 DEFAULT CHARACTER SET utf8;

@ Just Memo
semi  synchronize replication, sync_binlog sync_relay_log 

http://bryan7.tistory.com/101

MyStory - My experience in learning English.

Playing games in English for learning English.
 - It helped me nothing. but I enjoyed playing game.

Reading CNN news for learning English.
 - It has helped me to have some knowledge.
 - I can also have ability to read in English more.

Reading comic book for learning English.
 - It helped me a few. but if you read loudly, It will help you a lot.

Watching TV drama with subscript for learning English.
 - It also helped me nothing.  It was my mistake.
 - But I have watched movies without subscript, that has helped me listening.
 - I recommend you to use Youtube.

Meeting a native speaker for learning English.
 - This is the best way.
 - But if you don't study English usually, It is useless.

Studying with English books and reading loudly for learning English.
 - This is also the best way.
 - I have been doing this way when I have time.

Linux - How to change DHCP for STATIC - 3

@ This is a DHCP set automatically when it is installed.


@ This is a static IP address changed.