Troubleshooting - When a data is putted into the Hive.

@  The permission is cause that the following is error.
@ You'd better change date on the log file.
--------------------------------------------------------------------
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
(88 bytes are truncated)

Hive - Install Parquet into Hive

@ 1. Get the source download
$ git clone https://github.com/Parquet/parquet-mr.git
@ 2. Change the tag
$ git checkout parquet-1.0.0
@ 3. You only need to command the following like this.
$ mvn install -rf :parquet-hive -e -DskipTests=true
$ mv ./parquet-hive-1.0.0.jar /usr/local/hive/lib/
@ 4. You need the library under the lib
$ cd /usr/local/hive/lib
$ for f in parquet-avro parquet-cascading parquet-column parquet-common parquet-encoding parquet-generator parquet-hadoop parquet-hive parquet-pig parquet-scrooge parquet-test-hadoop2 parquet-thrift
> do
> curl -O https://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/${f}/1.2.5/${f}-1.2.5.jar
> done
> curl -O https://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/parquet-format/1.0.0/parquet-format-1.0.0.jar
 
Thank you
http://cmenguy.github.io/blog/2013/10/30/using-hive-with-parquet-format-in-cdh-4-dot-3/

Hive - Install on CentOs

@1. Download the hive(user:root, place:/usr/local/src)
$ wget http://ftp.riken.jp/net/apache/hive/stable/hive-0.11.0.tar.gz

@ 2.Next you need to unpack the tarball(user:root, place:/usr/local/src)
$ tar xvf ./hive-0.11.0.tar.gz

@ 3. Move the hive unpacked(user:root, place:/usr/local/src)
$ mv ./hive-0.11.0 ../hive

@ 4. Change own(user:root, place:/usr/local)
$ chown -R hadoop.hadoop ./hive/

@5. Set the environment variable(user:root, place:/usr/local)
$ vim /etc/profile
export HIVE_HOME=/usr/local/hive
@OR (user:hadoop, place:/home/hadoop)
export HIVE_HOME=/usr/local/hive
export PATH=$HIVE_HOME/bin:$PATH

@6. You must create /tmp and /hive
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /hive
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /hive

@7. You must change the path on Hive(user:hadoop, place:/usr/local/hive/conf)
$ cp ./hive-default.xml.template ./hive-default.xml
# /user/hive/warehouse -> /hive
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/hive</value>
  <description>location of default database for the warehouse</description>
</property>

@8. Execute hive
$ $HIVE_HOME/bin/hive

Hive - How to stop a job in Hive.

$ hadoop job -kill <job_id>.

Hive - A QL for NULL

SELECT count(distinct s1.uu_id)
FROM (
    SELECT distinct uu_id FROM table1
    WHERE dt >= "2014-01-01" AND dt < "2014-01-02"
   ) s1
   LEFT OUTER JOIN (
    SELECT distinct uu_id FROM table1
    WHERE dt >= "2014-01-01" AND dt < "2014-01-02"
    AND activity = 'spend'
) s2
  ON (s1.uu_id = s2.uu_id)
JOIN (
    SELECT distinct uu_id FROM table1
    WHERE dt = "2014-01-02"
    AND activity = 'spend'
) s3
  ON (s1.uu_id = s3.uu_id)
WHERE s2.uu_id IS NULL

Troubleshooting - Hive

@When you see the following error.
----------------------------------------------------------------
FAILED: SemanticException [Error 10035]: Column repeated in partitioning columns
----------------------------------------------------------------
@Solution
sudo -u hdfs hive -e "CREATE TABLE table_temp (time string, aaa string, bbb string, dt string) partitioned by(dt string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS SEQUENCEFILE;"

Hive - join

@Join