Using Pig to Load and Store data from HBase

Lets first store data from HDFS to our HBase Table. For this we will be using

org.apache.pig.backend.hadoop.hbase
Class HBaseStorage

public HBaseStorage(String columnList) throws org.apache.commons.cli.ParseException,IOException)

Warning: Make sure that your PIG_CLASSPATH refers to all the library files in HBASE,HADOOP and ZOOKEEPER. Doing this will save you countless hours of debugging.

Lets Create a HBase table for the data given below named as testtable.

Make sure that your first column is the ROWKEY while doing an insert to HBase table.



Lets Create a Table for this data in HBase.

>> cd $HBASE_HOME\bin
>> ./hbase shell

This will take you to your HBase shell

>> create 'testtable','cf'
>> list 'testtable'
>> scan 'testtable'

Now lets fire up grunt shell

Type in the following commands in the grunt shell

and TaDa....

Pig Casting and Schema Management

Pig is quite flexible when schema need to be manipulated.

Consider this data set



Suppose we needed to define schema after some processing we could cast the columns with their data types



That all for today folks.

Cheers!

Pig

I will posting series of posts on Pig.

I personally feel amazed at the simplicity and the power of this language.

Pig Cheat Sheet:

http://mortar-public-site-content.s3-website-us-east-1.amazonaws.com/Mortar-Pig-Cheat-Sheet.pdf

Cheers!

Krishna

Distributed Cache - Pig

I had been trying to use Distributed-Cache in Pig.
After a lot of trial and errors behold SUCCESS!
Lets get to the meat.

Lets go through the steps.
a)Create an Eval UDF
b)Initialize Distributed Cache using getCachedFiles()
c)Initialize the Data Structure using step b.
d)Finally apply your logic on the data.

Word Count In Pig

Figuring out Github. Will post the code there once its done. :)

A = LOAD 'Nameipfile' using TextLoader();
B = FOREACH  A GENERATE FLATTEN(TOKENIZE($0)) as word;
C= FOREACH ( GROUP B by word ) GENERATE, GROUP  as word, COUNT($1) as ct;
Dump C;