Being a developer it a pain to configure hadoop every time.
I wanted a process that is easily reproducible.
So at a high level steps involved are:
Make sure you have vagrant installed
Config : 3 node , 3.4GB, 1 core each
1 master, 2 slaves
Create a vagrant file (Make sure you have more than 10GB Ram)
after creating this file all you need to do is 'vagrant up'
Step 1: FDQN
http://en.wikipedia.org/wiki/Fully_qualified_domain_name
change vi /etc/hosts
prepend with following entries
10.10.0.53 hdp.hadoop1.com hadoop1
10.10.0.54 hdp.hadoop2.com hadoop2
10.10.0.55 hdp.hadoop3.com hadoop3
(In All three nodes)
you should be able to ssh using domain names + 'hostname -f' returns fdqn for that machine
Install NTP and NTPD in all nodes
Use the script below to prepare your cluster
./prepare-cluster.sh hosts.txt
And Vola . Just run the commands below.
PS : Make sure you had entries to the hosts in Windows. Remember to check if the hostname is pingable from windows.
ping hdp.hadoop1.com
URL : hdp.hadoop1.com:8080
