Installing Hadoop Cluster with Cloudera Manager

This article shows how fast and easy it may be to install Hadoop cluster with Cloudera Manager. There are three major steps to follow:

1. Prepare hosts
2. Install Cloudera Manager
3. Install Cluster

Since the first two steps may be carried out either manually (in details showcases how to prepare hosts and install Cloudera Manager) or through Vagrant automatization (lets you play with Cloudera Manager due to Vagrant script that automates host preparation and CM installation), this article will cover both approaches.

Deploying, configuring and running a Hadoop cluster manually is rather time- and cost-consuming and may even lead to market position loss if company's business operations depend directly on the technical operations speed; that's why specialized tools are preferable.

Luckily, that's where software comes into play. To make things simpler, here's a helping hand to creating a fully distributed Hadoop cluster with Cloudera Manager to get down to real life practice. Mind though, if you need a dev environment, just download a pre-prepared virtual machine with a pseudo distributed cluster inside.

This step by step guide covers the main highlights of the process (full documentation of Cloudera Manager including security, cluster optimization and fine tuning could be found here). Let's get this Hadoop cluster installed.

Before we start, here's a short glossary that might come in handy:

• CDH – Cloudera Distribution including Hadoop. CDH includes Hadoop and other applications that are usually used along, e.g. Flume, HBase, Hive, Impala, Kafka, Pig, Spark, Sqoop, etc.
• Cloudera Manager – a tool for Apache Hadoop administration including such operations as installation, upgrading, host commission/decommission, monitoring
• Vagrant – a tool for building complete development environments

Used software versions include:

CDH - 5
Cloudera Manager - 5.7
Vagrant - 1.8.1
OS - Centos 6.7
VitrualBox - 4.3.28

Ground Plan

To install a Hadoop cluster:

1. Prepare servers
2. Install Cloudera Manager
3. Install Cloudera Manager Agents and CDH
4. Install Hadoop cluster

Vagrant

If you are going to practice on your own workstation (not on real servers), it's recommended that the first two steps are automated with Vagrant. Note: make sure you have at least 16 GB of RAM.

To prepare cluster servers and to install Cloudera Manager, do the following:

• Install VirtualBox
• Install Vagrant
• Get vagrant files
  $> git clone https://github.com/vbash/vagrant-cludera-manager.git


• Provision and run the cluster
  $> vagrant up

To make changes in your hosts file and make the virtual machines available by theirs hostnames, Vagrant may ask for the password of a current user. If Vagrant has provisioned and run the cluster successfully, skip steps 1 and 2 (preparing servers and installing Cloudera Manager).

1. Prepare servers

Skip this step if Vagrant was used. Go to the step 3.

For this minimal cluster, 4 servers are needed (minimal requirements for a non-production cluster):

• 1 x 8Gb RAM (Cloudera Manager + most important Hadoop Services)
• 3 x 1.2Gb RAM (Data Nodes)

Cloudera Manager Documentation provides the following instruction for each node in a soon-to-be cluster:

1. Disable Selinux
2. Setup NTP
3. Disable firewall
4. Define host names

These steps are automated in the Vagrant section above. If you prepare your servers manually, use the instructions below. Note: all the instructions are tested for CentOS 6.7. Use appropriate user manual for another OS.

1.1. Disable Selinux

Run the following in the command line:

$> sudo sed -i 's/^\(SELINUX\s*=\s*\).*$/\1disabled/' /etc/selinux/config

This command modifies Selinux's config file disabling Selinux service.

1.2. Setup NTP

NTP service is required to keep system clock on each server synchronized with global time and with each other. Do the following to setup it:

$> sudo yum -y install ntp
$> sudo chkconfig ntpd on
$> sudo service ntpd start
$> sudo hwclock --systohc

1.3. Disable Firewall

$> sudo chkconfig iptables off

1.4. Define host names

1.4.1. Edit /etc/hosts

The /etc/hosts file should have the following inside it:

127.0.0.1 localhost
10.211.55.101 cloudera-1
10.211.55.102 cloudera-2
10.211.55.103 cloudera-3
10.211.55.104 cloudera-4

1.4.2. Define system hostname

Run the following command on each host with a corresponding name (cloudera-1, cloudera-2, cloudera-3, cloudera-4):

$> sudo hostname cloudera-1

1.4.3. Edit /etc/sysconfig/network

The /etc/sysconfig/network file should look as follows:

NETWORKING=yes
HOSTNAME=cloudera-1
RES_OPTIONS="single-request-reopen"

1.4.4. Restart the network service

Restart the network service on each server to apply the changes:

$> sudo service network restart

2. Install Cloudera Manager

Skip this step if Vagrant was used. Go to the step 3.

This step is also automated with Vagrant. If you install it manually, follow the instruction below.

Installation could be divided into the following steps:

1. Install MySql database
2. Instal and run Cloudera Manager server

MySql an Cloudera Manager will be installed on cloudera-1 server (8GB of RAM).

2.1. Install MySql database

Install MySql server:

$> yum -y install mysql-server mysql
$> chkconfig mysqld on

Edit MySql configuration /etc/my.cnf in according to Cloudera recommendations:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550

#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

Start MySql:

$> service mysqld start

Create a script that sets up MySql security settings:

#!/bin/bash

yum -y install expect
SECURE_MYSQL=$(expect -c "
set timeout 10
spawn mysql_secure_installation
expect \"Enter current password for root (enter for none):\"
send \"\r\"
expect \"Change the root password?\"
send \"n\r\"
expect \"Remove anonymous users?\"
send \"y\r\"
expect \"Disallow root login remotely?\"
send \"n\r\"
expect \"Remove test database and access to it?\"
send \"y\r\"
expect \"Reload privilege tables now?\"
send \"y\r\"
expect eof
")

echo "$SECURE_MYSQL"

Save this script as mysqlsec.sh, then make it executable and run it:

$> chmod a+x mysqlsec.sh
$> ./mysqlsec.sh

Create Cloudera Manager database and user:

$> mysql -u root -e "create database scm" mysql
$> mysql -u root -e "grant all on *.* to 'scm'@'%' identified by 'scm' with grant option;" mysql

2.2. Install Cloudera Manager

Setup Cloudera repository:

$> wget -O /etc/yum.repos.d/cloudera-manager.repo http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo
$> yum -y update

Install packages:

$> yum -y install oracle-j2sdk1.7 cloudera-manager-server cloudera-manager-daemons
$> yum -y install mysql-connector-java

Prepare Cloudera Manager Database:

$> /usr/share/cmf/schema/scm_prepare_database.sh mysql -h localhost scm scm scm

Start Cloudera Manager server:

$> service cloudera-scm-server start

Wait a bit while Cloudera Manager is starting its services and web server.

3. Install Cloudera Manager Agents and CDH

Go to http://cloudera-1:7180

This is Cloudera Manager login page. Use admin/admin as login/password

Then read and accept the license agreement and choose "Cloudera Enterprise Data Hub Edition Trial" on the next page.

After that you'll be offered to setup a new cluster. Use the following pattern to search all the nodes for the new cluster:

cloudera-[1-4]

That will find all 4 servers we have prepared before.

Select all of them and press "Continue" button. Accept the default CDH repository settings. "Continue". Accept installing Java:

Do not choose the single user mode. Just press "Continue".

Provide Cloudera Manager with a root password for all the servers. If you have used Vagrant approach for servers preparation the password is "vagrant". If you prepared your servers manually, use the password you created:

Then just wait for Cloudera Manager to finish agents and CDH installation.

Press "Continue" and wait for distribution and activation.

Press "Continue" and wait for Cluster Inspector to finish the inspection.

4. Install Hadoop cluster

On the next page choose Core Hadoop installation.

Then you can choose the cluster roles distribution across the cluster. Accept the default options.

Then you have to define SQL server for the services. If you have used Vagrant for server preparation use the following parameters:

Host Name: cloudera-1
Database Type: MySQL
Database Name: scm
Username: scm
Password: scm

In case you prepared SQL server manually use your own parameters.

Press "Continue".

On the page with changes review accept the default settings and press "Continue".

Wait for the Cloudera Manager to setup the cluster roles.

The cluster role deployment diagram looks as follows:

When cluster is installed you can see it in Cloudera Manager (http://cloudera-1:7180/cmf/home):

Now you can monitor the cluster state, add and remove new services in this cluster, change configurations, identify problems in the cluster and so on. The yellow signs shown near the services are warnings that can be ignored now but should be analyzed and fixed if you are going to bring the cluster in production.

Summary

Cloudera Manager makes creation and maintenance of Hadoop clusters significantly easier than if they have been managed manually. Due to this instruction it is possible to create a Hadoop cluster in less than one hour when manual configuration and deployment could take a few hours or even days. Also, it's super convenient having all tools you need in one place. For more Cloudera Manager tricks visit cloudera.com