Building Your Own Cloud Computing Network

We have talked about cloud computing on more than one occasion last year. It

has since developed into a buzz word today and lot of big companies are talking

about and deploying it in their enterprises. The main benefit of Cloud Computing

that we have focused on till now is that it gives a more customizable and

granular control over the hardware when we go for a hosted application approach.

We have also talked about some of the pioneers who have deployed Cloud Computing

for providing such on-demand hosted services to their customers. The more well

known ones are Amazon's EC2 and Google's AppEngine. Probably Amazon was the

first one to release their Cloud offerings to the public and hence has become

quite popular in this domain.

If you have read our old issues, all of this would not be news for you. This

time we talk about something completely different. We thought, if Cloud is such

a great technology and gives such a great granular and detailed control over the

Datacenter resources, then why enterprises don't deploy their own cloud setups

and reap those benefits instead of going and buying them from some third party

service provider? And while doing our first level of research we figured out

that, yes, it is actually very much possible to deploy your own cloud in a data

center and by just using commodity hardware and open source middleware. The

benefit you get out of such an approach is immense. We shall also talk about

some of those benefits in this article. But before we go into details, let's

refresh ourselves with what Cloud computing is all about.

Cloud for Enterprises



There are thousands of documents which explain what Cloud Computing is, and it
is such a vast topic that understanding it in great detail requires a lot of

effort. But for those IT Managers and CIOs who don't have that much of time to

invest for learning, we can explain it in a nutshell as 'a cluster of

Virtualization Servers.'

The easiest way of understanding cloud is by understanding why is a cloud

required in the first place. The emergence of cloud computing has happened in

data centers because of the inherent drawback in virtualization infrastructure.

There's a need for true on the fly, on-demand scalability which no

virtualization platform can provide today. You might disagree with me on this

statement because generally virtualization is supposed to be a great enabler for

on the fly resource allocation. But the point to note over here is that a

Virtual machine (VM) cannot be scaled beyond the resources available on the Host

machine. In case the resource requirement of a VM increases beyond the resources

available on the host machine, one needs to migrate it on another virtual

machine with requisite resources. And to top it of, if the resource requirement

of the VM was temporary, then there is no proactive mechanism which can throw

back the VM to the old Host machine, thus saving the resources of the new Host

machine.

And this is where Cloud Computing comes to light. It is also sometimes

referred to as Eclectic Computing because it gives the ability to acquire and

release resources from a unified pool of hardware depending on requirement. To

understand it in detail, let's imagine a scenario where you need to build a web

server which will receive 1000 hits per day for 25 days in a month and 1,00,000

hits for the other five days.

Now if you want to get this server, you have to make sure that the server can

take the maximum possible load. So if you want to host this server elsewhere,

you have to sign an service level agreement (SLA) for 1,00,000 hits per day,

whereas your actual average requirement is far lesser than this. So you end up

paying for something which you don't actually need.



Even if you want to run this server in house, then also you have to buy a server
which can take the load of 1,00,000 hits per day and not the one which can only

take a load of 1000 hits per day, which would be a waste of money.

Just imagine a scenario where each and every server in your data center is

connected via a single middleware which converts it into a huge pool of unified

resources in terms of processors and RAM, etc and you run your servers as

virtual machines on top of it. And depending upon load at a given point of time,

VMs can just acquire the available resources in the pool (to a certain defined

max limit), use it and release it when the job is done. That's what Elastic

Computing is all about. And let's say the complete cloud is utilized to its max,

then all you have to do is to plug in one or more new free servers depending on

the resource requirement and it will automatically add up to the pool of

existing resources.

So, with this thought, we are going to deploy our own Cloud Computing

Infrastructure and run some VMs on top of it. We will be using Amazon's EC2

client for connecting and using the Cloud we create. In other words, we can even

say that we are going to build an EC2 compatible cloud.

Here we go...



To begin with, let's first identify what all we need to build a Cloud. Of

course we need some servers, which will be the nodes of the Cloud, and then we

need a Controller Server, which will manage the complete cloud. To connect all

of these, we will require a network, preferably a Gbps network, and we will need

a clustering and a virtualization middleware.

Here all the software we are going to use is open source or free and is

available for download. And if you are going to do a test setup of cloud for

understanding and research, only two servers will be more than enough for you.

You don't need to worry about spending a huge amount of money in this

deployment. You can just do a test run in your backyard.

Installing the controller



The software which we are going to use as the Cloud middleware is called
Eucalyptus and is an Open Source project managed by UCSB. Eucalyptus stands for

“Elastic Utility Computing Architecture for Linking Your Programs To Useful

Systems”. A long name that!

There are two ways by which you can deploy Eucalyptus. One is a lengthy

process where you install multiple Linux machines, install Java and Xen on them,

and then Install the Client components of Eucalyptus. Then install a separate

Linux machine and install the controller components of Eucalyptus. Next you

configure and use it (if you still have energy left for doing that!).

Else, there is a simpler approach. Install a ROCKS clustering front end with

Java and Xen roll, then add Eucalyptus Roll to it manually. Then let the ROCKS'

TFTP server take over the node deployments and you are done. Once the nodes are

installed, you can just download Amoazon's EC2 client and start using it.

After booting the central management machine with Rocks 5.0 DVD, type frontend to start installation

I am sure, you must be preferring the second option more and that's what I

also did. Now let's look at deploying Eucalyptus using ROCKS. For those who

don't know what ROCKS is, here's a small intro: it is a Clustering/Grid

middleware something similar to OSCAR and comes with a brilliant node monitoring

tool called Ganglia (which we have talked about before) and TFTP based remote

deployment support for installing nodes.

Installing ROCKS front end



Download ROCKS 5.0 Boot, Core and OS Roll DVD from http://www.rocksclusters.org/wordpress/?page_id=82.
Select the right architecture before downloading. We went with a 64-bit model,

which is most likely the desired architecture for a Cloud. But a word of caution

here! Don't get carried away with the instinct of downloading the latest and the

greatest because on the same site, ROCKS version 5.1 DVD is also available which

doesn't work with Eucalyptus. And no document anywhere says that. I first did

that and wasted a complete day in trying to deploy Eucalyptus for Rocks 5.1.

Once you have download the DVD ISO and burnt it, get a machine (can be any

machine with preferably one GB of RAM, P-IV and above with two network cards).

Now, connect one network card with your network switch which preferably has a

DHCP server and an Internet connection. We will call this the “production

network.” And connect the other network card to an isolated network (preferably

a Gbps network); where you are going to connect all your Cloud node machines. We

shall call this is Cloud network. Make sure there is no DHCP server running on

the Cloud network because Rocks frontend is going to run its own DHCP server,

pre-configured for node's remote installation.

Make sure to select Rolls shown in the image while installing Rocks for your Cloud

Now put in the DVD and fire up the installation, once the boot screen comes,

type “frontend” and hit enter. This will start the installation for ROCKS'

management server. After a couple of booting minutes, you will see a screen

which asks you to select the Rolls you want to install on the Rocks server.

There are two ways of installing the rolls: one is through the DVD and the other

is through the Internet. Select the CD/DVD option and you will see a page with a

list of available roles. Select the rolls shown in the image in the previous

page. Don't forget to specifically include the base, Xen, Java, OS and Ganglia

roles.

Once you have selected the rolls, click on the Submit button and you will be

redirected to the previous page. Now hit next and you will see a screen which

will ask you to fill in details for creating a self certificate for the HTTPS

web access of the Rocks front end. Enter your location information and proceed.

Running 'rocks list roll' will show you the list of Rolls installed. Here, check if Eucalyptus Roll is installed or not.

Once this is done, the installer will ask you standard information such as

password, timezone, etc. and it will ask you how to partition the server hard

drive. It is good to do a automatic partitioning to avoid any confusion. Just

make sure you don't have any data on the hard disk. After gathering this

information, the system will start copying the file and depending on your system

resources it will take from 10 to 20 minutes to finish the install.

Adding the Eucalyptus Roll



Once the ROCKS installation is over, it is time to install the Eucalyptus Roll
on the Rocks front end. For this you have to download the eucalyptus roll ISO

from http://eucalyptus.cs.ucsb.edu/downloads/31 and copy it to the front end

system's '/' folder. Now add the eucalyptus role by running the following

command:

# rocks add roll clean=1 /

$eucalyptus-1.3.x86_64.disk1.iso

Now enable the roll by running the following command:

#rocks enable roll eucalyptus

Once this is done, you can check if the eucalyptus roll is installed and

enabled or not, by running the following command:

# rocks list roll

This should show something like the visual above.

Now you have to modify the TFTP based remote installation of the node in such

a way that when rocks does the remote installation, the eucalyptus node

components also gets installed on the nodes. To do so run the following command:

# cd /home/install && rocks-dist dist

This command will take some time to run and will build the new package list

for remote installation of nodes.

In the inter-ether command screen select the VM Container option to start the installation of Eucalyptus nodes.

Installing nodes



Now this is the time to install nodes over the network. For this, run the Rocks
command “insert-ether” from a terminal. It will open up a window with a list of

device types. Select the last option which says “vm-container” and hit enter. A

blank window will open. Now boot up all the nodes and make sure they are all

connect to the cloud network and their boot preference is set to Network/PXE

boot. Once you do so you will start seeing entries being added to the blank

window of “insert-ether”. You will see one entry each for the number of nodes

you have booted.

On the nodes, the remote installation will start automatically and the

complete installation will happen with any manual intervention. You can even

have your nodes headless and see the installation screen remotely through a VNC

client with the host name of the nodes shown in the insert-ether window.

Once the nodes have been installed with Eucalyptus client, you have to now

install and configure the Eucalyptus web management interface and manager on the

front end/management machine. For this run the following commands. These

commands will install and configure all eucalyptus server side rpms:

#kroll eucalyptus > build.sh



#sh ./build.sh

Once this is done, reboot all your node machines. Don't know why, but I faced

a lot of trouble getting all nodes attached to the cloud till the time I

rebooted all of them once. Once the reboot process has been done, you can say

your EC2 enabled Eucalyptus cloud is up and running. Now this is the time to

start using it.

Adding images



Now your cluster is ready but you need some virtual machines to run on top of
it. For doing this, you have to upload some Linux OS images to it. For getting

your hands dirty, you can use a very tiny Linux image which is available from

Eucalyptus' website. You can download it from http://eucalyptus.cs.ucsb.edu/releases/euca-ttylinux.tgz.

once downloaded, untar and unzip it and run the following command:

#/opt/eucalyptus/usr/sbin/euca add_image --disk-image

ttylinux.img --kernel-image vmlinuz-2.6.16.33-xen --image-name ttylinux

And the image will be updated to the cloud.

Once your nodes start booting up you will see them getting registered in this window one after the other.

Using the Cloud with Amazon's EC2 client



If you are comfortable with Amazon's EC2 client then you can just skip this
section. But if you are not, then read on. To start with, first get a client

machine, any Windows or Linux machine will do. But the process will vary a bit.

We did it on a Linux machine.

First download Amazon's EC2 toolset and APIs from

http://s3.amazonaws.com/ec2-downloads/ec2-api-tools-1.3-24159.zip. Now unzip it.

But before using the commands first generate and download a certificate for your

cloud's account, so that you can authenticate the machine from the system that

is running the EC2 client. To do so, open the following link:

https://ip.of.the.cloud.front.end:8443

Change “ip.of.the.cloud. front. end” with the IP or FQDN of the Cloud's

frontend. A login page will appear. Login through it using admin as username and

password. Once you login for the first time to the Eucalyptus page, you will be

asked to change the Admin password. Change it and proceed and you will see the

admn page. Now from here, you can add more users and then login through them to

get the certificate or you can just download the Admin's certificate to get

started. To download the Admin's certificate, click on “Generate Certificate”

button on the page. It will download a zipped file. Once downloaded, unzip it to

~/.euca folder.

Now move EC2 Tools folder to opt and run the following commands to register

the EC2 and certificate paths, and the cloud's website:

export EC2_HOME=/opt/ec2-api-tools-1.3-24159



export PATH=$PATH:$EC2_HOME/bin


export EC2_PRIVATE_KEY=~/.euca/euca2-*-pk.pem


export EC2_CERT=~/.euca/euca2-*-cert.pem


export EC2_URL=http:// ip.of.the.cloud.front.end:8773/services/Eucalyptus

Now, you will be able to run the following command to see how many cores you

have in your cloud and how many are free for you:

#ec2-describe-availability-zones

Now you have to create a private key, so that you can use it to connect to an

instance when you run them, through SSH. To do so, run:

ec2-add-keypair key >> key.private

Now run the following command to get the EMI identifier for the image/s you

have added:

#ec2-describe-images

It will return a value like emi-xxxxxxx. Now run the following command to

start instances of the image on the cloud:

#ec2-run-instances emi-xxxxxxx -k key

And the linux image will boot up on the cloud. You can access it through SSH.

But for that you will need its IP which you can get by running the following

command:

#ec2-describe-instances

To connect to the image using SSH and the private key you just created, run

the following command:

ssh -i key.private root@ip-addresses-of-the-image

This screen shows that our 'Cloud' is Up with all 24 cores out of the 24 available free.

And you are done. In case you still have any doubts or want to provide your

feedback on this article, please visit http://forums. pcquest.com, a 3 lakh

strong community of techies, constantly discussing new tech and services.