In this tutorial, we will install the Hortonworks Data Platform (HDP) HDP 3.0.1/ HDP2.6.5 Sandbox Through Docker on AWS on an single node EC2 Instance.
Apache Hadoop is a layered structure to process and store massive amounts of data. In our case, Apache Hadoop will be recognized as an enterprise solution in the form of HDP. At the base of HDP exists our data storage environment known as the Hadoop Distributed File System. When data files are accessed by Hive, Pig or another coding language, YARN is the Data Operating System that enables them to analyze, manipulate or process that data. HDP includes various components that open new opportunities and efficiencies in healthcare, finance, insurance and other industries that impact people.
1.Login in to the AWS Console and search the EC2 (Virtual Server in the Cloud) services and we launch a service.
2.Boot an EC2 instance with Amazon Linux 2 AMI:
Use >= 16GB of RAM (t2.xlarge or above).
3.We keep the default settings
- Add >=20GB storage.
4 For the HDP 3.0.1 we require more than 30 GB storage
5 We addd key tags key
Name an value
- Configure security Group: Add rule for [Type=All TCP, Source=My IP]. It is important to restrict access only to your IP.
6 We choose our key pair or create new one.
- Keep note the key pair you are using for the EC2 instance.
- Note down the public ip of this instance. This is the IP address you are going to use to access HDP services and web UIs.
7 After the EC2 instance is up and running, select the instance, click
Specify the path to your key-pair file and connect. I used to connect via ssh on the terminal.
8 Enter to the EC2 instance via ssh and install git with
sudo yum install -y git.
9 Check out the scripts with
git clone https://github.com/ruslanmv/HDP-Sandbox-AWS.git,
Enter to the HDP-Sanbox-AWS folder
10 Install docker on the EC2 instance with
bash install_docker.sh. Logout the SSH client and login again
docker info to confirm docker is working without
12 In the GitHub repository of the installation, that I have here
There are two HDP versions 3.0.1, and 2.6.5 that we can install on AWS Cloud.
You should choose one, in this tutorial I selected the 2.6.5 because Hive View is not present in HDP 3.0.1 or Ambari 2.7.
a) HDP 3.0.1
Install HDP through docker with
b) HDP 2.6.5
Install HDP through docker with
It will take a while to install.
The advantage of the HDP 2.6.5 is that you have Hive View
When it is finished you will have the following screen
You can check with the command
docker ps to see if is running
14 After it finishes, access Ambari through
http://your-ec2-public-ip:8080/ to confirm it is working.
You can see the ip4 public ip and paste it on the web browser
by default the credentials are:
Username : maria_dev
Just after deployed the server the services are starting
and after few minutes the services should be ready ( something like 7 minutes)
Great finally is running our server on the AWS Cloud with HDP.
Just as an example let us visualize a dataset in Hive
We want to use a datset from the GroupLens which has collected and made available rating data sets from the MovieLens web site (http://movielens.org). The data sets were collected over various periods of time, depending on the size of the set. Just a test we will use the MovieLens 100K movie ratings. Stable benchmark dataset. 100,000 ratings from 1000 users on 1700 movies. Released 4/1998
We can extract the all the files of the zip file.
There are two files that we will use, u.data nd u.item
The next step is open Hive View
We prooceed to upload file from local,
We specify we click on settings for the file type and we choose the Field Delimitar
and we select the file
And we rename the columns and table name
and we click upload table
Now we repeat the same procedure, we click on settings for the file type and we choose the filed delimiter
and now we choose the file
and rename the columns and table name
and we upload the table
Query in Hive
After the tables are uploaded we can open the query editor and execute the following SQL command and execute it.
In addition we can visualize the data easily , we choose the x, y vaiables the columns that we want to visualize.
Great now, we are able to plot features of a dataset by using Hive.
Important , after use the aws instance, if you wont use the server,please terminate the instance to avoid charges.
HDP Sandbox Port Forwards
- 2122 -> 22 HostSSH
- 2200 -> 22 HostSSH2
- 2222 -> 2222 DockerSSH
- 7777 -> 7777 Streaming Analytics Manager
- 8585 -> 8585 Streams Messaging Manager
- 7788 -> 7788 Schema Registry
- 8000 -> 8000 Storm Logviewer
- 9995 -> 9995 Zeppelin1
- 9996 -> 9996 Zeppelin2
- 9088 -> 9088 NiFi Protocol
- 61080 -> 61080 NiFi Registry
- 8886 -> 8886 AmbariInfra
- 61888 -> 61888 Log Search
- 10500 -> 10500 HS2v2
- 4040 -> 4040 Spark
- 4200 -> 4200 AmbariShell
- 8983 -> 8983 SolrAdmin
- 42080 -> 80 Apache
- 42111 -> 42111 nfs
- 8020 -> 8020 HDFS
- 8040 -> 8040 nodemanager
- 8032 -> 8032 RM
- 8080 -> 8080 ambari
- 8443 -> 8443 Knox
- 8744 -> 8744 StormUI
- 1080 -> 1080 Splash Page
- 8993 -> 8993 Solr
- 10000 -> 10000 HS2
- 10001 -> 10001 HS2Http
- 10002 -> 10002 HiveJDBCJar
- 30800 -> 30800 DAS
- 11000 -> 11000 Oozie
- 15000 -> 15000 Falcon
- 19888 -> 19888 JobHistory
- 50070 -> 50070 WebHdfs
- 50075 -> 50075 Datanode
- 50095 -> 50095 Accumulo
- 50111 -> 50111 WebHcat
- 16010 -> 16010 HBaseMaster
- 16030 -> 16030 HBaseRegion
- 60080 -> 60080 WebHBase
- 6080 -> 6080 XASecure
- 18080 -> 18080 SparkHistoryServer
- 8042 -> 8042 NodeManager
- 21000 -> 21000 Atlas
- 8889 -> 8889 Jupyter
- 8088 -> 8088 YARN
- 2181 -> 2181 Zookeeper
- 9090 -> 9090 Nifi
- 4557 -> 4557 NiFi DistributedMapCacheServer
- 6627 -> 6627 Storm Nimbus Thrift
- 9000 -> 9000 HST
- 6667 -> 6667 Kafka
- 9091 -> 9091 NiFi UI HTTPS
- 2202 -> 2202 Sandbox SSH 2
- 8188 -> 8188 YarnATS
- 8198 -> 8198 YarnATSR
- 9089 -> 9089 Druid1
- 8081 -> 8081 Druid2
- 2201 -> 2201 SSH HDP CDA
The default Ambari credential is
maria_dev: maria_dev . The default AmbariShell login credential is
Congratulations! We have installed and executed Hive visualizations on an EC2 instance using Cloudera HDP Sandbox.
For further information of HDP you can follow the official tutorials: