CLUSTER

On a first stage, four virtual machines were deployed on a private OpenNebula cloud environment, placed on CSUC facilities. That configuration allowed to get expertise in the installation and management of CDH, the popular Apache Hadoop distribution. Some programming and performing analysis were executed, which revealed the need to go further and deploy the system using physical machines.

Cadi cluster

HP CP4000

• 64 Opteron 275 dual core
  • 2.2 GHz, 256 KB L1 cache, 2 MB L2 cache
• 256 GB RAM
• 4.5 TB disk
• 281.6 Gflops

SOFTWARE

• CentOS 6.5
• Cloudera Manager 5.0.0
  • Cloudera Express
  • oracle-j2sdk 1.7.0+update45-1
  • PostgreSQL 8.4.20

Upgrade to 5.4.1

• Backup databases
  • pg_dump -h localhost -p 7432 -U scm > /tmp/scm_server_db_backup.$(date +m%d)
    • password from /etc/cloudera-scm-server/db.properties
• Stop Hue, then stop Hive
• Stop the Cloudera Management Service
• Stop any running commands
• sudo service cloudera-scm-server stop
• sudo service cloudera-scm-server-db stop
• sudo service cloudera-scm-agent stop
• wget http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo
  • Alternatively, vi /etc/yum.repos.d/cloudera-manager.repo

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64           	  
name=Cloudera Manager
baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/
gpgkey = http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera    
gpgcheck = 1

• sudo yum clean all
• sudo yum upgrade 'cloudera-*'
• sudo service cloudera-scm-server-db start
• sudo service cloudera-scm-server start
• sudo service cloudera-scm-agent start
• Log in to the Cloudera Manager Admin Console and select "Yes, I would like to upgrade the Cloudera Manager Agent packages now"
• Select "Matched Release for this Cloudera Manager Server"
• Check "Install Oracle Java SE Development Kit (JDK)"
• Wait for installation

Cesc Julbe (erased user) - 2015-06-04

Comments