hadoop_hbase安装_恢复_实验文档

学云计算技术就来三通it学院安装hadoop系列软件目录版本兼容性说明.........................................................................................................................................................................12011.12.31hadoop集群基本安装过程记录.............................................................................................................................12011.01.18星期三hbase与zookeeper...................................................................................................................................5hbase多master运行实验[2012.03.16-03.19]..........................................................................................................................9hadoopnamenode恢复实验[2012.03.16-03.19].....................................................................................................................11故障解决方法...........................................................................................................................................................................16hadoophbase配置文件[2012.03.19].......................................................................................................................................17版本兼容性说明hadoop版本：0.20.205.0zookeeper版本：3.3.3hbase版本：0.90.42011.12.31hadoop集群基本安装过程记录本次使用三台机器，IP地址分别是：10.236.0.11、10.236.0.12、10.236.0.661.环境变量设置三台机器上都做如下配置：cat/root/.bash_profileexportJAVA_HOME=/usr/java/jdk1.6.0_26exportJRE_HOME=/usr/java/jdk1.6.0_26/jreexportCLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexportPATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH:$HOME/bin#exportMAVEN_OPTS=-Xmx1024mexportJAVA_OPTS=-Xmx1024MexportHADOOP_HOME=/opt/hadoopexportPATH=$HADOOP_HOME/bin:$PATHunsetUSERNAME2.准备文件drwxrwsr-x9avahi-autoipdusers4096Jun152011eclipse-rw-r--r--1rootroot220979370Jul1813:02eclipse-jee-indigo-linux-gtk.tar.gzlrwxrwxrwx1rootroot13Dec3110:46hadoop-hadoop-0.21.0drwxrwxr-x105005004096Aug172010hadoop-0.21.0学云计算技术就来三通it学院:46hbase-hbase-0.90.4drwxr-xr-x8rootroot4096Dec3110:45hbase-0.90.4-rw-r--r--1rootroot37161251Oct2923:00hbase-0.90.4.tar.gzlrwxrwxrwx1rootroot15Dec3110:46zookeeper-zookeeper-3.3.3drwxr-xr-x10100010004096Feb242011zookeeper-3.3.3-rw-r--r--1rootroot9639959Dec1517:31zookeeper-3.3.3.tar.gz3.准备ip地址与主机名映射关系主机是10.236.0.66，三台机器的/etc/hosts文件均为：10.236.0.66test6610.236.0.11SERVER-000110.236.0.12SERVER-0002127.0.0.1localhost.localdomainlocalhost4.ssl设置在namenode上执行：[root@test66~]#ssh-keygen，然后一路回车。[root@test66~]#cp/root/.ssh/id_rsa.pub/root/.ssh/authorized_keys[root@test66~]#cat/root/.ssh/authorized_keysscp/root/.ssh/authorized_keysserver-0001:/root/.ssh/scp/root/.ssh/authorized_keysserver-0002:/root/.ssh/验证namenode到datanodessh登录不再需要输入密码。sshserver-0001sshserver-0002ssh10.236.0.11ssh10.236.0.12在每个datanode上执行创建公钥，将其内容追加入namenode的/root/.ssh/authorized_keys：ssh-keygencat/root/.ssh/id_rsa.pub追加后namenode的内容含有3个公钥。验证datanode到namenodessh登录不再需要输入密码，datanode之间还是要密码的（不是必要的）。5.配置hadoop三个机器的hadoop版本、安装位置、配置，应该一样，下面是配置过程：5.1.conf/hadoop-env.sh修改hadoop-env.sh文件中的JAVA_HOME为正确设置5.2.配置文件修改core-site.xml创建临时目录mkdir/opt/hadoop_tmphadoop.tmp.dir其它临时目录的基础目录fs.default.namenamenode文件系统类型和端口号dfs.name.dirnamenode持久存储名字空间及事务日志的本地文件路径dfs.data.dirDatanode存放块数据的本地文件路径5.3.配置文件修改hdfs-site.xmldfs.replication数据备份的数量，默认35.4.配置文件修改mapred-site.xmlmapred.job.trackerjobtracker_IP:9001学云计算技术就来三通it学院配置masters和slavesmasters：加入namenode的IP地址或主机名slaves：加入所有datanode的IP地址或主机名5.6.将配置好的hadoop目录从namenode复制到各个datanode。scp-rconfserver-0001:/opt/hadoop/scp-rconfserver-0002:/opt/hadoop/5.7.配置文件内容参考[root@test66conf]#morecore-site.xml?xmlversion=1.0??xml-stylesheettype=text/xslhref=configuration.xsl?!--Putsite-specificpropertyoverridesinthisfile.--configurationpropertynamehadoop.tmp.dir/namevalue/opt/hadoop_tmp/value/propertypropertynamefs.default.name/namevaluehdfs://test66:9000/valuedescriptionspecifynamenodeandthedefaultfilesystem/description/property/configuration[root@test66conf]#cathdfs-site.xml?xmlversion=1.0??xml-stylesheettype=text/xslhref=configuration.xsl?!--Putsite-specificpropertyoverridesinthisfile.--configurationpropertynamedfs.replication/namevalue1/valuedescriptionreplicationfactor/description/propertypropertynamedfs.block.size/namevalue67108864/valuedescriptionblocksize,unitisByte,defaultis64MB/description/property/configuration[root@test66conf]#catmapred-site.xml?xmlversion=1.0??xml-stylesheettype=text/xslhref=configuration.xsl?!--Putsite-specificpropertyoverridesinthisfile.--configurationpropertynamemapred.job.tracker/namevaluetest66:9001/valuedescriptionjobtracker/description/property/configuration[root@test66conf]#catmasterstest66[root@test66conf]#catslavestest66server-0001server-0002学云计算技术就来三通it学院启动hadoop5.8.1.格式化namenode[root@test66bin]#whereishadoophadoop:/opt/hadoop-0.21.0/