hadoop大数据应用开发环境配置

Hadoop大数据应用开发环境配置一、运行环境计算机名IP角色备注Kingo-mn192.168.0.230主节点(namenode),masters负责总管分布式数据的调度。Kingo-ms192.168.0.231副节点(JobTracker),masters分解任务的执行。资源控制器Kingo-d1192.168.0.232数据节点1(Datanode1)Slaves/TaskTracker，负责分布式数据存储以及任务的执行。Kingo-d2192.168.0.238数据节点2(Datanode2)Kingo-d3192.168.0.239数据节点3(Datanode3)说明修改/etc/sysconfig/network文件可以修改主机名；主机名不能含“_”。Ip主机名程序进程192.168.0.230Kingo-mnJdkHadoophbaseNamenodeDFSZKFailoverControllerHamsterresourceManager192.168.0.231Kingo-msJdkHadoophbaseNamenodeSecondaryNamenodeJobTrackerDFSZKFailoverControllerHamsterresourceManager192.168.0.232Kingo-d1JdkHadoopZookeeperhbaseDatanodenodeManagerJournalNodeQuorumPeerMainHregionServerTaskTracker192.168.0.238Kingo-d2JdkHadoopZooKeeperhbaseDatanodenodeManagerJournalNodeQuorumPeerMainHregionServerTaskTracker192.168.0.239Kingo-d3JdkHadoopZookeeperHbaseDatanodenodeManagerJournalNodeQuorumPeerMainHregionServerTaskTracker二、创建帐号四台机器均创建kingo用户组和bigdata帐号，密码统一为Kingo123。#groupaddkingo#useradd-d/home/bigdata-gkingobigdata#passwdbigdata//设置密码为Kingo123#cd/home#chgrp-Rkingobigdata//设置目录所属组#chmod-Rg=rwxbigdata//设置目录所属组的权限为读、写、执行三、绑定hostname与IP#vi/etc/hosts，增加内容如下：192.168.0.230kingo-mn192.168.0.231kingo-ms192.168.0.238kingo-rm192.168.0.232kingo-d1192.168.0.239kingo-d2四、设置节点之间的免密码通信1、向各节点复制Hosts文件scp/etc/hostskingo-ms:/etc/hostsscp/etc/hostskingo-d1:/etc/hostsscp/etc/hostskingo-d2:/etc/hostsscp/etc/hostskingo-d3:/etc/hosts2、在各节点上生成各自SSH秘钥对#ssh-keygen-tdsa-P''-f~/.ssh/id_dsa#cd~/.ssh#catid_dsa.pubauthorized_keys#chmod600authorized_keys3、复制公钥到某一节点#ssh-copy-id-i~/.ssh/id_dsa.pubkingo-mn4、复制密钥到各个节点。在kingo-mn上执行如下命令：#scp/root/.ssh/authorized_keyskingo-ms:/root/.ssh/;scp/root/.ssh/authorized_keyskingo-d1:/root/.ssh/;scp/root/.ssh/authorized_keyskingo-d2:/root/.ssh/;scp/root/.ssh/authorized_keyskingo-d3:/root/.ssh/五、安装JDK把jdk-7u80-linux-x64.rpm安装包复制到/bigdata/setup文件夹下，执行：#rpm–ivhjdk-7u80-linux-x64.rpmJdk安装到默认目录/usr/java/jdk1.7.0_80下。#vi/etc/profile加入如下配置信息：exportJAVA_HOME=/usr/java/jdk1.7.0_80exportCLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jarexportPATH=.:$JAVA_HOME/bin:$PATH#sourceprofile//执行profile内容，确认是否配置成功#java–version//查看是否JDK安装成功六、设置hadoop环境变量#vi/etc/profileexportHADOOP_HOME=/home/bigdata/hadoop-2.7.1exportPATH=.:$HADOOPHOME/sbin:$HADOOP_HOME/bin:$JAVAHOME/bin:$PATHexportHADOOP_LOG_DIR=/home/bigdata/hadoop-2.7.1/logsexportYARN_LOG_DIR=$HADOOP_LOG_DIRexportHADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexportHADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib#hadoopversion//查看hadoop是否安装成功七、配置hadoop1、配置hadoop2.7.1/etc/hadoop下的hadoop-env.sh、yarn-env.sh、mapred-env.sh修改JAVA_HOME值：exportJAVA_HOME=/usr/java/jdk1.7.0_802、配置hadoop2.7.1/etc/hadoop/slaveskingo-d1kingo-d2kingo-d33、配置hadoop2.7.1/etc/hadoop/core-site.xmlconfigurationpropertynamefs.defaultFS/namevaluehdfs://kingo-mn:9000//valuedescription设定namenode的主机名及端口/description/propertypropertynamehadoop.tmp.dir/namevalue/home/bigdata/tmp/hadoop-${user.name}/valuedescription存放临时文件的目录/description/propertypropertynamehadoop.proxyuser.hadoop.hosts/namevalue*/value/propertypropertynamehadoop.proxyuser.hadoop.groups/namevalue*/value/property/configuration4、配置hadoop2.7.1/etc/hadoop/hdfs-site.xmlconfigurationpropertynamedfs.namenode.http-address/namevaluekingo-mn:50070/valuedescriptionNameNode获得fsimage和edits/description/propertypropertynamedfs.namenode.secondary.http-address/namevaluekingo-ms:50090/valuedescriptionSecondNameNode获得最新的fsimage/description/propertypropertynamedfs.namenode.rpc-address/namevaluekingo-mn:9000/valuedescriptionkingo-mn/description/propertypropertynamedfs.replication/namevalue3/valuedescription设定HDFS存储文件的副本个数，默认为3/description/propertypropertynamedfs.namenode.name.dir/namevaluefile:///home/bigdata/hadoop-2.7.1/hdfs/name/valuedescriptionnamenode用来持续存放命名空间和交换日志的本地文件系统路径/description/propertypropertynamedfs.datanode.data.dir/namevaluefile:///home/bigdata/hadoop-2.7.1/hdfs/data/valuedescriptionDataNode在本地存放块文件的目录列表，用逗号分隔/description/propertypropertynamedfs.namenode.checkpoint.dir/namevaluefile:///home/bigdata/hadoop-2.7.1/hdfs/namesecondary/valuedescription设置secondary存放临时镜像的本地文件系统路径，如果这是一个用逗号分隔的文件列表，则镜像将会冗余复制到所有目录，只对secondary有效/description/propertypropertynamedfs.webhdfs.enabled/namevaluetrue/value/propertypropertynamedfs.stream-buffer-size/namevalue131072/valuedescription默认是4KB，作为hadoop缓冲区，用于hadoop读hdfs的文件和写hdfs的文件，还有map的输出都用到了这个缓冲区容量，对于现在的硬件很保守，可以设置为128k(131072),甚至是1M(太大了map和reduce任务可能会内存溢出)/description/propertypropertynamedfs.namenode.checkpoint.period/namevalue3600/valuedescription两次checkpoints之间的间隔，单位为秒，只对secondary有效/description/property/configurationpropertynamedfs.permissions.enabled/namevaluefalse/valuedescriptionIftrue,enablepermissioncheckinginHDFS.Iffalse,permissioncheckingisturnedoff,butallotherbehaviorisunchanged.Switchingfromoneparametervaluetotheotherdoes