hadoop大数据应用开发环境配置

整理文档很辛苦,赏杯茶钱您下走!

免费阅读已结束,点击下载阅读编辑剩下 ...

阅读已结束,您可以下载文档离线阅读编辑

资源描述

Hadoop大数据应用开发环境配置一、运行环境计算机名IP角色备注Kingo-mn192.168.0.230主节点(namenode),masters负责总管分布式数据的调度。Kingo-ms192.168.0.231副节点(JobTracker),masters分解任务的执行。资源控制器Kingo-d1192.168.0.232数据节点1(Datanode1)Slaves/TaskTracker,负责分布式数据存储以及任务的执行。Kingo-d2192.168.0.238数据节点2(Datanode2)Kingo-d3192.168.0.239数据节点3(Datanode3)说明修改/etc/sysconfig/network文件可以修改主机名;主机名不能含“_”。Ip主机名程序进程192.168.0.230Kingo-mnJdkHadoophbaseNamenodeDFSZKFailoverControllerHamsterresourceManager192.168.0.231Kingo-msJdkHadoophbaseNamenodeSecondaryNamenodeJobTrackerDFSZKFailoverControllerHamsterresourceManager192.168.0.232Kingo-d1JdkHadoopZookeeperhbaseDatanodenodeManagerJournalNodeQuorumPeerMainHregionServerTaskTracker192.168.0.238Kingo-d2JdkHadoopZooKeeperhbaseDatanodenodeManagerJournalNodeQuorumPeerMainHregionServerTaskTracker192.168.0.239Kingo-d3JdkHadoopZookeeperHbaseDatanodenodeManagerJournalNodeQuorumPeerMainHregionServerTaskTracker二、创建帐号四台机器均创建kingo用户组和bigdata帐号,密码统一为Kingo123。#groupaddkingo#useradd-d/home/bigdata-gkingobigdata#passwdbigdata//设置密码为Kingo123#cd/home#chgrp-Rkingobigdata//设置目录所属组#chmod-Rg=rwxbigdata//设置目录所属组的权限为读、写、执行三、绑定hostname与IP#vi/etc/hosts,增加内容如下:192.168.0.230kingo-mn192.168.0.231kingo-ms192.168.0.238kingo-rm192.168.0.232kingo-d1192.168.0.239kingo-d2四、设置节点之间的免密码通信1、向各节点复制Hosts文件scp/etc/hostskingo-ms:/etc/hostsscp/etc/hostskingo-d1:/etc/hostsscp/etc/hostskingo-d2:/etc/hostsscp/etc/hostskingo-d3:/etc/hosts2、在各节点上生成各自SSH秘钥对#ssh-keygen-tdsa-P''-f~/.ssh/id_dsa#cd~/.ssh#catid_dsa.pubauthorized_keys#chmod600authorized_keys3、复制公钥到某一节点#ssh-copy-id-i~/.ssh/id_dsa.pubkingo-mn4、复制密钥到各个节点。在kingo-mn上执行如下命令:#scp/root/.ssh/authorized_keyskingo-ms:/root/.ssh/;scp/root/.ssh/authorized_keyskingo-d1:/root/.ssh/;scp/root/.ssh/authorized_keyskingo-d2:/root/.ssh/;scp/root/.ssh/authorized_keyskingo-d3:/root/.ssh/五、安装JDK把jdk-7u80-linux-x64.rpm安装包复制到/bigdata/setup文件夹下,执行:#rpm–ivhjdk-7u80-linux-x64.rpmJdk安装到默认目录/usr/java/jdk1.7.0_80下。#vi/etc/profile加入如下配置信息:exportJAVA_HOME=/usr/java/jdk1.7.0_80exportCLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jarexportPATH=.:$JAVA_HOME/bin:$PATH#sourceprofile//执行profile内容,确认是否配置成功#java–version//查看是否JDK安装成功六、设置hadoop环境变量#vi/etc/profileexportHADOOP_HOME=/home/bigdata/hadoop-2.7.1exportPATH=.:$HADOOPHOME/sbin:$HADOOP_HOME/bin:$JAVAHOME/bin:$PATHexportHADOOP_LOG_DIR=/home/bigdata/hadoop-2.7.1/logsexportYARN_LOG_DIR=$HADOOP_LOG_DIRexportHADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexportHADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib#hadoopversion//查看hadoop是否安装成功七、配置hadoop1、配置hadoop2.7.1/etc/hadoop下的hadoop-env.sh、yarn-env.sh、mapred-env.sh修改JAVA_HOME值:exportJAVA_HOME=/usr/java/jdk1.7.0_802、配置hadoop2.7.1/etc/hadoop/slaveskingo-d1kingo-d2kingo-d33、配置hadoop2.7.1/etc/hadoop/core-site.xmlconfigurationpropertynamefs.defaultFS/namevaluehdfs://kingo-mn:9000//valuedescription设定namenode的主机名及端口/description/propertypropertynamehadoop.tmp.dir/namevalue/home/bigdata/tmp/hadoop-${user.name}/valuedescription存放临时文件的目录/description/propertypropertynamehadoop.proxyuser.hadoop.hosts/namevalue*/value/propertypropertynamehadoop.proxyuser.hadoop.groups/namevalue*/value/property/configuration4、配置hadoop2.7.1/etc/hadoop/hdfs-site.xmlconfigurationpropertynamedfs.namenode.http-address/namevaluekingo-mn:50070/valuedescriptionNameNode获得fsimage和edits/description/propertypropertynamedfs.namenode.secondary.http-address/namevaluekingo-ms:50090/valuedescriptionSecondNameNode获得最新的fsimage/description/propertypropertynamedfs.namenode.rpc-address/namevaluekingo-mn:9000/valuedescriptionkingo-mn/description/propertypropertynamedfs.replication/namevalue3/valuedescription设定HDFS存储文件的副本个数,默认为3/description/propertypropertynamedfs.namenode.name.dir/namevaluefile:///home/bigdata/hadoop-2.7.1/hdfs/name/valuedescriptionnamenode用来持续存放命名空间和交换日志的本地文件系统路径/description/propertypropertynamedfs.datanode.data.dir/namevaluefile:///home/bigdata/hadoop-2.7.1/hdfs/data/valuedescriptionDataNode在本地存放块文件的目录列表,用逗号分隔/description/propertypropertynamedfs.namenode.checkpoint.dir/namevaluefile:///home/bigdata/hadoop-2.7.1/hdfs/namesecondary/valuedescription设置secondary存放临时镜像的本地文件系统路径,如果这是一个用逗号分隔的文件列表,则镜像将会冗余复制到所有目录,只对secondary有效/description/propertypropertynamedfs.webhdfs.enabled/namevaluetrue/value/propertypropertynamedfs.stream-buffer-size/namevalue131072/valuedescription默认是4KB,作为hadoop缓冲区,用于hadoop读hdfs的文件和写hdfs的文件,还有map的输出都用到了这个缓冲区容量,对于现在的硬件很保守,可以设置为128k(131072),甚至是1M(太大了map和reduce任务可能会内存溢出)/description/propertypropertynamedfs.namenode.checkpoint.period/namevalue3600/valuedescription两次checkpoints之间的间隔,单位为秒,只对secondary有效/description/property/configurationpropertynamedfs.permissions.enabled/namevaluefalse/valuedescriptionIftrue,enablepermissioncheckinginHDFS.Iffalse,permissioncheckingisturnedoff,butallotherbehaviorisunchanged.Switchingfromoneparametervaluetotheotherdoes

1 / 25
下载文档,编辑使用

©2015-2020 m.777doc.com 三七文档.

备案号:鲁ICP备2024069028号-1 客服联系 QQ:2149211541

×
保存成功