搜索
一、Hadoop安装1、解压文件后的目录配置之前，先在本地文件系统创建以下文件夹：mkdir-p/home/hadoop/tmp/home/dfs/data/home/dfs/name4、进去hadoop配置文件目录[root@masterhadoop-2.6.0]#cdetc/hadoop/4.1、配置hadoop-env.sh文件--修改JAVA_HOME#Thejavaimplementationtouse.exportJAVA_HOME=/opt/jdk1.8.0_664.2、配置yarn-env.sh文件--修改JAVA_HOME#someJavaparametersexportJAVA_HOME=/opt/jdk1.8.0_664.3、配置slaves文件--增加slave节点slave1slave24.4、配置core-site.xml文件--增加hadoop核心配置（hdfs文件端口是9000、file:/home/spark/opt/hadoop-2.6.0/tmp、）configurationpropertynamefs.defaultFS/namevaluehdfs://master.hadoop:9000/value/propertypropertynameio.file.buffer.size/namevalue131072/value/propertypropertynamehadoop.tmp.dir/namevaluefile:/home/hadoop/tmp/valuedescriptionAbaseforothertemporarydirectories./description/propertypropertynamehadoop.proxyuser.root.hosts/namevalue*/value/propertypropertynamehadoop.proxyuser.root.groups/namevalue*/value/property!--指定zookeeper地址--propertynameha.zookeeper.quorum/namevaluemast1:2181,mast2:2181,mast3:2181/value/property/configuration4.5、配置hdfs-site.xml文件--增加hdfs配置信息（namenode、datanode端口和目录位置）configurationpropertynamedfs.namenode.secondary.http-address/namevaluemaster.hadoop:9001/value/propertypropertynamedfs.namenode.name.dir/namevaluefile:/home/dfs/name/value/propertypropertynamedfs.datanode.data.dir/namevaluefile:/home/dfs/data/value/propertypropertynamedfs.replication/namevalue3/value/propertypropertynamedfs.webhdfs.enabled/namevaluetrue/value/property!--开启NameNode故障时自动切换--propertynamedfs.ha.automatic-failover.enabled/namevaluetrue/value/property!--配置失败自动切换实现方式--propertynamedfs.client.failover.proxy.provider.ns/namevalueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value/property/configuration4.6、配置mapred-site.xml文件--增加mapreduce配置（使用yarn框架、jobhistory使用地址以及web地址）configurationpropertynamemapreduce.framework.name/namevalueyarn/value/propertypropertynamemapreduce.jobhistory.address/namevaluemaster.hadoop:10020/value/propertypropertynamemapreduce.jobhistory.webapp.address/namevaluemaster.hadoop:19888/value/property/configuration复制代码4.7、配置yarn-site.xml文件--增加yarn功能configurationpropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.nodemanager.aux-services.mapreduce.shuffle.class/namevalueorg.apache.hadoop.mapred.ShuffleHandler/value/propertypropertynameyarn.resourcemanager.address/namevaluemaster.hadoop:8032/value/propertypropertynameyarn.resourcemanager.scheduler.address/namevaluemaster.hadoop:8030/value/propertypropertynameyarn.resourcemanager.resource-tracker.address/namevaluemaster.hadoop:8035/value/propertypropertynameyarn.resourcemanager.admin.address/namevaluemaster.hadoop:8033/value/propertypropertynameyarn.resourcemanager.webapp.address/namevaluemaster.hadoop:8088/value/property/configuration5、将配置好的hadoop文件copy到另一台slave机器上$scp-rhadoop-2.6.0/root@slave1:/opt/四、验证1、格式化namenode:./bin/hdfsnamenode-format2、重新格式化hdfs系统的方法：删除namenode上存储hdfs名字空间元数据，datanode上数据块的物理存储位置，namenode上本地的hadoop临时文件夹重新执行命令：hadoopnamenode-format格式化完毕。注意：原来的数据全部被清空了。产生了一个新的hdfs。3、查看集群状态：jpsmaster中运行Slave中安装过程中如果重新格式化了hdfs要在slave中也删除对应的文件夹data\name,否则slave上的datanode无法启动。$./bin/hdfsdfsadmin-report-------------------------------------------------Livedatanodes(1):Name:10.126.45.56:50010(S1PA222)Hostname:S1PA209DecommissionStatus:NormalConfiguredCapacity:52101857280(48.52GB)DFSUsed:823296(804KB)NonDFSUsed:6352347136(5.92GB)DFSRemaining:45748686848(42.61GB)DFSUsed%:0.00%DFSRemaining%:87.81%ConfiguredCacheCapacity:0(0B)CacheUsed:0(0B)CacheRemaining:0(0B)CacheUsed%:100.00%CacheRemaining%:0.00%Xceivers:1Lastcontact:MonJan0516:44:50CST2015对应整个集群4解决本地库问题如果你是hadoop2.6的可以下载下面这个：下载完以后，解压到hadoop的native目录下，覆盖原有文件即可。操作如下：tar-xhadoop-native-64-2.4.0.tar-Chadoop/lib/native/7、查看hdfs：、查看RM：、运行wordcount程序9.1、创建input目录：[spark@S1PA11hadoop-2.6.0]$mkdirinput9.2、在input创建f1、f2并写内容[spark@S1PA11hadoop-2.6.0]$catinput/f1Helloworldbyejj[spark@S1PA11hadoop-2.6.0]$catinput/f2HelloHadoopbyeHadoop9.3、在hdfs创建/tmp/input目录[spark@S1PA11hadoop-2.6.0]$./bin/hadoopfs-mkdir/tmp15/01/0516:53:57WARNutil.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform...usingbuiltin-javaclasseswhereapplicable[spark@S1PA11hadoop-2.6.0]$./bin/hadoopfs-mkdir/tmp/input15/01/0516:54:16WARNutil.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform...usingbuiltin-javaclasseswhereapplicable9.4、将f1、f2文件copy到hdfs/tmp/input目录[spark@S1PA11hadoop-2.6.0]$./bin/hadoopfs-putinput//tmp15/01/0516:56:01WARNutil.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform...usingbuiltin-javaclasseswhereapplicable9.5、查看hdfs上是否有f1、f2文件[spark@S1PA11hadoop-2