java连接hdfs ha和调用mapreduce jar示例
时间:2021-08-27 09:59:19|栏目:JAVA代码|点击: 次
Java API 连接 HDFS HA
复制代码 代码如下:
public static void main(String[] args) {
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://hadoop2cluster");
conf.set("dfs.nameservices", "hadoop2cluster");
conf.set("dfs.ha.namenodes.hadoop2cluster", "nn1,nn2");
conf.set("dfs.namenode.rpc-address.hadoop2cluster.nn1", "10.0.1.165:8020");
conf.set("dfs.namenode.rpc-address.hadoop2cluster.nn2", "10.0.1.166:8020");
conf.set("dfs.client.failover.proxy.provider.hadoop2cluster", "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
FileSystem fs = null;
try {
fs = FileSystem.get(conf);
FileStatus[] list = fs.listStatus(new Path("/"));
for (FileStatus file : list) {
System.out.println(file.getPath().getName());
}
} catch (IOException e) {
e.printStackTrace();
} finally{
try {
fs.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Java API调用MapReduce程序
复制代码 代码如下:
String[] args = new String[24];
args[0] = “/usr/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar";
args[1] = "wordcount";
args[2] = "-D";
args[3] = "yarn.resourcemanager.address=10.0.1.165:8032";
args[4] = "-D";
args[5] = "yarn.resourcemanager.scheduler.address=10.0.1.165:8030";
args[6] = "-D";
args[7] = "fs.defaultFS=hdfs://hadoop2cluster/";
args[8] = "-D";
args[9] = "dfs.nameservices=hadoop2cluster";
args[10] = "-D";
args[11] = "dfs.ha.namenodes.hadoop2cluster=nn1,nn2";
args[12] = "-D";
args[13] = "dfs.namenode.rpc-address.hadoop2cluster.nn1=10.0.1.165:8020";
args[14] = "-D";
args[15] = "dfs.namenode.rpc-address.hadoop2cluster.nn2=10.0.1.166:8020";
args[16] = "-D";
args[17] = "dfs.client.failover.proxy.provider.hadoop2cluster=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider";
args[18] = "-D";
args[19] = "fs.hdfs.impl=org.apache.hadoop.hdfs.DistributedFileSystem";
args[20] = "-D";
args[21] = "mapreduce.framework.name=yarn";
args[22] = "/input";
args[23] = "/out01";
RunJar.main(args);


阅读排行
- 1Java Swing组件BoxLayout布局用法示例
- 2java中-jar 与nohup的对比
- 3Java邮件发送程序(可以同时发给多个地址、可以带附件)
- 4Caused by: java.lang.ClassNotFoundException: org.objectweb.asm.Type异常
- 5Java中自定义异常详解及实例代码
- 6深入理解Java中的克隆
- 7java读取excel文件的两种方法
- 8解析SpringSecurity+JWT认证流程实现
- 9spring boot里增加表单验证hibernate-validator并在freemarker模板里显示错误信息(推荐)
- 10深入解析java虚拟机




