HDFS Java Client实现目录与文件的相关功能(HDFS java client implements the functions related to directories and files)

HDFS文件系统的相关操作

  • 创建目录;
  • 上传本地文件至指定目录;
  • 查看指定目录下的文件列表;
  • 读取指定文件的内容并在终端显示;
  • 下载指定文件;
  • 指定目录下完成指定文件类型的多个文件合并;
  • 文件的移动;
  • 文件的删除;

代码示例

  • 文件的上传:
        String source = "/my/local/store/aa.mp4";
        String destination = "hdfs://122.51.241.109:9000/data/hdfs01.mp4";
        InputStream in;
        try {
            in = new BufferedInputStream(new FileInputStream(source));
            //HDFS读写的配置文件
            Configuration conf = new Configuration();
            //生成一个文件系统对象
            FileSystem fs = FileSystem.get(URI.create(destination), conf);
            //生成一个输出流
            OutputStream out = fs.create(new Path(destination));
            IOUtils.copyBytes(in, out, 4096, true);
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }    
  • 文件的下载:
      try {
            String srcFile = "hdfs://122.51.241.109:9000/data/hdfs01.mp4";
            Configuration conf = new Configuration();

            FileSystem fs = FileSystem.get(URI.create(srcFile), conf);
            FSDataInputStream hdfsInStream = fs.open(new Path(srcFile));

            String storePath = "/my/local/store/";
            Path hdfsPath = new Path(storePath);
            if (fs.mkdirs(hdfsPath)) {
                BufferedOutputStream outputStream = new BufferedOutputStream(new FileOutputStream(storePath + "hdfs01" +
                        ".mp4"));
                IOUtils.copyBytes(hdfsInStream, outputStream, 4096, true);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }

参考文献:

https://blog.csdn.net/weixin_43042683/article/details/120943721

————————

HDFS file system related operations

  • Create a directory;
  • Upload local files to the specified directory;
  • View the list of files in the specified directory;
  • Read the contents of the specified file and display it on the terminal;
  • Download the specified file;
  • Merge multiple files of the specified file type in the specified directory;
  • Movement of files;
  • Deletion of documents;

Code example

  • File upload:
        String source = "/my/local/store/aa.mp4";
        String destination = "hdfs://122.51.241.109:9000/data/hdfs01.mp4";
        InputStream in;
        try {
            in = new BufferedInputStream(new FileInputStream(source));
            //HDFS读写的配置文件
            Configuration conf = new Configuration();
            //生成一个文件系统对象
            FileSystem fs = FileSystem.get(URI.create(destination), conf);
            //生成一个输出流
            OutputStream out = fs.create(new Path(destination));
            IOUtils.copyBytes(in, out, 4096, true);
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }    
  • File download:
      try {
            String srcFile = "hdfs://122.51.241.109:9000/data/hdfs01.mp4";
            Configuration conf = new Configuration();

            FileSystem fs = FileSystem.get(URI.create(srcFile), conf);
            FSDataInputStream hdfsInStream = fs.open(new Path(srcFile));

            String storePath = "/my/local/store/";
            Path hdfsPath = new Path(storePath);
            if (fs.mkdirs(hdfsPath)) {
                BufferedOutputStream outputStream = new BufferedOutputStream(new FileOutputStream(storePath + "hdfs01" +
                        ".mp4"));
                IOUtils.copyBytes(hdfsInStream, outputStream, 4096, true);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }

reference:

https://blog.csdn.net/weixin_43042683/article/details/120943721