Saturday, April 13, 2013

List Files from hdfs/Hadoop Recursively using java


import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;

/**
 *
 * @author Shashwat Shriparv
 * @email  dwivedishashwat@gmail.com
 * @web    helpmetocode.blogspot.com
 */
public class RecursivelyPrintFilesOnHDFS {

    public static void main(String[] args) throws IOException, InterruptedException, URISyntaxException {
        printFilesRecursively("hdfs://master1:9000/");
    }

    public static void printFilesRecursively(String Url) throws IOException {
        try {

Write file to HDFS/Hadoop Read File From HDFS/Hadoop Using Java


import java.io.File;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;

/**
 *
 * @author    Shashwat Shriparv
 * @email     dwivedishashwat@gmail.com
 * @Web       helpmetocode.blogspot.com
 */
public class WritetoHDFSReadFromHDFSWritToLocal {
    private static byte[] buffer;
    private static int bytesRead;

    public static void main(String[] args) throws IOException, InterruptedException, URISyntaxException {
      

Thursday, April 4, 2013

Insert string after each N lines in a file

We can do this as follows:

awk '1;!(NR%<Number after which the line has to be insserted>){print "String to be inserted";}' origionalfiletoprocess >outfilewithinsertedstring

Eg:

awk '1;!(NR%100){print "Shashwat Shriparv";}' filecontainingtxt>outputfilewithnewinsertedlines

This command will read filecontainingtxt and will insert string Shashwat Shriparv after 100 lines the the output fill will be outputfilewithnewinsertedlines

Featured Posts

#Linux Commands Unveiled: #date, #uname, #hostname, #hostid, #arch, #nproc

 #Linux Commands Unveiled: #date, #uname, #hostname, #hostid, #arch, #nproc Linux is an open-source operating system that is loved by millio...