import java.io.IOException; import java.net.URI; import java.net.URISyntaxException; import java.util.logging.Level; import java.util.logging.Logger; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hdfs.DistributedFileSystem; /** * * @author Shashwat Shriparv * @email dwivedishashwat@gmail.com * @web helpmetocode.blogspot.com */ public class RecursivelyPrintFilesOnHDFS { public static void main(String[] args) throws IOException, InterruptedException, URISyntaxException { printFilesRecursively("hdfs://master1:9000/"); } public static void printFilesRecursively(String Url) throws IOException { try {
All the question that scared me now i am trying to scare them .. so that they cant scare others :)
Saturday, April 13, 2013
List Files from hdfs/Hadoop Recursively using java
Write file to HDFS/Hadoop Read File From HDFS/Hadoop Using Java
import java.io.File; import java.io.IOException; import java.net.URI; import java.net.URISyntaxException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hdfs.DistributedFileSystem; /** * * @author Shashwat Shriparv * @email dwivedishashwat@gmail.com * @Web helpmetocode.blogspot.com */ public class WritetoHDFSReadFromHDFSWritToLocal { private static byte[] buffer; private static int bytesRead; public static void main(String[] args) throws IOException, InterruptedException, URISyntaxException {
Thursday, April 4, 2013
Insert string after each N lines in a file
We can do this as follows:
awk '1;!(NR%<Number after which the line has to be insserted>){print "String to be inserted";}' origionalfiletoprocess >outfilewithinsertedstring
Eg:
awk '1;!(NR%100){print "Shashwat Shriparv";}' filecontainingtxt>outputfilewithnewinsertedlines
This command will read filecontainingtxt and will insert string Shashwat Shriparv after 100 lines the the output fill will be outputfilewithnewinsertedlines
Tuesday, April 2, 2013
Subscribe to:
Posts (Atom)
Featured Posts
#Linux Commands Unveiled: #date, #uname, #hostname, #hostid, #arch, #nproc
#Linux Commands Unveiled: #date, #uname, #hostname, #hostid, #arch, #nproc Linux is an open-source operating system that is loved by millio...
-
Hadoop is a batch processing system, and Hadoop jobs tend to have high latency and incur substantial overhead in job submission and sched...
-
Print numbers in order : #!/bin/bash for i in $(seq 0 4) do for j in $(seq $i -1 0) do echo -n $j done echo done Will gi...