Some usefull commands of hadoop file system

Here is the list of some commands other than basic commands like ls,copyFromLocal,put which you can use for getting various details of your HDFS files and directories

1). hadoop fs  -df [-h] [<path> …] :
Shows the capacity, free and used space of the filesystem. If the filesystem has
multiple partitions, and no path to a particular partition is specified, then
the status of the root partitions will be shown.

-h Formats the sizes of files in a human-readable fashion rather than a number
of bytes.

Ex:

 

2). hadoop fs -du [-s] [-h] <path> … :
Show the amount of space, in bytes, used by the files that match the specified
file pattern. The following flags are optional:

-s Rather than showing the size of each individual file that matches the
pattern, shows the total (summary) size.
-h Formats the sizes of files in a human-readable fashion rather than a number
of bytes.

Note that, even without the -s option, this only shows size summaries one level
deep into a directory.

 

 

3). hadoop fs -count [-q] [-h] <path> … :
Count the number of directories, files and bytes under the paths
that match the specified file pattern. The output columns are:
DIR_COUNT FILE_COUNT CONTENT_SIZE FILE_NAME or
QUOTA REMAINING_QUOTA SPACE_QUOTA REMAINING_SPACE_QUOTA
DIR_COUNT FILE_COUNT CONTENT_SIZE FILE_NAME
The -h option shows file sizes in human readable format.

4). hadoop fs -find <path> … <expression> … :
Finds all files that match the specified expression and
applies selected actions to them. If no <path> is specified
then defaults to the current working directory. If no
expression is specified then defaults to -print.

The following primary expressions are recognised:
-name pattern
-iname pattern
Evaluates as true if the basename of the file matches the
pattern using standard file system globbing.
If -iname is used then the match is case insensitive.

 

5). hadoop fs -test -[defsz] <path> :
Answer various questions about <path>, with result via exit status.
-d return 0 if <path> is a directory.
-e return 0 if <path> exists.
-f return 0 if <path> is a file.
-s return 0 if file <path> is greater than zero bytes in size.

-z return 0 if file <path> is zero bytes in size, else return 1.

 

6). hadoop fs -stat [format] <path> … :
Print statistics about the file/directory at <path> in the specified format.
Format accepts filesize in blocks (%b), group name of owner(%g), filename (%n),
block size (%o), replication (%r), user name of owner(%u), modification date
(%y, %Y)

-skipTrash option bypasses trash, if enabled, and immediately deletes <src>
-f If the file does not exist, do not display a diagnostic message or
modify the exit status to reflect an error.
-[rR] Recursively deletes directories


7).$ sudo -u hdfs hadoop dfsadmin -report
It provides complete installation details of your cluster

Leave a Reply

Your email address will not be published. Required fields are marked *