Some More Unix Stuff!!!

UNIX commands & useful links

DNS (Domain Name Service)
host string	Perform forward or reverse lookup on string
dig @nameserver string	Lookup string's host info from nameserver
System Hardware Information
grep -i memtotal /proc/meminfo	Find total amount of RAM in the system
dmidecode -q	Display DMI/SMBIOS information
cat /proc/cpuinfo	Display CPU information
egrep '(vmx|svm)' /proc/cpuinfo	See if a processor supports hardware virtualization
lspci -tv	Display PCI information
lsusb -tv	Display USB information
hdparm -i /dev/sda	Display disk information for sda
Quick HTML Editing Site Wide
find . -type f -print -exec sed -i -e 's|X|Y|g' {} \;	Replace all X's with Y's in all files this directory and below.
perl -pi -w -e 's/X/Y/g;' *.txt	Replace all X's with Y's in all files ending with *.txt
find -name '*' | xargs grep 'string'	Print filename and all lines containing 'string'
SSH (Secure Shell)
scp -p -r [USER]@[HOST]:file /dir	Recursively copy while preserving permissions
ssh-copy-id [USER]@[HOST]	Install public key for trusted login on HOST
Troubleshooting
ps -auxf	Full process list with Child Process Tree
netstat -tup	List all active network connections and state
lsof -i :80	Show information related to http (port 80) connections
lsof -u userid	List open files by a particular userid
iftop	List active network connections consuming bandwidth
iptraf	Monitor IP Traffic
htop	Like top, but with an h
mtr	My Traceroute - An improved traceroute
multitail	Tail mutiple files at the same time
siege	A http/web stress test utility
Samba / Windows File Sharing
smbtree	Scan local network for windows shares
Isolate DoS (Denial of Service) Attacks with netstat
netstat -an | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort –nr
netstat -anp | grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort –nr
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort –nr
netstat -ntu | grep -v TIME_WAIT | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort –nr
Bit Torrent Clients
Rtorrent	Bit Torrent Client
aria2	Bit Torrent Client

http://www.nivindel.com/blog/54-unix-advanced-command-reference.html

An interview question in unix, which is asked by most of the companies

Well, i was asked a question(which was last one) write a shell script which renames all *.txt files in a directory to *.sh ?

Immediately, mv command came into my mind as it does in many peoples' mind.Then i wrote like:

find . -name "*.txt" -exec mv *.txt *.sh {}\; 

however this was wrong. He told me to explain how the above would work and i realized this was not correct.

then i tried something else like:

find . -name "*.txt" -exec ls {}\; | perl -e '`rename *.txt *.sh`'

I wasn't pretty much sure that this would work.He told thats fine if i could  try without perl.

I asked for some hints and he reminded me of xargs & sh.Then i tried like:

find . -name "*.txt" -exec ls {}\;| xargs -n1 sh -c 'mv $1 `basename $1 .txt`.sh' -

and then i got selected in the unix rounds... :-) :-) 

xargs is really a great unix command like sed & awk.It is quite often used for parallel processing.Never miss this if you  like to play around with shell scripting!

Usage of unix tools Find,Sed and awk

Sed, Awk and Find are the excellent unix tools.Following are few problem statements where the use of these tools made the work easy.Have a glimpse if these are useful to you as well.

1.       Dump only CPU usage and process of the server to a file.

      Top –n 1 –d  |  sed ‘1,6d’ |   awk ‘{print  $8 $NF }’ > dump.txt

2.       Find a pattern “EMC2” which exists in multiple files under a current directory and replace the pattern with “CISCO” in all the files.

      Find .  –type f    | xargs sed –i ‘s/EMC2/CISCO/g’

      OR find  . –type f   -exec sed –i ‘s/EMC2/CISCO/g’ {} \;

3.       Calculate the number of lines between a line containing pattern “XYZ” & a line containing pattern “ABC” in a file.

Sed –n ‘/XYZ/,/ABC/p’ | wc –l

4.       List the top 3 largest files in a sub-directory.

      find  /root/dir/  -type f -exec ls -las {} \; | sort -n -r | head -3

5.       Calculate the number of Normal users created in a unix server         

       last | awk –F”:” ‘$4  > 1000 {print}’| wc -l
 
6.  Replace "root" by "admin" and "user" by "superuser" in a file only from line number 10 to the end of file.
    sed -i '10,/^END/{ s/root/admin/g  s/user/superuser/g  p}' filename
 
7. To extract data from beginning to every fourth line:
sed -n '1~4P'  

Search for those files and directories that occupies maximum space under a directory.

du - akd /your_directory_path | sort -nr | more

Change owner and group for all files and directories in a directory sub-tree to owner "parvez" and group "users."

    $ find /subtree/path -exec chown parvez:users {} \;

To find a pattern in a file inside a directory tree

find . -type f -print | xargs grep "pattern" {} \;

Issue a  telnet session in a shell script (Useful in automating of changing password to a number of servers or running the same script on a number of servers)

#!/bin/csh

setenv PASSWD `cat ~/.passwd`

foreach server_name (`cat hostlist`)

echo "connecting to server $server_name"

(sleep 5;echo $USERID;\

sleep 5;echo $PASSWD;\

sleep 2;echo passwd;\

sleep 2;echo $OLDPASSWD;\

sleep 2;echo $NEWPASSWD;\

sleep 2;echo $NEWPASSWD;\

sleep 2;exit;) | telnet $server_name | tee -a $server_name.log

end

Issue a ftp session in a shell script:

ftp -nv $SERVERNAME <> ftpout_file

user $USERID $PASSWD

mkdir /home/$SERVERNAME/$USERID/configure_web

cd /home/$SERVERNAME/$USERID/configure_web

lcd $HOMEDIR/webfile

ascii

put configure_web.sh

put p3p_plugin.bak

binary

put link_logo_bl.gif

quit

EOF

Editing file through shell script:

# Test_file.txt

This is my path /www/machine/instance/config/doc

It is very cold out here.

I am going to change this file with a new line

a.       I want to change /www/machine/instance/config/doc to /var/opt/mbs.

Add the following lines in a script:

vi test_file.txt <

:%s!/www/machine/instance/config/doc!/var/opt/mbs!g

:wq

EOF

Alternatively you can put the above two paths in a variable also.

oldpath=/www/machine/instance/config/doc; export oldpath

Newpath=/var/opt/mbs; export newpath

vi test_file.txt <

:%s!$oldpath!$newpath!g

:wq

EOF

The new test_file.txt will look like:

# Test_file.txt

This is my path /var/opt/mbs

It is very cold out here.

I am going to change this file with a new line

b. I want to add a line ‘I ll put on a sweater.’ Below the line ‘It is very cold out here.’

Create a separate file:

# add_below.sed

/It is very cold out here./a\

I ll put on a sweater.

Put this line in a script.

sed -f add_below.sed  test_file.txt > test_file.txt_tmp && mv test_file.txt_tmp test_file.txt

The new test_file.txt will look like:

# Test_file.txt

This is my path /var/opt/mbs

It is very cold out here.

I ll put on a sweater.

I am going to change this file with a new line

c.       I want to add a line ‘Let us have coffee.’ Above the line ‘It is very cold out here.’

Create a separate file:

# add_above.sed

/It is very cold out here./i\

Let us have coffee.

Put this line in a script.

sed -f add_above.sed  test_file.txt > test_file.txt_tmp && mv test_file.txt_tmp test_file.txt

The new test_file.txt will look like:

# Test_file.txt

This is my path /var/opt/mbs

Let us have coffee

It is very cold out here.

I ll put on a sweater.

I am going to change this file with a new line

d. I want to change a line ‘Let us have coffee.’ to ‘Can we have tea.’

Create a separate file:

# change_line.sed

/Let us have coffee./c\

Can we have tea.

Put this line in a script.

sed -f change_line.sed  test_file.txt > test_file.txt_tmp && mv test_file.txt_tmp test_file.txt

The new test_file.txt will look like:

# Test_file.txt

This is my path /var/opt/mbs

Can we have tea

It is very cold out here.

I ll put on a sweater.

I am going to change this file with a new line

e. Replace a block of text with new text.

I want to replace ‘am going to change this file’ with ‘have to replace a part of the sentence’ in the line ‘I am going to change this file with a new line’

Create a saperate file:

# replace_text.sed

s|\(am\).*\(file\)|\1 have to replace a part of the sentence \2|g

Put this line in a script.

sed -f replace_text.sed  test_file.txt > test_file.txt_tmp && mv test_file.txt_tmp test_file.txt

The new test_file.txt will look like:

# Test_file.txt

This is my path /var/opt/mbs

Can we have tea

It is very cold out here.

I ll put on a sweater.

I have to replace a part of the sentence with a new line

                                               

NOTE: While creating the XXXX.sed file add a blank extra line at the end of the file.

Shell and Command Questions

In this section, I will cover some of the common questions that arise in regard to the shell itself. Also included are a few questions regarding the execution of commands.

Why does #!/bin/sh have to be the first line of my scripts?
In Chapter 2, “Script Basics,” I stated that #!/bin/sh must be the first line in your script to ensure that the correct shell is used to execute your script. This line must be the first line in your shell script because of the underlying mechanism used by a shell to execute commands.

---

When you ask a shell to execute the command $ date,the shell uses the system call exec to ask the UNIX kernel to execute the command you requested. For those readers who are not familiar with the term system call, a system call is a C language function built

---

The shell passes the name of the command that should be executed to the exec system call. This system call reads the first two characters in a file to determine how to execute the command. In the case of shell scripts, the first two characters are #!, indicating that the script needs to be interpreted by another program instead of executed directly. The rest of the line is treated as the name of the interpreter to use.
Usually, the interpreter is /bin/sh, but you can also specify options to the shell on this line. Sometimes options such as -x or -nv are specified to enable debugging. This also enables you to write scripts tuned for a particular shell such as ksh or bash by using /bin/ksh or /bin/bash instead of /bin/sh.

How can I access the name of the current shell in my initialization scripts?
In your shell initialization scripts, the name of the current shell is stored in the variable $0.
Users who have a single .profile that is shared by sh, ksh, and bash use this variable in conjunction with a case statement near the end of this file to execute additional shell specific startup.
For example, I use the following case statement near the end of my .profile to set up the prompt, PS1, differently depending on the shell I am using:

case "$0" in

    *bash) PS1="\t \h \#$ " ;;

    *ksh) PS1="`uname -n` !$ " ;;

    *sh) PS1="`uname -n`$ " ;;

esac

export PS1

I have specified the shells as *bash, *ksh, and *sh, because some versions of UNIX place the - character in front of login shells, but not in front of other shells.

How do I tell whether the current shell is interactive or noninteractive?
Some scripts will need the capability to determine whether they are running in an interactive shell or noninteractive shell.
Usually, this is restricted to your shell initialization scripts because you don’t want to perform a full-blown initialization every time these scripts execute. Some other examples include scripts that can run from the at or cron commands.
Two common methods can determine whether a shell is interactive:

•  test -t or [ -t ]

•  tty -s

Both commands exit with zero status if STDIN is connected to a terminal. For example, the commands

$ if [ -t ] ; then echo interactive ; fi

and

$ if tty -s ; then echo interactive ; fi

produce the same result if the current shell is interactive:

interactive

On modern versions of UNIX both forms work equally well. On some older versions of UNIX the test -t command was not available, so the tty -s command had to be used.

How do I discard the output of a command?
Sometimes you will need to execute a command, but you don’t want the output displayed on the screen. In these cases you can discard the output by redirecting it to the file /dev/null:

command > /dev/null

Here command is the name of the command you want to execute. The file is a special file (called the bit bucket) that automatically discards all its input.
To discard both output of a command and its error output, use standard redirection to redirect STDERR to STDOUT:

command > /dev/null 2>&1

How can I display messages on STDERR?
You can display a message on to STDERR by redirecting STDIN into STDERR as follows:

echo message 1>&2

Here message is the message you want to display.
If you are interested in shell functions that perform additional formatting please consult Chapter 21, “Problem Solving with Functions,” which covers several shell functions that display messages on to STDERR.

How can I check whether a command was successful?
A command is successful if it exits with a status of zero. A nonzero exit code indicates that an error has occurred.
To check the exit code of the most recent command you executed, use the variable $?. For example:

grep root /etc/passwd > /dev/null 2>&1

if [ $? -ne 0 ] ; then echo "No one is in charge!" ; fi

Here you execute a grep command and then check the exit status of this command using the value stored in $?.

How do I determine whether the shell can find a particular command?
You can check to make sure that the shell can find a command or shell function by using the type command covered in Chapter 18, “Miscellaneous Tools”:

type name > /dev/null 2>&1 ; if [ $? -ne 0 ] ; then list ; fi

Here name is the name of the command you want check for, and list is the list of commands to execute if the shell does not know about name. Usually list is used to determine a fallback command.
The type command is a builtin in sh and bash. In ksh, it is usually an alias, whence -v.

How do I determine whether job control is available in the shell?
Job control, covered in Chapter 6, “Processes,” is the shell feature that enables you to control background processes based on a job ID. This feature is not available in the Bourne shell, sh. It is available in ksh and bash.
A common method used to check whether job control is enabled is to check whether the jobs command is defined:

if type jobs > /dev/null 2>&1 ; then

    echo "We have job control"

fi

This check is effective in most cases because the jobs command is not available in most versions of the Bourne shell.
Unfortunately, some versions of UNIX such as Sun Solaris, include a version of the Bourne shell that has a built-in command called jobs. On these systems when the shell is invoked as /bin/sh, the jobs command exists but does nothing. If the shell is invoked as /bin/jsh (as in job control shell), the jobs command behaves normally.

Variable and Argument Questions

In this section, I will examine some questions related to variables and their use in shell scripts. I will also cover some questions related to command line arguments.

How can I include functions and variable definitions from one file into another file?
To include functions and variable definitions defined in one file into another file you need to use the. command as follows:

. file

Here file is the name of the file you want to include. I covered this topic in Chapter 22, “Problem Solving with Shell Scripts.”

Is it possible to consider each argument to a shell script one at a time?
You can do this using a for loop:

for arg in "$@"

do

    list

done

Here the variable arg will be set to each argument in turn. The specified list of commands, list, will be executed for each argument.
You use $@ in this example for the arguments instead of $*, because $@ preserves the quoting used when the command was issued. The difference between $@ and $* was discussed in Chapter 12, “Parameters.”

How can I forward all the arguments given to my script to another command?
A common task for shell programmers is writing a wrapper script for command. A wrapper script might need to define a set of variables or change the environment in some way before a particular command starts executing.
When writing wrapper scripts, you will need to forward all the arguments given to your script to a command. Usually, the following is sufficient:

command "$@"

Here command is the name of the command you want to execute.
The one problem with this is that if no arguments were specified to your script, some versions of the shell will expand “$@” to “”. If no arguments were specified, you want to execute command, not command “”.To avoid this problem, use the form:

command ${@:+"$@"}

Here you are using one of the forms of variable substitution discussed in Chapter 8, “Substitution.” In this case, you check to see whether the variable $@ has a value. If it does, you substitute the value “$@” for it. If your script was not given any command line arguments, $@ will be null; thus no value will be substituted.

How do I use the value of a shell variable in a sed command?
The simplest method to use variables in a sed command is to enclose your sed command in double quotes (“) instead of single quotes (‘). Because the shell performs variable substitution on double-quoted strings, the shell will substitute the value of any variables you specify before sed executes.
For example, the command

sed "/$DEL/d" file1 > file2

deletes all the lines in file1 that contain the value stored in the variable $DEL.

How do I check to see whether a variable has a value?
There are several methods for determining this. The simplest is the if statement:

if [ -z "$VAR" ] ; then list ; fi

Here VAR is the name of the variable, and list is the command to execute if VAR does not contain a value. Usually, list initializes VAR to some default value.
You can initialize variables more succinctly using variable substitution. For example, the previous if statement can be written as

: ${VAR:=default}

Here default is the default that should be assigned to VAR, if VAR does not have a value.
If you need to execute a set of commands to obtain a default value, use command substitution with the backquote (`) operator to obtain the value that should be substituted:

: ${VAR:=`default`}

Here default is a list of commands to execute. If VAR does not have a value, the output of these commands will be assigned to it.

File and Directory Questions

In this section, I will look at some questions about files and directories. These questions include issues with specific commands and examples that illustrate the usage of commands to solve particular problems. 

How do I determine the full pathname of a directory?
Shell scripts that work with directories often need to determine the full pathname of a directory to perform the correct operations on these directories.
You can determine the full pathname of a directory by using the cd and pwd commands:

FULLPATH=`(cd dir ; pwd)`

Here dir is the name of a directory. This command changes directories to the specified directory, dir, and then displays the full pathname of the directory using the pwd command. Because command substitution is used, the full pathname is assigned to the variable FULLPATH.
Because the cd command changes the working directory of the current shell, you execute it in a subshell. Thus the working directory of the shell script is unchanged.

How do I determine the full pathname of a file?
Determining the full pathname of a file is slightly harder than determining the full pathname of a directory. You need to use the dirname and basename commands in conjunction with the cd and pwd commands to determine the full pathname of a file:

CURDIR=`pwd`

cd `dirname file`

FULLPATH="`pwd`/`basename file`"

cd $CURDIR

Here file is the name of a file whose full pathname you want to determine. First, you save the current path of the current directory in the variable CURDIR. Next, you change to the directory containing the specified file, file.
Then you join the output of the pwd command and the name of the file determined using the basename command to get the full pathname. This value gets stored in the variable FULLPATH. Finally you change back to the original directory.

How can locate a particular file?
The structure of the UNIX directory tree sometimes makes locating files and commands difficult. To locate a file, often you need to search through a directory and all its subdirectories. The easiest way to do this is to use the find command:

find dir -name file -print

Here dir is the name of a directory where find should start its search, and file is the name of the file it should look for.
The name option of the find command also works with the standard filename substitution operators covered in Chapter 8. For example, the command

find dir -name "*txt" -print

displays a list of all the files in the directory dir and all its subdirectories that end with the string txt.

How can I grep for a string in every file in a directory?
When you work on a large project involving many files, remembering the contents of the individual files becomes difficult. It is much easier to look through all the files for a particular piece of information.
You can use the find command in conjunction with the xargs command to look for a particular string in every file contained within a directory and all its subdirectories:

find dir -type f -print | xargs grep "string"

Here dir is the name of a directory in which to start searching, and string is the string to look for. Here you specify the -type option to the find command so that only regular files are searched for the string.

How do I remove all the files in a directory matching a particular name?
Some editors and programs create large numbers of temporary files. Often you need to clean up after these programs, to prevent your hard drive from filling up.
To generate the list of files to delete, you can use the find command. Most of the time you can combine the find command with the xargs command, but in this case the filenames can contain one or more spaces. Spaces can confuse the xargs command, so you need to use a for loop instead:

OLDIFS="$IFS"

IFS='

'

for FILE in `find . -type f -name "*string*" -print`

do

    rm "$FILE"

done

IFS="$OLDIFS"

Here string is a string that should be part of the name of each file you want to delete. So that the for loop is set to the correct value of FILE in each iteration, IFS needs to be set to the newline character.
If you do not change IFS, filenames that contain spaces will be interpreted as multiple files instead of as a single file.

What command can I use to rename all the *.aaa files to *.bbb files?
In DOS and Windows, you can rename all the *.aaa files in a directory to *.bbb by using the rename command as follows:

rename *.aaa *.bbb

In UNIX you can use the mv command to rename files, but you cannot use it to rename more than one file at the same time. To do this, you need to use a for loop:

OLDSUFFIX=aaa

NEWSUFFIX=bbb

for FILE in *."$OLDSUFFIX"

do

    NEWNAME=`echo "$FILE" | sed -e "s/${OLDSUFFIX}\$/$NEWSUFFIX/"`

    mv "$FILE" "$NEWNAME"

done

Here you generate a list of all the files in the current directory that end with the value of the variable OLDSUFFIX. Then you use sed to modify the name of each file by removing the value of OLDSUFFIX from the filename and replacing it with the value of NEWSUFFIX. You use the $ character in your sed expression to anchor the suffix in OLDSUFFIX to the end of the line. You do this to make sure the pattern that is replaced is really a filename suffix.
After you have a new name, you rename the file from its original name, stored in FILE, to the new name stored, stored in NEWNAME.
To prevent a potential loss of data, you might need to modify this loop to specify the -i option to the mv command. For example, if the files 1.aaa and 1.bbb exist prior to executing this loop, after the loops exits, the original version of 1.aaa will be overwritten when 1.bbb is renamed as 1.aaa. If mv -i is used, you will be prompted before 1.bbb is renamed:

mv: overwrite 1.aaa (yes/no)?

You can answer no to avoid losing the information in this file. The actual prompt produced by mv might be different on your version of UNIX.

What command can I use to rename all the aaa* files to bbb* files?
The technique used in the last question can be used to solve this problem as well. In this case you will use the variables OLDPREFIX to hold the prefix a file currently has and NEWPREFIX to hold the prefix you want the file to have.
As an example, you can use the following for loop to rename all files that start with aaa to start with bbb instead:

OLDPREFIX=aaa

NEWPREFIX=bbb

for FILE in "$OLDPREFIX"*

do

    NEWNAME=`echo "$FILE" | sed -e "s/^${OLDPREFIX}/$NEWPREFIX/"`

    mv "$FILE" "$NEWNAME"

done

How can I set my filenames to lowercase?
When you transfer a file from a Windows or DOS system to a UNIX system, the filename ends up in all capital letters. You can rename these files to lowercase using the following command:

for FILE in *

do

    mv -i "$FILE" `echo "$FILE" | tr '[A-Z]' '[a-z]'` 2> /dev/null

done

You are using the mv -i command here in order to avoid overwriting files. For example, if the files APPLE and apple both exist in a directory you do not want to rename the file APPLE.

How do I eliminate carriage returns (^M) in my files?
If you transfer text files from a DOS machine to a UNIX machine, you might see a ^M before the end of each line. This character corresponds to a carriage return.
In DOS a newline is represented by the character sequence \r\n, where \r is the carriage return and \n is newline. In UNIX a newline is represented by \n. When text files created on a DOS system are viewed on UNIX, the \r is displayed as ^M.
You can strip these carriage returns out by using the tr command as follows:

tr -d '\015' < file > newfile

Here file is the name of the file that contains the carriage returns, and newfile is the name you want to give the file after the carriage returns have been deleted.
Here you are using the octal representation \015 for carriage return because the escape sequence \r will not be correctly interpreted by all versions of tr.