Tuesday, 16 October 2012

Beginner's Introduction to Shell Script: PART 4

Beginner's Introduction to Shell Script: PART 1
Beginner's Introduction to Shell Script: PART 2
Beginner's Introduction to Shell Script: PART 3

Beginner's Introduction to Shell Script: PART 4
4. File and Archiving Commands

4.1. Archiving

Originally a Tape ARchiving program, 

2. ar
Creation and manipulation utility for archives, mainly used for binary object file libraries.

3. rpm

The Red Hat Package Manager,

  3.1 rpm -qf identifies which package a file originates from.

   [sankar@new-host ~]$ rpm -qf /bin/ls

   3.2  rpm -qa gives a complete list of all installed rpm packages on a given system.
         An rpm -qa package_name lists only the package(s) corresponding to package_name.
   [sankar@new-host ~]$ rpm -qa zip

4. cpio

This specialized archiving copy command (copinput and output) is rarely seen any more, having been supplanted by tar/gzip.

5. rpm2cpio

This command extracts a cpio archive from an rpm one.

6. pax
The pax portable archive exchange toolkit facilitates periodic file backups and is designed to be cross-compatible between various flavors of UNIX. It was designed to replace tar and cpio.

pax -wf daily_backup.pax ~/linux-server/files 
#  Creates a tar archive of all files in the target directory.
#  Note that the options to pax must be in the correct order --
#+ pax -fw     has an entirely different effect.

pax -f daily_backup.pax
#  Lists the files in the archive.

pax -rf daily_backup.pax ~/bsd-server/files
#  Restores the backed-up files from the Linux machine
#+ onto a BSD one.

4.2. Compression

1. gzip(gunzip)

The standard GNU/UNIX compression utility, replacing the inferior and proprietary compress.
          The zcat filter decompresses a gzipped file to stdout.The zcat command is equivalent to gzip -dc.

2. bzip2(bunzip2)
An alternate compression utility, usually more efficient (but slower) than gzip, especially on large files.
          Similar to the zcat command, bzcat decompresses a bzipped2-ed file to stdout.

4.3. File Information

1. file

A utility for identifying file types. The command file file-name 
          the magic numbers found in /usr/share/magic/etc/magic, or /usr/lib/magic, depending on the Linux/UNIX distribution.

The -f option causes file to run in batch mode, 

2. which

which command gives the full path to "command." This is useful for finding out whether a particular command or utility is installed on the system.

[sankar@new-host ~]$ which rm

3. whereis
Similar to which, above, whereis command gives the full path to "command," but also to its manpage.

[sankar@new-host ~]$ whereis rm
rm: /bin/rm /opt/sql:/bin/rm /usr/share/man/man1/rm.1.gz /usr/share/man/man1p/rm.1p.gz

4. whatis
whatis command looks up "command" in the whatis database. This is useful for identifying system commands and important configuration files. Consider it a simplified man command.

[sankar@new-host ~]$ whatis rm
rm                   (1p)  - remove directory entries
rm                   (1)  - remove files or directories

5. vdir
Show a detailed directory listing. The effect is similar to ls -lb.

[sankar@new-host myscript]$ vdir
total 8
drwxrwxr-x 3 sankar sankar 4096 Sep  4 11:11 perlscript
drwxrwxr-x 4 sankar sankar 4096 Oct 16 11:24 shell\ script

6. locateslocate
The locate command searches for files using a database stored for just that purpose. The slocate command is the secure version of locate.

find files by name.

       [sankar@new-host ~]$ locate perlscript

7. getfaclsetfacl
These commands retrieve or set the file access control list -- the ownergroup, and file permissions.

[sankar@new-host ~]$ getfacl *
# file: 1Licensing\040Details\040of\040DCI\040Chennai.xls
# owner: sankar
# group: sankar

# file: aaa
# owner: sankar
# group: sankar

[sankar@new-host ~]$ setfacl -m u:sankar:rw aaa     
# file: aaa
# owner: sankar
# group: sankar

8. readlink
Disclose the file that a symbolic link points to.

[sankar@new-host ~]$ readlink /usr/bin/awk

4.4. Comparison

1. diff
diff: flexible file comparison utility. It compares the target files line-by-line sequentially. 
          The diff command returns an exit status of 0 if the compared files are identical,

 [sankar@new-host shell script]$ cat file1.data
100 Shoes
200 Laces
300 Socks
[sankar@new-host shell script]$ cat file2.data
100 Shoes
100 $40.00
200 $1.00
300 $2.00

[sankar@new-host shell script]$ diff file1.data file2.data
< 200 Laces
< 300 Socks
> 100 $40.00
> 200 $1.00
> 300 $2.00

Note:Use zdiff to compare gzipped files.

2. sdiff
Compare and/or edit two files in order to merge them into an output file.

[sankar@new-host shell script]$ sdiff file1.data file2.data

100 Shoes                                                       100 Shoes

                                                              > 100 $40.00

                                                              > 200 $1.00

                                                              > 300 $2.00


                                                              > 100 Shoes

200 Laces                                                       200 Laces
300 Socks                                                       300 Socks

3. patch: flexible versioning utility

 patch -p1 <patch-file
# Takes all the changes listed in 'patch-file'
# and applies them to the files referenced therein.
# This upgrades to a newer version of the package.

4. cmp
The cmp command is a simpler version of diff, above. Whereas diff reports the differences between two files, cmp merely shows at what point they differ.
          Use zcmp on gzipped files.

5. comm
Versatile file comparison utility. The files must be sorted for this to be useful.
   $comm file-1 file-2 outputs three columns:

  • column 1 = lines unique to file-1
  • column 2 = lines unique to file-2
  • column 3 = lines common to both.

    [sankar@new-host shell script]$ cmp file1.data file2.data
    file1.data file2.data differ: byte 11, line 2

    This command is useful for comparing "dictionaries" or word lists -- sorted text files with one word per line.

    4.5. Utilities

    1. basename
    Strips the path information from a file name, printing only the file name

    [sankar@new-host shell script]$ basename /usr/bin/sort

    2. dirname

    Strips the basename from a filename, printing only the path information.
         [sankar@new-host shell script]$ dirname /usr/bin/sort

         3. splitcsplit
These are utilities for splitting a file into smaller chunks.
          The csplit command splits a file according to context, the split occuring where patterns are matched.
4.6. Encoding and Encryption

1. sumcksummd5sumsha1sum
These are utilities for generating checksums. A checksum is a number mathematically calculated from the contents of a file, for the purpose of checking its integrity.
For security applications, use the md5sum (message digest 5 checksum) command, or better yet, the newer sha1sum (Secure Hash Algorithm).

[sankar@new-host shell script]$ cksum /boot/vmlinuz-2.6.18-92.el5
1327335373 1806388 /boot/vmlinuz-2.6.18-92.el5

[sankar@new-host shell script]$ md5sum /boot/vmlinuz-2.6.18-92.el5
ce39b88a01d40745677be9ac7fb06da5  /boot/vmlinuz-2.6.18-92.el5

The cksum command shows the size, in bytes, of its target, whether file or stdout.
The md5sum and sha1sum commands display a dash when they receive their input from stdout.

bash$ md5sum testfile
e181e2c8720c60522c4c4c981108e367  testfile

bash$ sha1sum testfile
5d7425a9c08a66c3177f1e31286fa40986ffc996  testfile

2. uuencode
This utility encodes binary files (images, sound files, compressed files, etc.) into ASCII characters,

3. uuencode
This utility encodes binary files (images, sound files, compressed files, etc.) into ASCII characters,

4. openssl
This is an Open Source implementation of Secure Sockets Layer encryption.

5. shred
Securely erase a file by overwriting it multiple times with random bit patterns before deleting it.
          shred - overwrite a file to hide its contents, and optionally delete it

4.7. Miscellaneous

1. mktemp
Create a temporary file with a "unique" filename.

[sankar@new-host shell script]$ mktemp

2. make

Utility for building and compiling binary packages.

3. dos2unix
This utility, converts DOS-formatted text files  to UNIX format

5. Communications Commands

Information and Statistics

1. host
Searches for information about an Internet host by name or IP address, using DNS.

sankar@new-host tmp]$ host new-host
new-host.home has address

2. ipcalc
Displays IP information for a host. With the -h option, ipcalc does a reverse DNS lookup,

[sankar@new-host ~]$ ipcalc -h

3. nslookup
Do an Internet "name server lookup" on a host by IP address.

[sankar@new-host ~]$ nslookup

Non-authoritative answer:     name = NSG-Static-

Authoritative answers can be found from:
163.72.182.in-addr.arpa nameserver = bgpdnsdel.mantraonline.com.
163.72.182.in-addr.arpa nameserver = dnsblr.mantraonline.com.
163.72.182.in-addr.arpa nameserver = dnsbom.mantraonline.com.
163.72.182.in-addr.arpa nameserver = aaadel.mantraonline.com.
163.72.182.in-addr.arpa nameserver = dnsdel.mantraonline.com.
163.72.182.in-addr.arpa nameserver = dnsblr2.mantraonline.com.
aaadel.mantraonline.com internet address =
aaadel.mantraonline.com has AAAA address 2404:a800:0:b::9
dnsblr.mantraonline.com internet address =
dnsbom.mantraonline.com internet address =
dnsdel.mantraonline.com internet address =
dnsdel.mantraonline.com has AAAA address 2404:a800:0:b::7
dnsblr2.mantraonline.com        internet address =
bgpdnsdel.mantraonline.com      internet address =

4. dig
Domain Information Groper. Similar to nslookupdig does an Internet name server lookup on a host.

[sankar@new-host ~]$ dig -x

; <<>> DiG 9.3.4-P1 <<>> -x
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28297
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 8

;   IN      PTR

;; ANSWER SECTION: 86400 IN   PTR     NSG-Static-

163.72.182.in-addr.arpa. 86400  IN      NS      aaadel.mantraonline.com.
163.72.182.in-addr.arpa. 86400  IN      NS      bgpdnsdel.mantraonline.com.
163.72.182.in-addr.arpa. 86400  IN      NS      dnsdel.mantraonline.com.
163.72.182.in-addr.arpa. 86400  IN      NS      dnsbom.mantraonline.com.
163.72.182.in-addr.arpa. 86400  IN      NS      dnsblr.mantraonline.com.
163.72.182.in-addr.arpa. 86400  IN      NS      dnsblr2.mantraonline.com.

aaadel.mantraonline.com. 86400  IN      A
aaadel.mantraonline.com. 86400  IN      AAAA    2404:a800:0:b::9
dnsblr.mantraonline.com. 86400  IN      A
dnsbom.mantraonline.com. 86400  IN      A
dnsdel.mantraonline.com. 86400  IN      A
dnsdel.mantraonline.com. 86400  IN      AAAA    2404:a800:0:b::7
dnsblr2.mantraonline.com. 86400 IN      A
bgpdnsdel.mantraonline.com. 86400 IN    A

;; Query time: 1976 msec
;; WHEN: Tue Oct 16 16:30:25 2012
;; MSG SIZE  rcvd: 392

5. traceroute
Trace the route taken by packets sent to a remote host. This command works within a LAN, WAN, or over the Internet.

[sankar@new-host ~]$ traceroute
traceroute to (, 30 hops max, 40 byte packets
 1  Watson.home (  0.461 ms  0.540 ms  0.540 ms

6. ping
Broadcast an ICMP ECHO_REQUEST packet to another machine, either on a local or remote network. This is a diagnostic tool for testing network connections,

[sankar@new-host ~]$ ping localhost
PING localhost.localdomain ( 56(84) bytes of data.
64 bytes from localhost.localdomain ( icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from localhost.localdomain ( icmp_seq=2 ttl=64 time=0.056 ms
64 bytes from localhost.localdomain ( icmp_seq=3 ttl=64 time=0.055 ms
64 bytes from localhost.localdomain ( icmp_seq=4 ttl=64 time=0.056 ms

--- localhost.localdomain ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.043/0.052/0.056/0.009 ms

7. whois
Perform a DNS (Domain Name System) lookup. The -h option permits specifying which particular whois server to query.

[sankar@new-host ~]$ whois
[Querying whois.arin.net]
# Query terms are ambiguous.  The query is assumed to be:
#     "n"
# Use "?" to get help.

# The following results may also be obtained via:
# http://whois.arin.net/rest/nets;q=

NetRange: -

8. finger
Retrieve information about users on a network.

[sankar@new-host ~]$ finger
Login     Name       Tty      Idle  Login Time   Office     Office Phone
root      root       pts/1    6:58  Oct 16 09:42 (:1.0)
sankar              *:0             Oct 16 09:50
sankar              *pts/5          Oct 16 10:27 (:0)
xavier               pts/2    6:58  Oct 16 09:42 (:2.0)

9. chfn
Change information disclosed by the finger command.

10. vrfy
Verify an Internet e-mail address

6. Remote Host Access

1. sxrx
The sx and rx command set serves to transfer files to and from a remote host using the xmodem protocol.

[sankar@new-host ~]$ sx aaa
Give your local XMODEM receive command now.

2. uucpuuxcu
uucpUNIX to UNIX copy. This is a communications package for transferring files between UNIX servers.
         uuxUNIX to UNIX execute. Execute a command on a remote system. This command is part of the   uucp package.

         cuCall Up a remote system and connect as a simple terminal.

3. telnet
Utility and protocol for connecting to a remote host.

[sankar@new-host ~]$ telnet
telnet: connect to address Connection refused
telnet: Unable to connect to remote host: Connection refused

4. wget
The wget utility noninteractively retrieves or downloads files from a Web or ftp site.

5. lynx
The lynx Web and file browser can be used inside a script (with the -dump option) to retrieve a file from a Web or ftp site noninteractively.

6. rlogin
Remote login, initates a session on a remote host. This command has security issues, so use ssh instead.
7. rsh
Remote shell, executes command(s) on a remote host. This has security issues, so use ssh instead.
8. rcp
Remote copy, copies files between two different networked machines.
9. rsync
Remote synchronize, updates (synchronizes) files between two different networked machines.

10. ssh
Secure shell, logs onto a remote host and executes commands there. This secure replacement for telnetrloginrcp, and rsh uses identity authentication and encryption.

11. scp
Secure copy, similar in function to rcp, copies files between two different networked machines, but does so using authentication, and with a security level similar to ssh.

7. Local Network

1. write
This is a utility for terminal-to-terminal communication.

2. netconfig
A command-line utility for configuring a network adapter (using DHCP).

3. mail
Send or read e-mail messages.

[sankar@new-host ~]$ mail
No mail for sankar

4. mailto
Similar to the mail command, mailto sends e-mail messages from the command-line or in a script. However, mailto also permits sending MIME (multimedia) messages.

5. mailstats
Show mail statistics. This command may be invoked only by root.

root# mailstats
Statistics from Tue Jan  1 20:32:08 2008
  M   msgsfr  bytes_from   msgsto    bytes_to  msgsrej msgsdis msgsqur  Mailer
  4     1682      24118K        0          0K        0       0       0  esmtp
  9      212        640K     1894      25131K        0       0       0  local
  T     1894      24758K     1894      25131K        0       0       0
  C      414                    0

8. Terminal Control Commands

1. tput
Initialize terminal and/or fetch information about it from terminfo data.

[sankar@new-host ~]$ tput longname
xterm terminal emulator (X Window System)
Issuing a tput cup X Y moves the cursor to the (X,Y) coordinates in the current terminal.
Some interesting options to tput are:

  • bold, for high-intensity text
  • smul, to underline text in the terminal
  • smso, to render text in reverse
  • sgr0, to reset the terminal parameters (to normal), without clearing the screen

    2. infocmp

This command prints out extensive information about the current terminal. It references the terminfo database.

[sankar@new-host ~]$ infocmp
#       Reconstructed via infocmp from file: /usr/share/terminfo/x/xterm
xterm|xterm terminal emulator (X Window System),
        am, bce, km, mc5i, mir, msgr, npc, xenl,
        colors#8, cols#80, it#8, lines#24, pairs#64,
        bel=^G, blink=\E[5m, bold=\E[1m, cbt=\E[Z, civis=\E[?25l,
        clear=\E[H\E[2J, cnorm=\E[?12l\E[?25h, cr=^M,
        csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
        cud=\E[%p1%dB, cud1=^J, cuf=\E[%p1%dC, cuf1=\E[C,
        cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A,

3. reset
Reset terminal parameters and clear text screen.

4. clear
The clear command simply clears the text screen at the console or in an xterm.

5.  script
This utility records (saves to a file) all the user keystrokes at the command-line in a console or an xterm window.

6. resize
Echoes commands necessary to set $TERM and $TERMCAP to duplicate the size (dimensions) of the current terminal.

bash$ resize
set noglob;
 setenv COLUMNS '80';
 setenv LINES '24';
 unset noglob;

9. Math Commands

1. factor
          Decompose an integer into prime factors.

[sankar@new-host ~]$ factor 27417

27417: 3 13 19 37

2. bc
Bash can't handle floating point calculations,

[sankar@new-host ~]$ bc
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.

3. dc
The dc (desk calculator) utility is stack-oriented and uses RPN (Reverse Polish Notation). Like bc, it has much of the power of a programming language.

Miscellaneous Commands

1. run-parts
The run-parts command [1] executes all the scripts in a target directory, sequentially in ASCII-sorted filename order.

2. yes
In its default behavior the yes command feeds a continuous string of the character y followed by a line feed to stdout.
yes | fsck /dev/hda1 runs fsck non-interactively (careful!).
yes | rm -r dirname has same effect as rm -rf dirname (careful!).
[sankar@new-host ~]$ yes $BASH_VERSION

3. banner
Prints arguments as a large vertical banner to stdout,

4. printenv
Show all the environmental variables set for a particular user.

[sankar@new-host ~]$ printenv |grep HOME

5. lp
The lp and lpr commands send file(s) to the print queue, to be printed as hard copy

bash$ lp file1.txt or bash lp <file1.txt
It is often useful to pipe the formatted output from pr to lp.
bash$ pr -options file1.txt | lp
Formatting packages, such as groff and Ghostscript may send their output directly to lp.
bash$ groff -Tascii file.tr | lp
bash$ gs -options | lp file.ps
Related commands are lpq, for viewing the print queue, and lprm, for removing jobs from the print queue.

6. pathchk
This command checks the validity of a filename. If the filename exceeds the maximum allowable length (255 characters)

7. od
The od, or octal dump filter converts input (or files) to octal (base-8) or other bases.

8. hexdump
Performs a hexadecimal, octal, decimal, or ASCII dump of a binary file.

9. objdump
Displays information about an object file or binary executable in either hexadecimal form or as a disassembled listing (with the -d option).

bash$ objdump -d /bin/ls
/bin/ls:     file format elf32-i386

 Disassembly of section .init:

 080490bc <.init>:
  80490bc:       55                      push   %ebp
  80490bd:       89 e5                   mov    %esp,%ebp
  . . .

10. m4
A hidden treasure, m4 is a powerful macro processing filter, virtually a complete language.

11. xmessage
This X-based variant of echo pops up a message/query window on the desktop.

12. doexec
The doexec command enables passing an arbitrary list of arguments to a binary executable.

Beginner's Introduction to Shell Script: PART 5

No comments:

Post a Comment