Thursday 5 January 2012

shell script basics

Note:

Well,
Bash is intepreted line by line as it runs and depends on calling a lot of external progs (depending on what you want to do).
You often have to use temp files as intermediate storage for result sets.
It (shell) was originally designed to talk to the system and automate cmd sequences (shell files).

Perl is more like C, it's largely self contained with a huge library of free code and it's compiled , so it runs much faster, eg about 80-90% speed of C, but easier to program (eg variable sizes are dynamic).
Syntax is similar to C. 

1)UNIX Commands:

Although a shell script can make use of any unix commands here are a number of commands which are more often used than others. These commands can generally be described as commands for file and text manipulation.

Command syntax
Purpose
echo "some text"
write some text on your screen
ls
list files
wc -l file
wc -w file
wc -c file
count lines in file or
count words in file or
count number of characters
cp sourcefile destfile
copy sourcefile to destfile
mv oldname newname
rename or move file
rm file
delete a file
grep 'pattern' file
search for strings in a file
Example: grep 'searchstring' file.txt
cut -b colnum file
get data out of fixed width columns of text
Example: get character positions 5 to 9
cut -b5-9 file.txt
Do not confuse this command with "cat" which is something totally different
cat file.txt
write file.txt to stdout (your screen)




file somefile
describe what type of file somefile is
read var
prompt the user for input and write it into a variable (var)
sort file.txt
sort lines in file.txt
uniq
remove duplicate lines, used in combination with sort   since uniq removes only duplicated consecutive lines
Example: sort file.txt | uniq
expr
do math in the shell
Example: add 2 and 3
expr 2 "+" 3
find
search for files
Example: search by name:
find . -name filename -print
This command has many different possibilities and options. It is unfortunately too much to explain it all in this article.
tee
write data to stdout (your screen) and to a file
Normally used like this:
somecommand | tee outfile
It writes the output of somecommand to the screen and to the file outfile
basename file
return just the file name of a given name and strip the directory path
Example: basename /bin/tux
returns just tux
dirname file
return just the directory name of a given name and strip the actual file name
Example: dirname /bin/tux
returns just /bin
head file
print some lines from the beginning of a file
tail file
print some lines from the end of a file
sed
sed is basically a find and replace program. It reads text from standard input (e.g from a pipe) and writes the result to stdout (normally the screen). The search pattern is a regular expression (see references). This search pattern should not be confused with shell wildcard syntax. To replace the string linuxfocus with LinuxFocus in a text file use:
cat text.file | sed 's/linuxfocus/LinuxFocus/' > newtext.file
This replaces the first occurance of the string linuxfocus in each line with LinuxFocus. If there are lines where linuxfocus appears several times and you   want to replace all use:
cat text.file | sed 's/linuxfocus/LinuxFocus/g' > newtext.file
awk
Most of the time awk is used to extract fields from a text line. The default field separator is space. To specify a different one use the option -F.
 cat file.txt | awk -F, '{print $1 "," $3 }'
 
Here we use the comma (,) as field separator and print the first and third ($1 $3) columns. If file.txt has lines like:
Adam Bor, 34, India
Kerry Miller, 22, USA
 
then this will produce:
Adam Bor, India
Kerry Miller, USA
 
There is much more you can do with awk but this is a very common use.

Simple Stuff:

First, a convention. I'll list things for you to type in this format:
$ date
I will list the computer's reply like this:
Tue Dec 23 10:52:51 PST 2003

To put this another way, enter this example:
# whoami
root

Where are you?

As you may be aware, a Linux filesystem is in the form of a large tree with many branches called "subdirectories". When you issue a shell command,
$ pwd
/path/path/path

You can decide where you are in the tree. Type this example:
$ cd ~
$ pwd
/home/username


Listing Files:

Directories contain files, and you can list them in a simple, compact format:
$ ls
filename filename filename ...

Or you can list them in more detail:
$ ls -la
(detailed list, one file per line)

And, very important, to find out what a command's options are, use the "man" (manual) command:
$ man ls
(manual page for "ls")


To find files by name:
$ find . -name '*.jpg'
(list of files with .jpg suffix in current and all child directories)

To create a text diagram of the directory tree:
$ tree -d .
(diagram of the directory tree from the current directory)


Examining Files:

There are a number of things you can do to find out more about the files in the list. Here are just a few:
The "file" command tries to identify files by examining their contents:

$ file tux_small.png
tux_small.png: PNG image data, 128 x 151, 8-bit/color RGB, non-interlaced


The next example uses the obscurely named "cat" command. It prints the contents of a file. Unfortunately if the file's contents are not readable, they get printed anyway.
$ cat zipcodes.txt
(prints the entire contents of a file named "zipcodes.txt")


If a file is too long to be viewed on one page, you can say:
$ more zipcodes.txt
(prints file one screenful at a time)


You can also use "grep" to print only those parts of a file you are interested in:
$ grep 10001 zipcodes.txt
(prints only those lines that have the character string "10001" in them)


The "grep" command is very useful, unfortunately it has a difficult-to-remember name. Be sure to:
$ man grep



Pipelines and Redirection:

Enter this command:
$ echo "cherry apple peach"
cherry apple peach

$ echo "cherry apple peach" | tr " " "\n"
cherry
apple
peach

$ echo "cherry apple peach" | tr " " "\n" | sort
apple
cherry
peach


$ echo "cherry apple peach" | tr " " "\n" | sort -r
peach
cherry
apple

Remember: A pipeline ("|") takes the output of one command and makes it the input to another command.


  • Normally the output from commands is printed on the screen. But using the symbol ">", you can redirect the output to a file:
    $ date > RightNow.txt
    $ cat RightNow.txt
    Tue Dec 23 14:43:33 PST 2003
  • The above example used ">" to replace the content of any existing file having the name "RightNow.txt". To append new data to an existing file, use ">>" instead:
    $ date >> RightNow.txt
    $ cat RightNow.txt
    Tue Dec 23 14:43:33 PST 2003
    Tue Dec 23 14:46:10 PST 2003

  • Remember: Use ">" to overwrite any existing file, use ">>" to append to any existing file. In both cases, if no file exists, one is created.

  • Many commands have inputs as well as outputs. The input defaults to the keyboard, the output defaults to the screen.
  • To redirect the output to a file, use ">" or ">>" as shown above.
  • To make the output of a command serve as the input of another command, use "|".
  • To make the contents of a file serve as the input to a command, use "<":
    $ wc < RightNow.txt
    2 12 58
    As is so often the case in shell programming, there is at least one other way to produce the above result:
    $ cat RightNow.txt | wc
    2 12 58
Shell Script Basics:

To be executable, a shell script file must meet some conditions:

  • The file must have a special first line that names an appropriate command processor. For this tutorial, the following will work in most cases:
    #!/bin/bash
  • If this example doesn't work, you will need to find out where your Bash shell executable is located and substitute that location in the above example. Here is one way to find out:
    $ whereis bash

  • The file must be made executable by changing its permission bits. An example:
    $ chmod +x (shell script filename)
  • One normally executes a shell script this way:
    $ ./scriptname.sh
First Shell Script

1. Run your choice of editor and type the following lines:

$vi myscript.sh



 #!/bin/bash
 echo "Hello, world."


$ chmod +x myscript.sh

$ ./myscript.sh


Tests and Branching

Bash shell scripts can perform, and act on, various kinds of tests. This will be just the most basic introduction 

Here is an example of a test and branch:


if [ -e . ]
then
 echo "Yes."
else
 echo "No."
fi
              
              
Run the test script:
$ ./myscript.sh
Yes.


We created a test (the part of the script between "[" and "]") which tested whether a particular element existed ("-e"). Because the symbol "." in this context means the current directory, the test succeeded. Try replacing the "." with something that is not present in the current directory, example "xyz". See how the outcome changes.


It is important to realize that "[" is an alias for the command "test". The script could have been written as:


if test -e .
then
 echo "Yes."
else
 echo "No."
fi
              
              
$man test

test - check file types and compare values

Otherwise, EXPRESSION is true or false and sets exit status.  
It is one of:

       ( EXPRESSION )
              EXPRESSION is true

       ! EXPRESSION
              EXPRESSION is false

       EXPRESSION1 -a  EXPRESSION2
              both EXPRESSION1 and EXPRESSION2 are true

       EXPRESSION1 -o EXPRESSION2
             either EXPRESSION1 or EXPRESSION2 is true

-n STRING
              the length of STRING is nonzero

       
-z STRING
              the length of STRING is zero

STRING1 = STRING2
              the strings are equal

STRING1 != STRING2
              the strings are not equal

INTEGER1 -eq INTEGER2
              INTEGER1 is equal to INTEGER2

INTEGER1 -ge INTEGER2
              INTEGER1 is greater than or equal to INTEGER2

INTEGER1 -gt INTEGER2
              INTEGER1 is greater than INTEGER2

INTEGER1 -le INTEGER2
              INTEGER1 is less than or equal to INTEGER2

INTEGER1 -lt INTEGER2
              INTEGER1 is less than INTEGER2

INTEGER1 -ne INTEGER2
              INTEGER1 is not equal to INTEGER2

FILE1 -ef FILE2
              FILE1 and FILE2 have the same device and inode numbers

FILE1 -nt FILE2
              FILE1 is newer (modification date) than FILE2

FILE1 -ot FILE2
              FILE1 is older than FILE2

-b FILE
              FILE exists and is block special

-c FILE
              FILE exists and is character special

-d FILE
              FILE exists and is a directory

-e FILE
              FILE exists

-f FILE
              FILE exists and is a regular file

-g FILE
              FILE exists and is set-group-ID

-G FILE
              FILE exists and is owned by the effective group ID

-h FILE
              FILE exists and is a symbolic link (same as -L)

-k FILE
              FILE exists and has its sticky bit set

-L FILE
              FILE exists and is a symbolic link (same as -h)

-O FILE
              FILE exists and is owned by the effective user ID

-p FILE
              FILE exists and is a named pipe

-r FILE
              FILE exists and read permission is granted

-s FILE
              FILE exists and has a size greater than zero

-S FILE
              FILE exists and is a socket

-t FD  file descriptor FD is opened on a terminal

-u FILE
              FILE exists and its set-user-ID bit is set

-w FILE
              FILE exists and write permission is granted

-x FILE
              FILE exists and execute (or search) permission is granted

Note:when a test is conducted or a command returns a result value, the numerical value for "true" is 0, and "false" is 1. 

$ echo $(( 0 && 0 ))
0


$ echo $(( 1 && 0 ))
0

$ echo $(( 0 && 1 ))
0

$ echo $(( 1 && 1 ))
1


A couple of rules about logical operators used as branches:
  • If you write "test && command", the command will only be executed if the test succeeds.
  • If you write "test || command", the command will only be executed if the test fails.
Run these tests:
$ true && echo "Yes."
Yes.


$ false || echo "Yes."
Yes.
========================================================================


Loops and Repetition:

1. Here are some examples of loop operators:


for fn in *; #

do
   echo "$fn"
done

==>the "*" is expanded by the shell to a list of all the files in the current directory.

            

xinetd FAQ

xinetd FAQ



Q. What is xinetd ?
A. xinetd is a replacement for inetd, the internet services daemon.

Q: I am not a system administrator; what do I care about an inetd replacement ?
A: xinetd is not just an inetd replacement. Anybody can use it to start servers that don't require privileged ports because xinetd does not require that the services in its configuration file be listed in /etc/services.

Q. Is it compatible with inetd ?
A. No, its configuration file has a different format than inetd's one and it understands different signals. However the signal-to-action assignment can be changed and a program has been included to convert inetd.conf to xinetd.conf.


Q. Why should I use it ?
A. Because it is a lot better (IMHO) than inetd. Here are the reasons:
1) It can do access control on all services based on:
a. address of remote host
b. time of access
c. name of remote host
d. domain name of remote host
2) Access control works on all services, whether multi-threaded or single-threaded and for both the TCP and UDP protocols. All UDP packets can be checked as well as all TCP connections.
3) It provides hard reconfiguration:
a. kills servers for services that are no longer in the configuration file
b. kills servers that no longer meet the access control criteria
4) It can prevent denial-of-access attacks by
a. placing limits on the number of servers for each service (avoids process table overflows)
b. placing an upper bound on the number of processes it will fork
c. placing limits on the size of log files it creates
d. placing limits on the number of connection a single host can initiate
e. place limits on the rate of incoming connections
f. discontinue services if the load exceeds specified limit
5) Extensive logging abilities:
a. for every server started it can log:
i) the time when the server was started
ii) the remote host address
iii) who was the remote user (if the other end runs a RFC-931/RFC-1413 server)
iv) how long the server was running
(i, ii and iii can be logged for failed attempts too).
b. for some services, if the access control fails, it can log information about the attempted access (for example, it can log the user name and command for the rsh service)
6) No limit on number of server arguments
7) You can bind specifc services to specific IP's on your host machine


Q. Whom should I thank/blame for this program ?
A. panos@cs.colorado.edu originally wrote this program, but I am fielding bug reports at this time.

Q. What's up with 2.2.1 version of xinetd?
A. The most recent original version of xinetd was 2.1.1 with patches bringing it up to 2.1.8. Nick Hilliard created xinetd 2.2.1, based off an unreleased xinetd 2.2.0 by Panos. The copyright included with xinetd specified the required versioning to be the official release of xinetd (2.1.8 in this case) and a fourth version number tacked on to indicate the modification level. This is the versioning I have adopted. xinetd 2.1.8.X, which is available here, is not based off xinetd 2.2.0 or higher. It was created from the codebase of xinetd 2.1.8, although I have re-implemented some of the features introduced in xinetd-2.2.1.

Q. Where can I find the latest-and-greatest version ?
A. The xinetd source can be obtained from http://www.synack.net/xinetd

Q. Has anyone been able to get qmail working with xinetd?
A. yes, here is the entry info
service smtp
{
        flags           = REUSE NAMEINARGS
        socket_type     = stream
        protocol        = tcp
        wait            = no
        user            = qmaild
        server          = /usr/sbin/tcpd
        server_args     = /var/qmail/bin/tcp-env -R /var/qmail/bin/qmail-smtpd
}
Contributed by: Anthony Abby
This method will allow you to set environment variables and whatnot in /etc/hosts.allow. Although xinetd can be compiled with libwrap support, this doesn't mean it can completly replace tcpd's functionality. xinetd calls host_access(), which performs the access control documented in host_access(5) man page. This is a subset of the features offered by tcpd.

Q. What platforms is xinetd know to work on?
A. I have run it on Solaris 2.6 (sparc and x86), Linux, BSDi, and IRIX 5.3 and 6.2. The original package ran on SunOS 4 and Ultrix.

Q. How to do setup a chrooted environment for a service?
A. Here is the config file entry:
service telnet_chroot
{
        log_on_success  = HOST PID DURATION USERID
        log_on_failure         = HOST RECORD USERID
        no_access      = 152.30.11.93
        socket_type     = stream
        protocol        = tcp
        port           = 8000
        wait            = no
        user            = root
        server          = /usr/sbin/chroot
        server_args    = /var/public/servers /usr/libexec/telnetd
}
Contributed by: lburns@sasquatch.com

Q. How do I use itox?
A. itox reads in a regular inetd.conf file from stdin and writes an xinetd.conf file to stdout. In general, you use the command:
itox < /etc/inetd.conf > /etc/xinetd.conf
If your inetd.conf does not have explicit paths to each of the daemons, you must use the -daemon_dir option. Suppose all your daemons live in /usr/sbin, use the following command:
itox -daemon_dir=/usr/sbin < /etc/inetd.conf > /etc/xinetd.conf
itox is rather old and hasn't been updated for a while. xconv.pl is a perl script that is a little better about converting modern inetd.conf files. It's usage is similar to itox's.

Q. Does xinetd support libwrap (tcpwrappers)?
A. Yes. xinetd can be compiled with libwrap support by passing --with-libwrap as an option to the configure script. When xinetd is compiled with libwrap support, all services can use the /etc/hosts.allow and /etc/hosts.deny access control. xinetd can also be configured to use tcpd in the traditional inetd style. This requires the use of the NAMEINARGS flag, and the name of the real daemon be passed in as server_args. Here is an example for using telnet with tcpd:
service telnet
{
        flags       = REUSE NAMEINARGS
        protocol    = tcp
        socket_type = stream
        wait        = no
        user        = telnetd
        server      = /usr/sbin/tcpd
        server_args = /usr/sbin/in.telnetd
}
Q. Does xinetd support IPv6?
A. Yes. xinetd can be compiled with IPv6 support by adding the --with-inet6 option to the configure script. Access control is functional with IPv6. You can use ipv4 mapped addresses, or give normal dotted quad ipv4 addresses for access control, and xinetd will map them to ipv6 addresses.

Q. No services start with IPv6! What's the deal?
A. When you compile IPv6 support in, all sockets are IPv6 sockets. If your kernel doesn't understand what an IPv6 socket is, all attempts to create sockets will fail, and no services will start. Only compile xinetd with IPv6 support if your kernel supports IPv6.

Q. What's this setgroups(0, NULL) error?
A. By default, xinetd does not allow group permissions to the server processes, and it does this by setting the groups of the child process to nothing. Some BSD's have a problem with this. To avoid this error, put the directive groups = yes into your services. This says to allow the server process to have all the group privleges entitled to the user the server process is running as.

Q. Why can't telnetd start normally on Linux?
A. On some Linux distributions, the telnet daemon starts as a nonprivleged user, but the user belongs to groups that allow it to open new tty's, and to update utmp. By default, xinetd does not allow group permissions to the server process, so telnetd can fail to start properly. To get the server process to posess the proper groups, use the groups = yes directive for the telnet service. This will tell xinetd that it is OK for the server process to start with all the groups the user has access to.

Q. How do I use xinetd to wrap SSL around services
A. Use the program stunnel to wrap SSL around services. This can actually be used by an inetd.

Q. How do I setup a cvs server with xinetd?
A. A user wrote in with this suggestion:
cvspserver  stream tcp nowait root /usr/bin/cvs cvs --allow-root=/home/pauljohn/cvsroot  --allow-root=/home/pauljohn/cvsmisc pserver
If you want to make the same work under xinetd, you save a config file in /etc/xinetd.d called cvspserver, (where the last line tells it the names of your repositories):
service cvspserver
{
        socket_type         = stream
        protocol            = tcp
        wait                = no
        user                = root
        passenv             = 
        server              = /usr/bin/cvs
        server_args         = --allow-root=/home/pauljohn/cvsroot --allow-root=/home/pauljohn/cvsmisc pserver -f
}
All the other cvs setup stuff is the same. This seems to work, afaik.


remdy







                                                                                   










nagios close



script shell


Creating a configuration script

Learning objective

After completing this topic, you should be able to use sed in a UNIX shell script to edit a file.

Exercise overview

In this exercise, you're required to create a new configuration file and perform a sed substitution.
This involves the following tasks:
·         redirecting sed output
·         substituting a text string

Task 1: Redirecting sed output

Let's say that you want to adapt the configuration file named "hosts" on the primary GLEESON server for use on the secondary server, without changing the original file.

You decide to use sed to search and replace host addresses and to save the output as a new file called "en_hosts".

Step 1 of 2

Let's say that you want to change all instances of "190" to "192".
See if you can type the sed command that will substitute the 190 string.
 
sed MISSING CODE

Result

You enter 's/190/192/g' to specify the global substitution that will change all instances of "190" to "192."

Step 2 of 2

Let's say that you want to create a new configuration file containing the changed output.
Choose the code that you think will specify hosts as the input file and en_hosts as the output file.
 
$ sed 's/190/192/g' MISSING CODE
Options:
1.    en_hosts > hosts
2.    hosts > en_hosts
3.    en_hosts < hosts

Result

The code that specifies hosts as the input file and en_hosts as the output file is hosts > en_hosts
Option 1 is incorrect. This option uses en_hosts as the input file and hosts as the output file.
Option 2 is correct. The data on the left of the > redirector is written to the file on the right of the redirector.
Option 3 is incorrect. This command would redirect the hosts file as an argument to the command on the left of the redirector. Since en_hosts is a file, this command will generate an error.

Task 2: Substituting a text string

Step 1 of 2

Let's say that you need to change all instances of "gleeson" to "gleeson_assoc" in the en_hosts configuration file.
Choose the code that you think will substitute all the text strings that match "gleeson."
 
$ sed MISSING CODE
Options:
1.    's/^gleeson/gleeson_assoc/g'
2.    's/gleeson_assoc/gleeson/g'
3.    's/gleeson/gleeson_assoc/p'
4.    's/gleeson/gleeson_assoc/g'

Result

You enter 's/gleeson/gleeson_assoc/g' to specify the substitution that will change all instances of "gleeson" to "gleeson_assoc" in the en_hosts configuration file.
Option 1 is incorrect. This would only replace the string "gleeson" if it appeared at the start of a line.
Option 2 is incorrect. This would replace gleeson_assoc with gleeson, instead of vice versa.
Option 3 is incorrect. The g flag is required to ensure the each instance is replaced, not just the first. The p flag sends the results of each replacement to standard output.
Option 4 is correct. You use the s command with the g flag to perform global substitutions. The first term is the search term that is replaced with the second term.

Step 2 of 2

See if you can complete the sed command to output the new configuration information to screen.
 
$ sed 's/gleeson/gleeson_assoc/g' MISSING CODE

Result

You enter en_hosts to specify the filename of the new configuration file.



Understanding UNIX shell scripts

Learning objective

After completing this topic, you should be able to explain what a shell script is.

1. Command-line processing

The UNIX shell is a command-line interpreter that processes each command or combination of commands on a command line when you press Enter.

This example shows only a single command on the command line.
$ sort -r userlist
You can combine multiple commands on the same command line by creating a composite command.

This example shows a composite command comprising the
ls and less commands.
$ ls -l | less
You can use a number of special characters in the command line. A semicolon (;), for example, allows you to place more than one command statement in the same command line. When you enter the code, the shell executes the preceding commands only when it reaches the semicolon.

In this example, the shell executes the
cp command statement and then the cat command statement.
$ cp list list.bak; cat list.bak
Special characters you can use to manipulate commands in the command line include
·         backslash (\)
·         greater than (>)
·         less than (<)
·         pipe (|)
·         ampersand (&)
backslash (\)
The backslash (\) character prevents the shell from treating another character as a special character through a process called backslash escaping.

This allows you to split a command statement across multiple lines. When you place the backslash at the end of a line and then press Enter, you can continue the statement on the next line. The backslash prevents the shell from treating the Enter keystroke – or new line character – as a special character.

This example shows a long echo statement carried across three lines.

The code for this is

$ echo Long pieces of text may not always fit onto a single \
> line of the command line interface, so it becomes \
> necessary to split them across multiple lines using \
> backslashes.
greater than (>)
The greater-than character (>) allows you to direct the standard output of a command to a file or a device such as a printer instead of to the terminal screen.

This example will send the output of the
ls command to a file called userdirs.

The code for this is

$ ls -l /usr/home > userdirs
less than (<)
The less-than character (<) allows you to send the contents of a file to a command as its standard input.

This example sends input from a file called list to the
sort command.

The code for this is

$ sort -d < list
pipe (|)
The pipe character (|) allows you to direct the output of one command to the input of another command.

This example pipes the output from the
cat command as input to the grep command for further processing.

The code for this is

$ cat EasyNomad | grep 'South America'
ampersand (&)
An ampersand (&) character at the end of a command statement allows you to run commands in the background.

This example specifies that the
find command will run a long search process in the background.

The code for this is

$ find 'EasyNomad' &
[1] 48748
$ EasyNomad
If you want to use special characters in command-line text without the shell recognizing them as special characters, you have to enclose them in quotes or precede them with a backslash (\).

This example shows an echo command in which the echo text contains an ampersand. There's a backslash in front of the ampersand, which prevents the shell from treating it as a special character.
$ echo Tours \& Accommodation

Tours & Accommodation

$

Question

How do you think the shell processes the contents of a command line in order to execute it?
Options:
1.       By analyzing commands first and then proceeding to options and arguments
2.       By dividing the command line into segments and processing each segment
3.       By processing the command line from beginning to end, one character at a time
4.       By processing the command line from beginning to end, one word at a time

Answer

When you execute a command line, the shell looks for spaces and special characters and splits the command line into segments wherever these characters occur. It then processes each segment in turn.
The segments into which the shell divides a command line are called tokens. To execute a command line, the shell processes the first token and then each subsequent token in turn.
To begin processing a token, the shell checks whether it's a keyword, an alias, or an ordinary word.
If the token is a keyword that opens a substructure such as a function, conditional statement, or bracketed group of commands, the shell processes the substructure before moving on to the next token.

If a token is an alias, the shell replaces it with the command to which the alias is mapped.

If a token is an ordinary word such as a command or a filename, the shell processes it directly.
After comparing a token against the list of known keywords and aliases, the shell processes it using several stages of expansion and substitution.
Expansion and substitution takes place in the following sequence:
·         brace expansion
·         tilde expansion
·         parameter substitution
·         command substitution
·         arithmetic substitution
·         word splitting
·         pathname substitution
brace expansion
In brace expansion, the shell looks for braces ({}) – also called curly brackets – in the token. If braces are present, it expands their contents.

For example, the token
b{all,ook} expands into ball book.
tilde expansion
In tilde expansion, the shell looks for tildes (~) in the token. If a tilde is present, it replaces the tilde with the location of the current user's home directory.

For example, depending on the system configuration, the token
~vincep/file2 might expand into /usr/home/vincep/file2.
parameter substitution
In parameter substitution, the shell checks whether the token is a variable name preceded by a dollar sign ($). If it is, the shell replaces the token with the current value of the corresponding variable.

For example, if the value of the
SHELL parameter is /bin/ksh, the token $SHELL is replaced with /bin/ksh.
command substitution
In command substitution, the shell checks whether the token is a command enclosed in brackets and preceded by a dollar sign ($). If it is, the shell processes the command and replaces the token with the command's output.

For example, the token
$(type username) might be replaced with vincep.
arithmetic substitution
In arithmetic substitution, the shell checks whether the token is an arithmetic expression enclosed in double brackets and preceded by a dollar sign. If it is, the shell evaluates the expression and replaces the token with the result.

For example, the shell replaces the token
$((72/9)) with 8.
word splitting
In word splitting, the shell examines those parts of the command line that have resulted from previous stages of expansion and substitution. If any of these contain spaces or special characters, it splits them into tokens for processing.
pathname substitution
In pathname substitution, the shell looks for wildcard characters in the token. If it finds asterisks (*), question marks (?), or double slashes (//), it searches the current directory for filenames that match these wildcards and substitutes them for the token.

For example, depending on the files in the current directory, the token
f*.txt might expand into fares.txt flights.txt funding.txt.
After performing expansion and substitution, the shell processes subsequent tokens until it reaches the end of a command, denoted by a semicolon or a new line character.

Then it matches the command against its list of known functions, built-in commands, and pathnames.
Once the shell has identified which command it needs to execute, it executes the command to produce output.

It then moves on to the next command, processing its tokens in the same way.

Question

Which special character is used to run a command in the background?
Options:
1.       &
2.       |
3.       ;
4.       \

Answer

The ampersand symbol (&) is used to run a command in the background.
Option 1 is correct. The ampersand symbol (&) is used at the end of a command, to run a job in the background. Jobs in the background are assigned a job id number. You can you can use this number to foreground the job again.
Option 2 is incorrect. The pipe (|) special character allows you to use the output of the command on the left of the pipe as input for the command on the right of the pipe.
Option 3 is incorrect. The semi-colon (;) is used to combine separate commands on the same line. Commands on the right of the semi-colon are only interpreted once commands on the left have been interpreted and executed.
Option 4 is incorrect. The backslash (\) is used to prevent special characters from being interpreted in such a way that their literal values are used in stings. Also, you can continue your command on a new line by typing a backslash before pressing Enter.

Question

Choose the code that contains an example of command substitution.
Options:
1.       cat logfile | grep $(cat /etc/hostname)
2.       cd ~/documents
3.       echo server{1,2,3).easynomad.com > hosts
4.       ls * | grep "easynomad"

Answer

The code $(cat /etc/hostname) is an example of command substitution. The contents of the hostname file are used as a search term by the grep command.
Option 1 is correct. In this example of command substitution, the command cat /etc/hostname is processed before the grep command is executed, so that its output can be substituted into the grep command.
Option 2 is incorrect. This is an example of tilde substitution. The path ~/documents expands to the documents folder in the home directory of the current user.
Option 3 is incorrect. This is an example of brace expansion. The code server{1,2,3).easynomad.com is expanded as: server1.easynomad.com; server2.easynomad.com; server3.easynomad.com.
Option 4 is incorrect. This is an example of filename substitution. The code ls * lists every file in the current directory.

2. Command grouping

You can join commands on a command line in such a way that the second command executes only if the first command has executed successfully.

For example, you can use a first command to check whether a file exists and a second command to perform an operation on it if it exists.

Question

How do you think the shell knows whether a command has executed successfully?
Options:
1.       Because the command terminates
2.       Because the command's exit status is zero
3.       Because the command's standard error output is null
4.       Because the command's standard output contains no error messages

Answer

The shell knows that a command has executed successfully when the exit status of the command is zero.
To make one command conditional on another, you join the commands using a double ampersand (&&). The command after the && symbols executes only if the command before the && symbols produces a zero exit status – in other words, if it executes successfully.
In this example, the ls command checks whether the userlist file exists. Because it does exist, the ls command executes without errors ( so its exit state is zero. This causes the sort command to execute.
$ ls userlist && sort userlist

userlist

BAKER, Daniel

CARUSO, Maria

GARZA, Teresa

LOGAN, Greg

MANEROWSKI, Sarah

NOVAK, Nicholas

NOVIALLO, Glen

OSWALD, Sam

PASCUCCI, Vince

REILLY, Molly

STROTHER, Tanya

WADE, Debora

$
If you delete the userlist file and run the command again, the ls command encounters an error – so its exit state is non-zero. Because the sort command is conditional, the shell doesn't attempt to execute it.
$ ls userlist && sort userlist

ls: userlist: No such file or directory

$
You use a double pipe (||) to make a command conditional on the unsuccessful execution of the previous command.

In such a case, the second command executes only if the first command has a non-zero exit state.

In this example, the
ls command looks for a file called userlist. If it fails to find the file, the touch command creates it.
$ ls userlist || touch userlist

ls: userlist: No such file or directory

$
If the ls command executes successfully, this means that the file already exists. In this case, the touch command doesn't execute.
$ ls userlist || touch userlist

userlist

$
You can group commands using braces ({}). The shell treats any command block enclosed in braces as if it were a single command.

This allows you to redirect input and output to and from a group of commands.
In this example, the braces group the sort and grep commands into a code block so that the shell sorts input and then extracts any lines containing the word Mexico.
$ {sort | grep 'Mexico'}
You can redirect input and output to a command block as if it were a single command. In this example, the code specifies the flights file as input and the mex_flights file as output.
$ {sort | grep 'Mexico'} < flights > mex_flights

$
You can group commands using round brackets – often called parentheses – instead of braces. This causes the shell to spawn a subshell and execute the command block in the subshell.
Commands that execute in a subshell do not affect what's happening in the main shell.

This allows you to define variables that exist only for the lifetime of the subshell, and to change the working directory within the subshell without affecting the parent shell.
$ (sort | grep 'Mexico') < massivefile > mex_info

$

Question

You want to create a file named hostname and containing the text easy1.easynomad.com. However, you don't want to overwrite any existing file by that name.
Which line of code will enable you to do this?
Options:
1.       cat hostname || echo easy1.easynomad.com > hostname
2.       cat hostname && echo easy1.easynomad.com > hostname
3.       cat hostname >> echo easy1.easynomad.com > hostname
4.       cat hostname | echo easy1.easynomad.com > hostname

Answer

The use of the || ensures that the code that writes the output from the echo command to the hostname file will only execute if the attempt to list the hostname file fails.
Option 1 is correct. You use the double pipe to make a command conditional on the unsuccessful execution of a previous command.
Option 2 is incorrect. The && conditional execution symbol ensures that if the attempt to list the hostname file succeeds, it will get overwritten.
Option 3 is incorrect. The >> redirector is used to append output to a file.
Option 4 is incorrect. The I symbol pipes the output from one command into another command as input.

3. Storing commands in scripts

Command grouping is useful for executing relatively short command-line code that you need to run only once.

However, you may need to run larger pieces of code that include several lines or to use the same piece of code many times.

In such cases, it's advantageous to store the code in a file.
You can store blocks of shell commands in shell scripts.

The contents of shell scripts are stored as ordinary ASCII text.

Question

What do you think distinguishes shell script files from ordinary ASCII text files?
Options:
1.       They contain commands
2.       They have a specific filename suffix
3.       They have an introductory line of code that defines them as scripts
4.       You can execute them

Answer

Unlike ordinary ASCII text files, shell scripts contain commands, are executable, and have an introductory line of code that defines them as scripts.
You can read and edit ordinary text files, but you cannot execute them. However, you need to be able to execute shell scripts.

Therefore, you have to assign executable permissions on script files.
The first line in any shell script has to be a special line of code that specifies the particular shell program in which the script must run.

This is necessary because some commands run differently in different shell programs.
The shell identifier at the beginning of a shell script consists of a hash followed by an exclamation point (#!) – commonly called a shebang – and the absolute pathname of the shell program.

This example shows the first line of a script that uses the Korn shell.
#! /bin/ksh
This simple example of a script tests whether the directory /usr/shared/tours exists. If it doesn't, the script creates it. Then it creates a file called tourlist inside this directory and returns a message.
#! /bin/ksh

ls /usr/shared/tours || mkdir /usr/shared/tours

touch /usr/shared/tours/tourlist

echo tour directory and tourlist file created.
Once you've created a script and made it executable, you can use it as many times as you like. You can execute it directly from the command line or you can invoke it from inside other scripts.

Question

Identify the statements that correctly describe shell scripts.
Options:
1.       Shell scripts are ASCII text files
2.       Shell scripts need to be compiled prior to execution
3.       Shell scripts need to have executable permissions set
4.       The first line of a shell script is used to identify the command interpreter

Answer

Shell scripts are ASCII text files that need to have executable permissions set. The first line of a shell script identifies the command interpreter.
Option 1 is correct. Because shell scripts are simple ASCII text files, you can easily create them in a text editor such as vi or emacs.
Option 2 is incorrect. Shell scipts are interpreted by the command interpreter, so they don't contain binary code and aren't compiled.
Option 3 is correct. Because shell scipts are executed, either one or more of the owner, group, or other executable permissions must be set.
Option 4 is correct. The first line of a shell script consists of a hash symbol followed by an exclamation mark and the absolute path to the command interpreter that will be used to execute the script. For example:
#!/bin/bash

Summary

You can use special characters to join commands on a single command line, to redirect input and output, to run commands in the background, and to continue a command over multiple lines. You can prevent the shell from recognizing a special character by preceding it with a backslash. When you execute a command line, the shell splits it into tokens and processes each token in turn.

You can group commands using braces or brackets, which cause the shell to treat the commands as a single command. You can join two commands so that the second command will execute only if the first command executes successfully or only if it executes unsuccessfully.

You can store blocks of commands in a text file called a shell script and make this file executable. You can execute shell scripts directly from the command line and reuse them as often as necessary.




Using UNIX commands in a script

Learning objective

After completing this topic, you should be able to write a shell script that uses the shell's command execution and substitution capabilities.

Exercise overview

In this exercise, you're required to complete a script that will perform a search for certain types of files, output the search results to a file, and e-mail that file to the root user.

If an error occurs, the script should display a message on the screen.
This involves the following tasks:
·         using code blocks
·         using substitution
·         using redirection and conditional execution

Task 1: Using code blocks

Let's say that you're the root user and that you're editing a script in the vi editor.

Step 1 of 1

You want the first two lines of the script to act as a distinct code block that returns one exit value for both commands. The commands should execute in the current shell.
Which type of brackets should you use to achieve this?
 
MISSING CODE ls -l report 

MISSING CODE MISSING CODE 

localreports ;

mail -s "local reports" root 

MISSING CODE 

localreports ; MISSING CODE

MISSING CODE echo 

An error occurred during search operation
Options:
1.       Round brackets (())
2.       Square brackets ([])
3.       Braces ({})
4.       Less than and greater than signs (<>)

Result

You use braces ({}) to create a code block that will execute in the current shell.
Option 1 is incorrect. You use round brackets to perform command substitution, as in the variable assignment command shown here:
DATE=$(date)
Option 2 is incorrect. You use the square brackets to enclose conditional expressions, such as:
if [ $FEEDBACK = y ] then
echo "proceed"
else
exit
fi
Option 3 is correct. The syntax for a function – a sequence of code that is available to the shell into which it is read – is:
function_name() {commands}
Option 4 is incorrect. The <> special characters are used to perform redirection. They are also used as conditional operators (less-than and greater-than) within conditional expressions.

Task 2: Using substitution

Step 1 of 1

Suppose that you want to use the ls command to search for all files that begin with the word "report" and end in a number between 0 and 9.
Use substitution to complete the search criteria shown here.
 
{ ls -l report MISSING CODE MISSING CODE 

localreports ;mail -s "local reports" root MISSING CODE l

ocalreports ; }

MISSING CODE echo 

An error occurred during search operation

Result

You specify report[0-9] with the ls command to list all files that begin with the word "report" and end in a number between 0 and 9.

Task 3: Using redirection and conditionals

Step 1 of 3

Suppose that you want to redirect the output of the ls command to a file named localreports.
Type the symbol that will achieve this.
 
{ ls -l report[0-9] MISSING CODE localreports ;

mail -s "local reports" root MISSING CODE localreports ; }

MISSING CODE echo 

An error occurred during search operation

Result

You use the greater than symbol (>) to redirect output to a file.

Step 2 of 3

In the second line of the script, you want to redirect the contents of a mail from the localreports file.
Type the correct symbol to do this.
 
{ ls -l report[0-9] > localreports ;

mail -s "local reports" root MISSING CODE localreports ; }

MISSING CODE echo 

An error occurred during search operation

Result

You use the less than symbol (<) to redirect input from a file.

Step 3 of 3

You want the last line of the script to execute only if the code block produces an error.
Type the symbol that will do this.
 
{ ls -l report[0-9] > localreports ;

mail -s "local reports" root < localreports ; }

MISSING CODE echo 

An error occurred during search operation

Result

You use the double pipe symbol (||) to execute a command only if the preceding command ( or, in this case, a command block ( fails.
You've now successfully completed a script that will perform a search for specific files, output the search results to a file named localreports, and e-mail the localreports file to the root user. It will also display an error message if the code fails.
{ ls -l report[0-9] > localreports ;

mail -s "local reports" root < localreports ; }

|| echo 

An error occurred during search operation