script shell

Creating a configuration script

Learning objective

After completing this topic, you should be able to use sed in a UNIX shell script to edit a file.

Exercise overview

In this exercise, you're required to create a new configuration file and perform a sed substitution.

This involves the following tasks:

·         redirecting sed output

·         substituting a text string

Task 1: Redirecting sed output

Let's say that you want to adapt the configuration file named "hosts" on the primary GLEESON server for use on the secondary server, without changing the original file.

You decide to use sed to search and replace host addresses and to save the output as a new file called "en_hosts".

Step 1 of 2

Let's say that you want to change all instances of "190" to "192".

See if you can type the sed command that will substitute the 190 string.

sed MISSING CODE

Result

You enter 's/190/192/g' to specify the global substitution that will change all instances of "190" to "192."

Step 2 of 2

Let's say that you want to create a new configuration file containing the changed output.

Choose the code that you think will specify hosts as the input file and en_hosts as the output file.

$ sed 's/190/192/g' MISSING CODE

Options:

1.    en_hosts > hosts

2.    hosts > en_hosts

3.    en_hosts < hosts

Result

The code that specifies hosts as the input file and en_hosts as the output file is hosts > en_hosts

Option 1 is incorrect. This option uses en_hosts as the input file and hosts as the output file.

Option 2 is correct. The data on the left of the > redirector is written to the file on the right of the redirector.

Option 3 is incorrect. This command would redirect the hosts file as an argument to the command on the left of the redirector. Since en_hosts is a file, this command will generate an error.

Task 2: Substituting a text string

Step 1 of 2

Let's say that you need to change all instances of "gleeson" to "gleeson_assoc" in the en_hosts configuration file.

Choose the code that you think will substitute all the text strings that match "gleeson."

$ sed MISSING CODE

Options:

1.    's/^gleeson/gleeson_assoc/g'

2.    's/gleeson_assoc/gleeson/g'

3.    's/gleeson/gleeson_assoc/p'

4.    's/gleeson/gleeson_assoc/g'

Result

You enter 's/gleeson/gleeson_assoc/g' to specify the substitution that will change all instances of "gleeson" to "gleeson_assoc" in the en_hosts configuration file.

Option 1 is incorrect. This would only replace the string "gleeson" if it appeared at the start of a line.

Option 2 is incorrect. This would replace gleeson_assoc with gleeson, instead of vice versa.

Option 3 is incorrect. The g flag is required to ensure the each instance is replaced, not just the first. The p flag sends the results of each replacement to standard output.

Option 4 is correct. You use the s command with the g flag to perform global substitutions. The first term is the search term that is replaced with the second term.

Step 2 of 2

See if you can complete the sed command to output the new configuration information to screen.

$ sed 's/gleeson/gleeson_assoc/g' MISSING CODE

Result

You enter en_hosts to specify the filename of the new configuration file.

Understanding UNIX shell scripts

Learning objective

After completing this topic, you should be able to explain what a shell script is.

1. Command-line processing

The UNIX shell is a command-line interpreter that processes each command or combination of commands on a command line when you press Enter.

This example shows only a single command on the command line.

$ sort -r userlist

You can combine multiple commands on the same command line by creating a composite command.

This example shows a composite command comprising the ls and less commands.

$ ls -l | less

You can use a number of special characters in the command line. A semicolon (;), for example, allows you to place more than one command statement in the same command line. When you enter the code, the shell executes the preceding commands only when it reaches the semicolon.

In this example, the shell executes the cp command statement and then the cat command statement.

$ cp list list.bak; cat list.bak

Special characters you can use to manipulate commands in the command line include

·         backslash (\)

·         greater than (>)

·         less than (<)

·         pipe (|)

·         ampersand (&)

backslash (\)

The backslash (\) character prevents the shell from treating another character as a special character through a process called backslash escaping.

This allows you to split a command statement across multiple lines. When you place the backslash at the end of a line and then press Enter, you can continue the statement on the next line. The backslash prevents the shell from treating the Enter keystroke – or new line character – as a special character.

This example shows a long echo statement carried across three lines.

The code for this is

$ echo Long pieces of text may not always fit onto a single \
> line of the command line interface, so it becomes \
> necessary to split them across multiple lines using \
> backslashes.

greater than (>)

The greater-than character (>) allows you to direct the standard output of a command to a file or a device such as a printer instead of to the terminal screen.

This example will send the output of the ls command to a file called userdirs.

The code for this is

$ ls -l /usr/home > userdirs

less than (<)

The less-than character (<) allows you to send the contents of a file to a command as its standard input.

This example sends input from a file called list to the sort command.

The code for this is

$ sort -d < list

pipe (|)

The pipe character (|) allows you to direct the output of one command to the input of another command.

This example pipes the output from the cat command as input to the grep command for further processing.

The code for this is

$ cat EasyNomad | grep 'South America'

ampersand (&)

An ampersand (&) character at the end of a command statement allows you to run commands in the background.

This example specifies that the find command will run a long search process in the background.

The code for this is

$ find 'EasyNomad' &
[1] 48748
$ EasyNomad

If you want to use special characters in command-line text without the shell recognizing them as special characters, you have to enclose them in quotes or precede them with a backslash (\).

This example shows an echo command in which the echo text contains an ampersand. There's a backslash in front of the ampersand, which prevents the shell from treating it as a special character.

$ echo Tours \& Accommodation

Tours & Accommodation

$

Question

How do you think the shell processes the contents of a command line in order to execute it?

Options:

1.       By analyzing commands first and then proceeding to options and arguments

2.       By dividing the command line into segments and processing each segment

3.       By processing the command line from beginning to end, one character at a time

4.       By processing the command line from beginning to end, one word at a time

Answer

When you execute a command line, the shell looks for spaces and special characters and splits the command line into segments wherever these characters occur. It then processes each segment in turn.

The segments into which the shell divides a command line are called tokens. To execute a command line, the shell processes the first token and then each subsequent token in turn.

To begin processing a token, the shell checks whether it's a keyword, an alias, or an ordinary word.

If the token is a keyword that opens a substructure such as a function, conditional statement, or bracketed group of commands, the shell processes the substructure before moving on to the next token.

If a token is an alias, the shell replaces it with the command to which the alias is mapped.

If a token is an ordinary word such as a command or a filename, the shell processes it directly.

After comparing a token against the list of known keywords and aliases, the shell processes it using several stages of expansion and substitution.

Expansion and substitution takes place in the following sequence:

·         brace expansion

·         tilde expansion

·         parameter substitution

·         command substitution

·         arithmetic substitution

·         word splitting

·         pathname substitution

brace expansion

In brace expansion, the shell looks for braces ({}) – also called curly brackets – in the token. If braces are present, it expands their contents.

For example, the token b{all,ook} expands into ball book.

tilde expansion

In tilde expansion, the shell looks for tildes (~) in the token. If a tilde is present, it replaces the tilde with the location of the current user's home directory.

For example, depending on the system configuration, the token ~vincep/file2 might expand into /usr/home/vincep/file2.

parameter substitution

In parameter substitution, the shell checks whether the token is a variable name preceded by a dollar sign ($). If it is, the shell replaces the token with the current value of the corresponding variable.

For example, if the value of the SHELL parameter is /bin/ksh, the token $SHELL is replaced with /bin/ksh.

command substitution

In command substitution, the shell checks whether the token is a command enclosed in brackets and preceded by a dollar sign ($). If it is, the shell processes the command and replaces the token with the command's output.

For example, the token $(type username) might be replaced with vincep.

arithmetic substitution

In arithmetic substitution, the shell checks whether the token is an arithmetic expression enclosed in double brackets and preceded by a dollar sign. If it is, the shell evaluates the expression and replaces the token with the result.

For example, the shell replaces the token $((72/9)) with 8.

word splitting

In word splitting, the shell examines those parts of the command line that have resulted from previous stages of expansion and substitution. If any of these contain spaces or special characters, it splits them into tokens for processing.

pathname substitution

In pathname substitution, the shell looks for wildcard characters in the token. If it finds asterisks (*), question marks (?), or double slashes (//), it searches the current directory for filenames that match these wildcards and substitutes them for the token.

For example, depending on the files in the current directory, the token f*.txt might expand into fares.txt flights.txt funding.txt.

After performing expansion and substitution, the shell processes subsequent tokens until it reaches the end of a command, denoted by a semicolon or a new line character.

Then it matches the command against its list of known functions, built-in commands, and pathnames.

Once the shell has identified which command it needs to execute, it executes the command to produce output.

It then moves on to the next command, processing its tokens in the same way.

Question

Which special character is used to run a command in the background?

Options:

1.       &

2.       |

3.       ;

4.       \

Answer

The ampersand symbol (&) is used to run a command in the background.

Option 1 is correct. The ampersand symbol (&) is used at the end of a command, to run a job in the background. Jobs in the background are assigned a job id number. You can you can use this number to foreground the job again.

Option 2 is incorrect. The pipe (|) special character allows you to use the output of the command on the left of the pipe as input for the command on the right of the pipe.

Option 3 is incorrect. The semi-colon (;) is used to combine separate commands on the same line. Commands on the right of the semi-colon are only interpreted once commands on the left have been interpreted and executed.

Option 4 is incorrect. The backslash (\) is used to prevent special characters from being interpreted in such a way that their literal values are used in stings. Also, you can continue your command on a new line by typing a backslash before pressing Enter.

Question

Choose the code that contains an example of command substitution.

Options:

1.       cat logfile | grep $(cat /etc/hostname)

2.       cd ~/documents

3.       echo server{1,2,3).easynomad.com > hosts

4.       ls * | grep "easynomad"

Answer

The code $(cat /etc/hostname) is an example of command substitution. The contents of the hostname file are used as a search term by the grep command.

Option 1 is correct. In this example of command substitution, the command cat /etc/hostname is processed before the grep command is executed, so that its output can be substituted into the grep command.

Option 2 is incorrect. This is an example of tilde substitution. The path ~/documents expands to the documents folder in the home directory of the current user.

Option 3 is incorrect. This is an example of brace expansion. The code server{1,2,3).easynomad.com is expanded as: server1.easynomad.com; server2.easynomad.com; server3.easynomad.com.

Option 4 is incorrect. This is an example of filename substitution. The code ls * lists every file in the current directory.

2. Command grouping

You can join commands on a command line in such a way that the second command executes only if the first command has executed successfully.

For example, you can use a first command to check whether a file exists and a second command to perform an operation on it if it exists.

Question

How do you think the shell knows whether a command has executed successfully?

Options:

1.       Because the command terminates

2.       Because the command's exit status is zero

3.       Because the command's standard error output is null

4.       Because the command's standard output contains no error messages

Answer

The shell knows that a command has executed successfully when the exit status of the command is zero.

To make one command conditional on another, you join the commands using a double ampersand (&&). The command after the && symbols executes only if the command before the && symbols produces a zero exit status – in other words, if it executes successfully.

In this example, the ls command checks whether the userlist file exists. Because it does exist, the ls command executes without errors ( so its exit state is zero. This causes the sort command to execute.

$ ls userlist && sort userlist

userlist

BAKER, Daniel

CARUSO, Maria

GARZA, Teresa

LOGAN, Greg

MANEROWSKI, Sarah

NOVAK, Nicholas

NOVIALLO, Glen

OSWALD, Sam

PASCUCCI, Vince

REILLY, Molly

STROTHER, Tanya

WADE, Debora

$

If you delete the userlist file and run the command again, the ls command encounters an error – so its exit state is non-zero. Because the sort command is conditional, the shell doesn't attempt to execute it.

$ ls userlist && sort userlist

ls: userlist: No such file or directory

$

You use a double pipe (||) to make a command conditional on the unsuccessful execution of the previous command.

In such a case, the second command executes only if the first command has a non-zero exit state.

In this example, the ls command looks for a file called userlist. If it fails to find the file, the touch command creates it.

$ ls userlist || touch userlist

ls: userlist: No such file or directory

$

If the ls command executes successfully, this means that the file already exists. In this case, the touch command doesn't execute.

$ ls userlist || touch userlist

userlist

$

You can group commands using braces ({}). The shell treats any command block enclosed in braces as if it were a single command.

This allows you to redirect input and output to and from a group of commands.

In this example, the braces group the sort and grep commands into a code block so that the shell sorts input and then extracts any lines containing the word Mexico.

$ {sort | grep 'Mexico'}

You can redirect input and output to a command block as if it were a single command. In this example, the code specifies the flights file as input and the mex_flights file as output.

$ {sort | grep 'Mexico'} < flights > mex_flights

$

You can group commands using round brackets – often called parentheses – instead of braces. This causes the shell to spawn a subshell and execute the command block in the subshell.

Commands that execute in a subshell do not affect what's happening in the main shell.

This allows you to define variables that exist only for the lifetime of the subshell, and to change the working directory within the subshell without affecting the parent shell.

$ (sort | grep 'Mexico') < massivefile > mex_info

$

Question

You want to create a file named hostname and containing the text easy1.easynomad.com. However, you don't want to overwrite any existing file by that name.

Which line of code will enable you to do this?

Options:

1.       cat hostname || echo easy1.easynomad.com > hostname

2.       cat hostname && echo easy1.easynomad.com > hostname

3.       cat hostname >> echo easy1.easynomad.com > hostname

4.       cat hostname | echo easy1.easynomad.com > hostname

Answer

The use of the || ensures that the code that writes the output from the echo command to the hostname file will only execute if the attempt to list the hostname file fails.

Option 1 is correct. You use the double pipe to make a command conditional on the unsuccessful execution of a previous command.

Option 2 is incorrect. The && conditional execution symbol ensures that if the attempt to list the hostname file succeeds, it will get overwritten.

Option 3 is incorrect. The >> redirector is used to append output to a file.

Option 4 is incorrect. The I symbol pipes the output from one command into another command as input.

3. Storing commands in scripts

Command grouping is useful for executing relatively short command-line code that you need to run only once.

However, you may need to run larger pieces of code that include several lines or to use the same piece of code many times.

In such cases, it's advantageous to store the code in a file.

You can store blocks of shell commands in shell scripts.

The contents of shell scripts are stored as ordinary ASCII text.

Question

What do you think distinguishes shell script files from ordinary ASCII text files?

Options:

1.       They contain commands

2.       They have a specific filename suffix

3.       They have an introductory line of code that defines them as scripts

4.       You can execute them

Answer

Unlike ordinary ASCII text files, shell scripts contain commands, are executable, and have an introductory line of code that defines them as scripts.

You can read and edit ordinary text files, but you cannot execute them. However, you need to be able to execute shell scripts.

Therefore, you have to assign executable permissions on script files.

The first line in any shell script has to be a special line of code that specifies the particular shell program in which the script must run.

This is necessary because some commands run differently in different shell programs.

The shell identifier at the beginning of a shell script consists of a hash followed by an exclamation point (#!) – commonly called a shebang – and the absolute pathname of the shell program.

This example shows the first line of a script that uses the Korn shell.

#! /bin/ksh

This simple example of a script tests whether the directory /usr/shared/tours exists. If it doesn't, the script creates it. Then it creates a file called tourlist inside this directory and returns a message.

#! /bin/ksh

ls /usr/shared/tours || mkdir /usr/shared/tours

touch /usr/shared/tours/tourlist

echo tour directory and tourlist file created.

Once you've created a script and made it executable, you can use it as many times as you like. You can execute it directly from the command line or you can invoke it from inside other scripts.

Question

Identify the statements that correctly describe shell scripts.

Options:

1.       Shell scripts are ASCII text files

2.       Shell scripts need to be compiled prior to execution

3.       Shell scripts need to have executable permissions set

4.       The first line of a shell script is used to identify the command interpreter

Answer

Shell scripts are ASCII text files that need to have executable permissions set. The first line of a shell script identifies the command interpreter.

Option 1 is correct. Because shell scripts are simple ASCII text files, you can easily create them in a text editor such as vi or emacs.

Option 2 is incorrect. Shell scipts are interpreted by the command interpreter, so they don't contain binary code and aren't compiled.

Option 3 is correct. Because shell scipts are executed, either one or more of the owner, group, or other executable permissions must be set.

Option 4 is correct. The first line of a shell script consists of a hash symbol followed by an exclamation mark and the absolute path to the command interpreter that will be used to execute the script. For example:
#!/bin/bash

Summary

You can use special characters to join commands on a single command line, to redirect input and output, to run commands in the background, and to continue a command over multiple lines. You can prevent the shell from recognizing a special character by preceding it with a backslash. When you execute a command line, the shell splits it into tokens and processes each token in turn.

You can group commands using braces or brackets, which cause the shell to treat the commands as a single command. You can join two commands so that the second command will execute only if the first command executes successfully or only if it executes unsuccessfully.

You can store blocks of commands in a text file called a shell script and make this file executable. You can execute shell scripts directly from the command line and reuse them as often as necessary.

Using UNIX commands in a script

Learning objective

After completing this topic, you should be able to write a shell script that uses the shell's command execution and substitution capabilities.

Exercise overview

In this exercise, you're required to complete a script that will perform a search for certain types of files, output the search results to a file, and e-mail that file to the root user.

If an error occurs, the script should display a message on the screen.

This involves the following tasks:

·         using code blocks

·         using substitution

·         using redirection and conditional execution

Task 1: Using code blocks

Let's say that you're the root user and that you're editing a script in the vi editor.

Step 1 of 1

You want the first two lines of the script to act as a distinct code block that returns one exit value for both commands. The commands should execute in the current shell.

Which type of brackets should you use to achieve this?

MISSING CODE ls -l report 

MISSING CODE MISSING CODE 

localreports ;

mail -s "local reports" root 

MISSING CODE 

localreports ; MISSING CODE

MISSING CODE echo 

An error occurred during search operation

Options:

1.       Round brackets (())

2.       Square brackets ([])

3.       Braces ({})

4.       Less than and greater than signs (<>)

Result

You use braces ({}) to create a code block that will execute in the current shell.

Option 1 is incorrect. You use round brackets to perform command substitution, as in the variable assignment command shown here:
DATE=$(date)

Option 2 is incorrect. You use the square brackets to enclose conditional expressions, such as:
if [ $FEEDBACK = y ] then
echo "proceed"
else
exit
fi

Option 3 is correct. The syntax for a function – a sequence of code that is available to the shell into which it is read – is:
function_name() {commands}

Option 4 is incorrect. The <> special characters are used to perform redirection. They are also used as conditional operators (less-than and greater-than) within conditional expressions.

Task 2: Using substitution

Step 1 of 1

Suppose that you want to use the ls command to search for all files that begin with the word "report" and end in a number between 0 and 9.

Use substitution to complete the search criteria shown here.

{ ls -l report MISSING CODE MISSING CODE 

localreports ;mail -s "local reports" root MISSING CODE l

ocalreports ; }

MISSING CODE echo 

An error occurred during search operation

Result

You specify report[0-9] with the ls command to list all files that begin with the word "report" and end in a number between 0 and 9.

Task 3: Using redirection and conditionals

Step 1 of 3

Suppose that you want to redirect the output of the ls command to a file named localreports.

Type the symbol that will achieve this.

{ ls -l report[0-9] MISSING CODE localreports ;

mail -s "local reports" root MISSING CODE localreports ; }

MISSING CODE echo 

An error occurred during search operation

Result

You use the greater than symbol (>) to redirect output to a file.

Step 2 of 3

In the second line of the script, you want to redirect the contents of a mail from the localreports file.

Type the correct symbol to do this.

{ ls -l report[0-9] > localreports ;

mail -s "local reports" root MISSING CODE localreports ; }

MISSING CODE echo 

An error occurred during search operation

Result

You use the less than symbol (<) to redirect input from a file.

Step 3 of 3

You want the last line of the script to execute only if the code block produces an error.

Type the symbol that will do this.

{ ls -l report[0-9] > localreports ;

mail -s "local reports" root < localreports ; }

MISSING CODE echo 

An error occurred during search operation

Result

You use the double pipe symbol (||) to execute a command only if the preceding command ( or, in this case, a command block ( fails.

You've now successfully completed a script that will perform a search for specific files, output the search results to a file named localreports, and e-mail the localreports file to the root user. It will also display an error message if the code fails.

{ ls -l report[0-9] > localreports ;

mail -s "local reports" root < localreports ; }

|| echo 

An error occurred during search operation