Sed and AWK Scripts

1. Sed

sed - stream editor for filtering and transforming text

Sed is a non-interactive  stream editor. It receives text input, whether from stdin or from a file, performs certain operations on specified lines of the input, one line at a time, then outputs the result to stdout or to a file. Within a shell script, sed is usually one of several tool components in a pipe.

All the operations in the sed toolkit, we will focus primarily on the three most commonly used ones.
These are printing (to stdout), deletion, and substitution.

Table 1. Basic sed operators

Operator	Name	Effect
[address-range]/p	print	Print [specified address range]
[address-range]/d	delete	Delete [specified address range]
s/pattern1/pattern2/	substitute	Substitute pattern2 for first instance of pattern1 in a line
[address-range]/s/pattern1/pattern2/	substitute	Substitute pattern2 for first instance of pattern1 in a line, over address-range
[address-range]/y/pattern1/pattern2/	transform	replace any character in pattern1 with the corresponding character in pattern2, over address-range(equivalent of tr)
g	global	Operate on every pattern match within each matched line of input

Unless the g (global) operator is appended to a substitute command, the substitution operates only on the first instance of a pattern match within each line.

sed -e '/^$/d' $filename # The -e option causes the next string to be interpreted as an editing instruction. # (If passing only a single instruction to sed, the "-e" is optional.) # The "strong" quotes ('') protect the RE characters in the instruction # Operates on the text contained in file $filename.

Sed uses the -e option to specify that the following string is an instruction or set of instructions. If there is only a single instruction contained in the string, then this may be omitted.

sed -n '/xzy/p' $filename # The -n option tells sed to print only those lines matching the pattern. # Otherwise all input lines would print. # The -e option not necessary here since there is only a single editing instruction.

Table 2. Examples of sed operators

Notation	Effect
8d	Delete 8th line of input.
/^$/d	Delete all blank lines.
1,/^$/d	Delete from beginning of input up to, and including first blank line.
/Jones/p	Print only lines containing "Jones" (with -n option).
s/Windows/Linux/	Substitute "Linux" for first instance of "Windows" found in each input line.
s/BSOD/stability/g	Substitute "stability" for every instance of "BSOD" found in each input line.
s/ *$//	Delete all spaces at the end of every line.
s/00*/0/g	Compress all consecutive sequences of zeroes into a single zero.
/GUI/d	Delete all lines containing "GUI".
s/GUI//g	Delete all instances of "GUI", leaving the remainder of each line intact.

The usual delimiter that sed uses is /. However, sed allows other delimiters, such as %. This is useful when / is part of a replacement string,

Shell Wrappers

A wrapper is a shell script that embeds a system command or utility, that accepts and passes a set of parameters to that command.

Wrapping a script around a complex command-line simplifies invoking it.

1. This simple script removes blank lines from a file.

sed -e /^$/d "$1"
# Same as
#    sed -e '/^$/d' filename
# invoked from the command-line.

#  The '-e' means an "editing" command follows (optional here).
#  '^' indicates the beginning of line, '$' the end.
#  This matches lines with nothing between the beginning and the end --
#+ blank lines.
#  The 'd' is the delete command.

2. A script that substitutes one pattern for another in a file.

#  Here is where the heavy work gets done.
sed -e "s/$old_pattern/$new_pattern/g" $file_name

#  's' is, of course, the substitute command in sed,
#+ and /pattern/ invokes address matching.
#  The 'g,' or global flag causes substitution for EVERY
#+ occurence of $old_pattern on each line, not just the first.
#  Read the 'sed' docs for an in-depth explanation.

exit $?  # Redirect the output of this script to write to a file.

2. Awk

Awk is a full-featured text processing language with a syntax reminiscent of C. While it possesses an extensive set of operators and capabilities.

Awk breaks each line of input passed to it into fields. By default, a field is a string of consecutive characters delimited by whitespace.

Awk parses and operates on each separate field. This makes it ideal for handling structured text files -- especially tables -- data organized into consistent chunks, such as rows and columns.

[sankar@new-host ~]$ echo one two

one two

[sankar@new-host ~]$ echo one two | awk '{print $1}'

one

[sankar@new-host ~]$ echo one two|awk '{print $0}'

one two

the awk print command in action. The only other feature of awk we need to deal with here is variables. Awk handles variables similarly to shell scripts, though a bit more flexibly.

Reference Cards to Shell Script(Over View): PART 6

Reference Cards:

Table 1. Special Shell Variables:

Variable	Meaning
$0	Filename of script
$1	Positional parameter #1
$2 - $9	Positional parameters #2 - #9
${10}	Positional parameter #10
$#	Number of positional parameters
"$*"	All the positional parameters (as a single word) *
"$@"	All the positional parameters (as separate strings)
${#*}	Number of positional parameters
${#@}	Number of positional parameters
$?	Return value
$$	Process ID (PID) of script
$-	Flags passed to script (using set)
$_	Last argument of previous command
$!	Process ID (PID) of last job run in background

* Must be quoted, otherwise it defaults to $@.

Table 2. TEST Operators: Binary Comparison:

Operator	Meaning	-----	Operator	Meaning
Arithmetic Comparison			String Comparison
-eq	Equal to		=	Equal to
			==	Equal to
-ne	Not equal to		!=	Not equal to
-lt	Less than		\<	Less than (ASCII) *
-le	Less than or equal to
-gt	Greater than		\>	Greater than (ASCII) *
-ge	Greater than or equal to
			-z	String is empty
			-n	String is not empty
Arithmetic Comparison	within double parentheses (( ... ))
>	Greater than
>=	Greater than or equal to
<	Less than
<=	Less than or equal to

* If within a double-bracket [[ ... ]] test construct, then no escape \ is needed.

Table 3. TEST Operators: Files:

Operator	Tests Whether	-----	Operator	Tests Whether
-e	File exists		-s	File is not zero size
-f	File is a regular file
-d	File is a directory		-r	File has read permission
-h	File is a symbolic link		-w	File has write permission
-L	File is a symbolic link		-x	File has execute permission
-b	File is a block device
-c	File is a character device		-g	sgid flag set
-p	File is a pipe		-u	suid flag set
-S	File is a socket		-k	"sticky bit" set
-t	File is associated with a terminal
-N	File modified since it was last read		F1 -nt F2	File F1 is newer than F2 *
-O	You own the file		F1 -ot F2	File F1 is older than F2 *
-G	Group id of file same as yours		F1 -ef F2	Files F1 and F2 are hard links to the same file *
!	NOT (inverts sense of above tests)

* Binary operator (requires two operands).

Table 4. Parameter Substitution and Expansion:

Expression	Meaning
${var}	Value of var (same as $var)
${var-DEFAULT}	If var not set, evaluate expression as $DEFAULT *
${var:-DEFAULT}	If var not set or is empty, evaluate expression as $DEFAULT *
${var=DEFAULT}	If var not set, evaluate expression as $DEFAULT *
${var:=DEFAULT}	If var not set, evaluate expression as $DEFAULT *
${var+OTHER}	If var set, evaluate expression as $OTHER, otherwise as null string
${var:+OTHER}	If var set, evaluate expression as $OTHER, otherwise as null string
${var?ERR_MSG}	If var not set, print $ERR_MSG and abort script with an exit status of 1.*
${var:?ERR_MSG}	If var not set, print $ERR_MSG and abort script with an exit status of 1.*
${!varprefix*}	Matches all previously declared variables beginning with varprefix
${!varprefix@}	Matches all previously declared variables beginning with varprefix

* If var is set, evaluate the expression as $var with no side-effects.

Table 5. String Operations:

Expression	Meaning
${#string}	Length of $string
${string:position}	Extract substring from $string at $position
${string:position:length}	Extract $length characters substring from $string at $position [zero-indexed, first character is at position 0]
${string#substring}	Strip shortest match of $substring from front of $string
${string##substring}	Strip longest match of $substring from front of $string
${string%substring}	Strip shortest match of $substring from back of $string
${string%%substring}	Strip longest match of $substring from back of $string
${string/substring/replacement}	Replace first match of $substring with $replacement
${string//substring/replacement}	Replace all matches of $substring with $replacement
${string/#substring/replacement}	If $substring matches front end of $string, substitute $replacement for $substring
${string/%substring/replacement}	If $substring matches back end of $string, substitute $replacement for $substring
expr match "$string" '$substring'	Length of matching $substring* at beginning of $string
expr "$string" : '$substring'	Length of matching $substring* at beginning of $string
expr index "$string" $substring	Numerical position in $string of first character in $substring* that matches [0 if no match, first character counts as position 1]
expr substr $string $position $length	Extract $length characters from $string starting at $position [0 if no match, first character counts as position 1]
expr match "$string" '\($substring\)'	Extract $substring*, searching from beginning of $string
expr "$string" : '\($substring\)'	Extract $substring* , searching from beginning of $string
expr match "$string" '.*\($substring\)'	Extract $substring*, searching from end of $string
expr "$string" : '.*\($substring\)'	Extract $substring*, searching from end of $string

* Where $substring is a Regular Expression.

Table 6. Miscellaneous Constructs:

Expression	Interpretation
Brackets
if [ CONDITION ]	Test construct
if [[ CONDITION ]]	Extended test construct
Array[1]=element1	Array initialization
[a-z]	Range of characters within a Regular Expression
Curly Brackets
${variable}	Parameter substitution
${!variable}	Indirect variable reference
{ command1; command2; . . . commandN; }	Block of code
{string1,string2,string3,...}	Brace expansion
{a..z}	Extended brace expansion
{}	Text replacement, after find and xargs
Parentheses
( command1; command2 )	Command group executed within a subshell
Array=(element1 element2 element3)	Array initialization
result=$(COMMAND)	Command substitution, new style
>(COMMAND)	Process substitution
<(COMMAND)	Process substitution
Double Parentheses
(( var = 78 ))	Integer arithmetic
var=$(( 20 + 5 ))	Integer arithmetic, with variable assignment
(( var++ ))	C-style variable increment
(( var-- ))	C-style variable decrement
(( var0 = var1<98?9:21 ))	C-style ternary operation
Quoting
"$variable"	"Weak" quoting
'string'	'Strong' quoting
Back Quotes
result=`COMMAND`	Command substitution, classic style

This is a very brief introduction to the sed and awk text processing utilities.

sed: a non-interactive text file editor
awk: a field-oriented pattern processing language with a C-style syntax

Operator Precedence
Table 7. Operator Precedence

Operator	Meaning	Comments
		HIGHEST PRECEDENCE
var++ var--	post-increment, post-decrement	C-style operators
++var --var	pre-increment, pre-decrement
! ~	negation	logical / bitwise, inverts sense of following operator
**	exponentiation	arithmetic operation
* / %	multiplication, division, modulo	arithmetic operation
+ -	addition, subtraction	arithmetic operation
<< >>	left, right shift	bitwise
-z -n	unary comparison	string is/is-not null
-e -f -t -x, etc.	unary comparison	file-test
< -lt > -gt <= -le >= -ge	compound comparison	string and integer
-nt -ot -ef	compound comparison	file-test
== -eq != -ne	equality / inequality	test operators, string and integer
&	AND	bitwise
^	XOR	exclusive OR, bitwise
|	OR	bitwise
&& -a	AND	logical, compound comparison
|| -o	OR	logical, compound comparison
?:	trinary operator	C-style
=	assignment	(do not confuse with equality test)
*= /= %= += -= <<= >>= &=	combination assignment	times-equal, divide-equal, mod-equal, etc.
,	comma	links a sequence of operations
		LOWEST PRECEDENCE

Table 8. Job identifiers

Notation	Meaning
%N	Job number [N]
%S	Invocation (command-line) of job begins with string S
%?S	Invocation (command-line) of job contains within it string S
%%	"current" job (last job stopped in foreground or started in background)
%+	"current" job (last job stopped in foreground or started in background)
%-	Last job
$!	Last background process

Exit Codes With Special Meanings

Table . Reserved Exit Codes:

Exit Code Number	Meaning	Example	Comments
1	Catchall for general errors	let "var1 = 1/0"	Miscellaneous errors, such as "divide by zero" and other impermissible operations
2	Misuse of shell builtins (according to Bash documentation)	empty_function() {}	Missing keyword or command
126	Command invoked cannot execute	/dev/null	Permission problem or command is not an executable
127	"command not found"	illegal_command	Possible problem with $PATH or a typo
128	Invalid argument to exit	exit 3.14159	exit takes only integer args in the range 0 - 255 (see first footnote)
128+n	Fatal error signal "n"	kill -9 $PPID of script	$? returns 137 (128 + 9)
130	Script terminated by Control-C	Ctl-C	Control-C is fatal error signal 2, (130 = 128 + 2, see above)
255*	Exit status out of range	exit -1	exit takes only integer args in the range 0 - 255

Notes:

1.	Out of range exit values can result in unexpected exit codes. An exit value greater than 255 returns an exit code modulo 256. For example, exit 3809 gives an exit code of 225 (3809 % 256 = 225).
2.	An update of /usr/include/sysexits.h allocates previously unused exit codes from 64 - 78. It may be anticipated that the range of unallotted exit codes will be further restricted in the future.

A Detailed Introduction to I/O and I/O Redirection

A command expects the first three file descriptors to be available. 

The first, fd 0 (standard input, stdin), is for reading.
The other two (fd 1, stdout and fd 2,stderr) are for writing.

There is a stdin, stdout, and a stderr associated with each command. ls 2>&1 means temporarily connecting the stderr of the ls command to the same"resource" as the shell's stdout.


[sankar@new-host ~]$ cat /etc/passwd >&-
cat: standard output: Bad file descriptor

Standard Command-Line Options:

The two most widely-accepted options are:

-h or (--help) ==> Help: Give usage message and exit.

-v or (--version) ==> Version: Show program version and exit.

-r or -R (--recursive) ==> Recursive: Operate recursively (down directory tree).

-v or (--verbose) ==> Verbose: output additional information to stdout or stderr.

Important Files

startup files

These files contain the aliases and environmental variables made available to Bash running as a user shell and to all Bash scripts invoked after system initialization.

/etc/profile

Systemwide defaults, mostly setting the environment (all Bourne-type shells, not just Bash)

/etc/bashrc

systemwide functions and aliases for Bash

$HOME/.bash_profile

user-specific Bash environmental default settings, found in each user's home directory (the local counterpart to /etc/profile)

$HOME/.bashrc

user-specific Bash init file, found in each user's home directory (the local counterpart to /etc/bashrc). Only interactive shells and user scripts read this file. 

logout file

$HOME/.bash_logout

user-specific instruction file, found in each user's home directory. Upon exit from a login (Bash) shell, the commands in this file execute.

data files

/etc/passwd

A listing of all the user accounts on the system, their identities, their home directories, the groups they belong to, and their default shell. Note that the user passwords are not stored in this file, 

But in /etc/shadow in encrypted form.

system configuration files

/etc/sysconfig/hwconf

Listing and description of attached hardware devices. This information is in text form and can be extracted and parsed.

bash$ grep -A 5 AUDIO /etc/sysconfig/hwconf       
 class: AUDIO
 bus: PCI
 detached: 0
 driver: snd-intel8x0
 desc: "Intel Corporation 82801CA/CAM AC'97 Audio Controller"
 vendorId: 8086

Important System Directories

Sysadmins and anyone else writing administrative scripts should be intimately familiar with the following system directories.

• /bin
  Binaries (executables). Basic system programs and utilities (such as bash).
  
• /usr/bin 
  More system binaries.
  
• /usr/local/bin
  Miscellaneous binaries local to the particular machine.
  
• /sbin
  System binaries. Basic system administrative programs and utilities (such as fsck).
  
• /usr/sbin
  More system administrative programs and utilities.
  
• /etc
  Et cetera. Systemwide configuration scripts.
  Of particular interest are the /etc/fstab (filesystem table), /etc/mtab (mounted filesystem table), and the /etc/inittab files.
  
• /etc/rc.d
  Boot scripts, on Red Hat and derivative distributions of Linux.
  
• /usr/share/doc
  Documentation for installed packages.
  
• /usr/man
  The systemwide manpages.
  
• /dev
  Device directory. Entries (but not mount points) for physical and virtual devices.
  
• /proc
  Process directory. Contains information and statistics about running processes and kernel parameters.
  
• /sys
  Systemwide device directory. Contains information and statistics about device and device names. This is newly added to Linux with the 2.6.X kernels.
  
• /mnt
  Mount. Directory for mounting hard drive partitions, such as /mnt/dos, and physical devices. In newer Linux distros, the /media directory has taken over as the preferred mount point for I/O devices.
  
• /media
  In newer Linux distros, the preferred mount point for I/O devices, such as CD/DVD drives or USB flash drives.
  
• /var
  Variable (changeable) system files. This is a catchall "scratchpad" directory for data generated while a Linux/UNIX machine is running.
  
• /var/log
  Systemwide log files.
  
• /var/spool/mail
  User mail spool.
  
• /lib
  Systemwide library files.
  
• /usr/lib
  More systemwide library files.
  
• /tmp
  System temporary files.
  
• /boot
  System boot directory. The kernel, module links, system map, and boot manager reside here.
  
  
  

  
  

  History Commands

  
  

  Bash history commands:
  
  
  

  1. history
  2. fc
     
     

  
  

  bash$ history 1 mount /mnt/cdrom 2 cd /mnt/cdrom 3 ls ...

  
  Internal variables associated with Bash history commands:
  
  
  

  1. $HISTCMD
  2. $HISTCONTROL
  3. $HISTIGNORE
  4. $HISTFILE
  5. $HISTFILESIZE
  6. $HISTSIZE
  7. $HISTTIMEFORMAT (Bash, ver. 3.0 or later)
  8. !!
  9. !$
  10. !#
  11. !N
  12. !-N
  13. !STRING
  14. !?STRING?
  15. ^STRING^string^
      

  
  Unfortunately, the Bash history tools find no use in scripting.
  
  

  #!/bin/bash # history.sh # A (vain) attempt to use the 'history' command in a script. history # No output. var=$(history); echo "$var" # $var is empty. # History commands disabled within a script. bash$ ./history.sh (no output)

  
  
  Converting DOS Batch Files to Shell Scripts
  

  
  

  Table 1. Batch file keywords / variables / operators, and their shell equivalents
  
  
  

  Batch File Operator	Shell Script Equivalent	Meaning
  %	$	command-line parameter prefix
  /	-	command option flag
  \	/	directory path separator
  ==	=	(equal-to) string comparison test
  !==!	!=	(not equal-to) string comparison test
  |	|	pipe
  @	set +v	do not echo current command
  *	*	filename "wild card"
  >	>	file redirection (overwrite)
  >>	>>	file redirection (append)
  <	<	redirect stdin
  %VAR%	$VAR	environmental variable
  REM	#	comment
  NOT	!	negate following test
  NUL	/dev/null	"black hole" for burying command output
  ECHO	echo	echo (many more option in Bash)
  ECHO.	echo	echo blank line
  ECHO OFF	set +v	do not echo command(s) following
  FOR %%VAR IN (LIST) DO	for var in [list]; do	"for" loop
  :LABEL	none (unnecessary)	label
  GOTO	none (use a function)	jump to another location in the script
  PAUSE	sleep	pause or wait an interval
  CHOICE	case or select	menu choice
  IF	if	if-test
  IF EXIST FILENAME	if [ -e filename ]	test if file exists
  IF !%N==!	if [ -z "$N" ]	if replaceable parameter "N" not present
  CALL	source or . (dot operator)	"include" another script
  COMMAND /C	source or . (dot operator)	"include" another script (same as CALL)
  SET	export	set an environmental variable
  SHIFT	shift	left shift command-line argument list
  SGN	-lt or -gt	sign (of integer)
  ERRORLEVEL	$?	exit status
  CON	stdin	"console" (stdin)
  PRN	/dev/lp0	(generic) printer device
  LPT1	/dev/lp0	first printer device
  COM1	/dev/ttyS0	first serial port

  Batch files usually contain DOS commands. These must be translated into their UNIX equivalents in order to convert a batch file into a shell script.

  
  

  Table 2. DOS commands and their UNIX equivalents
  
  
  

  DOS Command	UNIX Equivalent	Effect
  ASSIGN	ln	link file or directory
  ATTRIB	chmod	change file permissions
  CD	cd	change directory
  CHDIR	cd	change directory
  CLS	clear	clear screen
  COMP	diff, comm, cmp	file compare
  COPY	cp	file copy
  Ctl-C	Ctl-C	break (signal)
  Ctl-Z	Ctl-D	EOF (end-of-file)
  DEL	rm	delete file(s)
  DELTREE	rm -rf	delete directory recursively
  DIR	ls -l	directory listing
  ERASE	rm	delete file(s)
  EXIT	exit	exit current process
  FC	comm, cmp	file compare
  FIND	grep	find strings in files
  MD	mkdir	make directory
  MKDIR	mkdir	make directory
  MORE	more	text file paging filter
  MOVE	mv	move
  PATH	$PATH	path to executables
  REN	mv	rename (move)
  RENAME	mv	rename (move)
  RD	rmdir	remove directory
  RMDIR	rmdir	remove directory
  SORT	sort	sort file
  TIME	date	display system time
  TYPE	cat	output file to stdout
  XCOPY	cp	(extended) file copy

•