CS330 Shell Scripts


Highlights of this lab:


Different Shells and Script Intro

There are several different shells, they offer their own advantages and disadvantages. For instance, some allow for auto completion using the tab key; others don't.

A few common shells are the following:

For more on shells, see http://docstore.mik.ua/orelly/linux/lnut/ch08_07.htm or http://www.injunea.demon.co.uk/pages/page203.htm.

For more details about tcsh, the default shell on Hercules/Linux, click here

To see what shells exist on your current Unix system, try the following command:

$ cat /etc/shells

From the command line, you can type various commands which have the same responses in each of these shells. For instance chmod, ls, cd, and touch.

Let's say that you want to type the same commands over and over again. If that it the case, it is good to create a shell script.

For example, you could create a basic shell script by:

  1. Typing the following lines into a file called people.sh:
    ls -l
    mkdir tom
    cd tom
    touch greta monica george
  2. Making the file executable:
    chmod +x people.sh
  3. Executing the file at the command line:
    % people.sh
GOOD HABIT- To better identify your shell scripts from other files you can add .sh as an extension (however, it is not required)

Executing people.sh will do the following: first do a long listing of the current directory, then make a directory called tom, and create files: greta, monica, and george under the tom directory.

We can also add some comments to this file. Any line preceded with # is a comment (or anything after the # is ignored):

#list the files in the current directory
ls -l
 
#make a directory called tom
mkdir tom
 
#change into the tom directory and create three files.
cd tom
touch greta monica george

I've led you to believe that all shells are created equal. That is not the case. There is a major split between Bourne-compatible shells (such as sh and bash) and csh-compatible shells (such as csh and tcsh).

These differences show up in such things as conditional statements and loops. The following table shows the basic syntax for an if and while statement in both tcsh and bash

tcsh bash
if (expression) then
    commands
endif 
if [expression]
then
    commands
fi
while (expression)
   commands
end
while [expression]
do
   commands 
done

The difference is very subtle, but it is enough to create problems.

For the remainder of this lab, we will be focusing on tcsh shell scripts.


Starting off with a She-bang

In the first section, we stated that there are several shells and corresponding shell scripts. The question is: how do you indicate which shell environment you are using to execute the commands?

The answer is: there is a special notation used on the first line of the shell script. The line begins with a number sign (#), then an exclamation point (!), followed by the full path name of the shell on the computer's file system.

For instance, to state that you will interpret the above commands with the tcsh shell (on our Linux system), you would add the following to the beginning of the file:

#!/bin/tcsh
ls -l
mkdir tom
cd tom
touch greta monica george

The location of this tcsh shell is dependent on the system. You can verify the location by using:

$ cat /etc/shells

Where did she-bang come from?

There are a couple of different stories:


tcsh Variables

What would a program be without variables? tcsh shells have their own way of handling variables.

Some rules about variables are the following:

There is a difference in shell scripts between the name of a variable and its value:

Assignment can be made in two major ways:

  1. with set and = ( Example:  set myvar=2 )
  2. in a loop ( Example:  foreach myvar (1 2 3) )

Note 1: When setting variables in tcsh shells, make sure that there you are consistent with spacing on both sides of the equal sign.
(for instance set var1=2 is good but set var1 =2 will not work)

Note 2: $myvar is actually a simplified form of ${myvar}. Some situations may require the longer usage.

An example of assigning and using variables is the following code (echo allows us to print to the screen):

#!/bin/tcsh
#file: variables.sh


set var1="My string with spaces"
echo "var1 is: " $var1
echo


echo "Enter a number: "
#notice how we read input from the keyboard
set var2=$<
echo "var2 is: " $var2
echo

foreach var3 (1 2 3)
        echo "var3 is: " $var3
end

A sample run would yield the following results:

% variables.sh
var1 is: My string with spaces
Enter a number:
30
var2 is: 30
var3 is: 1
var3 is: 2
var3 is: 3

Quotes

In the topic of quotes, we will discuss four special characters:

Double quotes

These do not interfere with variable substitution. They are sometimes referred to as weak quotes.

For instance:

set var1=20
echo "This is $var1"

Will print: This is 20

Single quotes

These cause the variable name to be used literally. They prevent the shell from interpreting special characters. This is sometimes known as full quoting or strong quoting.

For instance:

set var2=30
echo 'This is $var2'

Will print: This is $var2

Backslash character

The backslash is another way of preventing the shell from interpreting special characters. It only works on the character immediately following the backslash. We can also say that the backslash escapes the special meaning of the succeeding character.

For instance:

#Escape the normal meaning of the space character
set var3=The\ price\ is--
 
#Escape the normal meaning of a $
set var4=\$20.00
 
echo $var3 $var4

Will print : The price is-- $20.00

Note that without the escape from the space, the shell will try and interpret "price" and "is--" as commands.

Back quotes

These enable an expression to be evaluated as a command, and replaced with whatever the expression prints to its standard output. Back quotes are sometimes also known as back ticks.

For instance:

set var5=`date`
echo "The date is $var5"

Will print: The date is Fri Jan 14 13:16:45 CST 2011

Note that on the keyboard, the (`) key is usually on the same key as the (~)

Conditionals

There are two conditional structures:

The If Statement

The basic format for the if statement is the following:

if (expression) simple command 

OR

if (expression) then    
    ...  
else   
    ...  
endif  

Where expression has built-in operators. The operators can be classified into four different groups:

The following chart summarizes these operators:

Integer Comparisons
int1 > int2 Greater-than
int1 < int2 Less-than
int1 >= int2 Greater-than-or-equal-to
int1 <= int2 Less-than-or-equal-to
int1 == int2 Equal to
int1 != int2 Not-equal to
String Comparisons (Note: space on either side of operator)
str1 == str2 Equal strings
str1 != str2 Not-equal strings
str1 =~ glob-pattern Equal to glob-pattern
str1 !~ glob-pattern Not equal to glob-pattern
Logical Operations
expr1 && expr2 Logical AND (true if expr1 and expr2 are true)
expr1 || expr2 Logical OR (true if expr1 or expr2 are true)
! expr Logical NOT
File Tests (for more see: tcsh org)
-f file File exists and is a regular file
-z file File is empty
-s file File is not empty
-r file File is readable
-w file File can be written to, modified
-x file File is executable
-d file Filename is a directory name
-l file Filename is a symbolic link
-e file File exists
-b file Filename references a block special file

The Switch/Case Statement

The basic format for the switch/case statement is the following:

switch (string)
  case pattern1:
    commands...
    breaksw
  case pattern2:
    commands...
    breaksw
  default:
    commands...
    breaksw
endsw

Some sample code making use of if and case statements is the following

#!/bin/tcsh
#file: conditions.sh

if ( -f ./conditions.sh ) echo "file exists"

#prompt and then read from the keyboard
echo "Enter a letter "
set var1 = $<

if ($var1 == 'a') then
        echo "a selected"
else
        echo "a not selected"
endif

#prompt for a string and read it in
echo "Enter a string "
set var2 = $<

#check if it is hello or has bye at the end
switch ($var2)
        case [hH][eE][lL][lL][oO]:
                echo "hello found"
                breaksw
        case *[bB][yY][eE]:
                echo "bye found"
                breaksw
        default:
                echo "nothing found"
                breaksw
endsw

#prompt for a number and read it in
echo "Enter a number "
set var3 = $<

#print the range that the number appears in
if ( $var3 >= 1 && $var3 <= 9 ) then
        echo "1 to 9"
else if ( $var3 >= 10 &&  $var3 <= 19 ) then
        echo "10 to 19"
  else if ( $var3 > 19  ||  $var3 == 0 ) then
        echo "greater than 19 or equal to 0"
    else
        echo "other range"
    endif
  endif
endif

 

You can also use the exit status of commands in conditions. Be careful that you are checking the status variable directly after excuting the command. An example follows:

#!/bin/tcsh
#file conditions2.sh

set user=turkey

#-q is quiet mode (supresses output)
#need to set a different path for hercules

if ($HOST == hercules) then
        set cmd=/usr/xpg4/bin/grep
else
        set cmd=grep
endif

#execute the grep with output supressed
$cmd -q $user /etc/passwd

if ( $status == 0 ) then
        echo "$user has an account"
else
        echo "$user doesn't have an account"
endif

Note: When using an operator, ensure that there is a space on either side of that operator.


Loops

We will go over the syntax of two looping structures:

The While Loop

The basic format for the while loop is the following:

while (expr)
   commands
end

As you would expect, the while loop executes the commands as long as the condition is true.

The Foreach Loop

The basic format for the foreach loop is the following:

foreach var (wordlist)
   commands
end

The foreach loop iterates over all of the elements in a list. As it is iterating, var will be assigned the value of each item in the list in turn.

For both of these loops, break and continue may be used to terminate or continue the loop prematurely

The following is an example of these two looping structures:

 #file: loops.sh
set class_mailing_list='/net/share/aliases/CS330-001.201210'

#A while loop with user input
echo "Enter a letter or x to quit "
set var1=$<

while ( $var1 != "x" )
   echo "Enter a letter or x to quit "
   set var1=$<
end


#A for loop echoing the contents of file
if ( ! -f $class_mailing_list) then
   echo Class list not found.
   exit 1

else
   foreach var2 (`cat $class_mailing_list`)
      echo $var2
   end
endif

Variable and Substitution Modifiers

Sometimes it is useful to strip off parts of a file path or name (for example, get the extension or remove the extension on a filename). You can use modifiers in tsch to to that. The following table summarizes a few modifiers. The example column is based on:

set a = "/a/b/c.d.e.f.g"

Modifier Meaning Example Results
r Root $a:r /a/b/c.d.e.f
e Extension $a:e g
h Head (or Directory) $a:h /a/b
t Tail (or Filename) $a:t c.d.e.f.g
s Substitute $a:s/a/z/ /z/b/c.d.e.f.g

(taken from: http://www.grymoire.com/Unix/Csh.html#uh-88)

Note:

The following example changes all *.JPG files to *.jpg and truncates the home path

#!/bin/tcsh
# file: truncate.sh

#first create some bogus *.JPG files
touch me.JPG you.JPG harry.JPG

#view them
ls -l *.JPG

#rename them
foreach i ( *.JPG )
   set newname=$i:r
   echo $newname
   set newer=$newname.jpg
   mv $i $newer
     #or
   #mv $i $i:r.jpg
     #or
   #mv $i $i:s/JPG/jpg/
end

#display your current working directory and modify
set var1=$PWD
echo $var1
set newpath=$var1:s#/home/hercules/##
echo $newpath
A sample run does the following:
% truncate.sh
-rw------- 1 temp0 temp 0 Nov 25 01:37 harry.JPG
-rw------- 1 temp0 temp 0 Nov 25 01:37 me.JPG
-rw------- 1 temp0 temp 0 Nov 25 01:37 you.JPG
harry
me
you
/home/hercules/t/temp0/330
/t/temp0/330 

Arithmetic Evaluations

Sometimes you might perform arithmetic operations on variables. To do this, you use the @ in combination with an arithmetic expression.

You have two kinds of operators:

The following table summarizes some of these operators:

Arithmetic Operators
* multiplication
/ division
+ addition
- subtraction
% modulo-results in the remainder of a division
++ increment operator
-- decrement operator
Relational Operators
> greater-than
< less-than
>= greater-than-or-equal-to
<= less-than-or-equal-to
= equal in expr
== equal in let
!= not-equal
& logical AND
| logical OR
! logical NOT

An example of several of these are in the following sample code:

#!/bin/tcsh
# file: arithmetic.sh

@ num1 = 3
@ num2 = 10

echo $num1
echo $num2

@ res=$num1 + $num2
echo "num1 + num2 = $res"

@ res2 = $num1 * 5
echo "num1 * 5 = $res2"

@ res2 = $num1 % 5
echo "num1 % 5  = $res2"

#Decrement or increment
@ res2++                     #increment operator
#@ res2--                      #decrement operator

echo $res2


#Logical assignment
@ mybool=!($res2 == $res)

#@ mybool=($res2 > $res)
echo $mybool

A sample run of the program would look like this:

% arithmetic.sh
3
10
num1 + num2 = 13
num1 * 5 = 15
num1 % 5  = 3
4
1

Notes:


Command Line Arguments

When you run a shell program that has command line options, each of the options are stored in a positional parameter. The idea is much the same as the argv[] in C programming. For instance, the command name (the name of the shell program) is stored in a variable called 0, the first command line argument is stored in a variable called 1 (or argv[1]), the second argument is stored in a variable called 2 (or argv[2]), and so on.

The following table contains a list of built in variables that are related to the command line.

Variable Use
$# Stores the number of command line arguments that were passed to the shell program
$? Stores the exit value of the last executed command
$0 Stores the first word of the entered command, which is the name of the shell program
$* Stores all the arguments that were entered on the command line ("$1 $2 ...") also $argv or $argv[*]

A sample program using the positional parameters is the following:

#!/bin/tcsh
#command.sh

if ( $# < 3 ) then
        echo "Please include three command line arguments"
else
        echo "First one: $1"
        echo "Second one: $2"
        echo "Third one: $3"
endif

A sample run might do the following:

% command.sh testing this command
First one: testing
Second one: this
Third one: command

Pipes and Redirections

In shell scripts, you can use pipes and I/O redirections as you would on the command line. For instance, you can combine commands (such as awk and grep) with the pipe (|), and you can redirect standard output and/or standard error to files.

You have use of three default file descriptors:

The following chart summarizes some redirectors (taken from Linux in a Nutshell). Commands in blue are csh only, whereas commands in green are bash only. Note that bash's has more advanced abilities to redirect file descriptors to different places.

  Redirector   Function
> file Direct standard output to file.
< file Take standard input from file.
cmd1 | cmd2

Pipe; take standard output of cmd1 as standard input to cmd2.

>> file

Direct standard output to file; append to file if it already exists.

>| file
>! file
Force standard output to file even if noclobber is set.
n>| file

Force output from the file descriptor n to file even if noclobber is set.

<> file Use file as both standard input and standard output.
<< text

Read standard input up to a line identical to text (text can be stored in a shell variable). Input is usually typed on the screen or in the shell program. Commands that typically use this syntax include cat, echo, ex, and sed. If text is enclosed in quotes, standard input will not undergo variable substitution, command substitution, etc.

>& file Direct standard error as well as standard output to file.
>&! file Direct standard error as well as standard output to file even if noclobber is set.
n> file Direct file descriptor n to file.
n< file Set file as file descriptor n.
>&n Duplicate standard output to file descriptor n.
<&n Duplicate standard input from file descriptor n.
&>file Direct standard output and standard error to file.
<&- Close the standard input.
>&- Close the standard output.
n>&- Close the output from file descriptor n.
n<&- Close the input from file descriptor n.

The following bash script provides a sample of three of these uses:

#!/usr/local/bin/bash
# file: pipes.sh


#To redirect standard output to standard error:
#only for this one command
        echo
        echo "Redirecting stdout to stderr"
        echo "Usage error: see administrator" 1 >&2

#send the files found (stdout) to a file "filelist"
#send standard error to a file "no_access"
#try the find without the redirection to see what the output looks like
        echo
        echo "Using find..."
        echo "Redirecting stdout to \"filelist\" and stderr to \"no_access\""
        find / -name "linux" -print >filelist 2>no_access

#a pipeline with two filters
#prints out temp0's home directory
        echo
        echo "Using a pipe to find temp0's home directory"
        grep temp0 /etc/passwd | awk -F: '{print $7}'


References

 

Some references that I've been using:

Reasons for not using C-Shell for scripting

Really cool page with comparisons of different shells' syntax