START OF WEEK 5 of Linux
<<< FILE TIMESTAMP >>>
-The up to date file is: /usr/courses/cps393/dwoit/courseNotes/
-If you are viewing your own copy, check it is up to date
-If you are viewing from a link in the Course Outline, be aware it may be outdated.
<<< TESTING & DEBUGGING SHELL PGMS >>>
---------------------------------
use shell parameter -x to debug interactively
bash -x pgm #displays every command in pgm just before executing it,
#with variables replaced by their actual values
#bash -v pgm same except variables NOT replaced by values
> cat fromto.sh
#!/bin/bash
#takes 3 CLAs, Filename, S and E
#displays lines S-E inclusive of file Filename, but
#in reverse order (line E, line E-1, ... Line S)
#ASSUME proper CLAs, so no error checking.
File=$1 ; S=$2 ; E=$3
cat ${File} | head -$E | tail -$(($E-$S+1)) | tac
>
>
> fromto.sh gfile 2 3 #run it normally
ccc
BBBaaaaaaaaBBBBBBBBBBBB
>
> bash -x fromto.sh gfile 2 3 #run it using -x
+ File=gfile
+ S=2
+ E=3
+ cat gfile
+ head -3
+ tail -2
+ tac
ccc
BBBaaaaaaaaBBBBBBBBBBBB
>
Program findtrunc.sh is supposed to truncate its $1 to 8 characters,
and then print Found if the resulting (truncated) string is found
in file gfile, and Not Found otherwise:
> cat gfile
abc11111111abcd22222 xxx yyyy z
BBBaaaaaaaaBBBBBBBBBBBB
ccc
dddddddd
>
> findtrunc.sh 11111111234567
Found
> findtrunc.sh ddddXddd
Not Found
>
Program findtruncBAD.sh is supposed to do the same thing. But
it always prints "Not Found" no matter what $1 is.
>
> findtruncBAD.sh 11111111234567
Not Found
>
You look at the code for findtruncBAD.sh but can't see any problem:
#!/bin/bash
#Source: findtruncBAD.sh truncates $1 to 8 characters and
#searches for it in file gfile
# a msg is printed to indicate if Found/Not Found
trunc=$(echo "$1" | cut -c1-8)
if [ "$(grep trunc gfile)" ]
then echo Found
else echo Not Found
fi
To figure out the problem with findtruncBAD.sh, run it with bash -x
> bash -x findtruncBAD.sh 11111111234567
++ echo 11111111234567
++ cut -c1-8
+ trunc=11111111
++ grep trunc gfile
+ '[' '' ']' <--- the condition evaluated to NULL string!
+ echo Not Found so grep did not find trunc in gfile
Not Found
>
So we see grep did not find trunc in gfile. Why not?
Check that gfile contains what you think it does. Yes.
Check that grep 11111111 gfile DOES produce output when run in shell. Yes.
Something must be wrong with this line of code: grep trunc gfile
Ah ha. It is searching for the literal string 'trunc' in gfile, instead of
the value of the variable named trunc.
Line should be: grep $trunc gfile
A year later, you have forgotten how to run findtrunc.sh, so you try this:
> findtrunc.sh
but you get no output. It just hangs there.
To stop it:
^C and it terminates, or
^D and it prints "Not Found"
What is the problem with findtrunc.sh? Why does it hang?
Use -x to find out why:
> bash -x findtrunc.sh
++ echo ''
++ cut -c1-8
+ trunc=
++ grep gfile <---- hangs, because grep is searching for string
gfile from stdin. If type ^D signals
end of stdin, giving...
+ '[' '' ']' <---- since grep printed nothing on stdout
+ echo not found <---- since [ ] is false
Not Found
bash -x helps us tell where the program was hanging, and helps us find
the problem. In this case, there's no problem with the given code. The
problem is that the user forgot the argument to findtrunc.sh, so it was null
Could fix this by adding quotes around "$trunc" as in:
if [ "`grep "$trunc" gfile`" ]
But this is not ideal because it will print Found if $1 is null, since
null string is everywhere!
Better to fix this by adding a conditional that makes sure user has
supplied an argument. See findtruncGOOD.sh
-----------------------------------------------------
| Aside
| Might try to use read to populate variable trunc, as in:
| echo "$1" | cut -c1-8 | read trunc
| but can not work due to a bash "feature" (expl. in CPS590)
|
| However, could instead use a here-string, as in:
| read trunc <<< $(echo "$1" | cut -c1-8)
|
| Or, could use ( ) or { } to extend scope as in:
| echo "$1" | cut -c1-8 | ( read trunc
| if [ "`grep $trunc gfile`" ]
| then echo Found
| else echo not found
| fi
| )
|
| Or, could use a while, as in:
| echo "$1" | cut -c1-8 | while read trunc
| The loop would only run once, but would work OK.
|
| Or, could use process substitution, as in
| read trunc < <(echo "$1" | cut -c1-8)
| Essentially takes stdin from a temporary named pipe
| (expl in CPS590)
-----------------------------------------------------
HMWK:
Programs /usr/courses/cps393/dwoit/courseNotes/Programs/linux/errorPgms/primeGood*.sh
read user input (a number) and print Prime if that number is a prime number, and
Not Prime otherwise.
Try PrimeGood.sh with numbers from 1-10 to verify that only these are prime:
2, 3, 5, 7
Try primeBad.sh and notice it ALWAYS prints Not Prime for every input.
Use bash -x primeBad.sh to help you figure out why.
Use bash -x to figure out why primeBad1.sh also produces incorrect output.
<<< SHIFT >>>
-------------
-shifts positional parameters ($1 $2 $3 ...) one to the left.
so supposing $1 $2 $3 $4 were: a b c d
then after shift $1 $2 $3 $4 are: b c d ''
-shift n shifts n positions to the left
so supposing $1 $2 $3 $4 were: a b c d
then after shift 2 $1 $2 $3 $4 are: c d '' ''
#!/bin/bash
#Source: shiftex.sh
#pgm to print its args.
while [ "$1" ] #<--- stops when $1 is null
do
echo "$1" #<--- so spaces preserved
shift #<--- moves $2 to $1, $3 to $2 etc.
done
exit 0
Helpful when you need to loop over the tail of the CLAs.
e.g.,
> repl.sh str1 str2 f1 f2 f3 f4 f5 f6 f7
replaces str1 by str2 in all the files f1, f2, etc.
repl.sh can accomplish this by:
-Save $1 and $2 in variables and then
-shift 2
-for i ; do
HMWK:
Write a program that replaces one string with another in
one or more files. The 2 strings and the file(s) are given
on the command line:
>repl.sh abc "A B C" rf1 rf2 rf3 rf4
The program must use shift so that the commmand line arguments
shift 2 over, enabling it to use this loop to process the
files: for i in "$@" #either this
for i #or this
It must use sed -i to replace strings in place
(so the files are modified).
<<< XARGS >>>
---------
-to perform a command on a group of things from stdin
-helps you take output of one command, and pass it as an
argument to another command
e.g., list lines containing 'bash' from all files whose NAME
starts with 'f', where files are located anywhere in the
filetree under current directory
> find . -type f -name "f*" | xargs grep bash
#prints LINES in files f* that contain the string bash
#assuming all files starting with f were: ./f1, ./f2, and ./f3
#ends up running: grep bash ./f1 ./f2 ./f3
#Note that without xargs, it does something much different
> find . -type f -name "f*" | grep bash
#prints NAMES of files f* whose NAME also contains string bash,
#like: f1bashy.txt, fashybashy.sh, etc.
> find . -type d -empty | xargs rmdir
#remove empty directories from filetree under current dir
#assuming the only empty directories were ./a, ./b/c, ./d
#ends up running: rmdir ./a ./b/c ./d
#Note that without xargs, get error: rmdir: missing operand
#The -p option PROMPTS before doing the command.
#Assuming same as above, except adding -p (enter y or n)
> find . -type d -empty | xargs -p rmdir
rmdir ./a ./b/c ./d ?...y #<--user typed the y for "yes"
#now those 3 dirs are deleted (if type n rmdir does not run)
#If no command is given, xargs uses echo by default. e.g.,
> ls | xargs
#assuming current dir contains only files a, b, c, d
#ends up running: echo a b c d
#The -ni option says instead of sending stdin as one string,
#split the string into chunks of i words
> ls | xargs -n2
#assuming current dir contains only files a, b, c, d
#ends up running: echo a b ; echo c d
#both -n and -p together
> echo abc de fghi jkl mn | xargs -n2 -p
echo abc de ?...y
abc de
echo fghi jkl ?...n
echo mn ?...y
mn
>
#option -0 (zero) processes whitespace in items properly.
#often used with -print0 (zero) option of find (which produces
#output with whitespace properly preserved
#e.g., suppose files included these 3 files which all contained string bash:
# f0 f1 'f s'
#To find files starting with f which contain string bash:
> find . -type f -name "f*" -print0 | xargs -0 grep bash
#If don't use -print0 and -0 the whitespace causes errors
#To delimit input, xargs uses blanks and newlines
#-d changes input delimiter (a single character only)
> echo "abc.d.efghi jkl.m.nop" | xargs -d. -n2
abc d
efghi jkl m
nop
The -I option runs the command once for each item in stdin (like -n1)
except each time, {} in command is replaced with item:
ls | xargs -I{} cp {} {}.bak # make backup copy of
#all files in current dir.
#assuming current dir contains only files a, b, c
#ends up running: cp a a.bak ; cp b b.bak ; cp c c.bak
#could have done this with a loop but more typing:
# ls | while read file ; do cp $file ${file}.bak ; done
Note: if b is a dir get cp b b.bak -- error, but xargs just
prints msg on stderr and continues on with next item
e.g., what does this print on stdout?
(echo a b c ; echo d e ; echo f) | xargs -I{} echo :{}:
HMWK:
1. write a shell program called clean that removes all files called
"core" from your directory structure, and then removes all files that
have size 0 (0 bytes) from your dir structure. That is your WHOLE
dir structure, not just the current dir.
(note: "find" can find files that are zero bytes long)
HMWK:
1.write a shell program called avgs which will read lines from
stdin such as the following (where the dots indicate more lines
of the same kind:
SID LNAME I T1/20 T2/30
92876035 WANG S 15 26
95908659 CHIANG R 10 29
. . . . .
. . . . .
. . . . .
91234987 MYRTH R 15 16
Your shell program will print out a line such as:
Average of T1/20 is 17 and of T2/30 is 24
where 17 and 24 are the averages of the last 2 columns
respectively. Note that your program must keep a total
and count for each of the last 2 columns, and must not
include data from the first line in the totals and counts.
2. Modify avgs so that the "title" line, e.g.,
SID LNAME I T1/20 T2/30
above, could be located at ANY LINE of stdin (ie., not
necessarily the FIRST line, as above.)
3. Modify avgs so that the highest and lowest marks are
printed for each of T1 and T2, as well as averages.
Your program should produce output such as the following:
T1/20: Average: 17 Highest: 19 Lowest: 1
T2/30: Average: 24 Highest: 29 Lowest: 9
4. Modify avgs so that the output above is in PERCENT, e.g.,
T1/20: Average: 85% Highest: 95% Lowest: 5%
T2/30: Average: 80% Highest: 96% Lowest: 3%
If variable x contains "T1/20" you could use an echo piped to
a cut, and assign the result to another variable, or you
could use ${x#*/} which will match pattern "*/" against the
beginning of the contents of x, delete the shortest part that
matches, and return the rest. Thus declare -i y=${x#*/} would
assign 20 to y. Note that pattern */ means anything followed by
a "/"
5. Write a shell program called findext which will list directories
under a given path that contain files/dirs having the given extensions.
Program findext takes at least 2 command line arguments. Its usage is:
findext path ext1 ext2 ...
where "path" is the path to the start of the directory structure
you wish to search, and ext1, ext2, etc are extensions.
Program findext will print out the directories under path which
contain files/dirs that end in .ext1 .ext2 etc
Your program does not need to work properly if the user passes glob
constructs or special chars as arguments.
For example:
findext /home/dwoit/courses c scm
will list all directories under /home/dwoit/courses that contain
files/dirs named *.c and/or *.scm
If findext is sent less than 2 command line args, it should print
on stderr
Usage: findext path arg1 arg2
and exit with exitcode 1
(the word findext above should be printed using $0)
If it is passed 2 or more arguments, it should print a list of dirs
in which files/dirs of the form *.ext1 *.ext2 etc reside. Then it
should exit with exitcode 0.
Note that only the DIRECTORY names are printed. Therefore you will
have to chop off the trailing file/dir name (the part after the last "/")
You can do this with ${var%/*} which says to match pattern "/*" (slash
followed by anything) at the END of variable var; and delete the shortest
part that matches and return the rest. You will also find utilities
sort and uniq useful.
<<< BUILT-IN COMMANDS >>>
-------------------------
Saw some already: echo, exit, pwd, shopt, break, cd, test, read, declare, etc
(man bash and search for "^SHELL BUILTIN COMMANDS")
<< EVAL >>
eg) v1="cat fn | grep 5"
#suppose want to run the command in v1. Try:
$v1
#but get file fn catted to screen, followed by:
cat: '|': No such file or directory
cat: grep: No such file or directory
cat: 5: No such file or directory
#Why? Because bash thinks you want to cat 4 files: fn, |, grep, 5
#The "|" is not recognized as a pipe.
#solution:
eval $v1
eg) v1="cat fn | grep 5"
v2="grep 3 | more "
#we wish to execute the command: cat fn | grep 5 | grep 3 | more
#so we try this:
$v1 | $v2
#but it does not work. Get errors because shell does not
#recognize 1st and 3rd "|" as pipes (since part of variables).
#errors include:
grep: |: No such file or directory
grep: more: No such file or directory
eval "$v1 | $v2" #sends lines of fn with both "5" and "3" in
#them to more
eval: 2 passes:
1) makes substitutions
eval "cat fn | grep 5 | grep 3 | more"
2) result evaluated i.e, meta chars etc. interpreted properly
#!/bin/bash
#Source: printXth.sh
#Expects a bunch of command line arguments.
#Prompts user and reads an integer (call it X)
#then it prints the Xth command line argument
echo -n "enter an integer: "
read X
the_arg='$'${X}
#note the difference in output:
echo "command line arg number ${X} is: ${the_arg}"
eval echo "command line arg number ${X} is: ${the_arg}"
#!/bin/bash
#Source: looptail.sh
#loops over CLAs $3,$4,$5,... (skipping $1 $2)
#without using shift
#First, create the string: $3 $4 $5 ... $#
#Then loop over the VALUES of the string
#works if spaces in CLAs (solutions using 'read' will not)
declare -i j=3
loopStr=""
#loop for j from 3 to $# and each time add ' $j' to loopStr
while [ $j -le $# ] ; do
loopStr="${loopStr} \$$j"
j=j+1
done
#loopStr contains $3 $4 $5 ... $#
for i in $(eval echo $loopStr) ; do #eval replaces $i with value
echo $i
done
Recall {1..3} generates list 1 2 3
> echo {1..3}
1
2
3
>
But only if start and end match
> echo {1..R}
{1..R}
>
Useful in loops:
> for i in {1..4} ; do echo $i ; done
1
2
3
4
>
Want to do the same, but instead of 4 use value in $X
However, it does not produce the same output as above.
> X=4
> for i in {1..$X} ; do echo $i ; done
{1..4}
>
Fix this by using eval:
> for i in $(eval "echo {1..$X}" ) ; do echo $i ; done
1
2
3
4
>
HMWK:
Use eval to write a shell program called revargs which will
print out its command line args in reverse order.
White-space preservation not necessary.
Write a shell program the does not use eval to print its
command line args in reverse order.
<<< SHELL CMDS >>>
-------------------
most shell cmds (not builtin ones) in "bins"
/bin, /usr/bin , /usr/local/bin , even ~/bin
shell searches directories to find cmds.
searches those in "PATH"
to see path:
> echo $PATH
/usr/bin:/bin:/etc:/usr/local/bin::
^
|
current dir
To add (or change) default path, reset PATH in .bash_profile
(or, .profile if that is what you use instead)
Also add to .bashrc for non-interactive, non-login shells,
which you only get by opening a new terminal window from GUI
once you are already logged into your machine (your linux laptop,
a lab machine booted to linux). Putty windows are login shells,
so .bashrc not relevant. Note Mac OS X’s Terminal.app runs
.profile for both types of windows.
PATH=${PATH}:${HOME}/bin
export PATH
- Adds /home/dwoit/bin to end of current path (for prof)
(prof keeps some personal shell pgms in /home/dwoit/bin)
-export makes PATH global
(so all subsequent processes can use it.)
If 2 different cmds with same name, shell executes 1st one
it finds in path
i.e. if you write your own "rm" in ~dwoit/bin/rm to use instead
of /bin/rm you must put YOUR ~dwoit/bin before /bin
in PATH. However, the best way to accomplish getting
your own rm run is to make it a function or alias, which are
always done first
<<< SHELL VARIABLES >>>
------------------------
problem: referencing a variable before it is
assigned a value (i.e. programmer error)
it will be null and shell pgm may behave unexpectedly
e.g., grep "$var" fname
If $var unset, get grep "" fname, which prints all lines of fname
which is probably not what you want
set -u # if any variable unset, referencing it causes error message
set +u # no error message for unset var
fix above:
set -u
grep "$var" fname
-bash: var: unbound variable
can set DEFAULT values:
${var:=value} #if var empty, assign value; if !empty no new assignment
> unset X
> echo ${X:=abc}
abc
X=def
> echo ${X:=abc}
def
>
for more default values see man bash (search for := )
Can use set to set value of $1 $2, $3, etc. in shell program
or on command line
set "val1" "val2"... #sets $1 to "val1", $2 to "val2" ...
set -- #unsets them all
set - * #sets $1, $2, $3 ... to all files in current dir
<<< HERE DOCUMENTS >>>
#!/bin/bash
#Source: menupgm.sh
#prints a menu and executes selected choice
#reprints menu for another selection
#continues until user selects the exit option
#
#illustrates use of "here document"
# -any string could replace string "here" in pgm
# -uses following lines as stdin until
# the string "here" appears in col. 1
#
while [ true ]
do
clear
#print menu display
cat << here
MAIN MENU
1) Print working dir.
2) List all entries in current dir.
3) Print date & time
4) Display contents of file
5) Create backup files
X) Exit
here
#prompt for user input using continuation char
echo -e "Please enter selection ${LOGNAME}: \c"
read selection
case $selection in
1) pwd
;;
2) ls -1
;;
3) date
;;
4) echo -e "Enter a file name: \c"
read fname
if [ -f $fname ]
then
echo "Contents of $fname are: "
more $fname
else
echo "file $fname does not exist"
fi
;;
5) echo -e "Enter filenames: \c"
read fnames
for fn in $fnames
do
cp $fn $fn.bak
done
;;
X) exit 0
;;
*) echo -e "Invalid choice. Try again \a"
;;
esac
#pause before redisplaying menu
echo -e "\n\n Press return to continue \c"
read hold
done
exit 0
Can use a here-document interactively in shell:
> wc <<HERE
Linux> here we have
Linux> exactly 3 lines
Linux> of text
HERE
3 8 37
>
Here documents are expanded like double-quoted strings. Expands:
$var, $(command), $((X+1)), and backslash to protect \`$
> X=5
> cat <<HERE
Linux > value of X: $X
Linux > date is: $(date)
Linux > X+1 is: $((X+1))
Linux > HERE
value of X: 5
date is: Sat 16 Nov 2024 03:38:33 PM EST
X+1 is: 6
>
Can suppress expansion by single-quoting HERE (or whatever you use)
> cat <<'HERE'
Linux > value of X: $X
Linux > date is: $(date)
Linux > X+1 is: $((X+1))
Linux > HERE
value of X: $X
date is: $(date)
X+1 is: $((X+1))
>
<<< ARGUMENT PROCESSING >>>
---------------------------
commands usually have options of form -x, with some options
requiring an additional argument
eg) ls -F -t
eg) wc -lw fname1 fname2
eg) gcc -o pgm pgm.c
#!/bin/bash
#Source: argp.sh
#process single arguments which can be -a, -b or -f
#arg -f expects filename after it
#e.g., argp.sh -b -f myfile -a
while [ "$1" ] ; do
case $1 in
-a | -b)
echo arg is -a or -b
#process these options
;;
-f)
#process option -f filename
if [ "`echo $2 | cut -c1`" = "-" -o -z "$2" ] ; then
echo "missing filename for -f"
elif [ "$2" ] ; then
# process -f filename
echo arg is -f $2
shift
fi
;;
*)
echo "$1 invalid argument"
;;
esac
shift
done
exit 0
<<< GETOPTS >>>
----------------
getopts cmd can parse args of a shell pgm.
getopts reads arguments 1 by 1 & puts in c (you name it)
? is assigned to c if arg not in list you provide
#!/bin/bash
#Source: gopsPgm.sh
#parses command line arguments using getops command
#getopts list arg
#list is a list of valid options, e.g., below they
# are a,b,d or o
#the ":" after any char in list indicates that option
# requires an argument.
#so valid usage of gopsPgm.sh is:
# gopsPgm.sh -o dog
# gopsPgm.sh -o "dog mouse"
# gopsPgm.sh -d "abce" -b
# gopsPgm.sh -a -o dog -b
# gopsPgm.sh -ab -o dog -d mouse
#invalid are:
# gopsPgm.sh -o
# gopsPgm.sh -x
#in these cases, error messages are printed automatically and $c is
#set to ? so that it matches the last part (so both exit 1)
while getopts abd:o: c #when done returns false
do
case $c in
a | b) #do some processing here
echo "in case a|b, option is $c"
;;
d) # $OPTARG is now whatever came
# after the -d
echo "in case d, option is $c arg is $OPTARG "
;;
o) # $OPTARG is now whatever came
# after the -o
echo "in case o, option is $c arg is $OPTARG "
;;
\?) #error msg is automatically written
#from the getopts command
exit 1;; #STOPS gopsPgm.sh and sets $? to 1
esac
done
exit 0
vvvvvvvvvvvvvvv optional vvvvvvvvvvvv
Woit shows cleanFiles.sh and cleanFilesFixed.sh
^^^^^^^^^^^^^^^ optional ^^^^^^^^^^^^
END OF WEEK 5 (UNIX)
May do u5Lab now