Thursday, March 19, 2009

Shell Script - eliminate duplicate patterns in a list

Here is a shell script to eliminate duplicate patterns in a list of patterns.


#################################################
# Let the name of the file containing the list
# of patterns be named 'tst.parm'
#
# Name : uniq_patern.ksh
#################################################

#!/usr/bin/ksh

fil=tst.parm
touch new_list
for i in $(cat $fil)
do
grep -vi "$i" $fil > x
var=`echo $i`
if [ `/usr/xpg4/bin/grep -iq "$var" new_list; echo $?` -ne 0 ]
then
echo "$i" >> new_list
else
##echo "entry already exist" > /dev/null
echo "entry already exist"
fi
mv x fil
done

#########
# E N D #
#########

#################################################
# Now view the file 'new_list' for a list of
# patterns that are not duplicated
#################################################



Here is the parameter file 'tst.parm' that was passed to the script, the tst.parm file has a list of duplicated patterns



unix$> cat tst.parm
one
ONE
two
three
ONE
two
TWO
one
THREE
one



Once after executing the script uniq_patern.ksh, the result is stored in 'new_list' file.


unix$> cat new_list
one
two
three

No comments: