Bootable DVD/CDROM

Have you ever wanted to determine whether a DVD or CDROM is a bootable CDROM or DVD from within a shell script?  If you use GNU/Linux you could parse the output from the isoinfo utility which is part of the CDRecord tools suite.

What if the isoinfo utility is not available on your operating system?  In this case you have to resort to rolling you own compiled langauge utility or shell script.  To do this you need to understand where to find the specific bytes on a CDROM/DVD which indicate whether it is a bootable CDROM/DVD or not.  The relevant specification covering bootable CDROMs (and by extension bootable DVDs) is the El Torito specification which is an extension to the ISO 9660 specification.

Our first shell script uses the dd utility to read in the required bytes which are then sliced and diced using the xxd and cut utilities..  Sector 17 (hexadecimal 11, the Boot Record Volume) contains two bytes at a specific offset which are an absolute pointer to the Booting Catalog which in turn contains a specific byte which indicates whether a CDROM/DVD is bootable or not.

#!/bin/bash
#
# use dd to read in the required bytes
#

DEVICE="/dev/sr0"
VERBOSE=1

# get 2 bytes at absolute offset 34887
bchex=`dd if=${DEVICE} skip=$(( 0x800 * 0x11 + 0x47))  bs=1 count=2 conv=swab 2>/dev/null | xxd | cut -d" " -f2`

# get one byte
bcdec=`printf "%d" 0x${bchex}`
bootable=`dd if=${DEVICE}  bs=1 skip=$(( 0x800 * bcdec + 0x20))  count=1  2>/dev/null | xxd | cut -d" " -f2`

if [[ $bootable = 88 ]]; then
   [ $VERBOSE ] && printf "Bootable CDROM\n"
   exit 0
else
   [ $VERBOSE ] && printf "Not a bootable CDROM\n"
   exit 1
fi

The previous shell script is classical in its approach to the problem in that it uses a number of small specialized utilities to achieve the desired result.

Another way of achieving the same result is to use the extended I/O features of ksh93 as shown in the following shell script.

#!/bin/ksh93
#
# Cannot open DVD/CDROM for read/write - hence cannot seek 
# forward on file descriptor. Workaround is to close and 
# reopen the file descriptor.
#

DEVICE="/dev/sr0"
VERBOSE=1

# read 2 bytes
redirect 3< $DEVICE || exit 3
3<#(( 0x800 * 0x11 + 0x47 ))
read -u3 -N 1 bchex1
read -u3 -N 1 bchex2
redirect 3<&-

# calculate absolute offset
integer bcdec=$(printf "%d" 0x$(printf "%x%x\n"  "'$bchex2" "'$bchex1"))

# read 1 byte
redirect 3< $DEVICE || exit 3
3<#(( 0x800 * bcdec + 0x20 ))
read -u3 -N 1 bootable
redirect 3<&-

if [[ $(printf "%x" "'$bootable") = 88 ]]; then
   [ $VERBOSE ] && printf "Bootable CDROM\n"
   exit 0
else
   [ $VERBOSE ] && printf "Not a bootable CDROM\n"
   exit 1
fi

The advantage of this approach is that all operations are done within the existing shell process without the overhead of fork/execs to invoke external utilities such as dd etc.

Note that both of the above shell scripts assume that the CDROM or DVD being checked is either non-bootable or contains a single boot-image configuration.  They can easily be extended to check for a multiple boot-image configuration.  That is left as an exercise for you the reader.

For the record, these two shell scripts were tested on Fedora 10 using version M of ksh93.

KSH93 Bit Manipulation

When programmers think about bitwise manipulation, they usually think about using C or C++ to solve their problem since both programming languages provide a rich set of features which make it easy to perform bitwise manipulation.

However it is possible to just as easily perform such operations using the ksh93 shell.  This post will explain what bitwise manipulation and number conversion facilities are available in ksh93 and provide some hopefully useful utilities and examples.

Many programmers are unaware that ksh93 has builtin support for different numeral radices (AKA arithmetic bases) as shown in the following example.

$ print $(( 10 + 20 ))
30
$ print $(( 2#10 + 20 ))
22
$ print $(( 2#10 + 4#20 ))
10
$

where base 2 is denoted by 2#, base 4 by #4 and so on.  The general case syntax is [base#]n where base is a decimal number between 2 and 36 representing the arithmetic base and n is a number in that base.  You can mix and match bases within expressions.  As you would expect, the default base, i.e. if [base#] is omitted, is decimal(10).

Here is an example of how to convert decimal 255 to base 2, base 8 and base 16 respectively using the typeset command.

$ typeset -i2 a=255
$ echo $a
2#11111111
$ typeset -i8 a=255
$ echo $a
8#377
$ typeset -i16 a=255
$ echo $a
16#ff
$

Note that ksh93 displays the base prefix before each number unless it is a decimal number, i.e. base 10. 

The bitwise manipulation operators in ksh93 are AND, OR, XOR, SHIFT LEFT and SHIFT RIGHT.  There are no SHIFT ROTATE or COMPLEMENT operators.  The following is the list of supported bitwise operators in decreasing order of precedence which, by the way, is the same as in the C programming language.

<< >>	Bitwise shift left, shift right
&	Bitwise AND
^	Bitwise XOR
|	Bitwise OR

The following shell script includes both logical and bitwise operators together with the expected output for each statement.  Notice the difference in output between a logical and bitwise operator.

#!/bin/ksh93

integer -i2 a=5
integer -i2 b=6

# expected output 1
print $(( a < b ))
# expected output 0
print $(( a > b ))
# expected output 3
print $(( a ^ b ))
# expected output 4
print $(( a & b ))
# expected output 1
print $(( a && b ))
# expected output 7
print $(( a | b ))
# expected output 1
print $(( a || b ))

A useful feature of bitwise operators is that by using the shift left and shift right bitwise operators you can mimic multiplication or division by 2 or a power of 2.

$ x=4
$ print $((x << 1))
8
$ print $(( x << 2 ))
16
$ print $(( x >> 1 ))
2
$ print $(( x >> 2 ))
1

Here is a fairly simple shell script which uses numeric base conversion and bitwise operators to calculate the network and broadcast addresses for a given IP address and subnet mask.

#!/bin/ksh93

typeset -i2 mask=255

[[ $# != 2 ]] && {
   echo "Usage: $0 ipaddress subnetmask"
   exit 1
}

SaveIFS=$IFS
IFS=.
typeset -a IParr=($1)
typeset -a NMarr=($2)
IFS=$SaveIFS

typeset -i2 ip1=${IParr[0]}
typeset -i2 ip2=${IParr[1]}
typeset -i2 ip3=${IParr[2]}
typeset -i2 ip4=${IParr[3]}

typeset -i2 nm1=${NMarr[0]}
typeset -i2 nm2=${NMarr[1]}
typeset -i2 nm3=${NMarr[2]}
typeset -i2 nm4=${NMarr[3]}

echo
echo "       IP Address: $1"
echo "      Subnet Mask: $2"
echo "  Network Address: $((ip1 & nm1)).$((ip2 & nm2)).$((ip3 & nm3)).$((ip4 & nm4))"
echo "Broadcast Address: $((ip1 | (mask ^ nm1))).$((ip2 | (mask ^ nm2))).$((ip3 | (mask ^ nm3))).$(( ip4 | (mask ^ nm4)))"
echo

exit 0

Some sample output for this script.

$ ./calculate-address 10.150.12.1 255.255.255.0

       IP Address: 10.150.12.1
      Subnet Mask: 255.255.255.0
  Network Address: 10.150.12.0
Broadcast Address: 10.150.12.255

$ ./calculate-address 10.150.12.1 255.255.254.0

       IP Address: 10.150.12.1
      Subnet Mask: 255.255.254.0
  Network Address: 10.150.12.0
Broadcast Address: 10.150.13.255

The next example calculates the network and broadcast addresses for a Classless Inter-Domain Routing (CIDR) compliant IP address.  Note the use of the shift right operator in calculating the network mask.

#!/bin/ksh93

[[ $# != 1 ]] && {
   echo "Usage: $0 ipaddress/netmask"
   exit 1
}

SaveIFS=$IFS
IFS="./"
typeset -a IParr=($1)
IFS=$SaveIFS

typeset -i2 ip1=${IParr[0]}
typeset -i2 ip2=${IParr[1]}
typeset -i2 ip3=${IParr[2]}
typeset -i2 ip4=${IParr[3]}
typeset -i2 cidr=${IParr[4]}

typeset -i2 nm1=0 nm2=0 nm3=0 nm4=0

typeset -i quad=$(( cidr / 8 ))
sigbits=$(( cidr % 8 ))
if (( sigbits != 0 )); then
   slot=$(( 256 - ( 256 >> $sigbits ) ))
fi

for (( i=1; i < 5; i++ ))
do
   nameref nm=nm${i}
   if (( quad != 0 )); then
      nm=255
      (( --quad ))
   elif (( quad == 0 )); then
      nm=slot
      break
   fi
done

typeset -i2 mask=255

print
print "           IP Address: $((ip1)).$((ip2)).$((ip3)).$((ip4))"
print "    CIDR Netmask Mask: $((nm1)).$((nm2)).$((nm3)).$((nm4))"
print " CIDR Network (Route): $((ip1 & nm1)).$((ip2 & nm2)).$((ip3 & nm3)).$((ip4 & nm4))"
print "    Broadcast Address: $((ip1 | (mask ^ nm1))).$((ip2 | (mask ^ nm2))).$((ip3 | (mask ^ nm3))).$((ip4 | (mask ^ n
m4)))"
print

exit 0

Sample output:

$ ./calculate-address "192.168.21.12/18"

           IP Address: 192.168.21.12
    CIDR Network Mask: 255.255.192.0
 CIDR Network (Route): 192.168.0.0
    Broadcast Address: 192.168.63.255

While both the bash and zsh shells also provide a full set of bitwise operators, only ksh93 and zsh with their builtin support for numeric base conversions makes such bitwise operators truly useful in shell scripts.
 

KSH93 Extended Patterns

According to David Korn, a shell is primarily a string processing language.  Pattern matching is an important compoment of any such language and indeed Korn Shell 93 (ksh93) has excellent support for extended patterns as well as regular expressions.  Extended patterns can be thought of as class or type of extended regular expressions.  Both the bash and zsh shells have something similar but not as comprehensive.  However, as usual, extended patterns are documented quite tersely in the ksh93 man page.

The purpose of this post is to explain, with some examples, how to use the power of extended patterns in your ksh93 scripts.  It is assumed that you are reasonably familiar with basic regular expressions (BREs) as implemented in sed, grep or awk.  If you need a introductory tutorial on regular expressions, here is one at IBM developerWorks.

The following table shows the the basic pattern matching operators in ksh93.

?(pattern)	Match if found 0 or 1 times
*(pattern)	Match if found 0 or more times
+(pattern)	Match if found 1 or more times
@(pattern1|...)	Match if any of the patterns found
!(pattern)	Match if no pattern found

Note that an operator must preceed a pattern in ksh93 whereas in egrep, sed and awk, the operator is placed after the pattern.

Here is an example of how to use the above operators to modify the contents of the string str

str="Joe Mike and Dave are all good friends"

print ${str//a?(re)/_}
# output: Joe Mike _nd D_ve _ _ll good friends

print ${str//g*(o)/_}
# output: Joe Mike and Dave are all _d friends

print ${str//+(o)/_}
# output: J_ Mike and Dave are all g_d friends

print ${str//@(Joe|Mike|Dave)/_}
# output: _ _ and _ are all good friends

print ${str//@(Joe|Mike|g*(o))/_}
# output: _ _ and Dave are all _d friends

print ${str//!(Joe)/_}
# output: _

print ${str//!(Joe|Mike|Dave)/_}
# output: _

In the above example, I have shown the expected output as a comment below each print statement.  Here is another example which should futher clarify your understanding of these pattern operators.  Note particularly the output of of the last print statement.

str='An extended pattern expression'

print ${str//e/#}
print ${str//[^e]/#}
print ${str//+(e)/#}
print ${str//-(e)/#}
print ${str//?(e)/#}
print ${str//*(e)/#}
print ${str//!(e)/#}

which produces the following output.

An #xt#nd#d patt#rn #xpr#ssion
###e##e##e######e###e###e######
An #xt#nd#d patt#rn #xpr#ssion
An extended pattern expression
#A#n# #x#t#n#d#d# #p#a#t#t#r#n# #x#p#r#s#s#i#o#n#
#A#n# #x#t#n#d#d# #p#a#t#t#r#n# #x#p#r#s#s#i#o#n#
#

The following table show a number of pattern matching interval quantifiers.

{n}(pattern)	Match if found exactly n times where n is a non-negative number
{n,m}(pattern)	Match if found between n and m times where n and m are non-negative integers and n <= m

Here is an example of how to use the above interval quantifiers to match various strings.

print $(
   [[ aaaa == {4}(a) ]] || print $?
   [[ aaaa == {,4}(a) ]] || print $?
   [[ aaaa == {3,}(a) ]] || print $?
   [[ aaaa == {2,4}(a) ]] || print $?
   [[ abc == {1,4}(ab)c ]] || print $?
   [[ abcabc == {,2}(abc) ]] || print $?
   [[ abababcc == {1,4}(ab){1,2}(c) ]] || print $?
   [[ abc == {1,4}(ab){1,2}(c) ]] || print $?
   [[ abcdcdabcd == {3,6}(ab|cd) ]] || print $?
   [[ abcdcdabcde == {5}(ab|cd)e ]] || print $?
)

By default an extended pattern attempts to match the longest possible string consistent with generating the longest overall match.  This is known as a greedy or maximal match.  A non-greedy (or minimal) match is one that matches the shortest possible string.  Perl was the first scripting langauge to popularize non-geedy matching.  By the way, ksh93 and zsh are the only shells that support non-greedy matching.

You can use the '-' qualifier to indicate to the shell that you want to use non-geedy matching as shown in the table below.

?-(pattern)	Shortest match if found 0 or 1 times
*-(pattern)	Shortest match if found 0 or more times
+-(pattern)	Shortest match if found 1 or more times
@-(pattern1|…)	Shortest match if any of the patterns found
{n,m}-(pattern)	Shortest match if found between n and m times

Alternatively, you can use the ~(-g) subpattern to indicate to ksh93 that you want to use non-geedy matching.  The following examples show both methods.

str="bcdabdcbabcd"

print "    Greedy: ${str/+(*ab)/_}"
print "Non-greedy: ${str/+-(*ab)/_}"

str="heleelloo hello"

print "    Greedy: ${str//he*l/_}"
print "Non-greedy: ${str//~(-g)he*l/_}"
print "    Greedy: ${str//?(he*ll)/_}"
print "Non-greedy: ${str//~(-g)?(he*ll)/_}"
print "Non-greedy: ${str//?-(he*ll)/_}"
print "    Greedy: ${str//+(he*ll)/_}"
print "Non-greedy: ${str//+-(he*ll)/_}"
print "    Greedy: ${str//*(he*ll)/_}"
print "Non-greedy: ${str//*-(he*ll)/_}"
print "    Greedy: ${str//{1,2}(he*ll)/_}"
print "Non-greedy: ${str//~(-g){1,2}(he*l)/_}"

A pattern-list is a list of one or more patterns separated from each other by either a & or a |.  A & (denoting logical AND) means that all patterns must be matched whereas | (denoting logical OR) means that only one pattern need be matched.  Composite patterns can also be created as shown below.

?(pattern-list)	Optionally matches any one of the patterns
*(pattern-list)	Matches zero or more occurrences of the patterns.
+(pattern-list)	Matches one or more occurrences of the patterns.
{n}(pattern-list)	Matches exactly n occurrences of the patterns.
{m,n}(pattern-list)	Matches m to n occurrences of the patterns.  If m is omitted, 0 is used. If n is omitted at least m occurrences are matched.
@(pattern-list)	Matches exactly one of the patterns.
!(pattern-list)	Matches anything except one of the patterns.

Again, by default, matching is greedy.  Each pattern in the pattern-list attempts to match the longest string possible consistent with generating the longest overall match.  If more than one match is possible, the match starting closest to the beginning of the string will be chosen.  However, for each of the above compound patterns a − can be inserted in front of the ( to specify that the shortest match to the specified pattern-list should be used.

Finer grained control of extended pattern matching is possible using sub-patterns of the form ~(options:pattern-list) where :pattern-list is optional and options consists of one or more of the following option flags:

+	Enable following options (default)
-	Disable following options
E	Remainder of the pattern uses ERE pattern syntax
F	Remainder of the pattern uses fgrep-like pattern syntax.
G	Remainder of pattern uses BRE pattern syntax
K	Remainder of pattern uses ksh93 pattern syntax (default)
i	Case insensitive match
g	Greedy match (default)
l	Left anchor pattern.
r	Right anchor pattern.

If both options and :pattern-list are specified, then the specified options apply only to :pattern-list.  Otherwise, the specified options remain in effect until disabled by a subsequent ~(...) sub-pattern or at the end of the sub-pattern containing ~(...).

ksh93 provides a way to translate extended patterns into regular expressions and vice-versa by means of two printf options.

$ printf "%R\n" "*[!0-9]*'
[^0-9]
$ printf "%P\n" "([0-9]+\.){3}"
*{3}(+([0-9])\.)*
$

I hope that I have inspired you to go away and experiment on your own with some of the more advanced features of extended patterns in ksh93.  Once you master the syntax, you can significantly reduce the need for your scripts to invoke external utilities such as sed or awk simply to parse text strings.
 
Enjoy!
 
P.S.  All the examples included in this post were tested on ksh93t+ 12/10/2008.
 

KSH93 Stat Builtin

One of the builtin commands that is missing in ksh93, in my humble opinion, is a builtin similar to stat(1) which would return information about a file.  Here is my initial implementation of a stat builtin.  The output is a compound variable whose subvariables contain the contents of the various fields of the stat(2) structure.  If you are unfamilar with compound variables, see my previous post for an eluridation.

/*
**  FPMurphy  2009-01-03
**
**  License: Common Public License Version 1.0
**
*/

#pragma prototyped

#include "defs.h"
#include "builtins.h"
#include "path.h"
#include <tm.h>

/* macro to create subvariables */
#define CREATE_CVE(X,Y,Z) \
   strcpy(b,(X)); \
   np = nv_open(buf, shp->var_tree, NV_NOASSIGN|NV_VARNAME); \
   nv_putval(np, (char*)(Y), (Z) ); \
   nv_close(np)

static char strperms_buf[30];

static const
char sh_optstat[] =
   "[-?\n@(#)$Id: stat 2009-01-03 $\n]"
   "[-author?Finnbarr P. Murphy fpm at hotmail.com ]"
   "[-license?http://www.opensource.org/licenses/cpl1.0.txt]"
   "[+NAME? stat - get file status]"
   "[+DESCRIPTION?\bstat\b creates the compound variable \avar\a corresponding"
       " to the file given by the pathname \afile\a.  The elements of \avar\a"
       " are the names of fields in the \astat\a structure with the \bst_\b"
       " prefix removed, together with the basename of \afile\a.]"
   "\n"
   "\nvar file\n"
   "\n"
   "[+EXIT STATUS?]{"
       "[+0?Success.]"
       "[+>0?An error occurred.]"
   "}"
   "[+SEE ALSO?\bstat\b(1),\bstat\b(2)]"
;

/* stringify the permission bits */
static char *
strperms(char * p, mode_t mode)
{
   char ftype = '?';

   if (S_ISBLK(mode))  ftype = 'b';
   if (S_ISCHR(mode))  ftype = 'c';
   if (S_ISDIR(mode))  ftype = 'd';
   if (S_ISFIFO(mode)) ftype = '|';
   if (S_ISLNK(mode))  ftype = 'l';
   if (S_ISREG(mode))  ftype = '-';

   sfsprintf(p, 30, "\\0%010lo %c%c%c%c%c%c%c%c%c%c %c%c%c\0",
   mode, ftype,
   mode & S_IRUSR ? 'r' : '-',
   mode & S_IWUSR ? 'w' : '-',
   mode & S_IXUSR ? 'x' : '-',
   mode & S_IRGRP ? 'r' : '-',
   mode & S_IWGRP ? 'w' : '-',
   mode & S_IXGRP ? 'x' : '-',
   mode & S_IROTH ? 'r' : '-',
   mode & S_IWOTH ? 'w' : '-',
   mode & S_IXOTH ? 'x' : '-',
   mode & S_ISUID ? 'U' : '-',
   mode & S_ISGID ? 'G' : '-',
   mode & S_ISVTX ? 'S' : '-');

   return(p);
}


int
b_stat(int argc, char *argv[], void *extra)
{
   register Shell_t *shp = ((Shbltin_t*)extra)->shp;
   register Namval_t *np;
   register int n;
   struct stat statb;
   char buf[100];
   char *b;

   while (n = optget(argv, sh_optstat)) switch (n) {
      case ':':
         errormsg(SH_DICT, 2, "%s", opt_info.arg);
         break;
      case '?':
         errormsg(SH_DICT, ERROR_usage(2), "%s", opt_info.arg);
         break;
   }

   argc -= opt_info.index;
   argv += opt_info.index;
   if (argc!=2)
       errormsg(SH_DICT, ERROR_usage(2), optusage((char*)0));

   /* stat the file */
   if (stat(argv[1], &statb) < 0)
       errormsg(SH_DICT, ERROR_system(1), "%s: stat failed", argv[1]);

   strcpy(buf, argv[0]);
   b = buf;
   while (*b) b++;

   /* create compound variable */
   np = nv_open(buf, shp->var_tree, NV_NOASSIGN|NV_VARNAME|NV_ARRAY );
   if (!nv_isnull(np))
      nv_unset(np);
   nv_setvtree(np);
   nv_close(np);

   /* create compound variable elements */
   CREATE_CVE(".name", path_basename(argv[1]) , NV_RDONLY);
   CREATE_CVE(".atime", &statb.st_atime, NV_RDONLY|NV_INTEGER);
   CREATE_CVE(".ctime", &statb.st_ctime, NV_RDONLY|NV_INTEGER);
   CREATE_CVE(".mtime", &statb.st_mtime, NV_RDONLY|NV_INTEGER);
   CREATE_CVE(".uid", &statb.st_uid, NV_RDONLY|NV_INTEGER);
   CREATE_CVE(".gid", &statb.st_gid, NV_RDONLY|NV_INTEGER);
   CREATE_CVE(".size", &statb.st_size, NV_RDONLY|NV_INTEGER|NV_LONG);
   CREATE_CVE(".dev", &statb.st_dev, NV_RDONLY|NV_INTEGER);
   CREATE_CVE(".ino", &statb.st_ino, NV_RDONLY|NV_INTEGER|NV_LONG);
   CREATE_CVE(".nlink", &statb.st_nlink, NV_RDONLY|NV_INTEGER);
   CREATE_CVE(".mode", strperms(strperms_buf, statb.st_mode), NV_RDONLY);

   return(0);
}

This code was tested using ksh93t+.

You can embed a man page into the builtin as done in the above source code.  Most, if not all, of the ksh93 commands have such embedded man pages.

Here is the output from stat --help.

$ stat --help
Usage: stat [ options ] var file
$

and here is the output from stat --man.

$ stat --man
NAME
   stat - get file status

SYNOPSIS
  stat [ options ] var file

DESCRIPTION
  stat creates the compound variable var corresponding to the file given 
  by the pathname file. The elements of var are the names of fields in the 
  stat structure with the st_ prefix removed, together with the basename 
  of file.

EXIT STATUS
    0     Success.
    >0    An error occurred.

SEE ALSO
  stat(1),stat(2)

IMPLEMENTATION
  version         stat 2009-01-03
  author          Finnbarr P. Murphy fpm at hotmail.com
  license         http://www.opensource.org/licenses/cpl1.0.txt
$

Here is an example of using the stat builtin to get information about a file called tksh.

$ stat fileinfo ./tksh
$ print $fileinfo
( atime=1231192796 ctime=1231192291 dev=2065 gid=500 ino=50897 
mode='0100755 -rwxr-xr-x ---' mtime=1231192291 name=tksh nlink=1 size=2171604 uid=500 )
$ print ${fileinfo.atime}
1231192796
$ printf "%(%Y-%m-%d %H:%M:%S)T\n" "#${fileinfo.atime}"
2009-01-05 16:59:56
$

As you can see the stat builtin gets the file statistics using stat(2), creates a compound variable with the specified name, i.e. fileinfo, and then creates a series of subvariables (atime, ctime, mtime, etc. ) whose names correspond to the fields of the stat(2) structure.  This compound variable is then available to you to use as necessary within your shell script.  Rather than having to handle a series of variables, one per stat(2) structure field, you simply have to deal with single compound variable, i.e. fileinfo.

As always, email me if you have any questions.

KSH93 Compound Variables

Most shells, including bash, ksh88 and pdksh, implement a flat variable namespace.  According to Dave Korn, one of the lessons learned from UNIX is that a hierarchical namespace is better than a flat namespace.  For this reason, amongst others, a hierarchical variable namespace was implemented by him in ksh93, with . (dot) as the separator for each level of the hierarchy.  This expanded variable namespace enabled the implementation of an aggregate definition for a shell variable to include subvariables.  Such shell variables are called compound variables.  Not much appears to have been written about compound variables to date and, as usual, the ksh93 man page is terse on the subject.  This post will try to explain compound variables in detail and demonstrate how useful they can be in when dealing with structured data.

As in bash, ksh88, and pdksh, a variable in ksh93 is defined by a name=value pair.

$ myvar=10
$ print $myvar
10

Now consider the following commands which show how to declare a compound variable, define subvariables and interact with those subvariables.

$ typeset -C myvar       # declare 'myvar' as compound variable 
$ myvar.x=10             # set subvariable 'x' to 10
$ myvar.y=20             # set subvariable 'y' to 20
$ print $myvar           # print definition of compound variable
( x=10 y=20 )
$ print ${myvar.x}       # print value of 'x' subvariable
10
$ print ${myvar.y}       # print value of 'y' subvariable
20
$ yourvar=( x=10 y=5 )   # declare and define 'yourvar' compound variable
$ print $yourvar             
( x=10 y=5 )
$ print $(( yourvar.x * yourvar.y ))   # multiply 2 subvariables together.
50
$

A variable with a . (dot,period) in it’s name is called a subvariable.  However to create a subvariable, a variable whose name consists of everything up to the period must already exist.  Note that variable names that begin with .sh are reserved for use by ksh93.

Just as var='' initializes a simple variable, var=() does the same for a compound variable. This is not assigning a value to var; it is simply declaring that var is a compound variable and its value will be defined by all subvariables of the form var.*.

You can specify the type of specific subvariables.  For example, to create a compound variable with a subvariable named x of type integer.

$ var=( typeset -i x=2 )

You can have more than one level of subvariables in a compound variable.  However, you must be careful to first declare the compound variable otherwise you will get an error.

$ var=
$ var.x.y=2
/bin/ksh93t: var.x.y=2: no parent
$ var=()
$ var.x.y=2
$ print $var
( x=( y=2 ) )
$

You can use += word with compound variables provided the types are compatable.  When += word is applied to an arithmetic type, word is added to the current value.  When applied to a string variable, word is appended to the value.

$ var=( str="abc" typeset -i num=12 )
$ var.str+="def"
$ var.num+=12
$ print ${var.}
( typeset -i num=24 str=abcdef )
$

You may be wondering why the value of a compound variable is outputted in the form that it is.  The reason is that value of a compound variable is intended to be in a form ready for reinput by ksh93 as shown by the following example.

$ var=( x=2 y=4 )
$ print $var
( x=2 y=4 )
$ print -r "newvar=$var" > file
$ unset var
$ print $var

$ . ./file
$ print $newvar
( x=2 y=4)
$

You can copy compound variables using the eval builtin.

$ var=( x=2 y=3 )
$ print $var
( x=2 y=3 )
$ eval "newvar=$var"
$ print $newvar
( x=2 y=3 )
$

You can create a new compound variable from part of a existing compound variable.

$ var=( x=1 y=2 )
$ var.y.a=1
$ var.y.b=2
$ var.y.c=3
$ print $var
( x=1 y=2 y=( a=1 b=2 c=3 .=2 ) )
$ eval "newvar=${var.y.}"
$ print $newvar
( a=1 b=2 c=3 .=2 )
$ print -r -- "newvar=${newvar.}"
newvar=(
        a=1
        b=2
        c=3
        .=2
)
$

Compound variables can be exported but subvariables cannot.

$ testvar=( x=12 )
$ echo ${testvar.x}
12
$ export testvar.x
/bin/ksh93: export: testvar.x: invalid export name
$ export testvar
$ env | grep testvar
testvar=(x=12;)
$

One thing you need to be aware of is that ksh93 also uses the . (dot) notation to denote what are called discipline functions.  Shell variables in ksh93 can also behave as active objects rather than as simple storage units by having a one or more functions associated with a variable.  These functions are called discipline functions.  A discipline function is defined like any other function, except that the name for a discipline function is formed by using the variable name, followed by a . (dot), followed by the discipline name.  Any variable can have discipline functions defined that are invoked when the variable is referenced or assigned a value.  The default set of discipline functions in ksh93 is get, set, and unset.  Other discipline functions can be defined via a custom shared library.

Compound variables are currently a work in progress.  Some of the examples that I have shown above may not work in future versions of ksh93.  These examples were tested on ksh93t 2008-07-24.

Well, that is about all you need to get you started using compound variables.  I hope this post has given you some ideas about how useful such variables could be in your future ksh93 scripts.

KSH93 Auditing and Accounting

Korn Shell 93 (ksh93) is the only UNIX or GNU/Linux shell that I am aware of that, with proper setup, supports a modicum of per-user accounting and auditing.  This post attempts to explain these facilities and show you how to access and manipulate the resulting accounting and auditing records.

Per-user accounting has been a feature of ksh93 since the earliest days of this shell.  It is a fairly primative facility which writes out an (undocumented) record for each user command that is executed.

An auditing facility was added in July 2008.  This is somewhat more sophisticated than the accounting facility in that it is configurable and writes out a more detailed record either locally or to a remote system for each user command that is executed.  This facility can be used to monitor, track, record, and audit the activities of one or more users on an system, including system administrators.

Both facilities only work for interactive users.  Neither facility is enabled by default.  In fact, you have to go into the ksh93 source code, modify the main Makefile to enable certain compile time options, and then recompile the sources.  This is not a trivial exercise in many cases as rebuilding ksh93 also requires that you to rebuild a number of libraries.  Source code is available at the AT&T Research AST software download site.



To build ksh93 with these facilities enabled, you must build the ksh93 executable using either the ast-base or ast-open package together with the INIT package.  Just as the shared libraries are not built if you use the ast-ksh package, neither are the accounting and auditing facilities.  I have never investigated why this is so, but I am sure that Glenn Fowler and David Korn have their reasons.

You need to modify the compile time options in ../src/cmd/ksh93/Makefile as follows to enable one or both facilities.

SHOPT_ACCT ==     1        /* accounting */
SHOPT_ACCTFILE == 1        /* per user accounting info */
SHOPT_AUDIT ==    1        /* auditing */
SHOPT_AUDITFILE == "/etc/ksh_audit" /* auditing file */

After you have recompiled the sources, the new ksh executable is located in ../arch/..../bin/ subdirectory.  To see what options have actually been compiled into a particular executable, just print out the shell version string.

bash-3.2$ pwd
/home/fpm/ksh/build/arch/linux.i386-64/bin
bash-3.2$ ls -al ksh
-rwxr-xr-x 1 fpm fpm 1356931 2008-12-26 16:31 ksh
bash-3.2$ ./ksh
$ echo ${.sh.version}
Version AJLM 93t+ 2008-12-10
$ echo $KSH_VERSION
Version AJLM 93t+ 2008-12-10

The option string AJLM means that (A) auditing is enabled, (J) one SIGCHLD trap per completed job is supported, (L) per-user accounting is supported, and (M) multibyte characters are supported.

Per-user accounting is enabled using the SHACCT environmental variable.  To turn on per-user accounting, simply set SHACCT to the name of the file where you wish to store the accounting records.

export SHACCT="/tmp/ksh_acctfile"

Here is part of the resulting file. Note that the time is stored as hexadecimal seconds since the Epoch.

$ cat /tmp/ksh_acctfile
echo ${.sh.version}     fpm     495990d8
pwd     fpm     495990da
id      fpm     495990dd
date    fpm     495990e3
exit    fpm     495990e5

The following shell script can be used to access the records in this file and output them in a more useful format.

#!/bin/ksh93
ACCTFILE="/tmp/ksh_acctfile"

printf "DATE       TIME     LOGIN  COMMAMD\n\n"

# set IFS to TAB only
while IFS="     " read cmdstr name hexseconds
do
   longsecs=$(printf "%ld" "0x${hexseconds}")
   timestr=$(printf "%(%Y-%m-%d %H:%M:%S)T" "#${longsecs}")
   print $timestr, $name, "$cmdstr"
done < $ACCTFILE

Invoking this script gives the following output for the above accounting records.

$ ./parse_acctfile
DATE       TIME     LOGIN  COMMAMD

2008-12-29 22:09:12, fpm, echo ${.sh.version}
2008-12-29 22:09:14, fpm, pwd
2008-12-29 22:09:17, fpm, id
2008-12-29 22:09:23, fpm, date
2008-12-29 22:09:25, fpm, exit

Next we turn our attention to the auditing facility.  In addition to rebuilding ksh93 with the SHOPT_AUDIT option, you must create an audit configuration file on each system to tell ksh93 where to store the audit records and to specify which users are to be audited.  The default location for the configuration file is /etc/ksh_audit but that location be changed in the main ksh93 Makefile.  The configuration file should contain a line that defines the file to write the audit records to, followed by the UID of each user whose commands are to generate audit records.  Here is the configuration file used to generate the audit records for this part of this post.

$ cat /etc/ksh_audit
/tmp/ksh_auditfile;500

This configuration file specifies that audit records are to be written to /tmp/ksh_auditfile for the user who's UID is 500.  Note that the field delimiter is a semi-colon.

Here are the audit records stored in the /tmp/ksh_auditfile which match the accounting records shown previously in this post.  The field separator is a semi-colon.  The first field is the UID of the user executing the command.  The second field is the time in seconds since the Epoch.  The third field is the terminal device on which the command was executed, and the final field is the actual command executed by the user.

500;1230606552;/dev/pts/2; echo ${.sh.version}
500;1230606554;/dev/pts/2; pwd
500;1230606557;/dev/pts/2; id
500;1230606563;/dev/pts/2; date
500;1230606565;/dev/pts/2; exit

As before, here is a simple ksh93 script which parses this audit file, replaces the UID with the actual user's name and seconds since the Epoch with the actual data and time, and outputs the enhanced records in a comma separated value (CSV) format.

#!/bin/ksh93

AUDITFILE="/tmp/ksh_auditfile"

while IFS=";" read uid sec tty cmdstr
do
   name=""
   while IFS=":" read pwname pword pwuid rest
   do
      if [[ "$uid" == "$pwuid" ]]
      then
          name=$pwname
          break
      fi
   done < /etc/passwd

   timestr=$(printf "%(%Y-%m-%d %H:%M:%S)T" "#$sec")
   print "$timestr", $name, $uid, $tty, "$cmdstr"
done < $AUDITFILE

Here is the output for the above audit records.

2008-12-29 22:09:12, fpm, 500, /dev/pts/2,  echo ${.sh.version}
2008-12-29 22:09:14, fpm, 500, /dev/pts/2,  pwd
2008-12-29 22:09:17, fpm, 500, /dev/pts/2,  id
2008-12-29 22:09:23, fpm, 500, /dev/pts/2,  date
2008-12-29 22:09:25, fpm, 500, /dev/pts/2,  exit

If the underlying operating system supports networking using the /dev/udp/host/port syntax or the /dev/tcp/host/port syntax, audit records can be sent by ksh93 across a network to another system.  This mechanism could be used to store audit records on a secured centralized system to which only specific personnel have access.  As an example, the following audit configuration file line designates that audit records for the user who's UID is 500 should be sent using UDP to the syslog network port (514) on a remote system who's IP is 192.168.0.99.

/dev/udp/192.168.0.99/514;500

Here are the same audit records stored by the syslog daemon on the remote system.

2008-12-29 22:09:12 192,169.0.115 500;1230606552;/dev/pts/2; echo ${.sh.version}
2008-12-29 22:09:14 192.169.0.115 500;1230606554;/dev/pts/2; pwd
2008-12-29 22:09:17 192.169.0.115 500;1230606557;/dev/pts/2; id
2008-12-29 22:09:23 192.169.0.115 500;1230606563;/dev/pts/2; date
2008-12-29 22:09:25 192.169.0.115 500;1230606565;/dev/pts/2; exit

Depending on the configuration of the syslog daemon on your particular system, the first part of the record may contain more or less information or be formatted differently but the final part of the record, i.e. the audit record sent by ksh93 should be in the standard audit record format.

Note that while the auditing and accounting facilities within ksh93 can provide you with much useful information regarding the actions of one or more users on a system or systems, these facilities should not be regarded as providing enhanced security akin to the Trusted Computing Base (TCB).  There are many ways of circumventing these facilities.  For example, a knowledgeable user could switch to a different shell such as bash where their actions will not be recorded.  There are a number of other ways but I will not discuss them here.

Most of the information provided in this post is not documented in a single place anywhere that I can find by searching the Internet.  The ksh93 man page does not mention either the accounting or auditing facilities.  Even the ksh93 source code is somewhat vague.  I gleened most of this information by studying the code in ../src/cmd/ksh93/edit/history.c.  If I got anything wrong, please let me know so that I can update this post.

PowerShell Grows Up

I am excited about the emminent release of Microsoft Windows Powershell Version 2 CTP3 (Community Technology Preview Version 3) which is due "real soon now."   The first CTP for Powershell 2.0 was in November 2007 and there has been a lot of progress on the product since then.  See the permanant link at the side of my blog to access the Powershell developers blog.

Why am I, a UNIX/Linux developer, excited by Powershell V2?  After all it does not run on any UNIX or GNU Linux platform and microsoft has no plans to port itMto these platforms.  The main reason is that Powershell V2 supports non-compiled functions (known as script cmdlets) in scripts just like bash or ksh93 does while providing deep object orientated interfaces to the operating system and underlying platform.  Previously cmdlets had to be written in a compiled language such as C# or VB.NET in order to provide such functionality.  (Currently, the terminology for what is generally know as a function in a UNIX/GNU Linux shell script is called a script cmdlet in Powershell but this is apparantly on track to change to function in CTP3.)  Another reason is the introduction of a graphic interface to Powershell.

I have spend more than 25 years of my life working on or with UNIX, UNIX-like or GNU/Linux operating systems.  For a period of that time, I was responsible for maintenance and enhancement of various shells and hence still maintain an interest in this area.  Much progress has occured in UNIX/GNU Linux shells over that period of time with bash, zsh and ksh93 emerging as the de facto leaders and csh, tcsh, sh (as in the Steve Bourne shell) and ash slowly fading away into obscurity.  

Standardization of functionality for portable shell scripts has moved from simply "use the Bourne shell if you want your script to run everywhere" to written specifications such as POSIX which all three shells support.  Numerious attempts (dtksh, Wksh, tksh come to mind) have been made to develop shells with graphical interfaces but none has been really successful.  Currently work is afoot to provide deeper interfaces into the operating system and platform with the latest versions of ksh93 having support for compiled builtins, compound variables and quasi-objects (which I plan to write about soon in another post.)

While it may be tantamount to heresy to say this, but kudos to the Microsoft Powershell development team who have managed in my opinion to leapfrog all the UNIX//GNU Linux shells in terms of functionality with Powershell V2.  I think UNIX/GNU Linux shell developers could learn a lot from taking a hard look at its design and functionality.

KSH93 Extended I/0

The ksh93 exec command is a special overloaded built-in command that can be used to manipulate file descriptors or to replace the current shell with a new command.  Starting with version s of ksh93, which was released in 2006, ksh93 contains a number of very useful extensions to exec relating to file descriptors and input/output which most shell programmers are unaware of.

From the current ksh93 man page

<#((expr))    Evaluate  arithmetic  expression  expr   and
              position  file descriptor 0 to the resulting
              value bytes from the start of the file.  The
              variables  CUR  and EOF evaluate to the cur-
              rent offset and end-of-file  offset  respec-
              tively when evaluating expr.

>#((offset))  The  same  as  <#  except  applies  to  file
              descriptor 1.

<#pattern     Seeks forward to the beginning of  the  next
              line containing pattern.

<##pattern    The  same  as  <# except that the portion of
              the file that is skipped is copied to  stan-
              dard output.

This post is about the these four extensions.  It is assumed that you are familar with ksh93 and the standard shell redirection facilities that are available in most modern shells, e.g.

exec 1>file          # redirect stdout to file
exec 2>&1            # redirect stderr to same 
exec 1>&-            # close file descriptor

If not, a quick Web search will return plenty of tutorials on the subject.

As with most Unix and GNU/Linux man pages, the ksh93 man page is quite cryptic and concise so before we go any further I will attempt to describe what each extension can be used for.  The first extension is useful for determining the current offset within a file or can be used to return the size of a file by by returning the value of the offset of the end of the value.  (You can also get the current file offset with offset=$(fd<#).)  The second extension can be used to position a file descriptor anywhere within a file.  The third extension can be used to seek forward to the start of a line containing a pattern specified by you.  The final extension is useful in that it outputs all lines between the current offset and the start of a line containing a pattern specified by you.  A number of things should be noted.  First, these extensions only work with text files.  Second, newlines count towards the offset from the start of the file.  Another thing to watch out for is while it is not clear from the ksh93 man page, it appears you must open a file for both read and write is you want to perform random access.  Why this is so is unclear to me but you get odd results if you seek backwards in a file even if you are only reading the file.

Because exec is a special command, any failure in I/O redirection will cause the script that invokes it to immediately terminate.  This can be prevented by invoking exec from the command built-in.  Incidently, redirect is an predefined alias for command exec.

For our first example, we will demonstrate the how to use these extensions to perform some basic I/O operations on a simple text file.

#!/bin/ksh93

TMP=file.$$

# create the temporary file used by example
cat <<EOT >$TMP
aaa
bbb
ccc
ddd
eee
fff
EOT

# open file descriptor 3 for read/write
command exec 3<> $TMP || exit 1

# check file descriptor 3 position
print
print "At offset: $(3<#)"
if (($(3<#) != 0))
then
   print "Not at offset 0"
   exit 1
fi

# read in the first line and print it
read -u3
print $REPLY
print "At offset $(3<#) after reading line"
print

# search forward for string "ddd"
3<#"ddd"
print "At offset $(3<#) after search forward for 'ddd'"
read -u3
print $REPLY
print

# check that we are at offset 8 and, if so, read line
if (( $(3<# ((8))) != 8))
then
  print "Not at offset 8"
  exit 1
fi
print "At offset $(3<#) after specifying absolute offset of 8"
read -u3
print $REPLY
print

# go to end of file. We know by inspection that is at offset 24, so check.
if (( $(3<#((EOF))) != 4*6 ))
then
   print "Not at EOF"
   exit 1
fi
print "At offset $(3<#) after specifying EOF"
print

# backup one line i.e. 4 characters
3<#((CUR - 4))
print "At offset $(3<#) after backing up 4 characters"
read -u3
print $REPLY
print

redirect 3<&- || echo 'cannot close FD 3'

rm $TMP

The following is the output from this example

$ ./example1

At offset: 0
aaa
At offset 4 after reading line

At offset 12 after search forward for 'ddd'
ddd

At offset 8 after specifying absolute offset of 8
ccc

At offset 24 after specifying EOF

At offset 20 after backing up 4 characters
fff
$

Our next example will selectly output the lines of a file between 2 search strings 'test' and '**' and display these lines as a single line.

$!/bin/ksh93

TMP=file1.$$
OUT=file2.$$

# create the temporary file used by example
cat <<'EOT' >$TMP
aaa
test
bbb
ccc
ddd
eee
fff
**
ggg
EOT

# redirect stdout
exec 4>&1

exec 3< $TMP > $OUT

# seek to start of line containing "test"
3<#'test'
# advance 5 characters i.e. the start of next line
3<# ((CUR + 5))
# output all lines until EOF or "**" found
3<##'\*\**'

exec 3<&-

# swap back stdout
exec 1>&-
exec 1>&4
exec 4>&-

rm $TMP

# hack to output as a single line
out=$(<$OUT)
echo $out

rm $OUT

The following is the output when we run this shell script

$ ./example2
bbb ccc ddd eee fff
$

For our next example, consider the following fictitious file which contains the results of a series of pings to local systems.

Wed Nov  4 15:25:14 EST 2008
PING Host11 (10.0.201.51) 56(84) bytes of data.
--- Host11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4010ms
rtt min/avg/max/mdev = 7.400/41.641/59.604/17.946 ms
.
Wed Nov  4 15:25:18 EST 2008
PING Host12 (10.0.202.51) 56(84) bytes of data.
--- Host12 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4016ms
rtt min/avg/max/mdev = 43.804/55.694/67.728/9.359 ms
.
Wed Nov  4 15:25:22 EST 2008
PING Host13 (10.0.205.51) 56(84) bytes of data.
--- Host13 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 40.237/56.283/68.973/10.433 ms
.
Wed Nov  4 15:25:26 EST 2008
PING Host14 (10.0.201.52) 56(84) bytes of data.
--- Host14 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 36.053/65.964/96.782/20.642 ms

The requirement is to create a table showing min, loss and mdev values for each host pinged.  Here is our shell script

#!/bin/ksh93

redirect 3< file

# get and store value of EOF
eof=$(3<#((EOF)))

# check that we are at start of file i.e. offset 0
if (( $(3<#((0))) != 0 ))
then
  print "ERROR: not at offset 0"
  exit 1
fi

# loop though the file and build one line of output per host
while (( $(3<#((CUR))) < $eof ))
do
   3<#'PING*'
   read -A -u3 arr1
   3<#'*packet loss*'
   IFS=" %" read -A -u3 arr2
   IFS=" /" read -A -u3 arr3
   printf "%s %6.3f %3d %6.3f\n" ${arr1[1]} ${arr3[6]} ${arr2[5]} ${arr3[9]}
done

redirect 3<&-

Here is the output after this script is run against the above file

$ ./example3
Host11  7.400   0 17.946
Host12 43.804   0  9.359
Host13 40.237   0 10.433
Host14 36.053   0 20.642

Our final example will demonstrate how to replace a line and append additional lines to an existing file.  Note also that we specify the use of a file descriptor > 10 by means of the extended file descriptor {variable} syntax.

#!/bin/ksh93
TMP=file.$$

# create temporary file
cat <<EOT >$TMP
aaa
bbb
ccc
ddd
eee
fff
EOT

# open FD > 10 for read/write
command exec {n} <> $TMP || exit 1
print "File descriptor assigned is: ${n}"
print

# search forward for string "ddd"
{n}<#"ddd"

# replace this string with "DDD"
print -u${n} -f "%.3c\n" D

# go to end of file
redirect {n}>#((EOF))

# add 2 lines of text
print -u${n} -f "%.6c\n" G
print -u${n} -f "%.6c\n" H

redirect {n}>&- || echo 'cannot close FD'

cat $TMP

rm $TMP

Here is the output for this script.

$ ./example4
File descriptor assigned is: 11

aaa
bbb
ccc
DDD
eee
fff
GGGGGG
HHHHHH

Note that ksh93 does not support syntax of the form exec {n}<>&- since <&- and >&- both close the file descriptor n.  If you do the following in ksh93

exec {n}<&-
exec {n}>&-

you do not get an error.  On the other hand, if you do the same in zsh an error message is issued saying file descriptor n is already closed.

As you can see from the these examples, the extended capabilities of exec in ksh93 can be used to do file manipulation and transformation operations that were previously only possible using utilities such as sed, grep and awk.  Enjoy!

The Mysterious KSH93 Alarm Built-in

Sometimes it is useful to have part of a shell script run periodically, e.g. once a millisecond or every 10 minutes.  Although this is possible by starting a process in the background, ksh93 has an easier (but undocumented) feature which allow a script writer to set up interval timers.  This undocumented feature is a built-in called alarm and a corresponding discipline also called alarm. I am not going to attempt to explain what a discipline or a compund variable is in this post as I assume that you have read the ksh93 man page.  If you list your built-ins using the builtin command, you will see the alarm built-in listed along with the other built-ins.  According to Dave Korn, ksh93 has the ability to handle multiple timeout events using alarm but this feature remains undocumented since "I have not decided what interface I want for this functionality."

If you try to retrieve information about alarm, using the standard ksh93 builtin options (--man or --help) only a single usage line is outputted.

$ alarm --man
Usage: alarm [-r] [varname interval]
$

Dave Korn has stated that -r means repeat the alarm every interval, varname is the name of the variable which is invoked, and interval can be the absolute time from the Epoch or of the form +nseconds.   Milliseconds are also supposedly supported but I have not tested this functionality.

Internally, alarm is an ksh93 built-in which adds an alarm discipline to a function or variable. This built-in sets up an interval timer that will call the alarm discipline as needed.

For our first example, consider the following trivial example which uses alarm to display the time every 2 seconds

alarm -r mytime +2
function mytime.alarm { print -n "$(date +%H:%S)\r"; }
read dummy   # dummy wait

The first line specifies that the function mytime is to be called every 2 seconds.  The second line defines the mytime function i.e. print the current time in HH:SS format.  The third line is just a hack so that you can see the results of the previous 2 lines.

Note that ksh93 is very picky as regards how statements are written when discipline functions are used.  For example, if you place the mytime function definition before the alarm statement, ksh93 will produce an error, i.e. "mytime.alarm: invalid discipline function" and exit the script.

Next is an example that demonstrates how to abort a script after a waiting 10 seconds for input from a user.  Note that the ksh93 read command has a timeout option but we are not using it here.  This example also shows you how to remove an alarm using the unset built-in.

alarm -r abortscript +10

abortscript.alarm()
{
   print "Exceeded allowed time. Aborting script ......"
   kill -9 $$
}

print "waiting for input from read"
read dummy        # dummy wait - press return to continue

unset abortscript
print "abortscript unset. Press return to exit"

read dummy        # dummy wait - press return to continue

exit 0

Here is the output when this example is run.

$ ./example2
waiting for input from read
Exceeded allowed time. Aborting script ......
Killed
$
$ ./example2
waiting for input from read

abortscript unset. Press return to exit

$

My final example shows how alarm can be used to update a progress bar once a second on a terminal.

alarm -r progressbar +1

progressbar.alarm()
{
   tput cup 2 10
   (( ++progressbar.pvalue > 50 )) && {
      tput cup 2 10; tput el
      progressbar.pvalue=0
   }
   printf "Progress: %s" "${progressbar.pstr:1:${progressbar.pvalue}}"
}

progressbar.pvalue=0
progressbar.pstr="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

tput clear
read dummy?"Press RETURN to exit"

unset progressbar

exit 0

Again the order of things in the script is very important.  First we specify the function that alarm is going to use, next we define the function itself and then we set the compound variable values.

Use alarm with caution since Dave Korn has also stated that "alarm hasn't been documented because it is not working in all cases."  The examples shown in this post were tested using ksh93 version s.

KSH93 Custom Builtins II

This blog entry continues an earlier discussion of ksh93 custom builtins and includes a number of more advanced examples.

First of all, the development of custom builtins is easier when the libast header <shell.h> is included. It ensures that other necessary AST headers are included so that C prototypes are provided and calls to stdio routines are re-mapped to use the equivalent but safer sfio (AST Safe Fast Input Output) routines. The maintainers of ksh93 (David Korn and Glen Fowler) regard the standard stdio routines as having too many weaknesses to be used safely by ksh93.

Another useful feature is the lib_init() function. If your custom shared library contains lib_init(), this function is invoked with an argument value of 0 when the shared library is loaded. In addition, lib_init() can be used to load a custome builtin using sh_addbuiltin(). In this case there is no restriction on the name of a custom builtin function name, i.e. it does not have to start with "b_". Note that in the current version of ksh93, lib_init() can take a second argument, void *context, but this seems to do nothing.

Finally we will discuss memory management within custom builtins, how to include online documentation within a custom builtin and how to access internal shell data about variables and more.

Example 1   Use lib_init() to load a custom builtin.

#include <ast/shell.h>

int
fpm_goodbye(int argc, char *argv[], void *extra)
{
   if (argc != 2) {       
      printf("Usage: goodbye arg\n");
      return(2);
   }
   
   printf("Goodbye %s\n", argv[1]);
 
   return(0);
}

int
b_hello(int argc, char *argv[], void *extra
{
   if (argc != 2) {     
      printf("Usage: hello arg\n");
      return(2);
   }
    
   printf("Hello %s\n", argv[1]);

   return(0);
}

void
lib_init(int c, void *context)
{
   /* automatically load goodbye when library is loaded */
   sh_addbuiltin("goodbye", fpm_goodbye, 0);
}

Assuming you built a shared library called libhello.so which contains the above code, and placed this library in the shared library location, you can access the two custom builtins, hello and goodbye as follows:

$ builtin hello hello
$ hello Joe
Hello Joe
$ goodbye Joe
Goodbye Joe
$

To remove both custom builtins at the same time you must specify both on the command line as follows

$ builtin –d hello –d goodbye
$

Note that goodbye was not named internally as b_goodbye in this example but as fpm_goodbye since the “b_” function prefix naming requirement is relaxed when a custom builtin is loaded via lib_init() and sh_addbuiltin().

Example 2   Another way to load custom builtins.

If for some reason you do not wish to use the AST sfio routines, you should include <shcmd.h> and then call LIB_INIT(context) to initialize the shell context so that ksh93 knows that you are not using the sfio routines.

#include <ast/shcmd.h>
#include <stdio.h>

int 
b_goodbye(int argc, char *argv[], void *extra) 
{
   if (argc != 2) {       
      printf("Usage: goodbye arg\n");
      return(2);
   }
   
   printf("Goodbye %s\n", argv[1]);
 
   return(0);
}

int
b_hello(int argc, char *argv[], void *extra)
{
   if (argc != 2) {     
      printf("Usage: hello arg\n");
      return(2);
   }

   printf("Hello %s\n", argv[1]);

   return(0);
}

void
lib_init(int c, void *context)
{
   LIB_INIT(context);

   sh_addbuiltin("hello", b_hello, 0);
   sh_addbuiltin("goodbye", b_goodbye, 0);
}

Assuming that you build a shared library called libhello.so, both custom builtins are automatically loaded when the shared library is loaded, i.e.

$ builtin –f libhello.so

Example 3    Self-documentating custom builtins

One of the advantages of ksh93 over other shells is the fact that online help is provided for most commands using -?, --man or --help. Self-documenting code is supported in custom builtins but is very poorly documented. All custom builtins can generate their own manual page in several formats. While this documentation takes up space in the shared library and loaded image, the benefits probably outweight the cost.

#include <ast/shell.h>

#define SH_DICT "libgoodbye"

static const char usage_goodbye[] =
   "[-?\n@(#)$Id: goodbye 2008-04-06 $\n]"
   "[-author?Finnbarr P. Murphy <fpmATun-ixDOTcom>]"
   "[-licence?http://www.opensource.org/licenses/cpl1.0.txt]"
   "[+NAME?goodbye - output goodbye message]" 
   "[+DESCRIPTION?\bgoodbye\b outputs either a short or extended"    
   "message to stdout.]"
   "[x:xtended?Output extended message]"
   "\n"
   "\nname\n"
   "\n"
   "[+EXIT STATUS?] {"
      "[+0?Success.]"
      "[+>0?An error occurred.]"
   "}"
   "[+SEE ALSO?\bprint\b(1), \becho\b(1)]"
;

int
b_goodbye (int argc, char *argv[], void *extra)
{
   register int n, extend=0;

   while (n = optget(argv, usage_goodbye)) switch(n) {
      case 'x':
         extend=1;
         break;
      case ':':
         error(2, "%s", opt_info.arg);
         break;
      case '?':
         errormsg(SH_DICT, 
         ERROR_usage(2), "%s", opt_info.arg);
         break;
      }

      argc -= opt_info.index;
      argv += opt_info.index;

      if (argc != 1)
         errormsg(SH_DICT, 
         ERROR_usage(2), "%s", optusage((char *)0));

      if (extend)
         sfprintf(sfstdout, "Goodbye for now %s\n", *argv);
      else
         sfprintf(sfstdout, "Goodbye %s\n", *argv);

      return(0);
}

void
lib_init(int c, void *context)
{    
   sh_addbuiltin("goodbye", b_goodbye, 0);
}

Example 4   Memory management within custom builtins

This example shows how to encrypt a character string using a numeric key. It uses a relatively simple XOR operation to encrypt the string and is commonly known as the Vernam cipher or XOR encryption.

To prevent memory leaks in your custom builtins you should avoid using malloc() and calloc() and similar routines. The preferred way to obtain, free and manage memory space when required in a custom builtin is to use the libast stk(3) routines. Memory leaks can arise when a custom builtin does not free all of its allocated memory upon return or is interrupted by a signal such as SIGINT before it can do so.

This custom builtin uses the stkcopy() routine to get the required space. This memory is guaranteed to be freed by libast when the custom builtin exits, i.e. no additional code such as free() is required to free up the allocated memory.

#include <shell.h>
#include <stk.h>

#define SH_DICT "strcrypt"

static const char usage_strcrypt[] =
   "[-?\n@(#)$Id: strcrypt 2008-05-04 $\n]"
   "[-author?Finnbarr P. Murphy <fpmAThotmailDOTcom>]"
   "[-licence?http://www.opensource.org/licenses/cpl1.0.txt]"
   "[+NAME?strcrypt - encrypt string using numeric key]"
   "[+DESCRIPTION?\bencrypt\b a string using a numeric key.
        Note uses XOR to do encryption. Only works when
        numeric key used. Use same numeric key to decrypt.]"
   "[+OPTIONS?none.]"
   "\n"
   "\nstring key\n"
   "\n"
   "[+EXIT STATUS?] {"
        "[+0?Success.]"
        "[+>0?An error occurred.]"
   "}"
   "[+SEE ALSO?\bcrypt\b(2)]"
;

int
b_strcrypt(int argc, char *argv[], void *extra)
{
   int i, sl, pl, c;
   char *v, *s, *p;

   while (i = optget(argv, usage_strcrypt)) switch(i) {
      case ':':
         error(2, "%s",opt_info.arg);
         break;
      case '?':
         errormsg(SH_DICT, ERROR_usage(2),
            "%s", opt_info.arg);
         break;
    }
   argc -= opt_info.index;
   argv += opt_info.index;

   if (argc != 2)
      errormsg(SH_DICT, ERROR_usage(2), "%s",
         optusage((char *)0));

   s  = argv[0];
   sl = strlen(s);
   p  = argv[1];
   pl = strlen(p);

   /* copy string onto stack so it can be modified */
   if (!(v = (char *)stkcopy(stkstd, s))
      error(3, "stkcopy failed");

   for (i = 0; i < sl; i++) {
      c = s[i] ^ toupper(p[i%pl]);
      if (c != 0)
         v[i] = (char)c;
   }

   sfprintf(sfstdout,"%s", v);

   return(0);
}

With this custom builtin, a string of text (argument 1) is encrypted using the supplied numeric key (argument 2). To decrypt the encrypted string, simply invoke the custom builtin again using the encrypted string and same numeric key.

$ strcrypt “hello” 1478
YQ[T^
$ strcrypt “YQ[T^“ 1478
hello
$

Note that libast also contains a set of routines whose names start with stak and which provide similar functionality. However the stak(3) routines are marked deprecated and should not be used in custom builtins.

Example 5   Using name-value routines to access ksh93 internals

The purpose of this custom builtin is to set the value of a specified variable (argument 1) to the size of the specified file (argument 2).

$ statsize myfilesize /usr/bin/ksh
$ print ${myfilesize}
1157295
$

This example uses the libast nval(3) name-value routines to access the internals of ksh93 and set up a variable with the specifed name (i.e. myfilesize) and value of 1157295 as returned by stat(2) for the file /usr/bin/ksh. See the nval(3) documentation for more detailed information.

#include <shell.h>
#include <nval.h>

#define SH_DICT "statsize"

static const char usage_statsize[] =
   [-?\n@(#)$Id: stat 2008-05-03 $\n]"
   [-author?Finnbarr P. Murphy <fpmAThotmailDOTcom>]"
   [-licence?http://www.opensource.org/licenses/cpl1.0.txt]"
   [+NAME?statsize - assign size of file to variable]"
   [+DESCRIPTION?\bstat\b assigns the size of the specified file"
      "(in bytes) to the specified variable.]"
   "[+OPTIONS?none.]"
   "\n"
   "[+EXIT STATUS?] {"
      "[+0?Success.]"
      "[+>0?An error occurred.]"
   "[+SEE ALSO?\bstat\b(2)]"
;

int
b_statsize(int argc, char *argv[], void *extra)
{
   Namval_t *nvp = (Namval_t *)NULL;
   Shell_t *shp = (Shell_t *)NULL;
   struct stat st;
   long d;
   register int n;

   while (n = optget(argv, usage_statsize)) switch(n) {
      case ':':
        error(2, "%s", opt_info.arg);
        break;
      case '?':
        errormsg(SH_DICT, ERROR_usage(2), "%s", 
           opt_info.arg);
        break;
   }
   argc -= opt_info.index;
   argv += opt_info.index;

   if (argc != 2)
      errormsg(SH_DICT, ERROR_usage(2), "%s", 
         optusage((char*)0));

   /* get current shell context */
   shp = sh_getinterp();

   /* retrieve information about file */
   stat(argv[1], &st);
   /* assign size of file to long */
   d = (long) st.st_size;

   /* access the variables tree and add specified variable */
   nvp = nv_open(argv[0], shp->var_tree,     
          NV_NOARRAY|NV_VARNAME|NV_NOASSIGN);
   if (!nv_isnull(nvp))
      nv_unset(nvp);
   nv_putval(nvp, (char *)&d, NV_INTEGER|NV_RDONLY);
   nv_close(nvp);

   return(0);
}

This custom builtin can easily be extended to provide much more information about a file using command line options. I will leave it as an exercise for you the reader.

Example 6   Print out detailed information about a specified shell variable.

$ showvar HOME  
Value: /home/fpm, Flags: 12288 NV_EXPORT NV_IMPORT
$ integer i=123
$ showvar i
Value: 123, Flags: 10 NV_UINT64 NV_UTOL

See the libast header <nval.h> for detailed information about the different flags which can be associated with each variable. Note that some flags are overloaded so that they mean different things according to how they are OR’ed with other flags.

#include <shell.h>
#include <nval.h>

#define SH_DICT "showvar"

static const char usage_showvar[] =
   "[-?\n@(#)$Id: showvar 2008-05-04 $\n]"
   "[-author?Finnbarr P. Murphy <fpmAThotmailDOTcom>]"
   "[-licence?http://www.opensource.org/licenses/cpl1.0.txt]"
   "[+NAME?showvar - display variable details]"
   "[+DESCRIPTION?\bshowvar\b displays details about the  
                   specified variable.]"
   "[+OPTIONS?none.]"
   "\n"
   "\nvariable_name\n"
   "\n"
   "[+EXIT STATUS?] {"
        "[+0?Success.]"
        "[+>0?An error occurred.]"
   "}"
   "[+SEE ALSO?\bstat\b(2)]"
;

struct Flag {
   int flag;
   char *name;
};

/* Note: not a complete list of all possible flags */
struct Flag Flags[] = {
   NV_ARRAY,   "NV_ARRAY",
   NV_BINARY,  "NV_BINARY",
   NV_EXPORT,  "NV_EXPORT",
   NV_HOST,    "NV_HOST",
   NV_IMPORT,  "NV_IMPORT",
   NV_LJUST,   "NV_LJUST",
   NV_LTOU,    "NV_LTOU",
   NV_RAW,     "NV_RAW",
   NV_RDONLY,  "NV_RDONLY",
   NV_REF,     "NV_REF",
   NV_RJUST,   "NV_RJUST",
   NV_TABLE,   "NV_TABLE",
   NV_TAGGED,  "NV_TAGGED",
   NV_UTOL,    "NV_UTOL",
   NV_ZFILL,   "NV_ZFILL",
   0,(char *)NULL     
};

struct Flag IntFlags[] = {
   NV_LTOU|NV_UTOL|NV_INTEGER,   "NV_UINT64",
   NV_LTOU|NV_RJUST|NV_INTEGER,  "NV_UINT16",
   NV_RJUST|NV_ZFILL|NV_INTEGER, "NV_FLOAT",
   NV_UTOL|NV_ZFILL|NV_INTEGER,  "NV_LDOUBLE",
   NV_RJUST|NV_INTEGER,          "NV_INT16",
   NV_LTOU|NV_INTEGER,           "NV_UINT32",
   NV_UTOL|NV_INTEGER,           "NV_INT64",
   NV_RJUST,                     "NV_SHORT",
   NV_UTOL,                      "NV_LONG",   
   NV_LTOU,                      "NV_UNSIGN",
   NV_ZFILL,                     "NV_DOUBLE", 
   NV_LJUST,                     "NV_EXPNOTE",
   NV_INTEGER,                   "NV_INT32(NV_INTEGER)",
   0,(char *)NULL     
};

int
b_showvar(int argc, char *argv[], void *extra)
{
    Shell_t *shp = (Shell_t *)NULL;
    Namval_t *nvp = (Namval_t *)NULL;
    char *ptr = (char *)NULL;
    int i;
     
    while (i = optget(argv, usage_showvar)) switch(i) {
         case ':':
           error(2, "%s", opt_info.arg);
           break;
         case '?':
           errormsg(SH_DICT, ERROR_usage(2), "%s", opt_info.arg);
           break;
    }
    argc -= opt_info.index;
    argv += opt_info.index;

    if (argc != 1)
        errormsg(SH_DICT, ERROR_usage(2), "%s", optusage((char*)0));

    /* get current shell context */
    shp = sh_getinterp();

    if ((nvp = nv_search(*argv, shp->var_tree, 0)) == NULL) {
           errormsg(SH_DICT, ERROR_exit(1), 
              "%s: variable not found", *argv);
        return(1);
    }

    if ((ptr = nv_getval(nvp)) == NULL) {
        errormsg(SH_DICT, ERROR_exit(3), 
                "%s: variable is NULL", *argv);
        return(1);
    }

    sfprintf(sfstdout, 
             "Value: %s, Flags: %d", ptr, (int)nvp->nvflag);
    if ((int)nvp->nvflag & NV_INTEGER) {
       for (i=0; IntFlags[i].name != NULL; i++) {
          if ((int)nvp->nvflag & IntFlags[i].flag) {
              sfprintf(sfstdout, " %s", IntFlags[i].name);
              break;
          } 
       }
    } 
    for (i=0; Flags[i].name != NULL; i++) {
        if ((int)nvp->nvflag & Flags[i].flag)
            sfprintf(sfstdout, " %s", Flags[i].name);
    }

    sfprintf(sfstdout,"\n");
    nv_close(nvp);

    return(0);
}

Well, that is about all that you really need to know about ksh93 custom builtins in order to start developing your own custom builtins. For your information, these examples were tested on versions ksh93s and ksh93t. Good luck and feel free to ask questions. I will try and answer any relevant questions.