IEEE Std. 1003.1-2008

I am a big fan of the POSIX.1 standard.  The lastest version of this standard (IEEE Std. 1003.1-2008) was released in December 1st 2008.  The formal name for this version of the standard is Portable Operating System Interface (POSIX) Base Specifications, Issue 7.

Like its predecessors IEEE Std. 1003.1-2004 and IEEE Std. 1003.1-2001 it was developed by the Austin Common Standards Revision Group (CSRG), more commonly called the Austin Working Group, which is a joint technical working group established in 1999 to develop and maintain the core open systems interfaces.  The working group's stated approach to specification development is write once, adopt everywhere, with the deliverables being a set of specifications that simultaniously carry the IEEE POSIX designation, The Open Group's Technical Standard designation, and the ISO/IEC designation.  Thus the new version of the specifications is simultaneously ISO/IEC 9945, IEEE Std. 1003.1-2008 and The Open Group Base Technical Standard Specifications, Issue 7.

The standard describes the characteristics and facilities of an operating system that are of importance to application developers.  It intended, however, to be used by both application developers and system implementors.  More precisely, it defines a standard operating system interface and environment, including a shell and utilities to support application portability at the source code level.  It contains approximately 4000 pages and is divided into 4 volumes.  General definitions,terms, concepts, and interfaces common to all volumes of this standard, including utility conventions and C programming language header definitions, are included in the Base Definitions volume.  Definitions for system service functions and subroutines, language-specific system services for the C programming language, function issues, including portability, error handling, and error recovery, are included in the System Interfaces volume.  Definitions for a command interpreter (shell) language and various utilities for application developemnt and support are included in the Shell and Utilities volume. The Rational volume contains information that did not fit well into the other volumes, including historical information about the standard and why particular features were included, not included or removed by the working group.

The standard contains a base set of interfaces and functionality which is required in all compliant systems, together with a number of option groups.  Based on these options and option groups, profiles can be defined which further group the options which are needed for a particular application domain.  Some of the option groups contain a lot of functionality, others very little.  The size of the mandatory base set was significantly increased in this revision.  Ulrich Drepper has put up Web page about POSIX options.  It is Linux-centic but nevertheless is quite useful and worth looking at.

The following summarizes the major changes that were incorporated in the standard: Austin Group defect reports, IEEE Interpretations against IEEE Std 1003.1-2004, responses to ISO/IEC 9945 defect reports, Open Group corrigenda and resolutions, the Batch Environment Services and Utilities option was marked obsolescent, the <sys/timeb.h> and <ucontext.h> headers were removed, the Open Group Technical Standard, 2006, Extended API Set, parts 1 to 4 was incorporated and existing functionality was aligned with ISO/IEC 9899:1999 and ISO/IEC 9899:1999 Corregenda 2:2004.

The following functions were added:



ialphasort()dirfd()dprintf()
duplocale()faccessat()fchmodat()
fchownat()fdopendir()fexecve()
fmemopen()freelocale()fstatat()
futimens()getdelim()getline()
isalnum_l()isalpha_l()isblank_l()
iscntrl_l()isdigit_l()isgraph_l()
islower_l()isprint_l()ispunct_l()
isspace_l()isupper_l()iswalnum_l()
iswalpha_l()iswblank_l()iswcntrl_l()
iswctype_l()iswdigit_l()iswgraph_l()
iswlower_l()iswprint_l()iswpunct_l()
iswspace_l()iswupper_l()iswxdigit_l()
isxdigit_l()linkat()mbsnrtowcs()
mkdirat()mkdtemp()mkfifoat()
mknodat()newlocale()openat()
open_memstream()open_wmemstream()psiginfo()
psignal()renameat()scandir()
stpcpy()stpncpy()strcasecmp_l()
strcoll_l()strfmon_l()strncasecmp_l()
strndup()strnlen()strsignal()
strxfrm_l()symlinkat()tolower_l()
toupper_l()towctrans_l()towlower_l()
towupper_l()unlinwcsncasecmp()
wcsncasecmp_l()wcsnlen()wcsnrtombs()
wcsxfrm_l()wctrans_l()wctype_l()

The following functions were previously part of an option group but are now mandatory:





































aio_cancel()aio_error()aio_fsync()
aio_read()aio_return()aio_suspend()
aio_write()asctime_r()catclose()
catgets()catopen()clock_getres()
clock_gettime()clock_nanosleep()clock_settime()
ctime_r()dlclose()dlerror()
dlopen()dlsym()fchdir()
flockfile()fstatvfs()ftrylockfile()
funlockfile()getc_unlocked()getchar_unlocked()
getgrgid_r()getgrnam_r()getlogin_r()
getpgid()getpwnam_r()getpwuid_r()
getsid()getsubopt()gmtime_r()
iconv()iconv_close()iconv_open()
lchown()lio_listio()localtime_r()
mkstemp()mmap()mprotect()
munmap()nanosleep()nl_langinfo()
poll()posix_trace_timedgetnext_event()
pread()pthread_atfork()pthread_exit()















































pthread_attr_destroy()pthread_attr_getdetachstate()
pthread_attr_getguardsize()pthread_attr_getschedparam()
pthread_attr_init()pthread_attr_setdetachstate()
pthread_attr_setguardsize()pthread_attr_setschedparam()
pthread_barrier_destroy()pthread_barrier_init()
pthread_barrierd_cond_wait()pthread_condattr_destroy()
pthread_condattr_getclock()pthread_condattr_init()
pthread_condattr_setclock()pthread_create()
pthread_detach()pthread_equal()
pthread_getspecific()pthread_join()
pthread_key_create()pthread_keyhread_mutex_timedlock()
pthread_mutex_trylock()pthread_mutex_unlock()
pthread_mutexattr_destroy()pthread_mutexattr_gettype()
pthread_mutexattr_init()pthread_mutexattr_settype()
pthread_once()pthread_rwlock_destroy()
pthread_rwlock_init()pthread_rwlocryrdlock()
pthread_rwlock_trywrlock()pthread_rwlock_unlock()
pthread_rwlock_wrlock()pthread_rwlockattr_destroy()
pthread_rwlockattr_init()pthread_self()
pthread_setcancelstate()pthread_setcanceltype()
pthread_setspecific()pthread_spin_destroy()
pthre_unlock()pthread_testcancel()
putc_unlocked()putchar_unlocked()





















pwrite()rand_r()readdir_r()
sem_close()sem_destroy()sem_getvalue()
sem_init()sem_open()sem_post()
sem_timedwait()sem_trywait()sem_unlink()
sem_wait()sigqueue()sigqueue()
sigtimedwcmp()strdup()strerror_r()
strfmon()strncasecmp()strtok_r()
tcgetsid()timer_create()timer_delete()
timer_getoverrun()timer_gettime()timer_settime()
truncate()ttyname_r()waitid()

The following functions were marked as obsolescent:



















asctime()asctime_r()ctime()
ctime_r()gets()rand_r()
tmpnam()utime()_longjmp()
_setjmp()_tolower()_toupper()
ftw()getitimer()gettimeofday()
isascii()setitimer()setpgrp()
sighold()sigignore()sigpause()
sigrelse()sigset()siginterrupt()
tempnam()toascii()ulimit()

The following functions and symbols were removed:

















bcmp()bcopy()bsd_signal()
bzero()ecvt()fcvt()
ftime()gcvt()getcontext()
gethostbyaddr()gethostbyname()getwd()
h_errnoindex()makecontext()
mktemp()rindex()scalb()
setcontext()swapcontext()ualarm()
usleep()vfork()wcswcs()

Finally, if you want easy access to the specifications from your FireFox browser, you can install the Firefox Search Bar' plugin which I have posted at Mycroft Project.  It searchs the online version of Base Specification Issue 7 at the Open Group.  Like ECMA and a number of other standards development organizations, the Open Group has a policy of free access to their specifications online whereas the IEEE charges for access to their standards.

Well, this is my last post for 2008.  What a year it has been!  A Happy and Prosperous New Year to all.
 

Ruby 1.9.1 RC Released

In his usual understated way, Yuhi Sonoda announced the first Ruby 1.9.1 release candidate earlier today on the ruby-talk mailing list.

Hi, folks

Ruby 1.9.1 has been just released.

This is a release candidate of Ruby 1.9.1, which will be
the first stable version of Ruby 1.9 series. Try it early
and have a experience of modern, faster, with clearer
syntax, multilingualized and much improved Ruby world.

We have fixed 72 bugs and implemented some features:
http://redmine.ruby-lang.org/projects/ruby-19/issues?query_id=9

If you encounter a bug or a problem, please let us know
it via the official issue tracking system
(http://redmine.ruby-lang.org ).
I am a fan of Ruby and look forward to every release.  My first introduction to Ruby was way back in early days of 1997 just after Ruby 1.0 was released by Yukihiro Matsumoto (aka Matz) whose blog, or as the Japanese prefer to term it web diary is here.  Since then I have used it from time to time for various small scripts in the same why that I also use Perl.  I recommend it to people when asked what programming language they should learn first.  My only reservations are that traditionally Ruby tends to be somewhat slow compared to other popular scripting langauges and that there now any number of Ruby implementations which leads to some portability issues.  Performance has improved with every release and there is no reason nowadays not to use Ruby in production enviroments.

This release is significant in that it is the first release candidate of Ruby 1.9.1 which is targeted to be the first stable production-grade version of the Ruby 1.9 (aka Yarv) series.  Ruby 1.9 is intended to be a major evolation of the Ruby language akin to Python 3.0 or Perl 6.  Old crud and langauge features that were, in hindsight, poorly designed have been removed.  A number of new language features are implemented.  If you want more information about the changes in Ruby 1.9, a good reference to start with is Mauricio Fernandez's Changes in Ruby 1.9.  With this release, the language features are frozen except for multiligual support is some of the standard libraries which come with Ruby.  I have played with beta releases of Ruby 1.9 for some time now and am pleased with the language changes and especially the improved threading model.

As I said previously, there is not one single implementation of the Ruby language.  The implementation released today is regarded as the official Ruby language implementation in that it is maintained by Matz, Yuhi and a number of trusted lieutenants.  It is sometimes refered to as CRuby since the underlying language is C.

If you are unfamilar with the different Ruby implementations other than CRuby, the leading contenders, in no particular order, are Ruby Enterprise Edition, JRuby, Rubinius, MagLev and finally Microsoft's IronRuby which interestingly they are developing in a semi-open model.  You can actually contribute code and examine the source code!  For a comparision of the performance of the various implementations, have a look at Antonio Cangiano's Great Ruby Shootout 2008.  To interpret these results correctly, you need to read his followup post Reflections on the Ruby Shootout.

I still have some reservationa about the speed of code execution in Ruby.  While Ruby 1.9.1 is around twice as fast as Ruby 1.8.7, Ruby is still quite slow compared to other scripting languages such as Python or Perl even if it is somewhat faster than PHP.  Work still needs to be done to improve performance.

If you want to compare the performance of Ruby to other languages for various benchmarks, one place you can do so is at Debian's Alioth which claims to be the successor to Doug Bagley's The Great Computer Language Shootout amongst other things.  Other benchmarks worth looking at are here and here.  However, keep in mind the the golden rule of benchmarking, to wit - all benchmarks are flawed benchmarks.

You can download the source code for this release at ruby-lang.org.  If you find the ruby-lang.org FTP server is too slow, Peter Cooper has made the tarball available on Amazon S3 which should result in a much faster download.

For some reason, new releases of CRuby always seem to occur in December.  Anybody know why?

KSH93 Auditing and Accounting

Korn Shell 93 (ksh93) is the only UNIX or GNU/Linux shell that I am aware of that, with proper setup, supports a modicum of per-user accounting and auditing.  This post attempts to explain these facilities and show you how to access and manipulate the resulting accounting and auditing records.

Per-user accounting has been a feature of ksh93 since the earliest days of this shell.  It is a fairly primative facility which writes out an (undocumented) record for each user command that is executed.

An auditing facility was added in July 2008.  This is somewhat more sophisticated than the accounting facility in that it is configurable and writes out a more detailed record either locally or to a remote system for each user command that is executed.  This facility can be used to monitor, track, record, and audit the activities of one or more users on an system, including system administrators.

Both facilities only work for interactive users.  Neither facility is enabled by default.  In fact, you have to go into the ksh93 source code, modify the main Makefile to enable certain compile time options, and then recompile the sources.  This is not a trivial exercise in many cases as rebuilding ksh93 also requires that you to rebuild a number of libraries.  Source code is available at the AT&T Research AST software download site.



To build ksh93 with these facilities enabled, you must build the ksh93 executable using either the ast-base or ast-open package together with the INIT package.  Just as the shared libraries are not built if you use the ast-ksh package, neither are the accounting and auditing facilities.  I have never investigated why this is so, but I am sure that Glenn Fowler and David Korn have their reasons.

You need to modify the compile time options in ../src/cmd/ksh93/Makefile as follows to enable one or both facilities.
SHOPT_ACCT ==     1        /* accounting */
SHOPT_ACCTFILE == 1 /* per user accounting info */
SHOPT_AUDIT == 1 /* auditing */
SHOPT_AUDITFILE == "/etc/ksh_audit" /* auditing file */
After you have recompiled the sources, the new ksh executable is located in ../arch/..../bin/ subdirectory.  To see what options have actually been compiled into a particular executable, just print out the shell version string.
bash-3.2$ pwd
/home/fpm/ksh/build/arch/linux.i386-64/bin
bash-3.2$ ls -al ksh
-rwxr-xr-x 1 fpm fpm 1356931 2008-12-26 16:31 ksh
bash-3.2$ ./ksh
$ echo ${.sh.version}
Version AJLM 93t+ 2008-12-10
$ echo $KSH_VERSION
Version AJLM 93t+ 2008-12-10
The option string AJLM means that (A) auditing is enabled, (J) one SIGCHLD trap per completed job is supported, (L) per-user accounting is supported, and (M) multibyte characters are supported.

Per-user accounting is enabled using the SHACCT environmental variable.  To turn on per-user accounting, simply set SHACCT to the name of the file where you wish to store the accounting records.
export SHACCT="/tmp/ksh_acctfile"
Here is part of the resulting file. Note that the time is stored as hexadecimal seconds since the Epoch.
$ cat /tmp/ksh_acctfile
echo ${.sh.version} fpm 495990d8
pwd fpm 495990da
id fpm 495990dd
date fpm 495990e3
exit fpm 495990e5
The following shell script can be used to access the records in this file and output them in a more useful format.
#!/bin/ksh93
ACCTFILE="/tmp/ksh_acctfile"

printf "DATE TIME LOGIN COMMAMD\n\n"

# set IFS to TAB only
while IFS=" " read cmdstr name hexseconds
do
longsecs=$(printf "%ld" "0x${hexseconds}")
timestr=$(printf "%(%Y-%m-%d %H:%M:%S)T" "#${longsecs}")
print $timestr, $name, "$cmdstr"
done < $ACCTFILE
Invoking this script gives the following output for the above accounting records.
$ ./parse_acctfile
DATE TIME LOGIN COMMAMD

2008-12-29 22:09:12, fpm, echo ${.sh.version}
2008-12-29 22:09:14, fpm, pwd
2008-12-29 22:09:17, fpm, id
2008-12-29 22:09:23, fpm, date
2008-12-29 22:09:25, fpm, exit
Next we turn our attention to the auditing facility.  In addition to rebuilding ksh93 with the SHOPT_AUDIT option, you must create an audit configuration file on each system to tell ksh93 where to store the audit records and to specify which users are to be audited.  The default location for the configuration file is /etc/ksh_audit but that location be changed in the main ksh93 Makefile.  The configuration file should contain a line that defines the file to write the audit records to, followed by the UID of each user whose commands are to generate audit records.  Here is the configuration file used to generate the audit records for this part of this post.
$ cat /etc/ksh_audit
/tmp/ksh_auditfile;500
This configuration file specifies that audit records are to be written to /tmp/ksh_auditfile for the user who's UID is 500.  Note that the field delimiter is a semi-colon.

Here are the audit records stored in the /tmp/ksh_auditfile which match the accounting records shown previously in this post.  The field separator is a semi-colon.  The first field is the UID of the user executing the command.  The second field is the time in seconds since the Epoch.  The third field is the terminal device on which the command was executed, and the final field is the actual command executed by the user.
500;1230606552;/dev/pts/2; echo ${.sh.version}
500;1230606554;/dev/pts/2; pwd
500;1230606557;/dev/pts/2; id
500;1230606563;/dev/pts/2; date
500;1230606565;/dev/pts/2; exit
As before, here is a simple ksh93 script which parses this audit file, replaces the UID with the actual user's name and seconds since the Epoch with the actual data and time, and outputs the enhanced records in a comma separated value (CSV) format.
#!/bin/ksh93

AUDITFILE="/tmp/ksh_auditfile"

while IFS=";" read uid sec tty cmdstr
do
name=""
while IFS=":" read pwname pword pwuid rest
do
if [[ "$uid" == "$pwuid" ]]
then
name=$pwname
break
fi
done < /etc/passwd

timestr=$(printf "%(%Y-%m-%d %H:%M:%S)T" "#$sec")
print "$timestr", $name, $uid, $tty, "$cmdstr"
done < $AUDITFILE
Here is the output for the above audit records.
2008-12-29 22:09:12, fpm, 500, /dev/pts/2,  echo ${.sh.version}
2008-12-29 22:09:14, fpm, 500, /dev/pts/2, pwd
2008-12-29 22:09:17, fpm, 500, /dev/pts/2, id
2008-12-29 22:09:23, fpm, 500, /dev/pts/2, date
2008-12-29 22:09:25, fpm, 500, /dev/pts/2, exit
If the underlying operating system supports networking using the /dev/udp/host/port syntax or the /dev/tcp/host/port syntax, audit records can be sent by ksh93 across a network to another system.  This mechanism could be used to store audit records on a secured centralized system to which only specific personnel have access.  As an example, the following audit configuration file line designates that audit records for the user who's UID is 500 should be sent using UDP to the syslog network port (514) on a remote system who's IP is 192.168.0.99.
/dev/udp/192.168.0.99/514;500
Here are the same audit records stored by the syslog daemon on the remote system.
2008-12-29 22:09:12 192,169.0.115 500;1230606552;/dev/pts/2; echo ${.sh.version}
2008-12-29 22:09:14 192.169.0.115 500;1230606554;/dev/pts/2; pwd
2008-12-29 22:09:17 192.169.0.115 500;1230606557;/dev/pts/2; id
2008-12-29 22:09:23 192.169.0.115 500;1230606563;/dev/pts/2; date
2008-12-29 22:09:25 192.169.0.115 500;1230606565;/dev/pts/2; exit
Depending on the configuration of the syslog daemon on your particular system, the first part of the record may contain more or less information or be formatted differently but the final part of the record, i.e. the audit record sent by ksh93 should be in the standard audit record format.

Note that while the auditing and accounting facilities within ksh93 can provide you with much useful information regarding the actions of one or more users on a system or systems, these facilities should not be regarded as providing enhanced security akin to the Trusted Computing Base (TCB).  There are many ways of circumventing these facilities.  For example, a knowledgeable user could switch to a different shell such as bash where their actions will not be recorded.  There are a number of other ways but I will not discuss them here.

Most of the information provided in this post is not documented in a single place anywhere that I can find by searching the Internet.  The ksh93 man page does not mention either the accounting or auditing facilities.  Even the ksh93 source code is somewhat vague.  I gleened most of this information by studying the code in ../src/cmd/ksh93/edit/history.c.  If I got anything wrong, please let me know so that I can update this post.

PowerShell CTP3

There was an early Christmas present from the Windows PowerShell (AKA PoSH) Team.  The Community Technology Preview 3 (CTP3) of Windows PowerShell v2.0 was released on December 23rd just in time for Christmas.  The announcement is here.  As expected CTP3 builds on the new technology provided in CTP2 which was released in May 2008.  You can download CTP3 from the Microsoft Download Center.

Hemant Mahawar, Program Manager for PowerShell, summarized the CTP3 release as follows:

This release brings, among other things, performance improvements ... things will be faster/more efficient than before. PowerShell remoting now allows implicit remoting where command execution appears to be local even though they are remote. We have added over 60 new cmdlets in this release ... cmdlets for adding/removing/renaming computers, cmdlets for event logs, cmdlets for WS-Man functionality and even a WS-Man provider. The “graphical” host, Windows PowerShell ISE, now supports a graphical debugger, context sensitive F1 help and a programmable interface for you to party on.
I tested CTP3 on Vista Ultimate SP1.  The only issue I encountered when installing CTP3 was that fact that CTP3 did not honor the execution policy set by me in CTP2 contary to what was stated in the Release Notes.  Moreover, it was impossible to set the execution policy to unrestricted using Set-Executionpolicy.



After digging around in the registry hives, the problem became apparant.  The PowerShell execution policy is set correctly in [HKEY_LOCAL_MACHINE\Software\Microsoft\PowerShell\1\ShellIds\Microsoft.PowerShell\executionPolicy] but not in [HKEY_CURRENT_USER\Software\Microsoft\PowerShell\1\ShellIds\Microsoft.PowerShell\executionPolicy].  I found this registry entry had to manually changed from allsigned to unrestricted.

One major enhancement in this release relates to remoting and background jobs.  Both require that you install Windows Remote Management (WinRM) 2.0 CTP3.  Currently WinRM 2.0 CTP3 is supported only on Windows Vista SP1 and on Windows Server 2008.  For some reason that I do not yet understand background jobs, even the jobs only run on the local computer, rely on the remoting features of PowerShell.

The othere major enhancement relates to what was known as Script CmdLets in CTP2.  They have been renamed to advanced functions in CTP3.  Advanced functions are functions that have the same capabilities and behaviors as cmdlets but are written using the PowerShell scripting language instead of a compiled language such as C#.

There are two types of advanced functions, i.e. named functions and unnamed functions.   Both types use the CmdletBinding attribute to identify themselves as advanced functions that act similar to compiled cmdlets.  Both types can also be used within a script file.  The difference is that running a script file only declares a named function but an unnamed function is actually invoked.  The CmdletBinding attribute is similar to the Cmdlet attribute used in compiled cmdlet classes to identify the class as a cmdlet.  For further information on Advanced Functions, type man about_functions_advanced.

Here is a simple CTP3 script which defines a single advanced function named get-myrandom that returns a pseudo-random number.
#
# Sample PowerShell v2.0 CTP3 script
#

Set-StrictMode -Version Latest

DATA msgText {
#culture="en-US"
ConvertFrom-StringData -stringdata @'
msg001 = Thank you for trying this script.
'@
}

function get-myrandom {

<#
.Synopsis
Generate a random number
.Description
This is a simple function to demonstrate some of the
PowerShell v2.0 advanced functions capabilities by
generating a random number.
.Parameter Max
Maximum value of the generated random number
.Example
PS> get-myrandom 46
.InputType
[int]
.ReturnValue
[int]
.Link
about_functions
about_functions_advanced
about_functions_advanced_methods
about_functions_advanced_parameters
about_script_internationalization
.Notes
AUTHOR: Finnbarr P. Murphy
DATE: 12/27/2008
REQUIRES: v2.0 CTP3
#>

[CmdletBinding(SupportsShouldProcess=$true)]
PARAM(
[Parameter(Mandatory=$true)]
[ValidateRange(50,99)]
[int]
$max
)

Begin {
import-localizeddata -bindingVariable msgText
$random = new-object System.Random
}

Process {
$random.Next($max)
}

End {
$msgText.msg001
}
}
The DATA section contains text strings which are replaced with the appropriate translated strings if available depending on the locale (cultural) settings of your computer.  ConvertFrom-StringData converts the text strings into dictionary-like hash tables to facilitate translation.  The code import-localizeddata -bindingVariable msgText does the actual work of retrieving the translated text strings from the appropriate myrandom.psd1 file in the locale specific subdirectories as shown below.
C:\Scripts
C:\Scripts\myrandom.ps1
C:\Scripts\de-DE\myrandom.psd1
C:\Scripts\ar-SA\myrandom.psd1
C:\Scripts\zh-CN\myrandom.psd1
The text between the comment start and end tags is parsed and displayed as help text when get-myrandom is queried using man or get-helpThe pseudo random number generator is instantiated as an object in the Begin{} block.  The actual work of the function, as far as the user is concerned, is performed in the Process{} block which returns a generated pseudo-random number.  Finally the Process{} outputs a locale appropriate text string if available otherwise the default text string provided in the DATA section.

One small nit.  I noticed that the TechNet online documentation for PowerShell has not been updated to reflect CTP3.  Also much of the built-in help documentation supplied with CTP3 is quite frankly incorrect.   Hopefully the PowerShell team will clean up the documentation before long.

BTW, for those of you who do not have PowerShell installed on your computer, Microsoft TechNet has a PowerShell Scripting Online Virtual Lab which is available to anybody who wants to learn about and try out PowerShell scripting.  Currently it uses Powershell v1.0.

As always, enjoy!

Listen to My Posts

You can now listen my posts being read by either a male or female using an automatic text to speech translater developed by Vozme which uses the Festival Speech Synthesis System developed by the The Center for Speech Technology Research at the University of Edinburgh in Scotland.

Festival is multi-lingual though the British and American dialects of English are the most advanced.  Tools and documentation for building new voices are available from the Carnegie Mellon University FestVox project.

I have added two white buttoms to the bottom of posts, one marked Male and one marked Female.  Just select whichever voice you want to hear my post in.  You can also save the post as a MP3 recording.

Try the service out and let me know if you run into any problems.

Fairytale of New York

Well Christmas is fast approaching so I decided to experiment with adding audio and video to my blog.  If you are from the United Kingdom, Ireland, or much of western Europe you will be no doubt familar with a folk-rock group called The Pogues and their Christmas song called Fairytale of New York which was first released in 1987 as part of the album If I Should Fall from Grace with God. This song is a perennial favorite for many people at this time of the year.  It is a duet between Kirsty MacColl and Shane MacGowan.  Kirsty MacColl died in Cozumel Mexico in 2000 protecting one of her sons during a dive from a speeding powerboat which entered a resticted dive site.  Incidentially, Matt Dillon is the actor who played the New York policeman in the video which was released with the song.

YouTube is simply amazing.  I found over 20 different videos for Fairytale of New York.  Here is the original video, as they say, for your listening and viewing pleasure.





Here is the original video with subtitles if you want to read the lyrics.





I expect that Google is quietly working away on automating voice to subtitles.  Just like you can now view this post in over 30 different langauges, it would be nice if you could view subtitles for songs and videos in those same languages.

Finally, if you want see Shane MacGowan and Kirsty MacColl perform the song live together on stage.





Enjoy!  And a Merry Christmas to all.

XSLT Variable Arrays

I recently answered a question on a popular programmers forum about how to store and access an array of user-defined variables in a stylesheet and then loop though those variables.  I realized that many developers are not familar with the available techniques for doing this and decided to add an entry in my blog about this topic.

User-defined variable arrays within stylesheets are not part of the XSLT specification.  The usual way to handle this problem in XSLT 1.0 stylesheets is to define a user-defined top-level element which belongs to a non-null namespace which is different from the XSLT namspace.  These user-defined top-level elements are typically used to store error messages, lookup data, etc.  You can then access these user-defined elements from within your stylesheet by treating the stylesheet as an additional source document and loading it using the document() function with an empty string as the first argument.  An empty string is interpreted to mean the current stylesheet.

The following stylesheet demonstrates this method.
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"
xmlns:foo="http://foo.com" exclude-result-prefixes="foo">

<xsl:output method="text" encoding="utf-8"/>

<foo:vars>
<foo:var name="z1">A</foo:var>
<foo:var name="z2">B</foo:var>
<foo:var name="z3">C</foo:var>
<foo:var name="z4">D</foo:var>
</foo:vars>

<xsl:template match="/">
<xsl:for-each select="document('')/xsl:stylesheet/foo:vars/foo:var" >
<xsl:value-of select="." />
<xsl:if test="position() != last()">
<xsl:text>,</xsl:text>
</xsl:if>
</xsl:for-each>
<xsl:text>
</xsl:text>
</xsl:template>

</xsl:stylesheet>
If you are using XSLT 2.0, this method is no longer needed as simpler and more elegant methods are available to us.  For example, you can store and directly access the variables using the <xsl:variable> element as shown in the following stylesheet.
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0" >
<xsl:output method="text" encoding="utf-8"/>

<xsl:variable name="z1" select="'A'" />
<xsl:variable name="z2" select="'B'" />
<xsl:variable name="z3" select="'C'" />
<xsl:variable name="z4" select="'D'" />
<xsl:variable name="vars" select="$z1, $z2, $z3, $z4" />

<xsl:template match="/">
<xsl:for-each select="$vars" >
<xsl:value-of select="." />
<xsl:if test="position() != last()">
<xsl:text>,</xsl:text>
</xsl:if>
</xsl:for-each>
<xsl:text>
</xsl:text>
</xsl:template>

</xsl:stylesheet>
Another way to store and access this data in a XSLT 2.0 stylesheet is to use a global variable definition as shown below.
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0" >

<xsl:output method="text" encoding="utf-8"/>

<xsl:variable name="vars">
<var name="z1">A</var>
<var name="z2">B</var>
<var name="z3">C</var>
<var name="z4">D</var>
</xsl:variable>

<xsl:template match="/">
<xsl:for-each select="$vars/var" >
<xsl:value-of select="." />
<xsl:if test="position() != last()">
<xsl:text>,</xsl:text>
</xsl:if>
</xsl:for-each>
<xsl:text>
</xsl:text>
</xsl:template>

</xsl:stylesheet>
All three stylesheets output the same data.



You may be wondering about this image.  It is from the screen of my laptop.  I used xsltproc on Microsoft Vista SUA for the XSLT 1.0 transformation and Saxon 9 in Microsoft Powershell v2.0 CTP2 for the two XSLT 2.0 transformations.

Note that the XSLT 1.0 method described above still works in XSLT 2.0.  If stylesheet portability is an issue for you, I suggest you continue to use this method.

Google Page Translator

Today, I added a gadget to my blog so that readers can easily translate my posts into any one of thirty five languages using the Google Translate webpage language translation engine.  You will find it low on the left hand side of this webpage.  From the drop down list, just pick the language you want to convert the current post to and after a short delay a new webpage will appear with the post displayed in the selected language.  Note - you need to have scripts enabled in your browser for the Google gadget to display on this webpage.

Adding this functionality was more challanging than I initially assumed since I wanted to ensure that my posts remained useful to a reader when displayed in a foreign language.  This means that the translation engine needs to be told what words and sections to leave alone as is.  The Google documentation and FAQ for this gadget are unclear about how to do this for things like my display boxes so a round of experimentation was required.  It turns out that you can mark a entire webpage, a word or a section of a webpage as not to be translated using the follows tags.
<!-- use meta element to protect entire webpage -->
<meta name='google' value='notranslate'>
<!-- use span element to protect a word or two -->
<span class='notranslate'>Do not translate these words</span>
<!-- use div element to protect display boxes such this -->
<div class='notranslate'>Do not translate entire section</div>
Now that I have figured out how to protect specific text and code snippets from being translated by the Google translation engine, I plan to go back and modify my previous posts over a period of time to make them more robust for automated language translation.

This will be quite a project as I typically include a number of programming langauge reserved words and code snippets in my posts.  For example, it required over 100 edits to my previous post on Microsoft's Powershell before I was happy that that particular post was suitable for automated language translation.

More on PowerShell

Microsoft's PowerShell is radically different than shells on UNIX or GNU Linux systems in that Powershell can deal in objects rather than just plain text.

A concrete example may help you more quickly understand the difference.  Suppose you want to get and save information about all the files in a certain subdirectory.   We want to get not only the names of the files but as much metadata as possible relating to each file such as date of creation, date of modification, etc.  This information also needs to be stored in a single XML document.  To keep the size of this post manageable, our subdirectory contains only two files, i.e. file.xml and file.xsl, as shown below.



On a UNIX or GNU Linux system, this would be a somewhat difficult task to quickly accomplish.  Using Powershell, however, it becomes a simple one or two line task.

Because Powershell is fundamentally object-orientated, it regards a filesystem as being an object representated as a location exposed by provider.  More generally, a location can be a file directory, a registry hive, a certificate store or some other "thing" exposed by a provider.  The Powershell get-childItem cmdLet is used to retrieve information, including metadata, about items in a specified location.  Thus get-childItem can be used for a lot of different types of operations including enumerating directory contents, listing registry values or showing the current values of variables.  For our purposes, get-childItem exposes information about files in a specified subdirectory or the current directory if no directory is specified.

Here is a single line Powershell script which does what we want to do.   It retrieves metadata about the two files in the current directory using the get-childItem cmdLet, pipes that data to a cmdLet called ConvertTo-XML which in turn converts the metadata into a valid XML document.  The save method is then used to write this stream out to a file called file.out in the current directory.
(get-childItem | ConvertTo-XML -NoTypeInformation).save("$(convert-path "X:")\file.out")
Alternatively, we could store the retrieved metadata in a variable and then write it out to a file as shown below.  Similar to bash and ksh93, Powershell supports the concept of aliases and defines ls as one of the aliases for the get-childItem cmdLet,dir and gci being the other two.
$a=(ls | ConvertTo-XML)
$a.save("$(convert-path "X:")\file.out")
Here is the contents of file.out.  As you can see, quite a bit of information about each of the two files is exposed.
<?xml version="1.0"?>
<Objects>
<Object Type="System.IO.FileInfo">
<Property Name="PSPath" Type="System.String">Microsoft.PowerShell.Core\FileSystem::C:\PowerShell\file23.xml</Property>
<Property Name="PSParentPath" Type="System.String">Microsoft.PowerShell.Core\FileSystem::C:\PowerShell</Property>
<Property Name="PSIsContainer" Type="System.Boolean">False</Property>
<Property Name="BaseName" Type="System.String">file23</Property>
<Property Name="Mode" Type="System.String">-a---</Property>
<Property Name="Name" Type="System.String">file23.xml</Property>
<Property Name="Length" Type="System.Int64">294</Property>
<Property Name="DirectoryName" Type="System.String">C:\Windows\SUA\home\fpm\PowerShell</Property>
<Property Name="Directory" Type="System.IO.DirectoryInfo">C:\Windows\SUA\home\fpm\PowerShell</Property>
<Property Name="IsReadOnly" Type="System.Boolean">False</Property>
<Property Name="Exists" Type="System.Boolean">True</Property>
<Property Name="FullName" Type="System.String">C:\Windows\SUA\home\fpm\PowerShell\file23.xml</Property>
<Property Name="Extension" Type="System.String">.xml</Property>
<Property Name="CreationTime" Type="System.DateTime">12/17/2008 2:22:17 PM</Property>
<Property Name="CreationTimeUtc" Type="System.DateTime">12/17/2008 6:22:17 AM</Property>
<Property Name="LastAccessTime" Type="System.DateTime">12/17/2008 2:22:17 PM</Property>
<Property Name="LastAccessTimeUtc" Type="System.DateTime">12/17/2008 6:22:17 AM</Property>
<Property Name="LastWriteTime" Type="System.DateTime">12/17/2008 1:25:15 PM</Property>
<Property Name="LastWriteTimeUtc" Type="System.DateTime">12/17/2008 5:25:15 AM</Property>
<Property Name="Attributes" Type="System.IO.FileAttributes">Archive</Property>
</Object>
<Object Type="System.IO.FileInfo">
<Property Name="PSPath" Type="System.String">Microsoft.PowerShell.Core\FileSystem::C:\PowerShell\file23.xsl</Property>
<Property Name="PSParentPath" Type="System.String">Microsoft.PowerShell.Core\FileSystem::C:\PowerShell</Property>
<Property Name="PSChildName" Type="System.String">file23.xsl</Property>
<Property Name="BaseName" Type="System.String">file23</Property>
<Property Name="Mode" Type="System.String">-a---</Property>
<Property Name="Name" Type="System.String">file23.xsl</Property>
<Property Name="Length" Type="System.Int64">459</Property>
<Property Name="DirectoryName" Type="System.String">C:\Windows\SUA\home\fpm\PowerShell</Property>
<Property Name="Directory" Type="System.IO.DirectoryInfo">C:\Windows\SUA\home\fpm\PowerShell</Property>
<Property Name="IsReadOnly" Type="System.Boolean">False</Property>
<Property Name="Exists" Type="System.Boolean">True</Property>
<Property Name="FullName" Type="System.String">C:\Windows\SUA\home\fpm\PowerShell\file23.xsl</Property>
<Property Name="Extension" Type="System.String">.xsl</Property>
<Property Name="CreationTime" Type="System.DateTime">12/17/2008 2:22:17 PM</Property>
<Property Name="CreationTimeUtc" Type="System.DateTime">12/17/2008 6:22:17 AM</Property>
<Property Name="LastAccessTime" Type="System.DateTime">12/17/2008 2:22:17 PM</Property>
<Property Name="LastAccessTimeUtc" Type="System.DateTime">12/17/2008 6:22:17 AM</Property>
<Property Name="LastWriteTime" Type="System.DateTime">12/17/2008 2:40:27 PM</Property>
<Property Name="LastWriteTimeUtc" Type="System.DateTime">12/17/2008 6:40:27 AM</Property>
<Property Name="Attributes" Type="System.IO.FileAttributes">Archive</Property>
</Object>
</objects>
Now suppose requirements changed and we had to save the metadata as a comma seperated value (CSV) file.  Again this is easy to do in Powershell using the Export-CSV cmdLet.
get-childItem | Export-CSV ./file.out
Alternatively, as before, you could store the metadata first in a variable and then writing it out to file in the required format.
$a=(ls)
$a | Export-CSV ./file.out
Here is what file.out now contains.
"Microsoft.PowerShell.Core\FileSystem::C:\Windows\SUA\home\fpm\psh\file.xml","Microsoft.PowerShell.Core\FileSystem::C:\W
indows\SUA\home\fpm\psh","file.xml","X","Microsoft.PowerShell.Core\FileSystem","False", "file","-a---","file.xml","294",
"C:\Windows\SUA\home\fpm\psh","C:\Windows\SUA\home\fpm\psh","False","True","C:\Windows\SUA\home\fpm\psh\file.xml",".xml"
,"12/17/2008 2:22:17 PM","12/17/2008 6:22:17 AM","12/17/2008 2:22:17 PM","12/17/2008 6:22:17 AM","12/17/2008 1:25:15 PM"
,"12/17/2008 5:25:15 AM","Archive"
"Microsoft.PowerShell.Core\FileSystem::C:\Windows\SUA\home\fpm\psh\file.xsl","Microsoft.PowerShell.Core\FileSystem::C:\W
indows\SUA\home\fpm\psh","file.xsl","X","Microsoft.PowerShell.Core\FileSystem","False", "file","-a---","file.xsl","459",
"C:\Windows\SUA\home\fpm\psh","C:\Windows\SUA\home\fpm\psh","False","True","C:\Windows\SUA\home\fpm\psh\file.xsl",".xsl"
,"12/17/2008 2:22:17 PM","12/17/2008 6:22:17 AM","12/17/2008 2:22:17 PM","12/17/2008 6:22:17 AM","12/17/2008 2:40:27 PM"
,"12/17/2008 6:40:27 AM","Archive"
As you can see from these simple examples, it is possible to support objects in a command line shell.  I would like to see one or more of the UNIX/GNU Linux shells evolve to also support objects.  It a different paradign to what existing shell script writers on these platforms are accustomed to but it could enable more robust scripts to be written, simplify many existing scripts, and make scripts easier to understand and maintain.  David Korn has publicly said that this is a direction that he is thinking about for ksh93.  It will be interesting to see what he comes up with.

P.S. The above examples are based on Powershell v2.0 CTP2 and do not work on Powershell v1.0.

Grady Booch Podcasts

To anybody who has received formal software enginnering training, the name Grady Booch is instantly recognizable for his work and influence on software architecture, software engineering, and modeling.

He was an early advocate of formal software engineering, object orientated design and what we know today as software patterns.  The Booch Method, which he detailed in his book Object Oriented Analysis and Design is an object modeling language and software development methodology that was widely used in object-oriented analysis.  Along with with Ivar Jacobson and James Rumbaugh, he co-developed version 1.0 of the Unified Modeling Language.

Unless you are a member of the IEEE Computer Society, you are probably unaware the he recently recorded his On Architecture columns from IEEE Software as podcasts which are available for free download here .

So far there are 7 podcasts online; the latest being "The Irrelevance of Architecture."  I don't know how long they will be available for downloading but this is your chance to hear one of the greats talk about software architecture and development methodology.

PowerShell Grows Up

I am excited about the emminent release of Microsoft Windows Powershell Version 2 CTP3 (Community Technology Preview Version 3) which is due "real soon now."   The first CTP for Powershell 2.0 was in November 2007 and there has been a lot of progress on the product since then.  See the permanant link at the side of my blog to access the Powershell developers blog.

Why am I, a UNIX/Linux developer, excited by Powershell V2?  After all it does not run on any UNIX or GNU Linux platform and microsoft has no plans to port itMto these platforms.  The main reason is that Powershell V2 supports non-compiled functions (known as script cmdlets) in scripts just like bash or ksh93 does while providing deep object orientated interfaces to the operating system and underlying platform.  Previously cmdlets had to be written in a compiled language such as C# or VB.NET in order to provide such functionality.  (Currently, the terminology for what is generally know as a function in a UNIX/GNU Linux shell script is called a script cmdlet in Powershell but this is apparantly on track to change to function in CTP3.)  Another reason is the introduction of a graphic interface to Powershell.

I have spend more than 25 years of my life working on or with UNIX, UNIX-like or GNU/Linux operating systems.  For a period of that time, I was responsible for maintenance and enhancement of various shells and hence still maintain an interest in this area.  Much progress has occured in UNIX/GNU Linux shells over that period of time with bash, zsh and ksh93 emerging as the de facto leaders and csh, tcsh, sh (as in the Steve Bourne shell) and ash slowly fading away into obscurity.  

Standardization of functionality for portable shell scripts has moved from simply "use the Bourne shell if you want your script to run everywhere" to written specifications such as POSIX which all three shells support.  Numerious attempts (dtksh, Wksh, tksh come to mind) have been made to develop shells with graphical interfaces but none has been really successful.  Currently work is afoot to provide deeper interfaces into the operating system and platform with the latest versions of ksh93 having support for compiled builtins, compound variables and quasi-objects (which I plan to write about soon in another post.)

While it may be tantamount to heresy to say this, but kudos to the Microsoft Powershell development team who have managed in my opinion to leapfrog all the UNIX//GNU Linux shells in terms of functionality with Powershell V2.  I think UNIX/GNU Linux shell developers could learn a lot from taking a hard look at its design and functionality.

KSH93 Extended I/0

The ksh93 exec command is a special overloaded built-in command that can be used to manipulate file descriptors or to replace the current shell with a new command.  Starting with version s of ksh93, which was released in 2006, ksh93 contains a number of very useful extensions to exec relating to file descriptors and input/output which most shell programmers are unaware of.

From the current ksh93 man page
<#((expr))    Evaluate  arithmetic  expression  expr   and
position file descriptor 0 to the resulting
value bytes from the start of the file. The
variables CUR and EOF evaluate to the cur-
rent offset and end-of-file offset respec-
tively when evaluating expr.

>#((offset)) The same as <# except applies to file
descriptor 1.

<#pattern Seeks forward to the beginning of the next
line containing pattern.

<##pattern The same as <# except that the portion of
the file that is skipped is copied to stan-
dard output.
This post is about the these four extensions.  It is assumed that you are familar with ksh93 and the standard shell redirection facilities that are available in most modern shells, e.g.
exec 1>file          # redirect stdout to file
exec 2>&1 # redirect stderr to same
exec 1>&- # close file descriptor
If not, a quick Web search will return plenty of tutorials on the subject.

As with most Unix and GNU/Linux man pages, the ksh93 man page is quite cryptic and concise so before we go any further I will attempt to describe what each extension can be used for.  The first extension is useful for determining the current offset within a file or can be used to return the size of a file by by returning the value of the offset of the end of the value.  (You can also get the current file offset with offset=$(fd<#).)  The second extension can be used to position a file descriptor anywhere within a file.  The third extension can be used to seek forward to the start of a line containing a pattern specified by you.  The final extension is useful in that it outputs all lines between the current offset and the start of a line containing a pattern specified by you.  A number of things should be noted.  First, these extensions only work with text files.  Second, newlines count towards the offset from the start of the file.  Another thing to watch out for is while it is not clear from the ksh93 man page, it appears you must open a file for both read and write is you want to perform random access.  Why this is so is unclear to me but you get odd results if you seek backwards in a file even if you are only reading the file.

Because exec is a special command, any failure in I/O redirection will cause the script that invokes it to immediately terminate.  This can be prevented by invoking exec from the command built-in.  Incidently, redirect is an predefined alias for command exec.

For our first example, we will demonstrate the how to use these extensions to perform some basic I/O operations on a simple text file.
#!/bin/ksh93

TMP=file.$$

# create the temporary file used by example
cat <<EOT >$TMP
aaa
bbb
ccc
ddd
eee
fff
EOT

# open file descriptor 3 for read/write
command exec 3<> $TMP || exit 1

# check file descriptor 3 position
print
print "At offset: $(3<#)"
if (($(3<#) != 0))
then
print "Not at offset 0"
exit 1
fi

# read in the first line and print it
read -u3
print $REPLY
print "At offset $(3<#) after reading line"
print

# search forward for string "ddd"
3<#"ddd"
print "At offset $(3<#) after search forward for 'ddd'"
read -u3
print $REPLY
print

# check that we are at offset 8 and, if so, read line
if (( $(3<# ((8))) != 8))
then
print "Not at offset 8"
exit 1
fi
print "At offset $(3<#) after specifying absolute offset of 8"
read -u3
print $REPLY
print

# go to end of file. We know by inspection that is at offset 24, so check.
if (( $(3<#((EOF))) != 4*6 ))
then
print "Not at EOF"
exit 1
fi
print "At offset $(3<#) after specifying EOF"
print

# backup one line i.e. 4 characters
3<#((CUR - 4))
print "At offset $(3<#) after backing up 4 characters"
read -u3
print $REPLY
print

redirect 3<&- || echo 'cannot close FD 3'

rm $TMP
The following is the output from this example
$ ./example1

At offset: 0
aaa
At offset 4 after reading line

At offset 12 after search forward for 'ddd'
ddd

At offset 8 after specifying absolute offset of 8
ccc

At offset 24 after specifying EOF

At offset 20 after backing up 4 characters
fff
$
Our next example will selectly output the lines of a file between 2 search strings 'test' and '**' and display these lines as a single line.
$!/bin/ksh93

TMP=file1.$$
OUT=file2.$$

# create the temporary file used by example
cat <<'EOT' >$TMP
aaa
test
bbb
ccc
ddd
eee
fff
**
ggg
EOT

# redirect stdout
exec 4>&1

exec 3< $TMP > $OUT

# seek to start of line containing "test"
3<#'test'
# advance 5 characters i.e. the start of next line
3<# ((CUR + 5))
# output all lines until EOF or "**" found
3<##'\*\**'

exec 3<&-

# swap back stdout
exec 1>&-
exec 1>&4
exec 4>&-

rm $TMP

# hack to output as a single line
out=$(<$OUT)
echo $out

rm $OUT
The following is the output when we run this shell script
$ ./example2
bbb ccc ddd eee fff
$
For our next example, consider the following fictitious file which contains the results of a series of pings to local systems.
Wed Nov  4 15:25:14 EST 2008
PING Host11 (10.0.201.51) 56(84) bytes of data.
--- Host11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4010ms
rtt min/avg/max/mdev = 7.400/41.641/59.604/17.946 ms
.
Wed Nov 4 15:25:18 EST 2008
PING Host12 (10.0.202.51) 56(84) bytes of data.
--- Host12 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4016ms
rtt min/avg/max/mdev = 43.804/55.694/67.728/9.359 ms
.
Wed Nov 4 15:25:22 EST 2008
PING Host13 (10.0.205.51) 56(84) bytes of data.
--- Host13 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 40.237/56.283/68.973/10.433 ms
.
Wed Nov 4 15:25:26 EST 2008
PING Host14 (10.0.201.52) 56(84) bytes of data.
--- Host14 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 36.053/65.964/96.782/20.642 ms
The requirement is to create a table showing min, loss and mdev values for each host pinged.  Here is our shell script
#!/bin/ksh93

redirect 3< file

# get and store value of EOF
eof=$(3<#((EOF)))

# check that we are at start of file i.e. offset 0
if (( $(3<#((0))) != 0 ))
then
print "ERROR: not at offset 0"
exit 1
fi

# loop though the file and build one line of output per host
while (( $(3<#((CUR))) < $eof ))
do
3<#'PING*'
read -A -u3 arr1
3<#'*packet loss*'
IFS=" %" read -A -u3 arr2
IFS=" /" read -A -u3 arr3
printf "%s %6.3f %3d %6.3f\n" ${arr1[1]} ${arr3[6]} ${arr2[5]} ${arr3[9]}
done

redirect 3<&-
Here is the output after this script is run against the above file
$ ./example3
Host11 7.400 0 17.946
Host12 43.804 0 9.359
Host13 40.237 0 10.433
Host14 36.053 0 20.642
Our final example will demonstrate how to replace a line and append additional lines to an existing file.  Note also that we specify the use of a file descriptor > 10 by means of the extended file descriptor {variable} syntax.
#!/bin/ksh93
TMP=file.$$

# create temporary file
cat <<EOT >$TMP
aaa
bbb
ccc
ddd
eee
fff
EOT

# open FD > 10 for read/write
command exec {n} <> $TMP || exit 1
print "File descriptor assigned is: ${n}"
print

# search forward for string "ddd"
{n}<#"ddd"

# replace this string with "DDD"
print -u${n} -f "%.3c\n" D

# go to end of file
redirect {n}>#((EOF))

# add 2 lines of text
print -u${n} -f "%.6c\n" G
print -u${n} -f "%.6c\n" H

redirect {n}>&- || echo 'cannot close FD'

cat $TMP

rm $TMP
Here is the output for this script.
$ ./example4
File descriptor assigned is: 11

aaa
bbb
ccc
DDD
eee
fff
GGGGGG
HHHHHH
Note that ksh93 does not support syntax of the form exec {n}<>&- since <&- and >&- both close the file descriptor n.  If you do the following in ksh93
exec {n}<&-
exec {n}>&-
you do not get an error.  On the other hand, if you do the same in zsh an error message is issued saying file descriptor n is already closed.

As you can see from the these examples, the extended capabilities of exec in ksh93 can be used to do file manipulation and transformation operations that were previously only possible using utilities such as sed, grep and awk.  Enjoy!

Half-Precision Floating Point Format

Half precision floating point is a 16-bit binary floating-point interchange format.  It was not part of the original ANSI/IEEE 754 Standard for Binary Floating-Point Arithmetic published in 1985 but is included in the current version of the standard, IEEE 754-2008 (previously known as IEEE 754r) which was published last August.  See this Wikipedia article for background information.

Floating point formats defined by this standard are classified as either interchange or non-interchange.  In the standard storage formats are narrow interchange formats, i.e. the set of floating point values that can be stored by the specified binary encoding is a proper subset of wider floating point formats such as the 32-bit float and 64-bit double.  In particular, the standard defines the encodings for one binary storage floating-point format of 16 bits length and radix 2, and one decimal storage floating-point format of 32 bits length.  Note that these two formats are for storage only and are not defined for in-memory arithmetic operations.  The remainder of this post is about the 16-bit binary storage format which we will refer to as half precision.

Half precision (also known as a 1.5.10 or s10e5 minifloat) was added to the standard because it is the de facto storage format for certain floating-point values frequently used in modern graphics processing units (GPUs) where minimizing memory usage and bus traffic is a major challange and priority.&  It is used in several computer graphics environments including OpenGL, OpenEXR, and by hardware in MP3 decoders and nVidia graphic cards.  This format became popular because it can store a larger range of values than an int16 without requiring the bandwidth and storage space of a float type.  Typically this increased range of numbers is used to preserve more highlighting and shadow detail.  The minimum and maximum representable values are 2.98×10-8 and 65504 respectively.

I only know of one C or C++ compiler which supports half precision i.e. Sourcery G++ Lite.  It uses an __fp16 type to represent half precision with a number of limitations.  However, whether __fp16 becomes part of ISO C remains to be seen.&  The lastest version of the C++ ABI also provides some support for name mangling of half precisions.  The GNU debugger appears to have some limited support.  Ruby supports half precision (IEEE_binary16) using the float-formats package (but only for little endian platforms according to the float-formats README).  The Python structs module which is the logical home for half-precision support does not currently support this format.  A cursory search of CPAN did not reveal any modules with support for half precision in Perl.

The following is a small C program which demonstrates how to encode and decode the half precision binary format.  It is a based on some code from OGRE (Object-oriented Graphics Rendering Engine) header.
/*
** This program is free software; you can redistribute it and/or modify it under
** the terms of the GNU Lesser General Public License, as published by the Free
** Software Foundation; either version 2 of the License, or (at your option) any
** later version.
**
** IEEE 758-2008 Half-precision Floating Point Format
** --------------------------------------------------
**
** | Field | Last | First | Note
** |----------|------|-------|----------
** | Sign | 15 | 15 |
** | Exponent | 14 | 10 | Bias = 15
** | Fraction | 9 | 0 |
*/

#include <stdio.h>
#include <inttypes.h>

typedef uint16_t HALF;

/* ----- prototypes ------ */
float HALFToFloat(HALF);
HALF floatToHALF(float);
static uint32_t halfToFloatI(HALF);
static HALF floatToHalfI(uint32_t);

float
HALFToFloat(HALF y)
{
union { float f; uint32_t i; } v;
v.i = halfToFloatI(y);
return v.f;
}

uint32_t
static halfToFloatI(HALF y)
{
int s = (y >> 15) & 0x00000001; // sign
int e = (y >> 10) & 0x0000001f; // exponent
int f = y & 0x000003ff; // fraction

// need to handle 7c00 INF and fc00 -INF?
if (e == 0) {
// need to handle +-0 case f==0 or f=0x8000?
if (f == 0) // Plus or minus zero
return s << 31;
else { // Denormalized number -- renormalize it
while (!(f & 0x00000400)) {
f <<= 1;
e -= 1;
}
e += 1;
f &= ~0x00000400;
}
} else if (e == 31) {
if (f == 0) // Inf
return (s << 31) | 0x7f800000;
else // NaN
return (s << 31) | 0x7f800000 | (f << 13);
}

e = e + (127 - 15);
f = f << 13;

return ((s << 31) | (e << 23) | f);
}

HALF
floatToHALF(float i)
{
union { float f; uint32_t i; } v;
v.f = i;
return floatToHalfI(v.i);
}

HALF
static floatToHalfI(uint32_t i)
{
register int s = (i >> 16) & 0x00008000; // sign
register int e = ((i >> 23) & 0x000000ff) - (127 - 15); // exponent
register int f = i & 0x007fffff; // fraction

// need to handle NaNs and Inf?
if (e <= 0) {
if (e < -10) {
if (s) // handle -0.0
return 0x8000;
else
return 0;
}
f = (f | 0x00800000) >> (1 - e);
return s | (f >> 13);
} else if (e == 0xff - (127 - 15)) {
if (f == 0) // Inf
return s | 0x7c00;
else { // NAN
f >>= 13;
return s | 0x7c00 | f | (f == 0);
}
} else {
if (e > 30) // Overflow
return s | 0x7c00;
return s | (e << 10) | (f >> 13);
}
}

int
main(int argc, char *argv[])
{
float f1, f2;
HALF h;

printf("Please enter a floating point number: ");
scanf("%f", &f1);

h = floatToHALF(f1);
f2 = HALFToFloat(h);

printf("Results are: %f %f %#lx\n", f1, f2, h);
}
The following example shows one way to read a single floating point number (1.0) encoded in a half precision binary floating point two byte string into a float using Python.
import struct

def HalfToFloat(h):
s = int((h >> 15) & 0x00000001) # sign
e = int((h >> 10) & 0x0000001f) # exponent
f = int(h & 0x000003ff) # fraction

if e == 0:
if f == 0:
return int(s << 31)
else:
while not (f & 0x00000400):
f <<= 1
e -= 1
e += 1
f &= ~0x00000400
print s,e,f
elif e == 31:
if f == 0:
return int((s << 31) | 0x7f800000)
else:
return int((s << 31) | 0x7f800000 | (f << 13))

e = e + (127 -15)
f = f << 13

return int((s << 31) | (e << 23) | f)


if __name__ == "__main__":

# a half precision binary floating point string
FP16='\x00\x3c'

v = struct.unpack('H', FP16)
x = HalfToFloat(v[0])

# hack to coerce to float
str = struct.pack('I',x)
f=struct.unpack('f', str)

# print the resulting floating point
print f[0]
I hope this post provided you with some useful information on the interesting subject of the half precision floating point binary format.  I suspect within a few years most compilers and scripting langauges will support it.

XSLT Copy with Exception

A recent problem that was posed to me concerned how to copy the entire contents of an XML document with certain exceptions.  Turns out that the simplest way to handle this requirement in XSLT1.0 was to include the standard XSL identity template in my stylesheet and add another template to handle the exception.

A simple example will make things clearer.  Suppose we have the following XML document (which I shamelessly copied from W3Schools.com and modified to simplify) and we want to copy this document in its entirety except for details of CDs by a specific artist.
<?xml version="1.0"?>
<CATALOG>
<CD>
<TITLE>Empire Burlesque</TITLE>
<ARTIST>Bob Dylan</ARTIST>
<COUNTRY>USA</COUNTRY>
<COMPANY>Columbia</COMPANY>
<YEAR>1985</YEAR>
</CD>
<CD>
<TITLE>Hide your heart</TITLE>
<ARTIST>Bonnie Tyler</ARTIST>
<COUNTRY>UK</COUNTRY>
<COMPANY>CBS Records</COMPANY>
<YEAR>1988</YEAR>
</CD>
<CD>
<TITLE>Greatest Hits</TITLE>
<ARTIST>Dolly Parton</ARTIST>
<COUNTRY>USA</COUNTRY>
<COMPANY>RCA</COMPANY>
<YEAR>1982</YEAR>
</CD>
<CD>
<TITLE>One night only</TITLE>
<ARTIST>Bee Gees</ARTIST>
<COUNTRY>UK</COUNTRY>
<COMPANY>Polydor</COMPANY>
<YEAR>1998</YEAR>
</CD>
<CD>
<TITLE>Sylvias Mother</TITLE>
<ARTIST>Dr.Hook</ARTIST>
<COUNTRY>UK</COUNTRY>
<COMPANY>CBS</COMPANY>
<YEAR>1973</YEAR>
</CD>
<CD>
<TITLE>Maggie May</TITLE>
<ARTIST>Rod Stewart</ARTIST>
<COUNTRY>UK</COUNTRY>
<COMPANY>Pickwick</COMPANY>
<YEAR>1990</YEAR>
</CD>
</CATALOG>
Below is the stylesheet.  It accepts one parameter, i.e. the name, or part of the name, of an artist.  It copies all nodes and attributes to the output document except those nodes and attributes related to the artist whose name matches or is is a superset of the inputted string.
<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:param name="artist"></xsl:param>

<xsl:output method="xml"/>

<xsl:template match="node() | @*">
<xsl:copy>
<xsl:apply-templates select="@* | node()"/>
</xsl:copy>
</xsl:template>

<xsl:template match="/CATALOG/CD">
<xsl:if test="not(contains(./ARTIST,$artist))">
<xsl:copy-of select="." />
</xsl:if>
</xsl:template>
</xsl:stylesheet>
The first template is the standard identity template from the XSL 1.0 Transformations Recommendation.  It matches all attributes and all nodes being children of other nodes, and copies them to the output document.   The second template cancels the automatic copying of the <CD> nodes and children, checks to see that the name, or part of the name, of the artist does not match the search string, and only does a deep copy of the nodes and attributes to the output document if there is no match.

Assuming you are on a Linux system, you can use the command line utility xsltproc (which is part of the XSLT C library for GNOME) to transform the input document into the following output document when "Stewart" is passed as a parameter to the stylesheet.  BTW, note the use of Java style string quotes for the parameter string.
$ xsltproc -param artist "'Stewart'" file.xsl file.xml
Here is the output document. It is a copy of the input document except the "record" for the Rod Stewart's CD is not included.
<?xml version="1.0"?>
<CATALOG>
<CD>
<TITLE>Empire Burlesque</TITLE>
<ARTIST>Bob Dylan</ARTIST>
<COUNTRY>USA</COUNTRY>
<COMPANY>Columbia</COMPANY>
<YEAR>1985</YEAR>
</CD>
<CD>
<TITLE>Hide your heart</TITLE>
<ARTIST>Bonnie Tyler</ARTIST>
<COUNTRY>UK</COUNTRY>
<COMPANY>CBS Records</COMPANY>
<YEAR>1988</YEAR>
</CD>
<CD>
<TITLE>Greatest Hits</TITLE>
<ARTIST>Dolly Parton</ARTIST>
<COUNTRY>USA</COUNTRY>
<COMPANY>RCA</COMPANY>
<YEAR>1982</YEAR>
</CD>
<CD>
<TITLE>One night only</TITLE>
<ARTIST>Bee Gees</ARTIST>
<COUNTRY>UK</COUNTRY>
<COMPANY>Polydor</COMPANY>
<YEAR>1998</YEAR>
</CD>
<CD>
<TITLE>Sylvias Mother</TITLE>
<ARTIST>Dr.Hook</ARTIST>
<COUNTRY>UK</COUNTRY>
<COMPANY>CBS</COMPANY>
<YEAR>1973</YEAR>
</CD>
</CATALOG>
I hope this example will help you simplify your stylesheet the next time you have to copy a document with exceptions.  Functional programming is a very powerful paradigm but sometimes it is difficult to quickly formulate a simple and elegant solution if you have spent years programming using object-oriented or imperative paradigms.