Microsoft SUA JavaScript Shell

Recently I needed to test some JavaScript code on a Microsoft Vista Ultimate operating system using the command line. This is something that I have done before on Linux platforms using a JavaScript shell but had never done on a Microsoft platform. For those who are not familiar with a JavaScript shell, it is a command line interface to a JavaScript engine. Similar to Python or Ruby, the JavaScript shell has two modes of operation. You can use it as an interactive shell, in which you type JavaScript code at a prompt and get instant results, or you can invoke a JavaScript program.

The easiest way that I know of to build a JavaScript shell is to download and build the SpiderMonkey JavaScript engine which comes with a JavaScript Shell. SpiderMonkey is one of two JavaScript engines which the Mozilla software project supports. It is used in the Firefox browser and elsewhere. The other JavaScript engine is Rhino. Rhino is written using the Java language whereas SpiderMonkey is a pure C language implementation which conforms to ECMA-262 Edition 3. Included in SpiderMonkey is a JavaScript shell. Source code for both JavaScript engines is readily available on the Mozilla website.

For Microsoft Vista I had a choice of either building a native Windows executable which could be accessed via the Vista command prompt (similar to the old DOS prompt) or, because Vista Ultimate comes with Subsystem for Unix-based Applications (SUA), a shell which was accessable via a more familar Unix-like environment. I choice to go the SUA route.

Many people are unaware of Microsoft's SUA offering. Basicly it is a full-featured POSIX-compliant subsystem for Windows NT-based operating systems. For their own reasons, Microsoft limits the availablity of SUA to certain products such as Vista Enterprise and Ultimate editions and Windows Server 2008. It comes with hundreds of utilities including the GNU compiler collection, Korn shell, and development headers and libraries. See here for further information.

SUA is a subsystem which runs natively on top of the Windows kernel. which was orginally developed by Softway Systems. The original name for the product was OpenNT but that was later changed to Interix. Interix and most of the development team was acquired by Microsoft in 1999 and the product was renamed Services For Unix (SFU) in 2005. With the release of Microsft Vista, it was integrated into the operating system as a separate installable component and was renamed Subsystem for Unix-based Applications. There is an active user community around SUA which is hosted by Interop Systems and sponsored by Microsoft.

This is the Makefile which I hacked together to build a static js executable from the SpiderMonkey 1.8 sources for use with Microsoft Vista SUA.

JSVERSION = js-1.8.0

CC = gcc

OS_ARCH    := $(subst /,_,$(shell uname -s | sed /\ /s//_/))
OS_RELEASE := $(shell uname -r)
OS_CONFIG  := $(OS_ARCH)$(OS_RELEASE)
OBJDIR      = ./$(OS_CONFIG)

DEFINES     = -DXP_UNIX -DPOSIX_SOURCE -DHAVE_LOCALTIME_R

LIBDIR     := lib
INCLUDES   += -I$(OBJDIR)
RANLIB      = ranlib
LDFLAGS     = -lm

OPTIMIZER   =
INTERP_OPTIMIZER =

CFLAGS          =  $(OPTIMIZER) $(DEFINES) $(INCLUDES)
INTERP_CFLAGS   = $(INTERP_OPTIMIZER) $(DEFINES) $(INCLUDES)

PREDIRS += editline
DEFINES += -DEDITLINE

JS_CFILES =     jsapi.c jsarena.c jsarray.c jsatom.c jsbool.c jscntxt.c jsdate.c \
                jsdbgapi.c jsdhash.c jsdtoa.c jsemit.c jsexn.c jsfun.c jsgc.c jshash.c \
                jsinterp.c jsinvoke.c jsiter.c jslock.c jslog2.c jslong.c jsmath.c \
                jsnum.c jsobj.c jsopcode.c jsparse.c jsprf.c jsregexp.c jsscan.c \
                jsscope.c jsscript.c jsstr.c jsutil.c jsxdrapi.c jsxml.c prmjtime.c

JS_HFILES =     jsarray.h jsatom.h jsbool.h jsconfig.h jscntxt.h jsdate.h \
                jsemit.h jsexn.h jsfun.h jsgc.h jsinterp.h jsiter.h jslibmath.h \
                jslock.h jsmath.h jsnum.h jsobj.h jsopcode.h jsparse.h jsarena.h \
                jsclist.h jsdhash.h jsdtoa.h jshash.h jslong.h jstypes.h jsprvtd.h \
                jspubtd.h jsregexp.h jsscan.h jsscope.h jsscript.h jsstr.h jsutil.h \
                jsxdrapi.h jsxml.h

API_HFILES =    jsapi.h jsdbgapi.h

OTHER_HFILES =  jsbit.h jscompat.h jscpucfg.h jsotypes.h jsstddef.h jsopcode.tbl \
                jsproto.tbl js.msg jsshell.msg jskeyword.tbl prmjtime.h resource.h


ifdef JS_HAS_FILE_OBJECT
JS_CFILES += jsfile.c
JS_HFILES += jsfile.h
endif

LIB_CFILES  = $(JS_CFILES)
PROG_CFILES = js.c

LIB_OBJS  = $(addprefix $(OBJDIR)/, $(LIB_CFILES:.c=.o))
PROG_OBJS = $(addprefix $(OBJDIR)/, $(PROG_CFILES:.c=.o))

CFILES = $(LIB_CFILES) $(PROG_CFILES)
OBJS   = $(LIB_OBJS) $(PROG_OBJS)

LIBRARY = $(OBJDIR)/libjs.a
PROGRAM = $(OBJDIR)/js
TARGETS = $(PROGRAM)

EDIT_LIBRARY  = $(OBJDIR)/libedit.a
EDIT_DIR      = ./editline
EDIT_CFILES   = $(EDIT_DIR)/editline.c $(EDIT_DIR)/sysunix.c
EDIT_OBJS     = $(OBJDIR)/editline.o $(OBJDIR)/sysunix.o
EDIT_CFLAGS   = -DANSI_ARROWS -DHAVE_TCGETATTR -DHIDE -DUSE_DIRENT -DSYS_UNIX \
                -DHAVE_STDLIB -DUNIQUE_HISTORY $(DEFINES) -I$(EDIT_DIR)


define MAKE_OBJDIR
if test ! -d $(@D); then rm -rf $(@D); mkdir -p $(@D); fi
endef


all:
        $(MAKE) $(TARGETS)

$(PROGRAM): $(PROG_OBJS) $(LIBRARY) $(EDIT_LIBRARY)
        $(CC) -o $@ $(CFLAGS) $(PROG_OBJS) $(LIBRARY) $(EDIT_LIBRARY) $(LDFLAGS)

$(OBJDIR)/%.o: %.c %.h
        @$(MAKE_OBJDIR)
        $(CC) -o $@ -c $(CFLAGS) $*.c

$(OBJDIR)/editline.o: $(EDIT_DIR)/editline.c $(EDIT_DIR)/editline.h
        @$(MAKE_OBJDIR)
        $(CC) -o $@ -c $(EDIT_CFLAGS) $(EDIT_DIR)/editline.c

$(OBJDIR)/sysunix.o: $(EDIT_DIR)/sysunix.c $(EDIT_DIR)/editline.h
        @$(MAKE_OBJDIR)
        $(CC) -o $@ -c $(EDIT_CFLAGS) $(EDIT_DIR)/sysunix.c

$(OBJDIR)/%.o: %.c
        @$(MAKE_OBJDIR)
        $(CC) -o $(OBJDIR)/jscpucfg jscpucfg.c
        $(OBJDIR)/jscpucfg > $(OBJDIR)/jsautocfg.h
        $(CC) -o $(OBJDIR)/jskwgen jskwgen.c
        $(OBJDIR)/jskwgen $(OBJDIR)/jsautokw.h
        $(CC) -o $@ -c $(CFLAGS) $*.c

$(OBJDIR)/jsinterp.o: jsinterp.c jsinterp.h
        @$(MAKE_OBJDIR)
        $(CC) -o $@ -c $(INTERP_CFLAGS) jsinterp.c

$(EDIT_LIBRARY): $(EDIT_OBJS)
        $(AR) rv $@ $?
        $(RANLIB) $@

$(LIBRARY): $(LIB_OBJS)
        $(AR) rv $@ $?
        $(RANLIB) $@

clean:
        rm -rf $(OBJS)

clobber:
        rm -rf $(OBJDIR)

tarball:
        mkdir -p ./$(JSVERSION)/editline
        cp *.[ch] ./$(JSVERSION)
        cp *.tbl ./$(JSVERSION)
        cp *.js ./$(JSVERSION)
        cp *.msg ./$(JSVERSION)
        cp Makefile ./$(JSVERSION)
        cp README ./$(JSVERSION)
        cp editline/*.[ch] ./$(JSVERSION)/editline
        cp editline/README ./$(JSVERSION)/editline
        cp editline/editline.3 ./$(JSVERSION)/editline
        tar cvf $(JSVERSION).tar ./$(JSVERSION)
        gzip $(JSVERSION).tar
        rm -rf ./$(JSVERSION)

Note you need to use gcc 4.2 instead of gcc 3.2 otherwise you will get an undefined _JS_DHashTableOperate symbol due to an obsure gcc bug. However with gcc 4.2, you will see a large number of compiler warnings relating to alignment is greater than maximum object file. Fortunately you can safely ignore them. By the way, I choose to build a static executable because building shared libraries on SUA using gcc is somewhat problematic.



Once you have build an executable, a simple smoketest is to use the executable to execute the prefect.js JavaScript script which is included with the SpiderMonkey source code. You should get output similar to the following:

$ ./Interix6.0/js perfect.js

A number is 'perfect' if it is equal to the sum of its
divisors (excluding itself).

The perfect numbers up to 500 are:
6 = 1 + 2 + 3
28 = 1 + 2 + 4 + 7 + 14
496 = 1 + 2 + 4 + 8 + 16 + 31 + 62 + 124 + 248
That's all.
$

Well, this should be enough to enable you to build your own JavaScript shell on Microsoft's Subsystem for Unix-based Applications. Good Luck!
 

JavaScript E4X

In previous posts, I discussed the SpiderMonkey command line shell js and how to add support to it to enable full access (read, write, create, copy, delete, etc.) to the local filesystem via the File object and the NSPR library.

While rumaging around in the source code and documentation for js, I found that js partially supported the EX4 XML extension via a user configurable option.

This post looks at what it takes to load an XML document into js from your local filesystem, process it and write out the resulting document to your local filesystem using File objects and the E4X extension.

The ECMAScript for XML (E4X) (ECMA-357) specification adds native support for XML objects and XMLList objects to the JavaScript programming language.  This standard was first published in 2004 and was based on XML extensions provided in the BEA (now Oracle) Weblogic Workshop product.  These extensions were designed by Terry Lucas and John Schneider who led the ECMAScript for XML (E4X) initiative.

The basis idea behind E4X was that declarative languages such as XSL and XPATH are too complex for the average programmer to quickly learn and therefore a simpler way of accessing and manipulating XML documents was needed.  Personally I do not agree with that assertion.

As an aside, currently Schneider is founder and CEO at AgileDelta which developed the Efficient XML binary format specification which I plan to write about in a future post.  A W3C working group is currently developing the EXI specification which is based on the AgileDelta specification.

TO BE CONTINUED

JavaScript File Object

As you are probably aware JavaScript engines such as SpiderMonkey typically do not allow access to the local filesystem for reasons of security.  To enable developers to test the scripts from a command line, js includes the load() function which enables you to load one or more JavaScript scripts into the SpiderMonkey engine.  However this is not sufficient for our purposes as no means is provided to write to the filesystem.  Looking more closely at the source code, I noticed support for File objects.  This support is not enabled by default however.  It is not sufficient to simply recompile SpiderMonkey with this option enabled; you must also download and build the Netscape Portable Runtime (NSPR) library.  This library provides a platform-neutral API for system level and libc-like functions, and is used by a number of Mozilla projects and other third party software developers.  The current release is 4.7.3 and you can download it here.

There are some gotchas to building Spidermonkey with NSPR.  First of all, you need to successfully build NSPR.  The source code tarball for NSPR comes with the standard GNU autoconfigure tools.  If you are on a 64-bit system, you need to execute configure with the -enable-64bit option; otherwise the build will quickly fail.  You should then test the build by going to the test subdirectory, building the testsuite and executing it.  You also need to modify SpiderMonkey's Makefile.ref (I am assuming you are building SpiderMonkey 1.7 and not an earlier release) to include libnspr and the NSPR headers.  Two compile time defines are needed.  You can either define JS_HAS_FILE_OBJECT and JS_THREADSAFE in Makefile.ref or as command line arguments to make.  After than you, should be able to successfully build SpiderMonkey with native File object support.

Now that we have js build with support for File objects, what can we do with it.  Well, I guess we should start with the expected Hello World script.

js> File.output.writeln("Hello World");     
Hello World
true
js> File.output.writeln("Hello, world"); "OK" 
Hello, world
OK
js> File.output.writeln("Hello, world"); ""  
Hello, world

js>

Notice that true is outputted unless you append something else as shown above.  Here is another short example which demonstrates how to list the properties of the instance File object for the current directory.

js> dir = new File('.');
/home/fpm/js/.
js> for ( i in dir ) print(i);
length
parent
path
name
isDirectory
isFile
exists
canRead
canWrite
canAppend
canReplace
isOpen
type
mode
creationTime
lastModified
size
hasRandomAccess
hasAutoFlush
position
isNative
js>print(dir.path);
/home/fpm/js/.
js>print(dir.size)
4096
js>

The next example shows how to list some information about files in the current directory.

js> dir = new File('.');
/home/fpm/js/.
js> files = dir.list(); 'OK'
OK
js> for (i in files ) print (files[i].name + '  ' + files[i].size + '  ' + files[i].creationTime); 
music.xml  1081  Tue Jan 06 2009 17:37:14 GMT-0500 (EST)
xml.js  259  Tue Dec 30 2008 18:23:22 GMT-0500 (EST)
xml1.js  699  Tue Jan 06 2009 23:33:26 GMT-0500 (EST)
2.xml  96  Tue Jan 06 2009 22:41:37 GMT-0500 (EST)
3.xml  127  Wed Jan 07 2009 00:02:18 GMT-0500 (EST)
multiply.js  249  Tue Dec 30 2008 17:49:02 GMT-0500 (EST)
helloworld.js  88  Tue Dec 30 2008 17:12:50 GMT-0500 (EST)
hw.js  124  Thu Jan 01 2009 00:24:38 GMT-0500 (EST)
xml2.js  502  Wed Jan 07 2009 00:02:17 GMT-0500 (EST)
regex.js  143  Tue Dec 30 2008 18:10:55 GMT-0500 (EST)
1.xml  15  Tue Jan 06 2009 20:35:27 GMT-0500 (EST)
js>

In the above example, list is an instance method of the File object.  Using other File object instance methods, you can read data from a file and write data to a file.

In the following example, the script reads lines in from file.in and write the lines out to another file file.out, prepending each line with the corresponding line number.

#!/bin/js

var filein = new File("file.in");
filein.open("read", "text");

var fileout = new File('file.out');
fileout.open("write,create", "text");

var n=1;
while (data = filein.readln())
   fileout.writeln(n++ + ' ' + data);

filein.close();
fileout.close();

The File object provides two ways to access data inside a file: a text oriented access, based on characters, and a binary oriented access, based on bytes. In text mode, the maximum line length is 256 and the following encodings are supported: ASCII (text), UTF-8 and UCS-2.  The File object has a number of instance methods including read, readLn, readAll, write, writeLn and writeAll.  If you just want to copy the file in it's entirety, you can use the copyTo method.

var file = new File("file.in");
file.open("read", "text");

file.copyTo("file.out");

file.close();

Similar instance methods which work on files include remove to delete a file or a directory and removeTo to rename a file.

Now that I have shown you how to enable js to access the local filesystem, I would be amiss if I did not point out to you the following warning which is in the Javascript File Object Proposal.

Flaming disclaimer:   Do not play with the File object if
you are not prepared to have your hard drive erased,
smashed, and broken into little bits! It mostly works
right now, but no guarantees.

So far I have had no problem using the File object on my Fedora 10 64-bit platform but one never knows!  Proceed with caution.

JavaScript Shells

Recently I was working on a fairly complex JavaScript script relating to floating point conversions for a new Web page.  After a while I got tired on trying to debug the problem via a Web browser and decided to see if I could find a JavaScript shell, i.e. a standalone Javascript intrepreter just like Ruby's irb, Python's interactive prompt or the Korn shell, which could load and run JavaScript scripts from the command line without having to reload a Web page.

First, some background on the JavaScript langauge for those who are unfamilar with the details.  JavaScript is a complex full-featured weakly typed object- based functional programming language originally developed by Brendan Eich in 1995 while working on the Netscape Navigator browser.  It is most frequently used in client-side web applications but is also used to enable scripting access to embedded objects in other applications.

The langauge has been standardized in the ECMA-262 (ECMAScript) specification.  The first version of ECMAScript was published in June 1997, and was partially based on JavaScript v1.2.  The current version is Edition 3 (Dec 1999) and work is ongoing on the next edition.  Formally, Javascript is a dialect of ECMAScript whose langauge specification is controlled by the Mozilla Foundation.  There are other dialects including ActionScript which the scripting language used in Adobe Flash.  Javascript is still evolving as a language and several versions are in daily use.  The current version is JavaScript 1.8.

The JavaScript engine in Firefox is written in C.  It was orginally called Javascript Reference (JSRef) but nowadays is known as SpiderMonkey.  Other Mozilla products also use this engine and it is available to the public under a MPL/GPL/LGPL tri-license.  The current version, SpiderMonkey 1.7, conforms to JavaScript 1.8 which is a superset of ECMA-262 Edition 3. It consists of a library (or DLL) containing the JavaScript runtime (compiler, interpreter, decompiler, garbage collector, atom manager and standard classes) engine.  This codebase has no dependencies on the rest of the Mozilla codebase. The codebase also contains the routines for a simple user interface which can be linked to the runtime library in order to make a command line shell.

You can download the source code for SpiderMonkey 1.7 here.  Aternatively you can use wget,curl or ftp to download the tarball.  No build script is provided with this version of SpiderMonkey.  Here is how I downloaded, built and smoketest'ed the shell.

mkdir mozilla
cd mozilla
wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.7.0.tar.gz
tar xzf js-1.7.0.tar.gz
cd js/src
make -f Makefile.ref

If everything compiles correctly, you should then smoketest the JavaScript shell (js) by executing the following command:

./Linux_All_DBG.OBJ/js ./perfect.js 

If the 3 perfect numbers between 1 and 500 are printed and you are returned to your shell prompt without any error messages, all is well.  You should then copy the ./Linux_All_DBG.OBJ/js to /bin or /usr/local/bin to make it easier to use.

Unlike other programming languages, JavaScript does not have a concept of printing to STDOUT or reading from STDIN.  These functions, along with quit(), load() and a small number of other functions are provided within js.  They are not part of the Javascript runtime library.  The first thing you will notice is that everything is function-based.  To exit js, you do not type quit, instead you have to type quit().

Here is the standard Hello World example.

$ cat helloworld.js
//
// Hello World!
//

function helloWorld(name)
{
    print("Hello World, " + name);
}
$ /bin/js
js> load('helloworld.js')
js> helloWorld('Finnbarr')
Hello World, Finnbarr
js> helloWorld('Patricia')
Hello World, Patricia
js> quit()
$

Here is the same script called directly from ksh93.  This is possible because of the shebang (!#) sytax on the first line of the script.

$ cat hw.js
#!/bin/js

//
// Hello World!
//

function helloWorld(name)
{
    print("Hello World, " + name);
}

helloWorld('Finnbarr');
$ ./hw.js
Hello World, Finnbarr
$

The shell comes with a readline() function which enables you to ask a user to enter a value such a string or a number as the following example shows.

$ cat multiply.js
//
// multiply.js
//

function multiply()
{
    print("Enter a number:");
    var n1 = readline();
    print("Enter another one:");
    var n2 = readline();

    print("You entered " + n1 + " and " + n2 + ". The result is " + n1*n2);
}

$ /bin/js
js> load('multiply.js')
js> multiply()
Enter a number:
12
Enter another one:
10
You entered 12 and 10. The result is 120

You have full access to all functionality that is defined in the Javascript 1.8 specification.

$ cat regex.js
//
//  test regular expression
//

myRe = /d(b+)d/g;
myArray = myRe.exec("cdbbdbsbz");

print("The value of lastIndex is " + myRe.lastIndex);

$ /bin/js -f regex.js
The value of lastIndex is 5
$

You can also disassemble your JavaScript script using the dissrc() function.

js> dissrc(multiply)     

;-------------------------   7:     print("Enter a number:");
00000:   7  name "print"
00003:   7  pushobj
00004:   7  string "Enter a number:"
00007:   7  call 1
00010:   7  pop
;-------------------------   8:     var n1 = readline();
00011:   8  name "readline"
00014:   8  pushobj
00015:   8  call 0
00018:   8  setvar 0
00021:   8  pop
;-------------------------   9:     print("Enter another one:");
00022:   9  name "print"
00025:   9  pushobj
00026:   9  string "Enter another one:"
00029:   9  call 1
00032:   9  pop
;-------------------------  10:     var n2 = readline();
00033:  10  name "readline"
00036:  10  pushobj
00037:  10  call 0
00040:  10  setvar 1
00043:  10  pop
;-------------------------  11:  
;-------------------------  12:     print("You entered " + n1 + " and " + n2 + ". The result is " + n1*n2);
00044:  12  name "print"
00047:  12  pushobj
00048:  12  string "You entered "
00051:  12  getvar 0
00054:  12  add
00055:  12  string " and "
00058:  12  add
00059:  12  getvar 1
00062:  12  add
00063:  12  string ". The result is "
00066:  12  add
00067:  12  getvar 0
00070:  12  getvar 1
00073:  12  mul:0
00074:  12  add
00075:  12  call 1
00078:  12  pop
00079:  12  stop
js> quit()
$ 

A major limitation of js is that it cannot access files such as an XML documents, DOM objects nor upload or download files.  An alternative command line shell is available from Mozilla called xpcshell.  It is a XPConnect-enabled JavaScript command line shell where scripts running in it can access XPCOM functionality.  For those of you who are unfamilar with XPCOM, it is a cross platform component object model, similar to Microsoft's COM.  XPCOM is frequently used for unit testing.

Besides SpiderMonkey and TraceMonkey, there are a number of other standalone JavaScript shells available including wxjs and JSDB.  There also is the Rhino engine, created primarily by Norris Boyd, which is a JavaScript implementation written in Java that is ECMA-262 Edition 3 compliant.

I would be amiss if I did not point out that there are alternatives to using a command line shell such as js.  My personal favorite is Firebug which is a Firefox extension that provides a more advanced interactive shell, an advanced DOM inspector, a JavaScript debugger, a profiling tool and various other useful tools.

Plans for SpiderMonkey 1.8 appear to be shelved.  The Firefox development team are currently in the process of replacing SpiderMonkey with TraceMonkey which is based on a technique developed at UC Irvine called trace trees, builds on code and ideas shared with the Tamarin Tracing project, and adds just-in-time (JIT) native code compilation.  The net result is a seriously significant speed increase both in the browser chrome and Web page content.  As with SpiderMonkey, you can download the source code from the TraceMonkey mercurial repository and build your own command line shell.

After languishing for a number of years, JavaScript is becoming an increasing important language for Web applications. It is one of those languages that every Web programmer needs to understand in depth.  Being able to run and debug JavaScript scripts from the command line greatly assists in that understanding.

Enjoy!