Musings of an OS Plumber

ksh93 libshell

2009-12-17T17:47:00.007-05:00

In a number of posts about a year ago I discussed how to develop custom builtins for Korn Shell 93 (ksh93) using libshell and published APIs. You can also use these same APIs to access ksh93 functionality from within a C application. This post provides three simple examples of how to do this.

The first example simply takes two numbers as arguments on the command line, multiplies the numbers and prints out the result on stderr.

#include <shell.h>
#include <nval.h>

int main(int argc, char *argv[])
{
    Namval_t *np;
    Sfdouble_t res;

    Shell_t *shp = sh_init(argc, argv, 0);
    sh_trap("((xresult=$0*$1))", 0);

    np = nv_open("xresult", shp->var_tree, 0);
    res = nv_getnum(np);
    sfprintf(sfstderr,"%Lg\n", res);
    nv_close(np);

    return(0);
}

Here is how to compile this example using static libraries. I built these examples in my home directory but the ksh93 build environment is at /work/ksh93, hence the use of -I and -L options. Note the large number of AST (Advanced Software Toolkit) and ksh93 libraries you have to specify in order to resolve all the symbols.

gcc -o example1 -I/work/ksh93/arch/linux.i386-64/include/ast example1.c -L/work/ksh93/arch/linux.i386-64/lib -lshell -lm -ldll -lcmd  -ldl -last

For example:


$ ./example1 3 4 
12
$

The next example breaks the text string string into its component parts and prints them out to stdout. Note that "foo\\ bar" is one word, not two words.

#include <shell.h>
#include <stdio.h>
#include <nval.h>

int
main(int argc, char *argv[])
{
    char tmp[512];
    char string[] ="hello world 'my name is Finnbarr' foo\\ bar";

    sprintf(tmp, "set - %s && for i in \"$@\";\n do\n echo \"$i\"\ndone", string);

    Shell_t *shp = sh_init(argc, argv, 0);
    sh_trap(tmp, 0);

    return(0);
}

Here is the output:

./example2
hello
world
my name is Finnbarr
foo bar

The next example is similar to the previous example, except that it creates an array and then uses nv_getval() to access the contents of the created array and print the contents to stderr.

#include <shell.h>
#include <stdio.h>
#include <nval.h>

int 
shell(int argc, char* argv[], char *str)
{
    Namval_t *np, *np_sub;
    char tmp[512];

    sprintf(tmp, "set - %s && for i in \"$@\";\n do\n aname+=(\"$i\")\ndone", str);

    Shell_t *shp = sh_init(argc, argv, 0);
    sh_trap(tmp, 0);

    np = nv_open("aname", shp->var_tree, 0);
    nv_putsub(np, NULL, ARRAY_SCAN);
    np_sub = np;

    do {
        // copy out the arguments to wherever here. 
        fprintf(stderr, "%d: subscript='%s' value='%s'\n", np_sub, nv_getsub(np_sub), nv_getval(np_sub));
    } while (np_sub && nv_nextsub(np_sub)); 

    nv_close(np);

    return(0);
} 

int 
main(int argc, char *argv[])
{
    char string[] ="hello world 'my name is simon' foo\\ bar";

    shell(argc, argv, string);
}

Here is the output:

$ ./example1
37212336: subscript='0' value='hello'
37212336: subscript='1' value='world'
37212336: subscript='2' value='my name is simon'
37212336: subscript='3' value='foo bar'
$

As you can see, running shell script snippets from within a C application is not difficult. For more information on these APIs read the ksh93 shell(3) and nval(3) man pages.

Microsoft SUA JavaScript Shell

2009-11-10T18:50:00.024-05:00

Recently I needed to test some JavaScript code on a Microsoft Vista Ultimate operating system using the command line. This is something that I have done before on Linux platforms using a JavaScript shell but had never done on a Microsoft platform. For those who are not familiar with a JavaScript shell, it is a command line interface to a JavaScript engine. Similar to Python or Ruby, the JavaScript shell has two modes of operation. You can use it as an interactive shell, in which you type JavaScript code at a prompt and get instant results, or you can invoke a JavaScript program.

The easiest way that I know of to build a JavaScript shell is to download and build the SpiderMonkey JavaScript engine which comes with a JavaScript Shell. SpiderMonkey is one of two JavaScript engines which the Mozilla software project supports. It is used in the Firefox browser and elsewhere. The other JavaScript engine is Rhino. Rhino is written using the Java language whereas SpiderMonkey is a pure C language implementation which conforms to ECMA-262 Edition 3. Included in SpiderMonkey is a JavaScript shell. Source code for both JavaScript engines is readily available on the Mozilla website.

For Microsoft Vista I had a choice of either building a native Windows executable which could be accessed via the Vista command prompt (similar to the old DOS prompt) or, because Vista Ultimate comes with Subsystem for Unix-based Applications (SUA), a shell which was accessable via a more familar Unix-like environment. I choice to go the SUA route.

Many people are unaware of Microsoft's SUA offering. Basicly it is a full-featured POSIX-compliant subsystem for Windows NT-based operating systems. For their own reasons, Microsoft limits the availablity of SUA to certain products such as Vista Enterprise and Ultimate editions and Windows Server 2008. It comes with hundreds of utilities including the GNU compiler collection, Korn shell, and development headers and libraries. See here for further information.

SUA is a subsystem which runs natively on top of the Windows kernel. which was orginally developed by Softway Systems. The original name for the product was OpenNT but that was later changed to Interix. Interix and most of the development team was acquired by Microsoft in 1999 and the product was renamed Services For Unix (SFU) in 2005. With the release of Microsft Vista, it was integrated into the operating system as a separate installable component and was renamed Subsystem for Unix-based Applications. There is an active user community around SUA which is hosted by Interop Systems and sponsored by Microsoft.

This is the Makefile which I hacked together to build a static js executable from the SpiderMonkey 1.8 sources for use with Microsoft Vista SUA.


JSVERSION = js-1.8.0

CC = gcc

OS_ARCH    := $(subst /,_,$(shell uname -s | sed /\ /s//_/))
OS_RELEASE := $(shell uname -r)
OS_CONFIG  := $(OS_ARCH)$(OS_RELEASE)
OBJDIR      = ./$(OS_CONFIG)

DEFINES     = -DXP_UNIX -DPOSIX_SOURCE -DHAVE_LOCALTIME_R

LIBDIR     := lib
INCLUDES   += -I$(OBJDIR)
RANLIB      = ranlib
LDFLAGS     = -lm

OPTIMIZER   =
INTERP_OPTIMIZER =

CFLAGS          =  $(OPTIMIZER) $(DEFINES) $(INCLUDES)
INTERP_CFLAGS   = $(INTERP_OPTIMIZER) $(DEFINES) $(INCLUDES)

PREDIRS += editline
DEFINES += -DEDITLINE

JS_CFILES =     jsapi.c jsarena.c jsarray.c jsatom.c jsbool.c jscntxt.c jsdate.c \
                jsdbgapi.c jsdhash.c jsdtoa.c jsemit.c jsexn.c jsfun.c jsgc.c jshash.c \
                jsinterp.c jsinvoke.c jsiter.c jslock.c jslog2.c jslong.c jsmath.c \
                jsnum.c jsobj.c jsopcode.c jsparse.c jsprf.c jsregexp.c jsscan.c \
                jsscope.c jsscript.c jsstr.c jsutil.c jsxdrapi.c jsxml.c prmjtime.c

JS_HFILES =     jsarray.h jsatom.h jsbool.h jsconfig.h jscntxt.h jsdate.h \
                jsemit.h jsexn.h jsfun.h jsgc.h jsinterp.h jsiter.h jslibmath.h \
                jslock.h jsmath.h jsnum.h jsobj.h jsopcode.h jsparse.h jsarena.h \
                jsclist.h jsdhash.h jsdtoa.h jshash.h jslong.h jstypes.h jsprvtd.h \
                jspubtd.h jsregexp.h jsscan.h jsscope.h jsscript.h jsstr.h jsutil.h \
                jsxdrapi.h jsxml.h

API_HFILES =    jsapi.h jsdbgapi.h

OTHER_HFILES =  jsbit.h jscompat.h jscpucfg.h jsotypes.h jsstddef.h jsopcode.tbl \
                jsproto.tbl js.msg jsshell.msg jskeyword.tbl prmjtime.h resource.h


ifdef JS_HAS_FILE_OBJECT
JS_CFILES += jsfile.c
JS_HFILES += jsfile.h
endif

LIB_CFILES  = $(JS_CFILES)
PROG_CFILES = js.c

LIB_OBJS  = $(addprefix $(OBJDIR)/, $(LIB_CFILES:.c=.o))
PROG_OBJS = $(addprefix $(OBJDIR)/, $(PROG_CFILES:.c=.o))

CFILES = $(LIB_CFILES) $(PROG_CFILES)
OBJS   = $(LIB_OBJS) $(PROG_OBJS)

LIBRARY = $(OBJDIR)/libjs.a
PROGRAM = $(OBJDIR)/js
TARGETS = $(PROGRAM)

EDIT_LIBRARY  = $(OBJDIR)/libedit.a
EDIT_DIR      = ./editline
EDIT_CFILES   = $(EDIT_DIR)/editline.c $(EDIT_DIR)/sysunix.c
EDIT_OBJS     = $(OBJDIR)/editline.o $(OBJDIR)/sysunix.o
EDIT_CFLAGS   = -DANSI_ARROWS -DHAVE_TCGETATTR -DHIDE -DUSE_DIRENT -DSYS_UNIX \
                -DHAVE_STDLIB -DUNIQUE_HISTORY $(DEFINES) -I$(EDIT_DIR)


define MAKE_OBJDIR
if test ! -d $(@D); then rm -rf $(@D); mkdir -p $(@D); fi
endef


all:
        $(MAKE) $(TARGETS)

$(PROGRAM): $(PROG_OBJS) $(LIBRARY) $(EDIT_LIBRARY)
        $(CC) -o $@ $(CFLAGS) $(PROG_OBJS) $(LIBRARY) $(EDIT_LIBRARY) $(LDFLAGS)

$(OBJDIR)/%.o: %.c %.h
        @$(MAKE_OBJDIR)
        $(CC) -o $@ -c $(CFLAGS) $*.c

$(OBJDIR)/editline.o: $(EDIT_DIR)/editline.c $(EDIT_DIR)/editline.h
        @$(MAKE_OBJDIR)
        $(CC) -o $@ -c $(EDIT_CFLAGS) $(EDIT_DIR)/editline.c

$(OBJDIR)/sysunix.o: $(EDIT_DIR)/sysunix.c $(EDIT_DIR)/editline.h
        @$(MAKE_OBJDIR)
        $(CC) -o $@ -c $(EDIT_CFLAGS) $(EDIT_DIR)/sysunix.c

$(OBJDIR)/%.o: %.c
        @$(MAKE_OBJDIR)
        $(CC) -o $(OBJDIR)/jscpucfg jscpucfg.c
        $(OBJDIR)/jscpucfg > $(OBJDIR)/jsautocfg.h
        $(CC) -o $(OBJDIR)/jskwgen jskwgen.c
        $(OBJDIR)/jskwgen $(OBJDIR)/jsautokw.h
        $(CC) -o $@ -c $(CFLAGS) $*.c

$(OBJDIR)/jsinterp.o: jsinterp.c jsinterp.h
        @$(MAKE_OBJDIR)
        $(CC) -o $@ -c $(INTERP_CFLAGS) jsinterp.c

$(EDIT_LIBRARY): $(EDIT_OBJS)
        $(AR) rv $@ $?
        $(RANLIB) $@

$(LIBRARY): $(LIB_OBJS)
        $(AR) rv $@ $?
        $(RANLIB) $@

clean:
        rm -rf $(OBJS)

clobber:
        rm -rf $(OBJDIR)

tarball:
        mkdir -p ./$(JSVERSION)/editline
        cp *.[ch] ./$(JSVERSION)
        cp *.tbl ./$(JSVERSION)
        cp *.js ./$(JSVERSION)
        cp *.msg ./$(JSVERSION)
        cp Makefile ./$(JSVERSION)
        cp README ./$(JSVERSION)
        cp editline/*.[ch] ./$(JSVERSION)/editline
        cp editline/README ./$(JSVERSION)/editline
        cp editline/editline.3 ./$(JSVERSION)/editline
        tar cvf $(JSVERSION).tar ./$(JSVERSION)
        gzip $(JSVERSION).tar
        rm -rf ./$(JSVERSION)

Note you need to use gcc 4.2 instead of gcc 3.2 otherwise you will get an undefined _JS_DHashTableOperate symbol due to an obsure gcc bug. However with gcc 4.2, you will see a large number of compiler warnings relating to alignment is greater than maximum object file. Fortunately you can safely ignore them. By the way, I choose to build a static executable because building shared libraries on SUA using gcc is somewhat problematic.

Once you have build an executable, a simple smoketest is to use the executable to execute the prefect.js JavaScript script which is included with the SpiderMonkey source code. You should get output similar to the following:


$ ./Interix6.0/js perfect.js

A number is 'perfect' if it is equal to the sum of its
divisors (excluding itself).

The perfect numbers up to 500 are:
6 = 1 + 2 + 3
28 = 1 + 2 + 4 + 7 + 14
496 = 1 + 2 + 4 + 8 + 16 + 31 + 62 + 124 + 248
That's all.
$

Well, this should be enough to enable you to build your own JavaScript shell on Microsoft's Subsystem for Unix-based Applications. Good Luck!

Inverting Large Images Using CUDA

2009-10-10T00:15:00.019-04:00

This is a simple example of how to invert a very large image, stored as a vector using nVidia's CUDA programming environment and an EVGA GeForce GTX 260 graphics card.

A GeForce GTX 260 card should have only 24 cores on it, but the nVidia SDK deviceQuery utility reports that there are 27 multiprocessors on my GeForce GTX 260 graphics card and I am able to use all 27 of them.


$ deviceQuery
CUDA Device Query (Runtime API) version (CUDART static linking)
There is 1 device supporting CUDA

Device 0: "GeForce GTX 260"
  CUDA Driver Version:                           2.30
  CUDA Runtime Version:                          2.30
  CUDA Capability Major revision number:         1
  CUDA Capability Minor revision number:         3
  Total amount of global memory:                 938803200 bytes
  Number of multiprocessors:                     27
  Number of cores:                               216
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       16384 bytes
  Total number of registers available per block: 16384
  Warp size:                                     32
  Maximum number of threads per block:           512
  Maximum sizes of each dimension of a block:    512 x 512 x 64
  Maximum sizes of each dimension of a grid:     65535 x 65535 x 1
  Maximum memory pitch:                          262144 bytes
  Texture alignment:                             256 bytes
  Clock rate:                                    1.35 GHz
  Concurrent copy and execution:                 Yes
  Run time limit on kernels:                     Yes
  Integrated:                                    No
  Support host page-locked memory mapping:       Yes
  Compute mode:                                  Default (multiple host threads can use this device simultaneously)

The application is fairly trivial. It reads in an image file in PPM P6 format, stores the image in a vector of integers, one per pixel, with 24-bits used to store the RGB values. Depending on which command line option is selected ( -c invert using CPU, -g invert using GPU), it then inverts the image vector either using the CPU or the GPU (Graphical Processing Unit). Finally it writes the image out to an output file in PPM P6 format.

It uses a block size of 16 and a single kernel called myCudaInvertPixel which works on 16 rows of the image at the time via tiling.


#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
#include <climits>

#include <unistd.h>
#include <time.h>
#include <string.h>
#include <stdio.h>

#include <cuda.h>
#include <cuda_runtime_api.h>

#define BLOCK_SIZE 16
// #define DEBUG 0

// ------------ prototypes --------------
__global__ void myCudaInvertPixel(int *, int);


class CCL {

    int width;
    int height;

public:

    CCL() { width = height = 0; }
    ~CCL() { width = height = 0; }

    void cuda_ccl(vector& img, int w)
    {
        width = w;
        height = img.size()/w;

#ifdef DEBUG
        cout << "Image width=" << width << " height=" << height << endl;
#endif        
        int result = cuda_invert(img, width, height);

        return;
    }

private:
    int cuda_invert(vector&, int, int);

};


int
CCL::cuda_invert(vector& image, int width, int height)
{
    int deviceCount = 0;
    int driverVersion = 0, runtimeVersion = 0;
    int globalMem;
    int image_size = width * height * sizeof(int);

    int *cuda_image_ptr;
    cudaDeviceProp deviceProp;
    cudaError_t err;

    int strake_size = 0;
    int last_strake_size = 0;
    int strakes = 0;
    int this_strake_size = 0;
    int offset = 0;

    if (cudaGetDeviceCount(&deviceCount) != cudaSuccess) {
        printf("ERROR: cudaGetDeviceCount failed. CUDA Driver and Runtime version may be mismatched.\n");
        return 1;
    }

    if (deviceCount == 0) {
        printf("ERROR: No device supporting CUDA found\n");
        return 1;
    }

    cudaGetDeviceProperties(&deviceProp, 0);
    globalMem = deviceProp.totalGlobalMem;

#ifdef DEBUG 
    printf("Number of CUDA devices: %d\n\n", deviceCount);
    printf("Device %d: \"%s\"\n", 0, deviceProp.name);
#if CUDART_VERSION >= 2020
    cudaDriverGetVersion(&driverVersion);
    cudaRuntimeGetVersion(&runtimeVersion);
    printf("  CUDA Driver Version:                   %d.%d\n", driverVersion/1000, driverVersion%100);
    printf("  CUDA Runtime Version:                  %d.%d\n", runtimeVersion/1000, runtimeVersion%100);
#endif
    printf("  CUDA capability revision number:       %d.%d\n", deviceProp.major, deviceProp.minor);
#if CUDART_VERSION >= 2000
    printf("  Number of multiprocessors:             %d\n", deviceProp.multiProcessorCount);
    printf("  Number of cores:                       %d\n", 8 * deviceProp.multiProcessorCount);
    printf("  Maximum number of threads per block:   %d\n", deviceProp.maxThreadsPerBlock);
#endif
    printf("  Clock rate:                            %.2f GHz\n", deviceProp.clockRate * 1e-6f);
    printf("  Global memory:                         %d\n", globalMem);
    printf("  Shared memory per block:               %d\n", deviceProp.sharedMemPerBlock);
    printf("  Registers per block:                   %d\n\n", deviceProp.regsPerBlock);
#endif    

    cudaSetDevice(0);      // not needed - default is 0.

    int rowBytes = width * 4;                                                       // size of a row
    int maxrow  = BLOCK_SIZE;                                                       // max rows that can be handled in global memory
    strake_size = maxrow * rowBytes;                                                // memory to allocate
    strakes = image_size/strake_size + ((image_size % strake_size == 0) ? 0 : 1);   // there X strakes necessary
    last_strake_size = image_size - (strake_size * (strakes - 1));                  // figure out size of last strake (may be smaller)
#if DEBUG
    printf("  Image size %d\n", image_size);
    printf("  Image vector size %d\n", image.size());
    printf("  Row size %d\n", rowBytes);
    printf("  Maxximum rows %d\n", maxrow);
    printf("  Strake size %d\n", strake_size);
    printf("  Strakes %d\n", strakes);
    printf("  Last strake size %d\n\n", last_strake_size);
#endif

    // uncomment if you want to using CUDA's elapsed timer
    // float elapsed_time_ms = 0.0f;
    // cudaEvent_t  start, stop;
    // cudaEventCreate( &start);
    // cudaEventCreate( &stop);
    // cudaEventRecord(start, 0);

    dim3 dimBlock(BLOCK_SIZE, BLOCK_SIZE);
    dim3 dimGrid(((width*4)/dimBlock.x) + (((width*4)%dimBlock.x)?1:0), BLOCK_SIZE);

    cudaMalloc((void**)&cuda_image_ptr, strake_size);
    if ((err = cudaGetLastError()) != cudaSuccess) {
        fprintf(stderr, "ERROR: Cuda: %s\n", cudaGetErrorString(err));
        exit(2);
    } 

    for (int i = 0; i < strakes; i++) {
        if (i > 0 && i == strakes - 1) {
            this_strake_size = last_strake_size;
        } else {
            this_strake_size = strake_size;
        }
        offset = 0 + i *(strake_size/4);

        cudaMemcpy(cuda_image_ptr, &image[offset], this_strake_size, cudaMemcpyHostToDevice);
        if ((err = cudaGetLastError()) != cudaSuccess) {
            fprintf(stderr, "ERROR: Cuda: %s\n", cudaGetErrorString(err));
            exit(2);
        }

        // invoke CUDA kernel
        myCudaInvertPixel<<<dimGrid, dimBlock>>>(cuda_image_ptr, width);
        if ((err = cudaGetLastError()) != cudaSuccess) {
           fprintf(stderr, "ERROR: Cuda: %s\n", cudaGetErrorString(err));
           exit(2);
        }

        // not needed here - cudaThreadSynchronize();

        cudaMemcpy( &image[offset], cuda_image_ptr, this_strake_size, cudaMemcpyDeviceToHost);
        if ((err = cudaGetLastError()) != cudaSuccess) {
            fprintf(stderr, "ERROR: Cuda: %s\n", cudaGetErrorString(err));
            exit(2);
        }
    }

    cudaFree(cuda_image_ptr);

    // uncomment if you want to using CUDA's elapsed timer
    // cudaEventRecord(stop, 0);
    // cudaEventSynchronize(stop);
    // cudaEventElapsedTime( &elapsed_time_ms, start, stop);
    // printf("Elapsed time per CUDA: %f\n", elapsed_time_ms);

    return 0;
}


// CUDA kernel definition - invert pixel
__global__ void myCudaInvertPixel(int *pixel, int width)
{
    int ix = blockIdx.x * blockDim.x + threadIdx.x;
    int iy = blockIdx.y * blockDim.y + threadIdx.y;
    int idx = ix + iy*width;

    if (ix <= width || iy >= BLOCK_SIZE) return;

    pixel[idx] = 0x00FFFFFF - pixel[idx];
}


class Timer {

    private:

        timespec starttime, endtime;
        float elapsed;

        float difftime(timespec start, timespec end) {
            float temp;

            temp = (end.tv_sec - start.tv_sec)  +
                   (float) (end.tv_nsec - start.tv_nsec) / 1000000000.0;

            return temp;
        }

    public:

        Timer() { elapsed = 0.00; }

        void reset() { elapsed = 0.00; }

        void start() {
            elapsed = 0.00;
            clock_gettime(CLOCK_MONOTONIC, &starttime);
        }

        void stop() {
            clock_gettime(CLOCK_MONOTONIC, &endtime);
            elapsed = elapsed + difftime(starttime, endtime);
        }

        void restart() { clock_gettime(CLOCK_MONOTONIC, &starttime); }

        void print() { cout << "Elapsed time: " << elapsed << endl; }

};

class PPM {

  protected:

    int width;
    int height;
    int maxColor;
    vector <int> image;
    friend class CCL;

  public:

    PPM() {
        width = 0;
        height = 0;
        maxColor = 0;
    }

    int read(char *filename);

    int write(char* filename);

    void printsize() {
       cout << "Image height: " << height << endl;
       cout << "Image width: " << width <<< endl;
       cout << "Image vector size: " << image.size() << endl;
    }

    void invertpixelNoCuda() {
       for (vector<int>::iterator it = image.begin(); it != image.end(); ++it) {
           *it = 0x00FFFFFF - *it;
       }
    }

    void invertpixelCuda() {
       CCL ccl;
       ccl.cuda_ccl(image, width);
    }

    int getwidth() { return width; }

};


int
PPM::read(char *filename)
{
    FILE *fp = fopen(filename, "r");
    int rgb, x, y;
    unsigned char red, green, blue;
    char buf[256];
    int *pwidth = &width;
    int *pheight = &height;

    if (fp== NULL) return 1;

    do {
        fgets(buf, 256, fp);
    } while(buf[0] == '#');

    if (buf[0] != 'P' || buf[1] != '6') {
       fclose(fp);
       return 1;
    }

    do {
        fgets(buf, 256, fp);
    } while(buf[0] == '#');

    sscanf(buf, "%u %u", pwidth, pheight);
    fscanf(fp, "%u\n", &maxColor);

    image.resize(height*width);
    for (y = 0; y < height ; y++) {
        for (x = 0; x < width; x++) {
            fscanf(fp, "%c%c%c", &red, &green, &blue);
            rgb = ((int)blue & 0x000000ff) << 16 |
                  ((int)green & 0x000000ff ) << 8 |
                  ((int)red & 0x000000ff) | 0x00000000;
            image[y*width + x] = rgb;
        }
    }

    fclose(fp);
    return 0;
}


int
PPM::write(char* filename)
{
    FILE *fp;
    int  y, x;
    unsigned char red=0, green=0, blue=0;
    int rgb;

    if ((fp = fopen(filename, "w")) == NULL) {
        fprintf(stderr, "ERROR: Cannot open output file\n");
        return 1;
    }

    fprintf(fp, "P6\n");
    fprintf(fp, "%d %d\n", width, height);
    fprintf(fp, "255\n");

    for (y = 0; y < height; y++) {
       for (x = 0; x < width; x++) {
          rgb = image[y*width + x];
          blue = (rgb & 0x00FF0000) >> 16;
          green = (rgb & 0x0000FF00) >> 8;
          red = (rgb & 0x000000FF);
          fprintf(fp, "%c%c%c", red, green, blue);
       }
   }

   fclose(fp);

   return(0);
}


void
usage(char *name,
      int mode)
{
    cerr << "Usage: " << name << " [-c] [-d] [-g] infile outfile" << endl;
    exit(mode);
}


int
main(int argc,
     char *argv[])
{
    int c;
    int cflag = 0, dflag = 0, gflag = 0;
    PPM ppm;
    Timer timer;

    if (argc == 1)
        usage(argv[0], 0);

    opterr = 0;
    while ((c = getopt( argc, argv, "cdgh")) != -1) {
       switch (c) {
          case 'c':
             cflag = 1;
             break;
          case 'd':
             dflag = 1;
             break;
          case 'g':
             gflag = 1;
             break;
          case '?':
          case 'h':
             usage(argv[0], 0);
             break;
       }
    }

    if (optind != argc-2) {
        cerr << "ERROR: An input file and an output file must be specified" << endl;
        usage(argv[0], 1);
    }

    if (ppm.read(argv[optind++])) {
        cerr << "ERROR: Failed to read input file" << endl;
        exit(1);
    }

    timer.start();

    if (dflag) ppm.printsize();

    if (cflag) ppm.invertpixelNoCuda();

    if (gflag) ppm.invertpixelCuda();

    timer.stop();

    if (ppm.write(argv[optind])) {
        cerr << "ERROR: Failed to write output file" << endl;
        exit(1);
    }

    timer.print();

    exit(0);
}

Well that is all there is to this application. To compile it you need to have at least the nVidia CUDA 2.3 SDK installed and the latest nVidia graphics card driver.

If you are new to CUDA programming, I suggest you download the source code, compile it and play around with it on your own computer. The man pages for Linux do not work (they are all mixed up) but good up-to-date documentation can be found online at nVidia CUDA Library Documentation.

You might also want to play around with the block (dimBlock) and grid (dimGrid) dimensions. nVidia has a useful Microsoft Excel Occupancy Calculator Spreadsheet.

P.S. I tested the application on images up to a 32000 x 32000 pixels on RHEL5.4 using the CUDA 2.3 SDK and an EVGA GeForce GTX 260 graphics card with 896Mb of memory.

Connected Component Labeling 1

2009-10-03T11:11:00.021-04:00

Recently, I became interested in connected component labeling (CCL) algorithms because of a competition on TopCoder. Over the next few weeks I plan to explore various algorithms including parallel algorithms written to take advantage of nVidia GPUs via the CUDA programming model.

To simply things and enable me to concentrate on the algorithms, all images will be in PPM P6 format and will be loaded into a vector (image) of integers. The image vector will be processed by the CCL algorithm. The transformed image vector will then be written out to a PPM P6 file for viewing by an external application such as GIMP.

Here is an example program which contains a PPM class for reading, writing, inverting, and displaying some information about a PPM file. An image is read into the image vector with the red, green and blue components stored in specific bits of each integer. It also contains a simple timer class which uses clock_getttime() to retrieve the monotomic time.


#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
#include <unistd.h>
#include <time.h>

using namespace std;

class Timer {

    private:

        timespec starttime, endtime;
        float elapsed;

        float difftime(timespec start, timespec end)
        {
            float temp;

            temp = (end.tv_sec - start.tv_sec)  +
                   (float) (end.tv_nsec - start.tv_nsec) / 1000000000.0;

            return temp;
        }

    public:

        Timer() {
            elapsed = 0.00;
        }

        void reset() {
            elapsed = 0.00;
        }

        void start() {
            elapsed = 0.00;
            clock_gettime(CLOCK_MONOTONIC, &starttime);
        }

        void stop() {
            clock_gettime(CLOCK_MONOTONIC, &endtime);
            elapsed = elapsed + difftime(starttime, endtime);
        }

        void restart() {
            clock_gettime(CLOCK_MONOTONIC, &starttime);
        }

        void print() {
            cout << "Elapsed time: " << elapsed << endl;
        }
};


class PPM {

  protected:

    int width;
    int height;
    int maxColor;
    vector  image;

  public:

    PPM()
    {
        width = 0;
        height = 0;
        maxColor = 0;
    }

    int readfile(char *filename);

    int writefile(char* filename);

    void printsize()
    {
       cout << "Image height: " << height << endl;
       cout << "Image width: " << width << endl;
       cout << "Image vector size: " << image.size() << endl;
    }

    void invertpixel()
    {
       for (vector::iterator it = image.begin(); it != image.end(); ++it) {
           *it = 0x00FFFFFF - *it;
       }
    }
};

int
PPM::readfile(char* filename)
{
    ifstream fin;
    char buf[256];
    unsigned char red, green, blue;
    int i, j, rgb;

    fin.open(filename);
    if (!fin.is_open()) return 1;

    fin.getline(buf, 256);
    if (buf[0] != 'P' || buf[1] != '6') {
       fin.close();
       return 1;
    }
    while (fin.peek()=='#') fin.getline(buf, 256);

    fin >> width >> height >> maxColor;
    fin.getline(buf, 256);                // discard rest of line 

    image.resize(height*width);
    for (i = 0; i < height; i++) {
        for (j = 0; j < width; j++) {
            fin >> red >> green >> blue;
            rgb = ((int)blue & 0x000000ff) << 16 |
                  ((int)green & 0x000000ff ) << 8 |
                  ((int)red & 0x000000ff) | 0x00000000;
            image[i * width + j] = rgb;
        }
    }

    fin.close();
    return 0;
}

int
PPM::writefile(char* filename)
{
    ofstream fout;
    stringstream ss;
    unsigned char red, green, blue;
    int i, j, rgb;

    fout.open(filename);
    if (!fout.is_open()) return 1;

    fout << "P6" << endl;
    ss << width << " " << height;
    fout << ss.str() << endl;
    fout << "255" << endl;

    for (i = 0; i < height; i++) {
        for (j = 0; j < width; j++) {
            rgb = image[i * width + j];
            red = (rgb & 0x000000FF);
            green = (rgb & 0x0000FF00) >> 8;
            blue = (rgb & 0x00FF0000) >> 16;
            fout << red << green << blue;
        }
    }

    fout.close();
    return 0;
}
void
usage(char *name,
      int mode)
{
    cerr << "Usage: " << name << " [-d] infile outfile" << endl;
    exit(mode);
}

int
main(int argc,
     char *argv[])
{
    int c, result = 0;
    int dflag = 0, iflag = 0;
    PPM ppm;
    Timer timer;

    if (argc == 1)
        usage(argv[0], 0);

    opterr = 0;
    while ((c = getopt( argc, argv, "di")) != -1) {
       switch (c) {
          case 'd':
             dflag = 1;
             break;
          case 'i':
             iflag = 1;
             break;
          case '?':
          case 'h':
             usage(argv[0], 0);
             break;
       }
    }

    if (optind != argc-2) {
        cerr << "ERROR: An input file and an output file must be specified" << endl;
        usage(argv[0], 1);
    }


    timer.start();
    if (ppm.readfile(argv[optind++])) {
        cerr << "ERROR: Failed to read input file" << endl;
        exit(1);
    }

    if (dflag) ppm.printsize();

    if (iflag) ppm.invertpixel();

    if (ppm.writefile(argv[optind])) {
        cerr << "ERROR: Failed to write output file" << endl;
        exit(1);
    }

    timer.stop();
    timer.print();

    exit(0);
}

Here is an alternative way of reading and writing an image file using mmap()


#include <fcntl.h>
#include <sys/mman.h>
#include <string.h>
#include <stdio.h>

int 
PPM::readfile(char *filename)
{
    FILE *fp;
    struct stat statbuf;
    char buf[256];
    int *pwidth = &width;
    int *pheight = &height;

    if ((fp = fopen(filename, "r")) == NULL) return 1;

    do {
        fgets(buf, 256, fp);
    } while(buf[0] == '#');

    if (buf[0] != 'P' || buf[1] != '6') {
       fclose(fp);
       return 1;
    }
    
    do {
        fgets(buf, 256, fp);
    } while(buf[0] == '#');

    sscanf(buf, "%u %u", pwidth, pheight);
    fscanf(fp, "%u\n", &maxColor);

    image.resize(height*width);
    long offset = ftell(fp);
    int fd = fileno(fp);
    fstat(fd, &statbuf);

    unsigned char* data = (unsigned char*)mmap(0, statbuf.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
    if (data == MAP_FAILED) {
        fprintf(stderr, "ERROR: mmap failed. %s\n", strerror(errno));
        fclose(fp);
        return 1;
    }

    unsigned char* data_offset = (unsigned char*)(data + offset);
    unsigned char red, green, blue;
    int i = 0, o = 0;
    for (int y = 0; y < height * width; y++) {
        blue = data_offset[i++];
        green = data_offset[i++];
        red = data_offset[i++];
        image[o++] = red << 16 | green << 8 | blue;
    }

    munmap(data, statbuf.st_size);
    close(fd);

    return 0;
}

int
PPM::writefile(char* filename)
{
    unsigned int rgb;
    int result;

    int fd = open(filename, O_RDWR | O_CREAT | O_TRUNC, (mode_t)0600);
    if (fd == -1) {
        fprintf(stderr, "ERROR: Cannot create output file\n");
        return 1;
    }

    char buf[256];
    sprintf(buf, "P6\n%d %d\n255\n", width, height);
    int bufsize = strlen(buf);

    int mmap_size = bufsize + (height * width * 3);
    if (((result = lseek(fd, mmap_size - 1, SEEK_SET)) == -1) ||
        ((result = write(fd, "", 1)) == -1)) {
        fprintf(stderr, "ERROR: Cannot write null at end of output file\n");
        close(fd);
        return 1;
    }

    unsigned char* data = (unsigned char*)mmap(0, mmap_size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
    if (data == MAP_FAILED) {
        fprintf(stderr, "ERROR: mmap failed. %s\n", strerror(errno));
        close(fd);
        return 1;
    }

    int o = 0;
    for (int i = 0; i < bufsize; i++)
        data[o++] = buf[i];

    for (int i = 0; i < height * width; i++) {
        rgb = image[i];
        data[o++] = rgb & 0xFF;
        data[o++] = (rgb >> 8) & 0xFF;
        data[o++] = (rgb >> 16) & 0xFF;
    }

    munmap(data, mmap_size);
    close(fd);

    return(0);
}

Note the need to seek to the end of the new file and write a null before calling mmap() when writing a file. Also MAP_SHARED must be used instead of MAP_PRIVATE otherwise your file will contain nothing but zeros.

Well, that is all I have time for at present. In my next post on this subject I will implement a 4-connected thresholding union-find algorithm.

Coming Soon - Software Engineering PE

2009-09-19T09:37:00.006-04:00

In August 2009 the NCEES Board of Directors approved moving forward with the development of a Principles and Practice of Engineering examination, commonly known as the PE (Professional Engineer) exam, for the discipline of software engineering.

NCEES is the umbrella organization for the engineering and surveying jurisdictional licensing boards in the United States. It develops and administers examinations used for engineering licensure. The prerequisite for NCEES to consider initiating a PE examination in a new discipline includes written requests from no fewer than 10 state licensing boards that can demonstrate a need for the examination in their jurisdictions. The requests must include proof of such need, estimate of usage, and evidence that knowledge areas and skills are not adequately measured in an existing examination.

In addition, no new discipline can be added to the examination program unless there is an ABET-accredited degree in that discipline. The jurisdictional licensing boards requesting the examination were those with significant amounts of software engineering industry and those that have institutions granting ABET-accredited degrees in software engineering.

Over the past decade there have been several efforts to establish a path to professional practice licensure for software engineers in the United States. These efforts failed for various reasons. The main obstacle was the lack of a reasonable number of ABET-accredited programs offering an undergraduate degree in software engineering. This is no longer tha case as there are now a number of ABET-accredited software engineering undergraduate degree programs available in the United States.

In a report from the IEEE-USA Licensure and Registration committee, it was noted that software engineering professional licensure affects both US IEEE members and society at large. The central issue is one of protection of public safety, health and welfare, since the purpose of licensure is to establish competence to practice in an area and not to measure the level of an individual’s expertise above the competency threshold. A large number of technologies in existence today are directly impacted by software engineering, including safety critical artifacts such as real-time controllers and logistics systems. In addition other countries such as Canada require licensing of software engineers.

In July 2007, a group of stakeholder organizations including members of NSPE (National Society of Professional Engineers), IEEE, and TBPE (Texas Board of Professional Engineers) established the Software Engineering Licensure Consortium (SELC.) SELC developed a software engineering licensure needs document that was distributed to licensing boards. It also conducted a survey to determine need and interest in establishing a software engineering licensure path.

What comes next? The next phase is known as a Professional Activities and Knowledge Study (PAKS). The PAKS process will include the development of surveys and meetings with licensed engineers who practice software engineering that will ultimately result in a specification of the content for the software engineering licensure examination. Once the specification is developed, a committee of software engineers is formed to develop examination questions under the oversight of NCEES. After NCEES receives and approves the committee’s software engineering PE examination, it will be each up to individual licensing boards whether they license software engineers in their state or territory.

Professional licensure for an engineering discipline is established by each U.S. jurisdiction at their discretion. The existence of the software engineering PE examination does not automatically mean that each jurisdiction will approve a software engineer license since each board makes such a decision based on the needs (and politics) of their individual jurisdictions.

Hopefully the PAKS process will only take a year or two and the resulting examination will be approved by NCEES without delay. Meanwhile software engineers should start canvasing their state engineering licensing boards to approve licenses in this discipline once the software engineering PE examination is available.

Project Plymouth

2009-09-14T11:06:00.009-04:00

Plymouth is the codename for a freedesktop.org project started in 2007 by Ray Strobe of Redhat to develop a graphical application to display a flicker free animated splash screen during the boot process while logging console text output to a log file. Fedora 10 (Cambridge) was the first release of Fedora to contain Plymouth. Development work is actively ongoing and the current release is 0.71.

Plymouth is intended to be a replacement for rhgb (Red Hat Graphical Boot) which is currently used by Red Hat to provide a graphical boot display. If rhgb is part of the kernel command line, rhgb is started early in the boot process by /etc/sysinit. rhgb starts an X server for display :1 on one virtual terminal so that it avoids conflict with the regular X server which may be starting for display :0 on another virtual terminal. It also creates a Unix domain socket (/etc/rhgb/temp/rhgb-socket) so that boot scripts can communicate with it. As boot scripts execute, they can use rhgb-client to send messages to rhgb, which then updates the text and progress display. When the system is finished booting, rhgb-client is invoked with the --quit option to send a terminate request to rhgb. The user is then switched to the X server used by the display manager. Unfortunately the sequence of switching from text mode to rhgb's X server to text mode to the display manager's X server can cause significant screen flickering. Another major drawback of rhgb is that boot messages are not logged.

Plymouth is designed to work on systems with DRM (Direct Rendering Manager) kernel modesetting drivers. DRM is a component of the Direct Rendering Infrastructure project. It consists of two kernel modules, a generic DRM driver, and another which has support for the specific graphics card hardware. This pair of drivers allows a userspace client direct access to the graphics card hardware. See here for further information on DRM mode setting. The idea behind Plymouth is that very early on in the boot process the native video display mode for the system is set by a kernel mode setting driver. In turn Plymouth uses that mode, and that mode remains the same during the entire boot process up to and after an X server starts. For systems without kernel modesetting drivers, there is a fallback text mode which is the familiar tricolor blue/white/black progress bar. Plymouth also drops back to this text mode if the default plugin fails for whatever reason.

Kernel modesetting drivers are still in active development and somewhat buggy. Currently, only Radeon R500 and higher series graphics cards support kernel modesetting by default. There is work in progress to provide kernel modesetting support for R100 and R200 graphics cards. Intel kernel modesetting drivers exist but are not turned on by default. Support for kernel modesetting in the nVidia graphic cards via the Nouveau driver is still experimental. If you end up with nothing but a black screen during boot up, or a screen with nothing but random noise on it, try adding nomodeset to the kernel command line to disable kernel mode setting.

If there is no suitable kernel modesetting driver available for your particular graphics card or you want to set an explicit mode, you can add the string vga=XXX to the kernel command line. The kernel command line option vga=ask invokes the the built-in vesa framebuffer driver, displays a list of supported modes and asks you to select a mode. It then boots the kernel using this mode. The kernel command line option vga=mode, where mode is either a 4 digit hexadecimal with a leading zero and no letter 'x' or a 3 digit decimal number, enables you to set a specific mode.

How can you tell what particular modes are available and which will work best for you? This really depends on the type of graphics card that you have in your system, and the amount of video memory available. The only way is to experiment with different modes.

The following table shows the mode numbers you can input at the vga= prompt using hexadecimals

and here is the same table using decimal numbers.

Note that 8 bits = 256 colors, 15 bits [5:5:5] = 32,768 colors, 16 bits [5:6:5] = 65,536 colors and 24 bits [8:8:8] = 16.8 million colors. Additional modes are at the discretion of the graphics card manufacturer, as the VESA 2.0 specification only defines modes up to 0x31F. For more information about VESA modes, see this article about VESA BIOS Extension compliant graphic cards.

Plymouth works with themes which are analogous to screensavers that are displayed at boot time. Fedora 11 shipped with three graphical themes solar, fade-in and spinfinity, and two non-graphical themes text and details. The text theme is the default theme which is displayed if another theme fails for whatever reason.

The terminology and technology around themes and plugins has evolved as the project progressed. The version of Plymouth that shipped in Fedora 10 was based on a plugin system where each splash screen had to be coded from scratch. This problem was recognized and for Fedora 11 Plymouth went through a major rewrite whereby it now supports themes which in turn use standard plugins. Thus theme developers can now focus on the theme graphics rather than having to do raw coding.

Currently there are five themes in the Fedora repositories. Charge is the default theme for Fedora 11 (Leonidas). Spinfinity is a throbber that moves in a path shaped like the infinity sign. Fade-In shows the Fedora logo fading in and out in a star field. details shows the classic scrolling output from the boot process. text is the fallback bottom of the screen tricolor theme. Solar, my personal favorite to date and the default theme for Fedora 10, was not included in Fedora 11. It displays a planet with exploding pulsars.

To install all the Plymouth themes in the Fedora repositories:


# yum -y install plymouth-theme-*

Installed Plymouth themes can be listed use the plymouth-set-default-theme script:


# /usr/sbin/plymouth-set-default-theme --list
charge
details
fade-in
spinfinity
text

Theme files are stored in the /usr/share/plymouth/themes subdirectory.


# ls /usr/share/plymouth/themes/
charge  default.plymouth  details  fade-in  spinfinity  text

Note that default.plymouth is a symbolic link to the actual designed default theme.

There are two types of plugins: splash and control. There can only be one splash plugin in use at a time. A splash plugin is what draws the splash screen, asks for a password, displays messages, and more. A theme calls a splash plugin to do the actual work. For example, here is a listing of the files associated with the charge theme.


$ ls /usr/share/plymouth/themes/charge
box.png          progress-01.png  progress-07.png  progress-13.png  throbber-00.png  throbber-06.png  throbber-12.png
bullet.png       progress-02.png  progress-08.png  progress-14.png  throbber-01.png  throbber-07.png  throbber-13.png
charge.plymouth  progress-03.png  progress-09.png  progress-15.png  throbber-02.png  throbber-08.png  throbber-14.png
entry.png        progress-04.png  progress-10.png  progress-16.png  throbber-03.png  throbber-09.png  throbber-15.png
lock.png         progress-05.png  progress-11.png  progress-17.png  throbber-04.png  throbber-10.png
progress-00.png  progress-06.png  progress-12.png  progress-18.png  throbber-05.png  throbber-11.png

The theme configuration file that is read by plymouthd is the name of the theme with a .plymouth extension. In this case it is charge.plymouth.


$ cat /usr/share/plymouth/themes/charge/charge.plymouth
[Plymouth Theme]
Name=Charge
Description=A theme that features the shadowy hull of a Fedora logo charge up and and finally burst into into full form.
ModuleName=two-step

[two-step]
ImageDir=/usr/share/plymouth/themes/charge
HorizontalAlignment=.5
VerticalAlignment=.5
Transition=none
TransitionDuration=0.0
BackgroundStartColor=0x416fa7
BackgroundEndColor=0x4b83c1

This theme calls the two-step plugin to do the actual work of displaying the theme. The two-step plugin expects a certain number and type of image files with specific names. Various directives can be passed to plugins; the number and type being plugin specific. For example different type of transitions can be specified for the two-step plugin using the Transition directive, i.e. fade-over, cross-fade and merge-fade.

Some plugins did not ship with Fedora 11. One such plugin is the label plugin. It is not part of initrd but is loadable once the root filesystem is mounted. It is implicitly loaded when a splash plugin attempts to display text. After label is loaded, it uses pango and cairo to handle message localization.

Another such plugin is script which supports a scripting language for themes. It supports two basic objects, i.e. Image and Sprite. If you are familiar with Javascript or the C language you should be comfortable with the syntax and idiom. Note that the scripting language is undergoing rapid development at present with a view to making it more object orientated so you may have to read the git logs or the source code to figure out what is or is not supported.

For examples of scripted themes, I recommend you look at the sources for the Vizta or the Dandelion themes. These themes were both developed by Charlie Brie, a research assistant at the University of Manchester UK, who is the main developer behind the scripting language. If you want to try these themes out on Fedora 11, you will have to import the sources from the Plymouth Git tree, configure, rebuild and install on your system.

The two main binaries involved in Plymouth are /sbin/plymouthd, a daemon that does most of the actual work by displaying the splash screen and logging the boot session, and /bin/plymouth which is the interface to /sbin/plymouthd. Unfortunately no man page is supplied for /bin/plymouth but there is some information in the /usr/share/doc/plymouth-0.7.0 subdirectory. Both have a number of useful options.


$ /sbin/plymouthd --help
Boot splash control server
USAGE: plymouthd [OPTION...]
Options:
  --help                 This help message
  --attach-to-session    Redirect console messages from screen to log
  --no-daemon            Do not daemonize
  --debug                Output debugging information
  --mode=        Mode is one of: boot, shutdown

$ /bin/plymouth --help
Boot splash control client
USAGE: plymouth [OPTION...] [COMMAND [OPTION...]...]

Options:
  --help                         This help message
  --debug                        Enable verbose debug logging
  --newroot=             Tell boot daemon that new root filesystem is mounted
  --quit                         Tell boot daemon to quit
  --ping                         Check of boot daemon is running
  --sysinit                      Tell boot daemon root filesystem is mounted read-write
  --show-splash                  Show splash screen
  --hide-splash                  Hide splash screen
  --ask-for-password             Ask user for password
  --ignore-keystroke=    Remove sensitivity to a keystroke
  --update=              Tell boot daemon an update about boot progress
  --details                      Tell boot daemon there were errors during boot
  --wait                         Wait for boot daemon to quit

Available commands:
  ask-for-password   Ask user for password
  ask-question       Ask user a question
  message            Display a message
  watch-keystroke    Become sensitive to a keystroke
  pause-progress     Pause boot progress bar
  unpause-progress   Unpause boot progress bar
  report-error       Tell boot daemon there were errors during boot
  quit               Tell boot daemon to quit

Options for ask-for-password command:
  --command=             Command to send password to via standard input
  --prompt=              Message to display when asking for password
  --number-of-tries=    Number of times to ask before giving up (requires --command)
  --dont-pause-progress          Don't pause boot progress bar while asking

Options for ask-question command:
  --command=       Command to send the answer to via standard input
  --prompt=        Message to display when asking the question
  --dont-pause-progress    Don't pause boot progress bar while asking

Options for message command:
  --text=    The message text

Options for watch-keystroke command:
  --command=    Command to send keystroke to via standard input
  --keys=       Keys to become sensitive to

Options for quit command:
  --retain-splash    Don't explicitly hide boot splash on exit

In Fedora /usr/bin/rhgb-client is a symbolic link to /usr/bin/plymouth

One way to experiment with Plymouth is to invoke it from runlevel 2 or 3. For example, here is a simple script to display the default theme for 5 seconds, ask for a password, and ask for your name before finally quitting.


#!/bin/sh

# first check that we are in an appropriate runlevel
rlevel=$(runlevel | cut -d " " -f 2)
if [[ "$rlevel" != "2"  && "$rlevel" != "3" ]]
then
    echo "ERROR: You must be at runlevel 2 or 3"
    exit 1
fi

echo "Testing plymouth default theme ..."
plymouthd
sleep 1

# check if the plymouthd daemon is alive
plymouth --ping
if [[ $? -eq 1 ]]
then
    echo "ERROR: Plymouth daemon not running"
    exit 1
fi

# show the default splash screen for 5 seconds
plymouth  --show-splash
sleep 5

plymouth --ask-for-password
sleep 2

# using a command rather than an option
plymouth ask-question --prompt="What is your name?"
sleep 5

plymouth --quit

echo "Done ..."
exit 0

Note that not all plugins support every command and option at the present time. The above script works with the solar theme which uses the space-flares plugin. However this plugin does not support the message command for example. A useful option which is missing from plymouth would be an option to enumerate which commands were supported.

Plymouth is not really designed to be built from source by end users. For it to work correctly, it has to be integrated into the underlying distribution. Because it starts so early in the boot process, it needs to be added to a distribution's initrd (initial ram disk) and the distribution needs to interface with plymouthd to tell it how the boot is progressing. For example, here is the nash script in my Fedora 11 initrd. Notice how the Plymouth splash screen is called as soon as a console is available and also several other times in the script.


lsinitrd /boot/initrd-2.6.29.5-191.fc11.x86_64.img
.........................
#!/bin/nash
mount -t proc /proc /proc
setquiet
echo Mounting proc filesystem
echo Mounting sysfs filesystem
mount -t sysfs /sys /sys
echo Creating /dev
mount -o mode=0755 -t tmpfs /dev /dev
mkdir /dev/pts
mount -t devpts -o gid=5,mode=620 /dev/pts /dev/pts
mkdir /dev/shm
mkdir /dev/mapper
echo Creating initial device nodes
mknod /dev/null c 1 3
mknod /dev/zero c 1 5
mknod /dev/systty c 4 0
mknod /dev/tty c 5 0
mknod /dev/console c 5 1
mknod /dev/ptmx c 5 2
mknod /dev/fb c 29 0
mknod /dev/hvc0 c 229 0
mknod /dev/tty0 c 4 0
mknod /dev/tty1 c 4 1
mknod /dev/tty2 c 4 2
mknod /dev/tty3 c 4 3
mknod /dev/tty4 c 4 4
mknod /dev/tty5 c 4 5
mknod /dev/tty6 c 4 6
mknod /dev/tty7 c 4 7
mknod /dev/tty8 c 4 8
mknod /dev/tty9 c 4 9
mknod /dev/tty10 c 4 10
mknod /dev/tty11 c 4 11
mknod /dev/tty12 c 4 12
mknod /dev/ttyS0 c 4 64
mknod /dev/ttyS1 c 4 65
mknod /dev/ttyS2 c 4 66
mknod /dev/ttyS3 c 4 67
daemonize --ignore-missing /bin/plymouthd
/lib/udev/console_init tty0
plymouth --show-splash
echo Setting up hotplug.
hotplug
echo Creating block device nodes.
mkblkdevs
echo Creating character device nodes.
mkchardevs
echo Making device-mapper control node
mkdmnod
modprobe scsi_wait_scan
rmmod scsi_wait_scan
mkblkdevs
echo Scanning logical volumes
lvm vgscan --ignorelockingfailure
echo Activating logical volumes
lvm vgchange -ay --ignorelockingfailure  vg_ultra
resume /dev/mapper/vg_ultra-lv_swap
echo Creating root device.
mkrootdev -t ext4 -o defaults,ro /dev/mapper/vg_ultra-lv_root
echo Mounting root filesystem.
mount /sysroot
cond -ne 0 plymouth --hide-splash
echo Setting up other filesystems.
setuproot
loadpolicy
plymouth --newroot=/sysroot
echo Switching to new root and running init.
switchroot
echo Booting has failed.
sleep -1
init

During the boot progress the boot status is regularly updated with strings signifying what is happening. Plugins can listen to these if they choose to but they are generally ignored in the current plugins, and are only used for calculating the boot time estimation. In Fedora 11, For example the rc.sysinit script includes several calls to plymouth to hide or show the splash screen according to whether a password is needed to access a filesystem or a filesystem is to be relabeled by selinux.

So how does Plymouth know when to quit? Actually, it has no way of knowing. It just keeps on going until it receives a quit message. In the case of Fedora 11, the /etc/event.d/quit-plymouth script sends the quit message.


# quit-plymouth - script to stop boot splash
#
# This service triggers plymouth to quit when we reach the
# end of the boot cycle. We start on 'stopping rcX' to make sure
# this completes before the getty starts.
# prefdm handles quit differently, though.

start on runlevel S
start on stopping rc2
start on stopping rc3
start on stopping rc4

script
         /usr/bin/plymouth quit || :
end script

A special case is when a user boots to single user. In this case the /etc/event.d/rcS-sulogin script is executed.


# rcS-sulogin - "single-user" runlevel compatibility
#
# This task runs /bin/bash during "single-user" mode,
# then continues to the default runlevel.

start on runlevel S

stop on runlevel

console owner
script
        runlevel --set S >/dev/null || true
        plymouth --hide-splash || true
        exec /bin/bash
end script
post-stop script
        if [ "$1" = "S" ]; then
                runlevel=$(/bin/awk -F ':' '$3 == "initdefault" && $1 !~ "^#" { print $2 }' /etc/inittab)
                [ -z "$runlevel" ] && runlevel="3"
                exec telinit $runlevel
        fi
end script

What is not commonly known is that you can also use Plymouth to provide a splash screen during system shutdown or reboot. This is done in Fedora 11 via the /etc/event.d/plymouth-shutdown script. as shown below.


# plymouth-shutdown - put up shutdown splash
#
# This service triggers plymouth to put up a splash
# when leaving runlevel 5.

start on stopped prefdm

console output
script
        set $(runlevel || true)
        if [ "$2" != "0" ] && [ "$2" != "6" ]; then
                exit 0
        fi

        /sbin/plymouthd --mode=shutdown || exit 1
        /bin/plymouth --sysinit
        /bin/plymouth --show-splash
        if [ "$2" = "0" ]; then
                /bin/plymouth message --text="Shutting down..."
        elif [ "$2" = "6" ]; then
                /bin/plymouth message --text="Restarting..."
        fi
end script

Console boot messages are redirected to a pseudo-terminal which is created very early on in the boot process. These messages are buffered until filesystems are fully mounted. Then the buffer is dumped to /var/log/boot.. In either text or graphics mode, the boot messages are obscured. However you can see these messages at any time during boot sequence by hitting the ESC key.

One of the side effects of changing Plymouth themes is that you have to generate a new initrd image. Usually this is done using the mkinird script. However there is an alternative to doing this. You can modify your existing initrd image to remove any Plymouth-related files and create a second initrd image which contains just the Plymouth-related files. When you change a theme, only the Plymouth image needs to be generated. You have to modify grub.conf to load both images when booting. Here is a stanza from my grub.conf which does just that.


title Graphical Boot (Fedora 2.6.29.6-217.2.16.fc11.x86_64)
        root (hd0,1)
        kernel /vmlinuz-2.6.30.5-43.fc11.x86_64 ro root=/dev/mapper/vg_ultra-lv_root rhgb quiet nopat vga=0x37b 2
        initrd /initrd-2.6.30.5-43.fc11.x86_64.img  /initrd-plymouth.img

Here is a shell script which will generate the two images. It is based on existing scripts in the Plymouth codebase.


#!/bin/bash
#
#
#  FPMurphy  9/12/2009
#

[ -z "$TMPDIR" ] && TMPDIR="/var/tmp"

[ -z "$LIBEXECDIR" ] && LIBEXECDIR="/usr/libexec"
[ -z "$DATADIR" ] && DATADIR="/usr/share"
[ -z "$PLYMOUTH_PLUGIN_PATH" ] && PLYMOUTH_PLUGIN_PATH="$(plymouth --get-splash-plugin-path)"
[ -z "$PLYMOUTH_LOGO_FILE" ] && PLYMOUTH_LOGO_FILE="/usr/share/plymouth/bizcom.png"
[ -z "$PLYMOUTH_THEME_NAME" ] && PLYMOUTH_THEME_NAME=$(plymouth-set-default-theme)

[ -z "$PLYMOUTH_IMAGE_FILE" ] && PLYMOUTH_IMAGE_FILE="/boot/initrd-plymouth.img"
[ -z "$IMAGE_FILE" ] && IMAGE_FILE="/boot/initrd-$(uname -r).img"

if [ -z "$PLYMOUTH_POPULATE_SOURCE_FUNCTIONS" ]; then
    if [ -f "${LIBEXECDIR}/initrd-functions" ]; then
        PLYMOUTH_POPULATE_SOURCE_FUNCTIONS="${LIBEXECDIR}/initrd-functions"
    fi
    if [ -f "${DATADIR}/dracut/dracut-functions" ]; then
        PLYMOUTH_POPULATE_SOURCE_FUNCTIONS="${DATADIR}/dracut/dracut-functions"
    fi
fi

if [ -n "$PLYMOUTH_POPULATE_SOURCE_FUNCTIONS" ]; then
    source $PLYMOUTH_POPULATE_SOURCE_FUNCTIONS
fi

if [ " $(type -t inst) " != " function " ]; then
    echo "Need 'inst' function, try setting PLYMOUTH_POPULATE_SOURCE_FUNCTIONS to a file that defines it" 1>&2
    exit 1
fi

if [ " $(type -t set_verbose) " != " function " ]; then
    function set_verbose { true; }
fi

Function usage() {
    local output="/dev/stdout"
    local rc=0
    if [ "$1" == "error" ]; then
        output="/dev/stderr"
        rc=1
    fi

    echo "usage: plymouth_setup_initrds [ --verbose | -v ]" > $output
    exit $rc
}

verbose=false
INITRDDIR=""
while [ $# -gt 0 ]; do
    case $1 in
        --verbose|-v)
            verbose=true
            ;;
        --help|-h)
            usage normal
            ;;
        *)
            usage error
            break
            ;;
    esac
    shift
done
set_verbose $verbose || :

CURRENTDIR=`pwd`
INITRDDIR=`mktemp -d ${TMPDIR}/initrd.XXXXXX`
[ -z "$INITRDDIR" ] && {
     echo "mktemp failed"
     exit 1
}

mkdir -p ${INITRDDIR}${DATADIR}/plymouth/themes
inst /sbin/plymouthd $INITRDDIR /bin/plymouthd
inst /bin/plymouth $INITRDDIR
inst ${DATADIR}/plymouth/themes/text/text.plymouth $INITRDDIR
inst ${PLYMOUTH_PLUGIN_PATH}/text.so $INITRDDIR
inst ${DATADIR}/plymouth/themes/details/details.plymouth $INITRDDIR
inst ${PLYMOUTH_PLUGIN_PATH}/details.so $INITRDDIR
inst ${PLYMOUTH_LOGO_FILE} $INITRDDIR
inst /etc/system-release $INITRDDIR
if [ -z "$PLYMOUTH_THEME_NAME" ]; then
    echo "No default plymouth plugin is set" > /dev/stderr
    exit 1
fi

PLYMOUTH_MODULE_NAME=$(grep "ModuleName *= *" ${DATADIR}/plymouth/themes/${PLYMOUTH_THEME_NAME}/${PLYMOUTH_THEME_NAME}.plymouth | sed 's/ModuleName *= *//')

if [ ! -f ${PLYMOUTH_PLUGIN_PATH}/${PLYMOUTH_MODULE_NAME}.so ]; then
    echo "The default plymouth plugin (${PLYMOUTH_MODULE_NAME}) doesn't exist" > /dev/stderr
    exit 1
fi

inst ${PLYMOUTH_PLUGIN_PATH}/${PLYMOUTH_MODULE_NAME}.so $INITRDDIR

if [ -d ${DATADIR}/plymouth/themes/${PLYMOUTH_THEME_NAME} ]; then
    for x in ${DATADIR}/plymouth/themes/${PLYMOUTH_THEME_NAME}/* ; do
        [ ! -f "$x" ] && break
        inst $x $INITRDDIR
    done
fi

if [ -L ${DATADIR}/plymouth/themes/default.plymouth ]; then
    cp -a ${DATADIR}/plymouth/themes/default.plymouth $INITRDDIR${DATADIR}/plymouth/themes
fi

# generate the initrd-plymouth image
if [ -f ${PLYMOUTH_IMAGE_FILE} ]; then
     mv ${PLYMOUTH_IMAGE_FILE} ${PLYMOUTH_IMAGE_FILE}.bak
fi

echo "Generating image: $PLYMOUTH_IMAGE_FILE"
cd ${INITRDDIR}
rm -f lib*/{ld*,libc*,libdl*,libm*,libz*,libpthread*,libpng*,librt*}
rm -f usr/lib*/libpng*
find . | cpio -H newc --quiet -o | gzip -9 > ${PLYMOUTH_IMAGE_FILE}

cd ${CURRENTDIR}
rm -rf ${INITRDDIR}

#  now remove all plymouth items from regular initrd
INITRDDIR=`mktemp -d ${TMPDIR}/initrd.XXXXXX`
[ -z "$INITRDDIR" ] && {
     echo "mktemp failed"
     exit 1
}
cd ${INITRDDIR}
zcat ${IMAGE_FILE} | cpio -i

rm -f ${INITRDDIR}/bin/plymout*
rm -f ${INITRDDIR}/lib64/libply*
rm -f ${INITRDDIR}/usr/lib64/libply*
rm -rf ${INITRDDIR}/usr/lib64/plymouth
rm -rf ${INITRDDIR}/usr/share/plymouth

echo "Generating image: ${IMAGE_FILE}"
mv ${IMAGE_FILE} ${IMAGE_FILE}.bak
findall . | cpio -H newc --quiet -o  | gzip -9 > ${IMAGE_FILE}

cd ${CURRENTDIR}
rm -rf ${INITRDDIR}

exit 0

Here is a listing of the generated /boot/initrd-plymouth.img image.


$ lsinitrd /boot/initrd-plymouth.img
drwx------   6 root     root            0 Sep 12 16:43 .
drwxr-xr-x   2 root     root            0 Sep 12 16:43 etc
-rw-r--r--   1 root     root           29 May 11 18:45 etc/fedora-release
lrwxrwxrwx   1 root     root           14 Sep 12 16:43 etc/system-release -> fedora-release
drwxr-xr-x   4 root     root            0 Sep 12 16:43 usr
drwxr-xr-x   3 root     root            0 Sep 12 16:43 usr/lib64
drwxr-xr-x   2 root     root            0 Sep 12 16:43 usr/lib64/plymouth
-rwxr-xr-x   1 root     root        27242 Sep 12 01:42 usr/lib64/plymouth/details.so
-rwxr-xr-x   1 root     root        28471 Sep 12 01:42 usr/lib64/plymouth/text.so
-rwxr-xr-x   1 root     root        80032 Sep 12 01:42 usr/lib64/plymouth/space-flares.so
-rwxr-xr-x   1 root     root       200218 Sep 12 01:42 usr/lib64/libplybootsplash.so.2.0.0
lrwxrwxrwx   1 root     root           25 Sep 12 16:43 usr/lib64/libplybootsplash.so.2 -> libplybootsplash.so.2.0.0
drwxr-xr-x   3 root     root            0 Sep 12 16:43 usr/share
drwxr-xr-x   3 root     root            0 Sep 12 16:43 usr/share/plymouth
-rw-r--r--   1 root     root         5529 Sep 12 01:42 usr/share/plymouth/bizcom.png
drwxr-xr-x   5 root     root            0 Sep 12 16:43 usr/share/plymouth/themes
drwxr-xr-x   2 root     root            0 Sep 12 16:43 usr/share/plymouth/themes/details
-rw-r--r--   1 root     root           84 Sep 12 01:42 usr/share/plymouth/themes/details/details.plymouth
drwxr-xr-x   2 root     root            0 Sep 12 16:43 usr/share/plymouth/themes/text
-rw-r--r--   1 root     root           98 Sep 12 01:42 usr/share/plymouth/themes/text/text.plymouth
lrwxrwxrwx   1 root     root           20 Sep 12 16:43 usr/share/plymouth/themes/default.plymouth -> solar/solar.plymouth
drwxr-xr-x   2 root     root            0 Sep 12 16:43 usr/share/plymouth/themes/solar
-rw-r--r--   1 root     root          246 Sep 12 01:42 usr/share/plymouth/themes/solar/progress_bar.png
-rw-r--r--   1 root     root       355666 Sep 12 01:42 usr/share/plymouth/themes/solar/star.png
-rw-r--r--   1 root     root         1896 Sep 12 01:42 usr/share/plymouth/themes/solar/lock.png
-rw-r--r--   1 root     root          165 Sep 12 01:42 usr/share/plymouth/themes/solar/solar.plymouth
-rw-r--r--   1 root     root          296 Sep 12 01:42 usr/share/plymouth/themes/solar/bullet.png
-rw-r--r--   1 root     root          870 Sep 12 01:42 usr/share/plymouth/themes/solar/box.png
-rw-r--r--   1 root     root          350 Sep 12 01:42 usr/share/plymouth/themes/solar/entry.png
drwxr-xr-x   2 root     root            0 Sep 12 16:43 lib64
-rwxr-xr-x   1 root     root       293522 Sep 12 01:42 lib64/libply.so.2.0.0
lrwxrwxrwx   1 root     root           15 Sep 12 16:43 lib64/libply.so.2 -> libply.so.2.0.0
drwxr-xr-x   2 root     root            0 Sep 12 16:43 bin
-rwxr-xr-x   1 root     root        70256 Sep 12 01:42 bin/plymouth
-rwxr-xr-x   1 root     root       110319 Sep 12 01:42 bin/plymouthd

As you can see it only contains Plymouth-related files. It does not contain the label plugin because this is loaded in using dlopen() when needed.

You can debug Plymouth by adding plymouth:debug, plymouth:debug=file:, or plymouth:debug=file:path_to_log_file on the kernel command line. The default file is /var/log/plymouth-debug.log if logging to a file is specified but no file is specified, i.e. option two. Other kernel command line options include console=/dev/what_ever_works to override the default console (/dev/tty0) and plymouth:splash=name_of_theme to override the default theme.

There are also a number of key combinations such as CTRL-L to redraw the screen, CTRL-V to toggle debug mode and CTRL-T to enable text mode. Unfortunately I was unable to get any of these key combinations to work. However the ESC key worked as expected and toggled the display between detailed and the default theme.

Plymouth does all pixel manipulation in software. There is no GPU acceleration. It does not use MMX (Matrix Math Extension) or SSE (Streaming SIMD Extension) hence there is no CPU acceleration either. A plugin that loads a lot of images or does a lot of full screen updates will be slower than one that loads a few images and just updates small parts of the screen. Much of what has been written about Plymouth in the computing press implies that the goal of Plymouth is to provide a faster boot up experience but that is not an explicit design goal of Plymouth.

Well that is about all the useful information on Plymouth that I have time to write about at present. After reading this post, I hope you have a better understanding of Plymouth and how it relates to the boot sequence. Remember however that this project is in active development. and nothing is cast in stone. For example, the inclusion of Dracut, a replacement for nash, in Fedora 12 (Constantine) may affect how Plymouth is invoked. Themes and plug-ins are also rapidly evolving.

As Plymouth is implemented in other GNU/Linux distributions such as Ubuntu, expect to see a flourishing of graphical boot themes from independent authors. I look forward to that day.

Ruby D-Bus and Fedora 11

2009-09-07T23:40:00.001-04:00

Earlier this year I wrote a number of posts about monitoring and interacting with D-Bus using shell scripts. In this post I use Ruby to monitor and interact with D-Bus enabled applications. If you are unfamiliar with D-Bus, a good starting point is this Freedesktop.org tutorial by the authors of the D-Bus specification.

I used the standard out-of-the-box version of Ruby which comes with Fedora 11, i.e.


$ ruby --version
ruby 1.8.6 (2009-06-08 patchlevel 369) [x86_64-linux]

Those readers who are familiar with Ruby will recognize that this version of Ruby is quite old by Ruby standards and is in maintenance mode with Kirk Haines of Engine Yard being the lead maintainer. Hopefully a 1.9 version of Ruby will be included in the official repositories for Fedora 12.

I also installed the latest version (0.2.9 "I'm not dead") of ruby-dbus which comes as a compressed tarball rather than as a gem. This project was recently taken over by Martin Vidner, who works for Novell in their Prague development group, after a long period of no activity by the original developers. An introduction to and a tutorial on ruby-dbus is available here. It needs to be updated and expanded to include all the interfaces and methods but it is a good introduction to the general concepts.

Introspection is core to D-Bus scripting. Here is one way to introspect the D-Bus system bus and output the resulting introspection data in XML format. The introspection data format is specified by the D-Bus Specification. The introspect_data method does all the heavy lifting.


#!/usr/bin/ruby

require 'dbus'

bus = DBus::SystemBus.instance
xml = bus.introspect_data("org.freedesktop.DBus", "/org/freedesktop/DBus/Introspectable")
puts xml

Here is the output from my computer which is running Fedora 11:


<!DOCTYPE node PUBLIC "-//freedesktop//DTD D-BUS Object Introspection 1.0//EN"
"http://www.freedesktop.org/standards/dbus/1.0/introspect.dtd">
<node>
  <interface name="org.freedesktop.DBus.Introspectable">
    <method name="Introspect">
      <arg name="data" direction="out" type="s"/>
    </method>
  </interface>
  <interface name="org.freedesktop.DBus">
    <method name="Hello">
      <arg direction="out" type="s"/>
    </method>
    <method name="RequestName">
      <arg direction="in" type="s"/>
      <arg direction="in" type="u"/>
      <arg direction="out" type="u"/>
    </method>
    <method name="ReleaseName">
      <arg direction="in" type="s"/>
      <arg direction="out" type="u"/>
    </method>
    <method name="StartServiceByName">
      <arg direction="in" type="s"/>
      <arg direction="in" type="u"/>
      <arg direction="out" type="u"/>
    </method>
    <method name="UpdateActivationEnvironment">
      <arg direction="in" type="a{ss}"/>
    </method>
    <method name="NameHasOwner">
      <arg direction="in" type="s"/>
      <arg direction="out" type="b"/>
    </method>
    <method name="ListNames">
      <arg direction="out" type="as"/>
    </method>
    <method name="ListActivatableNames">
      <arg direction="out" type="as"/>
    </method>
    <method name="AddMatch">
      <arg direction="in" type="s"/>
    </method>
    <method name="RemoveMatch">
      <arg direction="in" type="s"/>
    </method>
    <method name="GetNameOwner">
      <arg direction="in" type="s"/>
      <arg direction="out" type="s"/>
    </method>
    <method name="ListQueuedOwners">
      <arg direction="in" type="s"/>
      <arg direction="out" type="as"/>
    </method>
    <method name="GetConnectionUnixUser">
      <arg direction="in" type="s"/>
      <arg direction="out" type="u"/>
    </method>
    <method name="GetConnectionUnixProcessID">
      <arg direction="in" type="s"/>
      <arg direction="out" type="u"/>
    </method>
    <method name="GetAdtAuditSessionData">
      <arg direction="in" type="s"/>
      <arg direction="out" type="ay"/>
    </method>
    <method name="GetConnectionSELinuxSecurityContext">
      <arg direction="in" type="s"/>
      <arg direction="out" type="ay"/>
    </method>
    <method name="ReloadConfig">
    </method>
    <method name="GetId">
      <arg direction="out" type="s"/>
    </method>
    <signal name="NameOwnerChanged">
      <arg type="s"/>
      <arg type="s"/>
      <arg type="s"/>
    </signal>
    <signal name="NameLost">
      <arg type="s"/>
    </signal>
    <signal name="NameAcquired">
      <arg type="s"/>
    </signal>
  </interface>
</node>

The following example uses the introspect method instead of the introspect_data method to enumerate a list of the available services using the ListActivatableNames method. I am going to assume that you are familiar with the Ruby language.so concepts such as looping and arrays do not need explanation.


require 'dbus'  
  
bus = DBus::SystemBus.instance  
proxy = bus.introspect("org.freedesktop.DBus", "/org/freedesktop/DBus/ListActivatableNames")  

bus.proxy.ListActivatableNames[0].each do |service|  
   puts "#{service}"  
end

This is the output:


org.freedesktop.DBus
org.freedesktop.DeviceKit.Disks
org.fedoraproject.Setroubleshootd
com.hp.hplip
org.fedoraproject.Config.Services
org.freedesktop.ConsoleKit
org.gnome.CPUFreqSelector
net.reactivated.Fprint
org.freedesktop.PackageKit
org.freedesktop.DeviceKit
org.freedesktop.NetworkManagerSystemSettings
org.gnome.ClockApplet.Mechanism
org.kerneloops.submit
org.freedesktop.PolicyKit
org.freedesktop.Gypsy
org.gnome.GConf.Defaults
fi.epitest.hostap.WPASupplicant
org.gnome.SystemMonitor.Mechanism
org.freedesktop.DeviceKit.Power
org.freedesktop.nm_dispatcher
org.opensuse.CupsPkHelper.Mechanism

You can query devices via the D-Bus interface to HAL as the following examples demonstrate.


#!/usr/bin/ruby

require 'dbus'  
  
bus = DBus::SystemBus.instance  
xml = bus.introspect_data("org.freedesktop.Hal", "/org/freedesktop/Hal/devices/computer")  
puts xml

Here is what is outputted when this script is executed:


<!DOCTYPE node PUBLIC "-//freedesktop//DTD D-BUS Object Introspection 1.0//EN"
"http://www.freedesktop.org/standards/dbus/1.0/introspect.dtd">
<node>
  <interface name="org.freedesktop.Hal.Device">
    <method name="GetAllProperties">
      <arg name="properties" direction="out" type="a{sv}"/>
    </method>
    <method name="SetMultipleProperties">
      <arg name="properties" direction="in" type="a{sv}"/>
    </method>
    <method name="GetProperty">
      <arg name="key" direction="in" type="s"/>
      <arg name="value" direction="out" type="v"/>
    </method>
    ......
    <signal name="InterfaceLockAcquired">
      <arg name="interface_name" type="s"/>
      <arg name="lock_holder" type="s"/>
      <arg name="num_locks" type="i"/>
    </signal>
    <signal name="InterfaceLockReleased">
      <arg name="interface_name" type="s"/>
      <arg name="lock_holder" type="s"/>
      <arg name="num_locks" type="i"/>
    </signal>
  </interface>
  .......
  <interface name="org.freedesktop.Hal.Device.CPUFreq">
    <method name="SetCPUFreqGovernor">
      <arg name="governor_string" direction="in" type="s"/>
    </method>
    <method name="SetCPUFreqPerformance">
      <arg name="value" direction="in" type="i"/>
    </method>
    <method name="SetCPUFreqConsiderNice">
      <arg name="value" direction="in" type="b"/>
    </method>
    <method name="GetCPUFreqGovernor">
      <arg name="return_code" direction="out" type="s"/>
    </method>
    <method name="GetCPUFreqPerformance">
      <arg name="return_code" direction="out" type="i"/>
    </method>
    <method name="GetCPUFreqConsiderNice">
      <arg name="return_code" direction="out" type="b"/>
    </method>
    <method name="GetCPUFreqAvailableGovernors">
      <arg name="return_code" direction="out" type="as"/>
    </method>
  </interface>
</node>

Here is an example of how to output the previous data in a more readable form using the interfaces method..


#!/usr/bin/ruby

require 'dbus'

bus = DBus::SystemBus.instance
proxy = bus.introspect("org.freedesktop.Hal", "/org/freedesktop/Hal/devices/computer")

proxy.interfaces.each do |interface|
   puts "Interface:  #{interface}"
   proxy[interface].methods.each do |key,value|
      puts "     Method: #{key}"
   end
   proxy[interface].signals.each do |key,value|
      puts "     Signal: #{key}"
   end

Here is an portion of the output: from executing this script.


Interface:  org.freedesktop.Hal.Device.CPUFreq
     Method: GetCPUFreqConsiderNice
     Method: GetCPUFreqGovernor
     Method: SetCPUFreqPerformance
     Method: GetCPUFreqAvailableGovernors
     Method: SetCPUFreqGovernor
     Method: GetCPUFreqPerformance
     Method: SetCPUFreqConsiderNice
Interface:  org.freedesktop.Hal.Device.SystemPowerManagement
     Method: Shutdown
     Method: SuspendHybrid
     Method: SetPowerSave
     Method: Hibernate
     Method: Reboot
     Method: Suspend
Interface:  org.freedesktop.Hal.Device
     Method: ReleaseInterfaceLock
     Method: ClaimInterface
     Method: Reprobe
     ........
     Method: SetPropertyBoolean
     Signal: InterfaceLockAcquired
     Signal: InterfaceLockReleased
     Signal: PropertyModified
     Signal: Condition
Interface:  org.freedesktop.DBus.Introspectable
     Method: Introspect

You can easily drill down to individual interfaces and methods. For example, here is one way to list the set of available CPU frequency governors.


#!/usr/bin/ruby
#
#  <interface name="org.freedesktop.Hal.Device.CPUFreq">
#    <method name="SetCPUFreqGovernor">
#      <arg name="governor_string" direction="in" type="s"/>
#    </method>
#    <method name="SetCPUFreqPerformance">
#      <arg name="value" direction="in" type="i"/>
#    </method>
#    <method name="SetCPUFreqConsiderNice">
#      <arg name="value" direction="in" type="b"/>
#    </method>
#    <method name="GetCPUFreqGovernor">
#      <arg name="return_code" direction="out" type="s"/>
#    </method>
#    <method name="GetCPUFreqPerformance">
#      <arg name="return_code" direction="out" type="i"/>
#    </method>
#    <method name="GetCPUFreqConsiderNice">
#      <arg name="return_code" direction="out" type="b"/>
#    </method>
#    <method name="GetCPUFreqAvailableGovernors">
#      <arg name="return_code" direction="out" type="as"/>
#    </method>
#  </interface>
#

require 'dbus'

bus = DBus::SystemBus.instance
proxy = bus.introspect("org.freedesktop.Hal", "/org/freedesktop/Hal/devices/computer")

interface = proxy["org.freedesktop.Hal.Device.CPUFreq"]
governors = interface.GetCPUFreqAvailableGovernors
puts "Available CPU Freq Governors:"
governors[0].each do |gov|
   puts "  #{gov}"
end

which produces the following output:


Available CPU Freq Governors:
  ondemand
  userspace
  performance

Lots of useful information can be be obtained from the HAL manager interface. The following script lists all the methods supported by the HAL manager interface and then uses the GetAllDevices method to enumerate the devices on my computer.


require 'dbus'

bus = DBus::SystemBus.instance
proxy = bus.introspect("org.freedesktop.Hal", "/org/freedesktop/Hal/Manager")

puts "HAL manager reports following methods:"
proxy.interfaces[0].each do |interface|
  proxy[interface].methods.each do |key, value|
    puts "   #{key}"
  end
end

interface = proxy["org.freedesktop.Hal.Manager"]
devices = interface.GetAllDevices
puts "\nHAL manager reports following devices:"
devices[0].each do |device|
   puts "  #{device}"
end

Here is a portion of the output when this script is executed


HAL manager reports following methods:
   AcquireGlobalInterfaceLock
   DeviceExists
   CommitToGdl
   GetAllDevicesWithProperties
   FindDeviceByCapability
   NewDevice
   SingletonAddonIsReady
   FindDeviceStringMatch
   GetAllDevices
   ReleaseGlobalInterfaceLock
   Remove

HAL manager reports following devices:
  /org/freedesktop/Hal/devices/net_ba_cf_03_4e_14_ca
  /org/freedesktop/Hal/devices/volume_part7_size_115326976
  /org/freedesktop/Hal/devices/computer
  /org/freedesktop/Hal/devices/storage_model_DVD_Writer_1070d
  ...............
  /org/freedesktop/Hal/devices/acpi_CPU0
  /org/freedesktop/Hal/devices/acpi_CPU1
  /org/freedesktop/Hal/devices/acpi_CPU2
  /org/freedesktop/Hal/devices/acpi_CPU3
  /org/freedesktop/Hal/devices/pci_8086_2940
  /org/freedesktop/Hal/devices/pci_8086_293e
  /org/freedesktop/Hal/devices/usb_device_1d6b_2_0000_00_1a_7_if0
  /org/freedesktop/Hal/devices/usb_device_1d6b_2_0000_00_1a_7
  /org/freedesktop/Hal/devices/pci_8086_293c
  /org/freedesktop/Hal/devices/usb_device_1d6b_1_0000_00_1a_2_if0
  /org/freedesktop/Hal/devices/usb_device_1d6b_1_0000_00_1a_2
  /org/freedesktop/Hal/devices/pci_8086_2939
  /org/freedesktop/Hal/devices/usb_device_1d6b_1_0000_00_1a_1_if0
  /org/freedesktop/Hal/devices/usb_device_1d6b_1_0000_00_1a_1
  /org/freedesktop/Hal/devices/pci_8086_2938
  /org/freedesktop/Hal/devices/usb_device_1d6b_1_0000_00_1a_0_if0
  /org/freedesktop/Hal/devices/usb_device_1d6b_1_0000_00_1a_0
  /org/freedesktop/Hal/devices/pci_8086_2937
  /org/freedesktop/Hal/devices/pci_8086_294c
  /org/freedesktop/Hal/devices/pci_10de_640
  /org/freedesktop/Hal/devices/pci_8086_29e1
  /org/freedesktop/Hal/devices/pci_8086_29e0

The next example uses the D-Bus interface to Tomboy, a desktop note-taking application, to create a simple note, display it for 5 seconds and then delete the note.


#!/usr/bin/ruby

require 'dbus'

bus = DBus::SessionBus.instance
service = bus.service("org.gnome.Tomboy")

tomboy = service.object("/org/gnome/Tomboy/RemoteControl")
tomboy.introspect
tomboy.default_iface = "org.gnome.Tomboy.RemoteControl"
note = tomboy.CreateNamedNote("My Note")[0]
tomboy.SetNoteContents(note, "Hello World")
tomboy.DisplayNote(note)
sleep 5
tomboy.DeleteNote(note)

Note the need to introspect the tomboy object and set the default interface using the default_iface method.

You can also add one or more tags to a Tomboy note and extract comprehensive metadata in XML format as shown in the following example.


#!/usr/bin/ruby

require 'dbus'

bus = DBus::SessionBus.instance
service = bus.service("org.gnome.Tomboy")

tomboy = service.object("/org/gnome/Tomboy/RemoteControl")
tomboy.introspect
tomboy.default_iface = "org.gnome.Tomboy.RemoteControl"
note = tomboy.CreateNamedNote("My Note")[0]
tomboy.SetNoteContents(note, "Hello World")
tomboy.AddTagToNote(note, "blog example")
tomboy.DisplayNote(note)
sleep 5
xml = tomboy.GetNoteCompleteXml(note)
tomboy.DeleteNote(note)

puts xml

Here is the XML that is outputted when this script is executed:


<?xml version="1.0" encoding="utf-16"?>
<note version="0.3" xmlns:link="http://beatniksoftware.com/tomboy/link" xmlns:size="http://beatniksoftware.com/tomboy/size" xmlns="http://beatniksoftware.com/tomboy">
  <title>My Note</title>
  <text xml:space="preserve"><note-content version="0.1">Hello World</note-content></text>
  <last-change-date>2009-09-07T15:44:10.2178650-04:00</last-change-date>
  <last-metadata-change-date>2009-09-07T15:44:10.2212100-04:00</last-metadata-change-date>
  <create-date>2009-09-07T15:44:10.2142940-04:00</create-date>
  <cursor-position>37</cursor-position>
  <width>450</width>
  <height>360</height>
  <x>1440</x>
  <y>0</y>
  <tags>
    <tag>blog example</tag>
  </tags>
  <open-on-startup>False</open-on-startup>
</note>

The next example show you how to monitor D-Bus messages and act upon certain messages. You can do this using D-Bus signals. Suppose, for example, you want to monitor activity relating to Tomboy notes such as when Tomboy notes are created. First of all you need to find out what D-Bus signals Tomboy supports. This is easy to do using introspection as shown in the following script.


#!/usr/bin/ruby

require 'dbus'

bus = DBus::SessionBus.instance
service = bus.service("org.gnome.Tomboy")

tomboy = service.object("/org/gnome/Tomboy/RemoteControl")
tomboy.introspect
tomboy.default_iface = "org.gnome.Tomboy.RemoteControl"
puts "List of supported signals:"
tomboy.signals.each do |key,value|
   puts "  #{key}"
end

Here is the output:


List of supported signals:
  NoteSaved
  NoteAdded
  NoteDeleted

As you can see the version of Tomboy on my computer emits three D-Bus signals - NoteAdded when a note is added, NoteSaved when a note is saved, and NoteDeleted when a note is deleted.

The following script outputs a message each time a Tomboy note is created, saved or deleted.


#!/usr/bin/ruby

require 'dbus'

bus = DBus::SessionBus.instance

match = DBus::MatchRule.new
match.type = "signal"
match.interface = "org.gnome.Tomboy.RemoteControl"
match.path = "/org/gnome/Tomboy/RemoteControl"

bus.add_match(match) do |msg, misc|
    puts "#{msg.member}  #{msg.params[0]}"
end

main = DBus::Main.new
main << bus
main.run

Here is what is outputted when a single Tomboy note is created, saved and then deleted.


NoteAdded  note://tomboy/56587548-7535-4e21-a4b3-3ee85ffa7756
NoteSaved  note://tomboy/56587548-7535-4e21-a4b3-3ee85ffa7756
NoteDeleted  note://tomboy/56587548-7535-4e21-a4b3-3ee85ffa7756

Well that's all for now. There is a lot more that you can do using ruby-dbus but I will leave it up to you to go explore the possibilities. The best place to discover what is possible is to study the ruby-dbussource code. It is fairly compact and terse but logically laid out.

RSS Ingester For XAM Reference VIM

2009-08-16T11:40:00.004-04:00

Recently the SNIA XAM SDK TWG (Software Development Kit Technical Working Group) released a new version (version 0.7) of the XAM SDK with lots of new features. Included in this SDK was the 5th code drop for the Reference VIM. Here is a summary of the major new functionality included in this code drop.

Support for XSet hold and release
Support for XSet autodelete and shred policies
Support for simple where clauses in level 1 queries
Support for retention policies including XSet import and export
Support for asynchronous methods
Support for level 2 queries

Also recently released were version 1.01 of the XAM technical positions (AKA specifications). If you are familiar with version 1.0 of these three documents you may just want to read the errata for each document which is also available on the website.

I decided to test the Reference VIM by writing a small Java application to ingest the RSS feed from my blog, display some information about each post and store this information in the Reference VIM database. This application uses an XML configuration file to store a list of the RSS feeds to be ingested and the popular ROME Java library to ingest and parse the individual RSS feeds. Rather then re-invent the world, it also uses a modified version of the Java source file ExampleBase.java which is supplied as part of the latest XAM SDK to create and authenticate the connection to the XAM library and Reference VIM.

Here is RSS2XAM..java which is meat of the application. As you can see, RSS2XAM extends ExampleBase. This is a useful paradigm for writing applications which use the Reference VIM.


//
// RSS2XAM - Ingest RSS feed and add to XAM Storage System
//           using SNIA XAM SDK V0.7 reference VIM
//

import java.net.URL;
import java.net.URLConnection;
import java.util.List;
import java.util.Calendar;
import java.io.File;

import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.*;

import com.sun.syndication.feed.synd.SyndFeed;
import com.sun.syndication.feed.synd.SyndEntry;
import com.sun.syndication.feed.synd.SyndImageImpl;
import com.sun.syndication.io.SyndFeedInput;
import com.sun.syndication.io.XmlReader;
import com.sun.syndication.feed.synd.SyndContentImpl;
import com.sun.syndication.feed.synd.SyndCategory;

import org.snia.xam.XSet;
import org.snia.xam.XSystem;
import org.snia.xam.XUID;

public class RSS2XAM
   extends ExampleBase
{

    private static void iterateRSSFeed(String rssFeedUrl, XSystem xsystem)
         throws Exception {

       URLConnection feedUrl = new URL(rssFeedUrl).openConnection();
       SyndFeedInput input = new SyndFeedInput();
       SyndFeed feed = input.build(new XmlReader(feedUrl));

       System.out.println("RSS feed: " + rssFeedUrl + " (" + feed.getFeedType() + ")");

       // ingest the feed, display info and create the XSets
       List list = feed.getEntries();
       for (int i = 0 ; i < list.size(); i++)
       {
           SyndEntry entry = (SyndEntry)list.get(i);

           String display = "Entry: " + i;
           display += "\n  Title: " + entry.getTitle();
           display += "\n  Link: " + entry.getLink();
           display += "\n  Author: " + entry.getAuthor();
           display += "\n  Date Published: " + entry.getPublishedDate();

           // get the list of categories 
           List catList = entry.getCategories();
           String categories = "";
           for (int j = 0 ; j < catList.size(); j++)
           {
               if (j > 0 ) categories += ", ";
               SyndCategory cat = (SyndCategory)catList.get(j);
               categories += cat.getName();
           }

           display += "\n  Categories: " + categories;
           System.out.println(display);

           XSet xset = xsystem.createXSet(XSet.MODE_UNRESTRICTED);
           xset.createProperty("com.fpmurphy.rss.title", false, entry.getTitle());
           xset.createProperty("com.fpmurphy.rss.link", false, entry.getLink());
           xset.createProperty("com.fpmurphy.rss.author", false, entry.getAuthor());
           xset.createProperty("com.fpmurphy.rss.categories", false, categories);
           Calendar cal=Calendar.getInstance();
           cal.setTime(entry.getPublishedDate());
           xset.createProperty("com.fpmurphy.rss.published_date", false, cal);

           XUID xuid  = xset.commit();
           System.out.println("  XSet XUID ==> " + xuid.toString() + "\n");
           xset.close();
       }
   }

   public static void main(String argv[]) {

      try {
          File file = new File("rss.xml");

          DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
          DocumentBuilder db = dbf.newDocumentBuilder();
          Document document = db.parse(file);
          document.getDocumentElement().normalize();

          // init, load Reference VIM, and authenticate 
          System.out.println("Connecting to Reference VIM");
          initLibrary();
          XSystem xsystem = connectToVIM(s_xri);
          authenticate(xsystem);

          // parse the rss.xml
          NodeList nodeList = document.getElementsByTagName("feed");
          int totalFeed = nodeList.getLength();
          System.out.println("Number of Feeds: " + totalFeed);

          for (int i = 0; i < nodeList.getLength(); i++) {
             Node feedNode = nodeList.item(i);
             if (feedNode instanceof Element) {
                Element child = (Element) feedNode;
                String check = child.getAttribute("check");
                String rssFeedUrl = child.getTextContent();

                // hand off to do the work
                iterateRSSFeed(rssFeedUrl, xsystem);
             }
          }
      } catch (Exception e) {
          e.printStackTrace();
      }
   }
}

Here is a copy of the modified exampleBase.java. If you compare it to the original source code in the XAM SDK (...//Java_Reference_VIM/examples/ExampleBase.java) you will see that most of the changes are minor simplifications.


/*
 * Copyright (c) 2009, Sun Microsystems, Inc.
 * Copyright (c) 2009, The Storage Networking Industry Association.
 *  
 * Redistribution and use in source and binary forms, with or without 
 * modification, are permitted provided that the following conditions are met:
 *  
 * Redistributions of source code must retain the above copyright notice, 
 * this list of conditions and the following disclaimer.
 *  
 * Redistributions in binary form must reproduce the above copyright notice, 
 * this list of conditions and the following disclaimer in the documentation 
 * and/or other materials provided with the distribution.
 *  
 * Neither the name of The Storage Networking Industry Association (SNIA) nor 
 * the names of its contributors may be used to endorse or promote products 
 * derived from this software without specific prior written permission.
 *  
 *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 
 *  AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
 *  IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 
 *  ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 
 *  LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 
 *  CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 
 *  SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 
 *  INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 
 *  CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
 *  ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 
 *  THE POSSIBILITY OF SUCH DAMAGE.
 */

import java.io.ByteArrayOutputStream;
import java.io.FileInputStream;
import java.util.Properties;
import org.snia.xam.XAMException;
import org.snia.xam.XAMLibrary;
import org.snia.xam.XSystem;
import org.snia.xam.base.XAMImplementation;
import org.snia.xam.util.SASLUtils;
import org.snia.xam.vim.reference.ReferenceAuthenticationStatus;
import org.snia.xam.vim.reference.utils.ReferenceSaslUtils;

public class ExampleBase {

    public static final String PROP_FILE = "rss2xam.props";
    public static final String XRI_PROP  = "xam.xri";
    public static final String CONFIG_PROP = "xam.vims";
    public static final String USER_PROP = "xam.username";
    public static final String PASS_PROP = "xam.password";

    protected static final String DEFAULT_USER = "test";
    protected static final String DEFAULT_PASS = "test";
    protected static String     s_pass;
    protected static String     s_user;
    protected static XAMLibrary s_xam;
    protected static String     s_xri;

    public static void initLibrary() throws Exception {
        System.out.println("\nInitializing VIM");

        Properties props = new Properties();
        String testPropFile = System.getProperty(PROP_FILE);
        if (testPropFile == null)
            testPropFile = PROP_FILE;

        System.out.println("Loading properties from file: " + testPropFile);
        props.load(new FileInputStream(testPropFile));

        s_xri = props.getProperty(XRI_PROP);

        s_user = props.getProperty(USER_PROP, DEFAULT_USER);
        s_pass = props.getProperty(PASS_PROP, DEFAULT_PASS);

        System.out.println("VIM Configuration contained in file: " + props.getProperty(CONFIG_PROP));
        s_xam = new XAMImplementation(props.getProperty(CONFIG_PROP));
    }

    public static XSystem connectToVIM(String xri) throws Exception {
        XSystem xsystem = null;
        System.out.println("Connection arguments: " + xri);
        xsystem = s_xam.connect(xri);
        return xsystem;
    }

    public static void authenticate(XSystem system) throws XAMException {
        String defMech = system.getString(XSystem.XAM_XSYSTEM_AUTH_SASL_DEFAULT);
        ByteArrayOutputStream response = new ByteArrayOutputStream(200);
        byte[] inputData = null;
        int retValue = 0;
        if (defMech.equals(ReferenceSaslUtils.SASL_MECHANISM_ANONYMOUS)) {
            retValue = system.authenticate(inputData, response);
        } else if (defMech.equals(SASLUtils.SASL_PLAIN)) {
            byte[] creds = ReferenceSaslUtils.encodeSASLPlain(null,
                    ReferenceAuthenticationStatus.TEST_USERNAME,
                    ReferenceAuthenticationStatus.TEST_PASSWORD);
            retValue = system.authenticate(creds, response);
        } else {
            throw new XAMException("Unknown SASL mechanism " + defMech);
        }

        if (retValue != XSystem.XAM_SASL_COMPLETE) {
            throw new XAMException("Failed to authenticate.");
        }
    }
}

Here is a copy of my rss2xam.props. I am storing the generated XSets at /home/fpm/rss/store but you can specify any directory you like by modifying the dir= directive on the second line of this file. Make sure that this directory exists and is writeable or you will get an exception when you invoke the application. I will leave it to you to modify the source to test that the specified directory exists and is writeable. As you can see, this file also specifies that the Reference VIM configuration file (vim.config) exists in the current directory. Again you are free to change the location to wherever you like.


xam.vims=./vim.config
xam.xri=snia-xam://SNIA_Reference_VIM!localhost?dir=/home/fpm/rss/store
xam.username=testuser
xam.password=testpasswd

Here is my vim.config: It is basically the same as comes with the XAM SDK. The only difference is that I create a new log every time the application is invoked rather than appending to the existing log.


.xam.config.vim.alias.SNIA_Reference_VIM=org.snia.xam.vim.reference.ReferenceVIM
xam_boolean..xam.log.append=false
xam_int..xam.log.max.size=200
xam_int..xam.log.verbosity=1
xam_int..xam.log.level=5

Here is a copy of my rss.xml. It contains a simple feed i.e. this blog. You can add more feeds like you like. The check attribute is for future use and is not used by this application.


<config>
   <feed check="10">http://blog.fpmurphy.com/feed</feed>
</config>

Here is the output when this application is complied and invoked.


$ javac RSS2XAM.java ExampleBase.java
$ java RSS2XAM
Connecting to Reference VIM
Initializing VIM
Loading properties from file: rss.props
VIM Configuration contained in file: ./vim.config
Connection arguments: snia-xam://SNIA_Reference_VIM!localhost?dir=/home/fpm/rss/store

Number of Feeds: 1
RSS feed: http://blog.fpmurphy.com/feed (rss_2.0)

Entry: 0
  Title: Amazon SimpleDB Typica Tutorial
  Link: http://blog.fpmurphy.com/2009/08/amazon-simpledb-typica-tutorial.html
  Author: fpmurphy
  Date Published: Tue Aug 04 12:35:14 GMT-05:00 2009
  Categories: AWS, Java, XAM
  XSet XUID ==> AAA6AwAqbu8xMjUwNDc2OTYwNzcwAcSLfePaTUlTnoaEkQ5kBOgDBBh2

Entry: 1
  Title: XAM Query Language
  Link: http://blog.fpmurphy.com/2009/07/xam-query-language.html
  Author: fpmurphy
  Date Published: Sun Jul 19 10:10:41 GMT-05:00 2009
  Categories: XAM, XSet, XStream, XSystem, SNIA
  XSet XUID ==> AAA6AwAqbH8xMjUwNDc2OTYwODExAjloAZmNdUUPto94Weh/11seuv3/

Entry: 2
  Title: Linux HPET Support
  Link: http://blog.fpmurphy.com/2009/07/linux-hpet-support.html
  Author: fpmurphy
  Date Published: Mon Jul 06 02:50:03 GMT-05:00 2009
  Categories: C, HPET, Linux
  XSet XUID ==> AAA6AwAquJ0xMjUwNDc2OTYwODMxAxk6BG7OQErMh3iqx3BMNwYGbeYJ

Entry: 3
  Title: XAM Mandated Fields
  Link: http://blog.fpmurphy.com/2009/06/xam-mandated-fields.html
  Author: fpmurphy
  Date Published: Mon Jun 29 21:07:04 GMT-05:00 2009
  Categories: XAM, SNIA
  XSet XUID ==> AAA6AwAqUWoxMjUwNDc2OTYwODUxBM5rT72XVEEeuQvgMU1G47JugiVN

Entry: 4
  Title: Atahualpa Theme
  Link: http://blog.fpmurphy.com/2009/06/blog-now-uses-wordpress-with-atahualpa-theme.html
  Author: fpmurphy
  Date Published: Wed Jun 24 16:14:55 GMT-05:00 2009
  Categories: Uncategorized
  XSet XUID ==> AAA6AwAqGhAxMjUwNDc2OTYwODY5BbZxkkmfZUMmnKAsE/THFApESwu7

Entry: 5
  Title: XAM Canonical Format
  Link: http://blog.fpmurphy.com/2009/06/xam-and-xop.html
  Author: fpmurphy
  Date Published: Sat Jun 20 17:15:00 GMT-05:00 2009
  Categories: XAM, XOP, SNIA
  XSet XUID ==> AAA6AwAq2AkxMjUwNDc2OTYwODg4Bhk1ODii0UOQlaAbfG+dvMwBUxFk

Entry: 6
  Title: Fedora 11 New Extended File Attributes Namespace
  Link: http://blog.fpmurphy.com/2009/06/fedore-11-extended-attibutes-namespace.html
  Author: fpmurphy
  Date Published: Mon Jun 15 09:31:00 GMT-05:00 2009
  Categories: Extended Attributes, Fedora 11, XAM, Linux
  XSet XUID ==> AAA6AwAqa8QxMjUwNDc2OTYwOTA3B3Mj+tqXN0YJjuftnzTjYW5uPlqR

Entry: 7
  Title: Fedora 11 nVidia Twinview Support
  Link: http://blog.fpmurphy.com/2009/06/fedora-11-nvidia-twinview-support.html
  Author: fpmurphy
  Date Published: Thu Jun 11 12:07:00 GMT-05:00 2009
  Categories: Fedora 11, Linux, Twinview, nVidia, Fedora
  XSet XUID ==> AAA6AwAq4JQxMjUwNDc2OTYwOTI5CNTrNQ44okBPnnJFleb57UFNj5t1

Entry: 8
  Title: Linux Security Capabilities
  Link: http://blog.fpmurphy.com/2009/05/linux-security-capabilities.html
  Author: fpmurphy
  Date Published: Thu May 28 10:14:00 GMT-05:00 2009
  Categories: Fedora 11, Linux, POSIX.1e, linux capabilities
  XSet XUID ==> AAA6AwAqURQxMjUwNDc2OTYwOTU1CTY+vTycKkMhgRj22XFJ+CpMdnKG

Entry: 9
  Title: Windows Parallel Filesystems
  Link: http://blog.fpmurphy.com/2009/05/windows-parallel-filesystems.html
  Author: fpmurphy
  Date Published: Tue May 26 12:27:00 GMT-05:00 2009
  Categories: Windows parallel filesystem, Windows
  XSet XUID ==> AAA6AwAq/jsxMjUwNDc2OTYwOTg0CuIRPc1YkUozqtuI+oCaeTkQx4n7

$

As you can see from the following listing, the Reference VIM creates an XML file and a subdirectory for each XSet that it creates as well as a Derby database called ReferenceVimDB. The name of each XML file and corresponding subdirectory is the XUID of the XSet.


$ ls store
ReferenceVimDB                                                     XSet_AAA6AwAqGhAxMjUwNDc2OTYwODY5BbZxkkmfZUMmnKAsE_THFApESwu7
XSet_AAA6AwAq2AkxMjUwNDc2OTYwODg4Bhk1ODii0UOQlaAbfG-dvMwBUxFk      XSet_AAA6AwAqGhAxMjUwNDc2OTYwODY5BbZxkkmfZUMmnKAsE_THFApESwu7.xml
XSet_AAA6AwAq2AkxMjUwNDc2OTYwODg4Bhk1ODii0UOQlaAbfG-dvMwBUxFk.xml  XSet_AAA6AwAq_jsxMjUwNDc2OTYwOTg0CuIRPc1YkUozqtuI-oCaeTkQx4n7
XSet_AAA6AwAq4JQxMjUwNDc2OTYwOTI5CNTrNQ44okBPnnJFleb57UFNj5t1      XSet_AAA6AwAq_jsxMjUwNDc2OTYwOTg0CuIRPc1YkUozqtuI-oCaeTkQx4n7.xml
XSet_AAA6AwAq4JQxMjUwNDc2OTYwOTI5CNTrNQ44okBPnnJFleb57UFNj5t1.xml  XSet_AAA6AwAquJ0xMjUwNDc2OTYwODMxAxk6BG7OQErMh3iqx3BMNwYGbeYJ
XSet_AAA6AwAqa8QxMjUwNDc2OTYwOTA3B3Mj-tqXN0YJjuftnzTjYW5uPlqR      XSet_AAA6AwAquJ0xMjUwNDc2OTYwODMxAxk6BG7OQErMh3iqx3BMNwYGbeYJ.xml
XSet_AAA6AwAqa8QxMjUwNDc2OTYwOTA3B3Mj-tqXN0YJjuftnzTjYW5uPlqR.xml  XSet_AAA6AwAqURQxMjUwNDc2OTYwOTU1CTY-vTycKkMhgRj22XFJ-CpMdnKG
XSet_AAA6AwAqbH8xMjUwNDc2OTYwODExAjloAZmNdUUPto94Weh_11seuv3_      XSet_AAA6AwAqURQxMjUwNDc2OTYwOTU1CTY-vTycKkMhgRj22XFJ-CpMdnKG.xml
XSet_AAA6AwAqbH8xMjUwNDc2OTYwODExAjloAZmNdUUPto94Weh_11seuv3_.xml  XSet_AAA6AwAqUWoxMjUwNDc2OTYwODUxBM5rT72XVEEeuQvgMU1G47JugiVN
XSet_AAA6AwAqbu8xMjUwNDc2OTYwNzcwAcSLfePaTUlTnoaEkQ5kBOgDBBh2      XSet_AAA6AwAqUWoxMjUwNDc2OTYwODUxBM5rT72XVEEeuQvgMU1G47JugiVN.xml
XSet_AAA6AwAqbu8xMjUwNDc2OTYwNzcwAcSLfePaTUlTnoaEkQ5kBOgDBBh2.xml
$

Here is the contents of one of these XML files. It corresponds to one post in my blog. You will see the set of properties the application created for this particular post towards the end of the file.


<?xml version="1.0" encoding="UTF-8"?>
<xsets xsi:schemaLocation="http://www.snia.org/2007/xam/export/XAMCanonicalXSetDefinition.xsd" xmlns="http://www.snia.org/2007/xam/export" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xop="http://www.w3.org/2004/08/xop/include">
<version>1.0.0</version>
<policies>
   <policy name=".xsystem.management.policy.list.org.snia.refvim.default.mgmt.policy" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="35">
      <string>org.snia.refvim.default.mgmt.policy</string>
   </policy>
</policies>
<xset>
<properties>
   <property name=".xset.hold" type="application/vnd.snia.xam.boolean" binding="false" readOnly="true" length="1">
      <boolean>false</boolean>
   </property>
   <property name=".xset.management.policy" type="application/vnd.snia.xam.string" binding="true" readOnly="true" length="35">
      <string>org.snia.refvim.default.mgmt.policy</string>
   </property>
   <property name=".xset.retention.base.enabled" type="application/vnd.snia.xam.boolean" binding="true" readOnly="true" length="1">
      <boolean>true</boolean>
   </property>
   <property name=".xset.retention.base.starttime" type="application/vnd.snia.xam.datetime" binding="true" readOnly="true" length="29">
      <date>2009-08-16T21:42:40.742-05:00</date>
   </property>
   <property name=".xset.retention.list.base" type="application/vnd.snia.xam.string" binding="true" readOnly="true" length="4">
      <string>base</string>
   </property>
   <property name=".xset.retention.list.event" type="application/vnd.snia.xam.string" binding="true" readOnly="true" length="5">
      <string>event</string>
   </property>
   <property name=".xset.time.access" type="application/vnd.snia.xam.datetime" binding="false" readOnly="true" length="29">
      <date>2009-08-16T21:42:40.742-05:00</date>
   </property>
   <property name=".xset.time.commit" type="application/vnd.snia.xam.datetime" binding="false" readOnly="true" length="29">
      <date>2009-08-16T21:42:40.742-05:00</date>
   </property>
   <property name=".xset.time.creation" type="application/vnd.snia.xam.datetime" binding="true" readOnly="true" length="29">
      <date>2009-08-16T21:42:40.724-05:00</date>
   </property>
   <property name=".xset.time.residency" type="application/vnd.snia.xam.datetime" binding="false" readOnly="true" length="29">
      <date>2009-08-16T21:42:40.742-05:00</date>
   </property>
   <property name=".xset.time.xuid" type="application/vnd.snia.xam.datetime" binding="false" readOnly="true" length="29">
      <date>2009-08-16T21:42:40.742-05:00</date>
   </property>
   <property name=".xset.xuid" type="application/vnd.snia.xam.xuid" binding="true" readOnly="true" length="42">
      <xuid>AAA6AwAqbu8xMjUwNDc2OTYwNzcwAcSLfePaTUlTnoaEkQ5kBOgDBBh2</xuid>
   </property>
   <property name="com.fpmurphy.rss.author" type="application/vnd.snia.xam.string" binding="false" readOnly="false" length="8">
      <string>fpmurphy</string>
   </property>
   <property name="com.fpmurphy.rss.categories" type="application/vnd.snia.xam.string" binding="false" readOnly="false" length="14">
      <string>AWS, Java, XAM</string>
   </property>
   <property name="com.fpmurphy.rss.link" type="application/vnd.snia.xam.string" binding="false" readOnly="false" length="69">
      <string>http://blog.fpmurphy.com/2009/08/amazon-simpledb-typica-tutorial.html</string>
   </property>
   <property name="com.fpmurphy.rss.published_date" type="application/vnd.snia.xam.datetime" binding="false" readOnly="false" length="29">
      <date>2009-08-04T12:35:14.000-05:00</date>
   </property>
   <property name="com.fpmurphy.rss.title" type="application/vnd.snia.xam.string" binding="false" readOnly="false" length="31">
      <string>Amazon SimpleDB Typica Tutorial</string>
   </property>
</properties>
<xstreams>
</xstreams>
</xset>
</xsets>

Well, that is all for now. As usual, I hope this post was useful to you and that you have increased your knowledge of XAM Storage Systems and how to use the XAM Reference VIM.

As the SNIA XAM SDK matures, the Reference VIM is becoming more useful for prototyping a XAM Storage System. Maybe one day, road warriors can use something like the Reference VIM to do useful work while disconnected from their corporate XAM Storage System with the Reference VIM automatically syncing up with the corporate XAM Storage System when a secure network connection becomes available.

Amazon SimpleDB Typica Tutorial

2009-08-04T23:55:00.000-04:00

While experimenting on how to prototype a Cloud XAM Storage System using Amazon Web Services (AWS), I experimented with the Amazon SimpleDB service using Java and the Typica AWS client library. I looked though the Internet for easy examples of how to use this library but found none that I liked - so here is my attempt at a tutorial on using the Typica library with the Amazon SimpleDB service.

To use the Typica library, you should download the latest version of the library. As of the date of this post it is typica.1.6. Install this library wherever you like; just make sure to add the location to your CLASSPATH. In Fedora, the standard location is /usr/share/java. In addition to this library, the following Java libraries (jars) must also be installed if not already installed and in your CLASSPATH.

commons-httpclient

commons-logging

commons-codec

jaxb

To run the examples you need an Amazon Web Services SimpleDB account . You will need a credit card for this. Usage fees apply for using the service but they are relatively low. At present SimpleDB users pay no usage fees on the first 25 Machine Hours, 1 GB of Data Transfer, and 1 GB of Storage consumed every month. In most use cases, approximately 2 million GET or SELECT API requests can be completed per month before incurring any usage fees. Note that you have to specifically sign up for each of the Amazon Web Service products, i.e. Simple Storage Service (S3), Elastic Compute Cloud (EC2), Simple Queue Service (SQS), SimpleDB, etc. However, if you have signed up for one AWS product, adding another product is just a couple of key clicks.

If you cannot sign up for an SimpleDB account, there are a couple of free alternatives which claim to be plug-compatible with SimpleDB. I have not yet tried any of these services and the examples in this post will obviously have to be modified somewhat to use these services. I will discuss this further in the first example below. M/Gateway Developments M/DB is probably the most well known of such services.

Before we go any further, here is a quick review of what SimpleDB is and is not. According to Amazon, SimpleDB is a web service which provides the core database functions of data indexing and querying in the cloud. It is not a relational database; it is more of a metadata store. It is a form of distributed database written in Erlang. It requires no schema and automatically indexes your data. It consists of domains, items and attributes. A domain is a collection of items. Items are little hash tables containing attributes i.e. key-value pairs. As an aside, M/DB supports unlimited attributes per item and unlimited number and size of domains.

SimpleDB supports both SOAP and REST (Representational State Transfer). The Typica library uses REST. SimpleDB REST calls are made using HTTP GET requests. The Action query parameter provides the method called and the URI specifies the target of the call. Additional call parameters are specified as HTTP query parameters. A response message is generated for every request message. The response is an XML document that conforms to a schema. Check out the Getting Started Guide and the Developer Guide for further information,

SimpleDB has some major limitations which you should be aware of. The principal ones are a maximum of 100 domains per account, 10GB maximum size for a domain, 256 attributes per item and a maximum attribute size of 1024 bytes. Queries also have limitations including a maximum of 2500 items returned per query, a 5 second maximum query runtime, and a maximum query response size of 1Mb for certain types of queries. See the SimpleDB Developers Guide for a full list of the limits.

Eventual Consistency is another interesting aspect of SimpleDB. Here is what Amazon have to say about this issue:

Amazon SimpleDB keeps multiple copies of each domain. When data is written or updated [...] and Success is returned, all copies of the data are updated. However, it takes time for the update to propagate to all storage locations. The data will eventually be consistent, but an immediate read might not show the change. Consistency is usually reached within seconds, but a high system load or network partition might increase this time. Repeating a read after a short time should return the updated data.

Whether this is a real issue in practice, I do not know but it sure sounds like weasel words to me!

For our Hello World example, we will just create a SimpleDB domain.


import com.xerox.amazonws.sdb.Domain;
import com.xerox.amazonws.sdb.ListDomainsResult;
import com.xerox.amazonws.sdb.SDBException;
import com.xerox.amazonws.sdb.SimpleDB;

public class SDBexample1 {

   public static void main(String[] args) {
      // NOTE - replace AAA with your access key and SSS with your secret key
      SimpleDB simpleDB = new SimpleDB("AAA","SSS");

      try {
          simpleDB.createDomain("example1");
          System.out.println("created example1 domain");

          simpleDB.createDomain("example2");
          System.out.println("created example2 domain");

          ListDomainsResult list = simpleDB.listDomains();

          Iterator iterator = list.getDomainList().iterator();
          while (iterator.hasNext())
          {
              Domain domain = (Domain) iterator.next();
              System.out.println("List domain: " + domain.getName());
              simpleDB.deleteDomain(domain.getName());
          }

          System.out.println("deleted all domains");
      } catch (SDBException ex) {
          System.err.println("message : " + ex.getMessage());
          System.err.println("requestID : " + ex.getRequestId());
      } catch (Exception e) {
              System.err.println("Error occured: " + e.getMessage());
              e.printStackTrace();
      }
   }
}

Documentation for all the classes and methods in the Typica library can be found online. The application connects to the Amazon SimpleDB service by means of the SimpleDB(accessIdString, secretKeyString) constructor. It then creates a new domain by called the createDomain(nameString) method. Any Typica SimpleDB exceptions are handled by the SDBException() exceptions constructor.

If you plan to use this example and the subsequent examples with the M/DB service, you will need to modify the SimpleDB constructor to supply the server host string and possibly the port number. See the Typica SimpleDB class documentation for further information.

Here is the output. when compiled and run.


$ javac SDBexample.java
$ java SDBexample
created example domain
$

For our next example, we will create two SimpleDB domains, list all our domains, and delete the domains.


import java.util.Iterator;

import com.xerox.amazonws.sdb.Domain;
import com.xerox.amazonws.sdb.ListDomainsResult;
import com.xerox.amazonws.sdb.SDBException;
import com.xerox.amazonws.sdb.SimpleDB;


public class SDBexample1 {

   public static void main(String[] args) {
      // NOTE - replace AAA with your access key and SSS with your secret key
      SimpleDB simpleDB = new SimpleDB("AAA","SSS");

      try {
          simpleDB.createDomain("example1");
          System.out.println("created example1 domain");

          simpleDB.createDomain("example2");
          System.out.println("created example2 domain");

          ListDomainsResult list = simpleDB.listDomains();

          Iterator iterator = list.getDomainList().iterator();
          while (iterator.hasNext())
          {
              Domain domain = (Domain) iterator.next();
              System.out.println("List domain: " + domain.getName());
              simpleDB.deleteDomain(domain.getName());
          }

          System.out.println("deleted all domains");
      } catch (SDBException ex) {
          System.err.println("message : " + ex.getMessage());
          System.err.println("requestID : " + ex.getRequestId());
      } catch (Exception e) {
              System.err.println("Error occured: " + e.getMessage());
              e.printStackTrace();
      }
   }
}

It uses the listDomains() method to return a list of all the current domains, the getDomainList() to set up iteration of the returned list of domains and the deleteDomain() method to delete a domain.

Here the the output from running this example.


created example1 domain
created example2 domain
List domain: example
List domain: example1
List domain: example2
deleted all domains

Note that it listed and deleted the example domain which we created in the previous example.

In the next example we use a properties file aws.properties to store the AWS accessid and secretkey strings. We encrypt communications between the application and AWS by adding a third parameter true to the SimpleDB() constructor. We also retrieve and print out the domain metadata.


import java.io.FileInputStream;
import java.io.IOException;
import java.io.FileNotFoundException;
import java.util.Iterator;
import java.util.Properties;

import com.xerox.amazonws.sdb.Domain;
import com.xerox.amazonws.sdb.DomainMetadataResult;
import com.xerox.amazonws.sdb.ListDomainsResult;
import com.xerox.amazonws.sdb.SDBException;
import com.xerox.amazonws.sdb.SimpleDB;

public class SDBexample3 {
   private static String AWS_PROPERTIES = "aws.properties";
   private static String AWS_ACCESSID = "aws.accessId";
   private static String AWS_SECRETKEY = "aws.secretKey";

   private static String domainName = "test1";
   private static String awsAccessId = null;
   private static String awsSecretKey = null;

   private static Properties props;

   public static void main(String[] args) {
      /* get the accessid and secretkey strings from the properties file*/
      try {
           props = new Properties();
           props.load(new FileInputStream(AWS_PROPERTIES));

           awsAccessId = props.getProperty(AWS_ACCESSID);
           if (awsAccessId == null || awsAccessId.trim().length() == 0)
           {
               System.out.println("Access key found or not set");
               System.exit((int) 1);
           }
           awsSecretKey = props.getProperty(AWS_SECRETKEY);
           if (awsSecretKey == null || awsSecretKey.trim().length() == 0)
           {
               System.out.println("Secret key either not found or not set");
               System.exit((int) 1);
           }
       } catch (FileNotFoundException e) {
          System.err.println("Could not find properties file");
          System.exit((int) 1);
       } catch (IOException e) {
          System.err.println("Could not access properties file");
          System.exit((int) 1);
       }

      /* encrypt communications */
      SimpleDB simpleDB = new SimpleDB(awsAccessId, awsSecretKey, true);
      try {
          simpleDB.createDomain(domainName);
          System.out.println("created " + domainName +  " domain");

          ListDomainsResult list = simpleDB.listDomains();

          Iterator iterator = list.getDomainList().iterator();
          while (iterator.hasNext())
          {
              Domain domain = (Domain) iterator.next();
              DomainMetadataResult metadata = domain.getMetadata();

              System.out.println("\nDomain Metadata for : " + domain.getName());
              System.out.println("     ItemCount: " + metadata.getItemCount());
              System.out.println("     AttributeNameCount: " + metadata.getAttributeNameCount());
              System.out.println("     AttributeValueCount: " + metadata.getAttributeValueCount());
              System.out.println("     ItemNamesSizeBytes: " + metadata.getItemNamesSizeBytes());
              System.out.println("     AttributeNamesSizeBytes: " + metadata.getAttributeNamesSizeBytes());
              System.out.println("     AttributeValuesSizeBytes: " + metadata.getAttributeValuesSizeBytes());
              System.out.println("     Timestamp: " + metadata.getTimestamp() + "\n");
          }

          simpleDB.deleteDomain(domainName);
          System.out.println("deleted " + domainName + " domain");
      } catch (SDBException ex) {
          System.err.println("message : " + ex.getMessage());
          System.err.println("requestID : " + ex.getRequestId());
      } catch (Exception e) {
              System.err.println("Error occured: " + e.getMessage());
              e.printStackTrace();
      }
   }
}

Here the the output from running this example. Note that most of the values are zero because we have not yet added any items or attributes.


created test1 domain

Domain Metadata for : test1
     ItemCount: 0
     AttributeNameCount: 0
     AttributeValueCount: 0
     ItemNamesSizeBytes: 0
     AttributeNamesSizeBytes: 0
     AttributeValuesSizeBytes: 0
     Timestamp: Sat Aug 08 19:18:28 GMT-05:00 2009

deleted test1 domain

The next example is our first example to get (create) an item using the getItem() method and add attributes to that item using the putAttrributes() method. It includes a 2 second delay (sleep) to guard against the eventual consistency issue before attempting to retrieve the domain metadata. I found that a Java exception was sometimes thrown unless I added this delay. The delay I picked was arbitrary; a smaller delay may suffice. It also includes a different way of retrieving the accessid and secretkey strings using getClassLoader().


import java.util.ArrayList;
import java.util.Iterator;
import java.util.Properties;
import java.util.List;

import com.xerox.amazonws.sdb.Item;
import com.xerox.amazonws.sdb.ItemAttribute;
import com.xerox.amazonws.sdb.Domain;
import com.xerox.amazonws.sdb.DomainMetadataResult;
import com.xerox.amazonws.sdb.ListDomainsResult;
import com.xerox.amazonws.sdb.SDBException;
import com.xerox.amazonws.sdb.SimpleDB;

public class SDBexample4 {
   private static String AWS_PROPERTIES = "aws.properties";
   private static String AWS_ACCESSID = "aws.accessId";
   private static String AWS_SECRETKEY = "aws.secretKey";

   private static String domainName = "test4";

   public static void main(String[] args) {

      try {
           /* get the accessid and secretkey strings from the properties file*/
           Properties props = new Properties();
           props.load(SDBexample4.class.getClassLoader().getResourceAsStream(AWS_PROPERTIES));

           String awsAccessId = props.getProperty(AWS_ACCESSID);
           String awsSecretKey = props.getProperty(AWS_SECRETKEY);

           if ((awsAccessId == null || awsAccessId.trim().length() == 0) ||
               (awsSecretKey == null || awsSecretKey.trim().length() == 0))
           {
               System.out.println("Either access key or Secret key not found or not set");
               System.exit((int) 1);
           }
          SimpleDB simpleDB = new SimpleDB(awsAccessId, awsSecretKey, true);

          Domain dom = simpleDB.createDomain(domainName);
          System.out.println("created " + dom.getName() +  " domain");

          // new item (called 'xset1')
          Item i = dom.getItem("xset1");

          List list = new ArrayList();

          list.add(new ItemAttribute("test1", "value1", false));
          list.add(new ItemAttribute("test2", "value2", false));
          list.add(new ItemAttribute("test3", "value3", false));

          // add the 3 attributes to xset1
          i.putAttributes(list);

          // sleep for 2 seconds to handle "eventual consistency'
          Long stoptime = 2000L;
          System.out.print("going to sleep for 2 seconds ");
          try {
              Thread.sleep(stoptime);
          } catch (InterruptedException e) {
               e.printStackTrace();
               System.exit((int) 1);
          }
          System.out.println("Done!");

          // get the domain metadata
          DomainMetadataResult metadata = dom.getMetadata();

          System.out.println("\nDomain Metadata for : " + dom.getName());
          System.out.println("     ItemCount: " + metadata.getItemCount());
          System.out.println("     AttributeNameCount: " + metadata.getAttributeNameCount());
          System.out.println("     AttributeValueCount: " + metadata.getAttributeValueCount());
          System.out.println("     ItemNamesSizeBytes: " + metadata.getItemNamesSizeBytes());
          System.out.println("     AttributeNamesSizeBytes: " + metadata.getAttributeNamesSizeBytes());
          System.out.println("     AttributeValuesSizeBytes: " + metadata.getAttributeValuesSizeBytes());
          System.out.println("     Timestamp: " + metadata.getTimestamp() + "\n");

          simpleDB.deleteDomain(domainName);
          System.out.println("deleted " + domainName + " domain");
      } catch (SDBException ex) {
          System.err.println("message : " + ex.getMessage());
          System.err.println("requestID : " + ex.getRequestId());
      } catch (Exception e) {
              System.err.println("Error occured: " + e.getMessage());
              e.printStackTrace();
      }
   }
}

Here is the output for this example:


created test5 domain
going to sleep for 2 seconds Done!

Domain Metadata for : test4
     ItemCount: 1
     AttributeNameCount: 3
     AttributeValueCount: 3
     ItemNamesSizeBytes: 5
     AttributeNamesSizeBytes: 15
     AttributeValuesSizeBytes: 18
     Timestamp: Sat Aug 08 21:48:07 GMT-05:00 2009

deleted test5 domain

The next example creates two items (xset1 and xset2) and adds 10 attributes to each item. It also outputs the request identifier and box usage for each of the two putAttributes requests. Every request generates a response message and these two items are included as part of this message. Box usage is the measure of machine resources consumed by a request. It does not include bandwidth or storage. It is reported as the portion of a machine hour used to complete a particular request.


import java.util.ArrayList;
import java.util.Iterator;
import java.util.Properties;
import java.util.List;

import com.xerox.amazonws.sdb.Item;
import com.xerox.amazonws.sdb.ItemAttribute;
import com.xerox.amazonws.sdb.Domain;
import com.xerox.amazonws.sdb.DomainMetadataResult;
import com.xerox.amazonws.sdb.ListDomainsResult;
import com.xerox.amazonws.sdb.SDBException;
import com.xerox.amazonws.sdb.SDBResult;
import com.xerox.amazonws.sdb.SimpleDB;

public class SDBexample5 {
   private static String AWS_PROPERTIES = "aws.properties";
   private static String AWS_ACCESSID = "aws.accessId";
   private static String AWS_SECRETKEY = "aws.secretKey";

   private static String domainName = "test5";

   public static void main(String[] args) {
     try {

           /* get the accessid and secretkey strings from the properties file*/
           Properties props = new Properties();
           props.load(SDBexample4.class.getClassLoader().getResourceAsStream(AWS_PROPERTIES));

           String awsAccessId = props.getProperty(AWS_ACCESSID);
           String awsSecretKey = props.getProperty(AWS_SECRETKEY);

           if ((awsAccessId == null || awsAccessId.trim().length() == 0) ||
               (awsSecretKey == null || awsSecretKey.trim().length() == 0))
           {
               System.out.println("Either access key or Secret key not found or not set");
               System.exit((int) 1);
           }

          SimpleDB simpleDB = new SimpleDB(awsAccessId, awsSecretKey, true);

          Domain dom = simpleDB.createDomain(domainName);
          System.out.println("created " + dom.getName() +  " domain");

          Item i = dom.getItem("xset1");

          List list = new ArrayList();

          list.add(new ItemAttribute("test10", "value10", false));
          list.add(new ItemAttribute("test11", "value11", false));
          list.add(new ItemAttribute("test12", "value12", false));
          list.add(new ItemAttribute("test13", "value13", false));
          list.add(new ItemAttribute("test14", "value14", false));
          list.add(new ItemAttribute("test15", "value15", false));
          list.add(new ItemAttribute("test16", "value16", false));
          list.add(new ItemAttribute("test17", "value17", false));
          list.add(new ItemAttribute("test18", "value18", false));
          list.add(new ItemAttribute("test19", "value19", false));

          SDBResult result = i.putAttributes(list);
          System.out.println("Item Identifier: " + i.getIdentifier());
          System.out.println("     Request ID: " + result.getRequestId());
          System.out.println("      Box Usage: " + result.getBoxUsage());
          i = dom.getItem("xset2");
          list = new ArrayList();

          list.add(new ItemAttribute("test20", "value20", false));
          list.add(new ItemAttribute("test21", "value21", false));
          list.add(new ItemAttribute("test22", "value22", false));
          list.add(new ItemAttribute("test23", "value23", false));
          list.add(new ItemAttribute("test24", "value24", false));
          list.add(new ItemAttribute("test25", "value25", false));
          list.add(new ItemAttribute("test26", "value26", false));
          list.add(new ItemAttribute("test27", "value27", false));
          list.add(new ItemAttribute("test28", "value28", false));
          list.add(new ItemAttribute("test29", "value29", false));

          result = i.putAttributes(list);
          System.out.println("Item Identifier: " + i.getIdentifier());
          System.out.println("     Request ID: " + result.getRequestId());
          System.out.println("      Box Usage: " + result.getBoxUsage());

          // sleep for 2 seconds to handle "eventual consistency'
          Long stoptime = 2000L;
          System.out.print("going to sleep for 2 seconds ");
          try {
              Thread.sleep(stoptime);
          } catch (InterruptedException e) {
               e.printStackTrace();
               System.exit((int) 1);
          }
          System.out.println("Done!");
 
          ListDomainsResult list1 = simpleDB.listDomains();
          Iterator iterator = list1.getDomainList().iterator();
          while (iterator.hasNext())
          {
              Domain domain = (Domain) iterator.next();
              DomainMetadataResult metadata = domain.getMetadata();

              System.out.println("\nDomain Metadata for : " + domain.getName());
              System.out.println("     ItemCount: " + metadata.getItemCount());
              System.out.println("     AttributeNameCount: " + metadata.getAttributeNameCount());
              System.out.println("     AttributeValueCount: " + metadata.getAttributeValueCount());
              System.out.println("     ItemNamesSizeBytes: " + metadata.getItemNamesSizeBytes());
              System.out.println("     AttributeNamesSizeBytes: " + metadata.getAttributeNamesSizeBytes());
              System.out.println("     AttributeValuesSizeBytes: " + metadata.getAttributeValuesSizeBytes());
              System.out.println("     Timestamp: " + metadata.getTimestamp() + "\n");
          }

      } catch (SDBException ex) {
          System.err.println("message : " + ex.getMessage());
          System.err.println("requestID : " + ex.getRequestId());
      } catch (Exception e) {
              System.err.println("Error occured: " + e.getMessage());
              e.printStackTrace();
      }
   }
}

Here is the output for this example:


created test5 domain
Item Identifier: xset1
     Request ID: 34e6df86-5f2b-4bcf-312d-22b595a87fe7
      Box Usage: 0.0000221907
Item Identifier: xset2
     Request ID: dd66f222-de70-3d06-8ce5-7842e5d22d1d
      Box Usage: 0.0000221907
going to sleep for 4 seconds Done!

Domain Metadata for : test5
     ItemCount: 2
     AttributeNameCount: 20
     AttributeValueCount: 20
     ItemNamesSizeBytes: 10
     AttributeNamesSizeBytes: 120
     AttributeValuesSizeBytes: 140
     Timestamp: Sun Aug 09 08:44:19 GMT-05:00 2009

For our final example, we will retrieve and print the attributes stored in the test5 domain xset1 item by the previous example.


import java.util.ArrayList;
import java.util.Iterator;
import java.util.Properties;
import java.util.List;

import com.xerox.amazonws.sdb.Item;
import com.xerox.amazonws.sdb.ItemAttribute;
import com.xerox.amazonws.sdb.Domain;
import com.xerox.amazonws.sdb.DomainMetadataResult;
import com.xerox.amazonws.sdb.ListDomainsResult;
import com.xerox.amazonws.sdb.SDBException;
import com.xerox.amazonws.sdb.SDBResult;
import com.xerox.amazonws.sdb.SimpleDB;

public class SDBexample6 {
   private static String AWS_PROPERTIES = "aws.properties";
   private static String AWS_ACCESSID = "aws.accessId";
   private static String AWS_SECRETKEY = "aws.secretKey";

   private static String domainName = "test5";

   public static void main(String[] args) {

      try {

           /* get the accessid and secretkey strings from the properties file*/
           Properties props = new Properties();
           props.load(SDBexample4.class.getClassLoader().getResourceAsStream(AWS_PROPERTIES));

           String awsAccessId = props.getProperty(AWS_ACCESSID);
           String awsSecretKey = props.getProperty(AWS_SECRETKEY);
          if ((awsAccessId == null || awsAccessId.trim().length() == 0) ||
               (awsSecretKey == null || awsSecretKey.trim().length() == 0))
           {
               System.out.println("Either access key or Secret key not found or not set");
               System.exit((int) 1);
           }

          SimpleDB simpleDB = new SimpleDB(awsAccessId, awsSecretKey, true);

          Domain dom = simpleDB.getDomain(domainName);
          System.out.println("opened " + dom.getName() +  " domain");

          Item i = dom.getItem("xset1");

          List attrs = i.getAttributes();
          for (ItemAttribute attr : attrs) {
              System.out.println(attr.getName() + " = " + attr.getValue());
          }

      } catch (SDBException ex) {
          System.err.println("message : " + ex.getMessage());
          System.err.println("requestID : " + ex.getRequestId());
      } catch (Exception e) {
              System.err.println("Error occured: " + e.getMessage());
              e.printStackTrace();
      }
   }
}

Here is the output from this example:


opened test5 domain
test10 = value10
test12 = value12
test11 = value11
test14 = value14
test13 = value13
test16 = value16
test15 = value15
test18 = value18
test17 = value17
test19 = value19

Well that is all of the examples for now. If you compile, run and play with these examples, you should end up with a good understanding of how to use the Typica library to access and manipulate Amazon SimpleDB. I plan to discuss how to query Amazon SimpleDB using this library in a future post.

Can the Typica library be used as part of a AWS Cloud VIM implementation? Absolutely. Can Amazon SimpleDB be used to implement a Cloud XAM Storage System? Not by itself. However if Amazon S3 is used to provide persistence for XStreams and Amazon SimpleDB is used to provide persistence for XSet metadata, the answer is yes to a basic prototype of a Cloud XAM Storage System but no for a production quality Cloud XAM Storage System due the eventual consistency issue as well as the current limitations in the number of attributes, size of domains, no datatyping and seconds a query may run.

XAM Query Language

2009-07-19T23:59:00.000-04:00

As well as providing vendor-independent means of creating, retrieving, modifying and deleting XSets, the SNIA XAM v1.0 specification also defines a query language (XAM QL), based on a subset of the SQL language, for selecting and retrieving the XUIDs of XSets based on content-defined criteria.

The set of reserved words for this query language is quite small: select, where, and, or, not, like, exists, binding, readonly, typeof, length, date, TRUE, FALSE, before, after, contains, and within. By design, XAM queries look like an SQL select statement. The query language is case insensitive and uses the ASCII character set.

Here is an example of a simple XAM query:

select ".xset.xuid" where "com.example.name" = ’Tuckers Plantation'

Version 1.0 of the XAM specification defines two levels of query language support, i.e. Level 1 and Level 2. Level 1 defines queries on properties and field attributes in XSets and is mandatory. Any XSet property value that is accessible to an application program via the XAM library can be queried. Level 2 extends Level 1 to support queries on XStreams and is optional. Both levels of query are accessed through a single, defined job type that all XAM providers must support. Since no vendor that I am aware of has actually implemented level 2 queries, the remainder of this post focuses on level 1 queries.

A XAM query statement consists of a mandatory select clause followed by an optional where clause. For XAM v1.0 the only valid select clause is select '.xset.xuid'. This specifies that the application is requesting a list of XUID values. For example

select ".xset.xuid "

will return a list of every XSet that is readable at the time of the query.

The where clause is used to specify a subset of XSets to be matched. For level 1 queries it is restricted to comparisons between XSet properties and literal values and/or field attributes and literal values.

select ".xset.xuid" where ".xam.time.xuid" > date(’2009-02-01T00:00:00.0’)

will return the list of all Xsets which were created on or after February 1st 2009.

The following table shows which field and literal types can be validly compared.
[table id=14/]
The XAM library validates that strings and strings liberals are conforming UTF-8 strings. Non-conforming UTF-8 string literals generate a XAM non-fatal query syntax error. Issues such as single versus multiple glyph characters and non printable characters are unspecified and are implementation and application dependent. String comparisons are case sensitive and comparison operators operate on a byte-by-byte basis. For relational operators the relationships are defined by the byte values. All data normalization is the responsibility of the application.

The supported escape sequences are the following. Any other escape sequence will generate an non-fatal error.
[table id=15/]
String literals must be quoted with single quotes. String literals which contain one or more single quote characters must escape each single quote character using a backslash. For example, the string literal of ’This is a string literal’ would be represented in a query as

select ".xset.xuid" where "com.example.string-property" = ’This is a string literal’

To use a single quote in a string, guard it with a backslash

select ".xset.xuid" where "com.example.claimed.ownership" = ’Tucker\’s’

All instances of field names in a XAM query string must be quoted with double quotes. If a field name contains double quote characters, each double quote character must be escaped using a backslash. For example, the field name for the xam_boolean property com.example.”qstring” should be represented in a query as

select ".xam.xuid" where "com.example.\"qstring\"" = TRUE

If a field name contains a backslash character, the backslash itself must be escaped with another backslash. For example, the field name for the xam_double property com.example.file\ratio should be represented in a query as

select ".xam.xuid" where "com.example.file\\ratio" = 100.1

The query language accepts date-time and XUID-valued literals using the selector functions date() and xuid() respectively. The date() function takes a properly formed date-time value, specified as a literal string which is consistent with the XAM date-time specification.

select ".xset.xuid" where ".xam.time.xuid" = date(’2009-06-01T00:00:00.0’)

The xuid() function expects an XUID which is in the form of a base64 encoded string literal. An improperly formed string literal generates a non-fatal error during query parsing.

Field attributes are accessed using the exists, typeof, readonly, binding, and length field attribute accessor functions. The exists() function tests for the existence of an named field attribute (property) in an XSet. It evaluates to TRUE if an XSet contains the named field otherwise it evaluates FALSE.

select ".xset.xuid" where exists("com.example.name")

The typeof() function returns the MIME type of a named field in an XSet which is a string-valued property.

select ".xset.xuid" where typeof("com.example.data") = ’text/plain’
select ".xset.xuid" where typeof("com.example.data") like ’text%’

This function can be used whenever an application could use a field reference to a string-valued property. Note that comparisons with any non-string literal value generate a non-fatal error during the parsing of the query.

The readonly() function evaluates to TRUE when a field in an XSet is marked as readonly.

select ".xset.xuid" where readonly("com.example.flag")
select ".xset.xuid" where not readonly("com.example.name")

The binding() function evaluates to TRUE when a field in an XSet is marked as binding.

select ".xset.xuid" where binding("com.example.case_id")
select ".xset.xuid" where not binding("com.example.subject")

The length() function returns the length, in bytes, of a named field.

select ".xset.xuid" where length("com.example.data") > 1024

When used on property fields this function returns the length as defined for stypes. Note that this function should not be used to compare the number of characters in a string as this comparison depends on the character encoding being used.

Turning now to logical operators. Subclauses within the where clause may be combined and modified by using the logical operators and, or and not. These operators are similar to their SQL counterparts.

The and operator requires both subclauses to evaluate to TRUE before including an XSet in the results. For example

select ".xset.xuid" where typeof("com.example.stream") = ’image/gif’ and length("com.example.stream") > 4096

selects only those XSets containing GIF images whose size is greater than 4096 bytes. The or operator evaluates to TRUE if either subclause evaluates to TRUE. For example

select ".xset.xuid" where typeof("com.example.stream") = ’image/jpeg’ or typeof("com.example.stream") = ’image/gif’

selects only those XSets containing a named stream of image type JPEG or GIF. The not operator negates a boolean expression. For example

select ".xset.xuid" where not binding("com.example.property")

selects all XSets with the nonbinding property com.example.property.

As in SQL certain operators take precedence over other operators in queries. XAM QL operator precedence is as follows:
[table id=16/]
Operators of the same precedence are evaluated left to right within a query. However operator precedence may be overridden using parentheses as shown by the following example.

select ".xset.xuid" where not "com.example.bool-prop" and  "com.example.int-prop" = 64
select ".xset.xuid" where not ("com.example.bool-prop" and "com.example.int-prop" = 64)

In the first example, the not operator only applies to the com.example.bool-prop property. In the second example, the not operator applies to both com.example.bool-prop and com.example.int-prop = 42.

Now that we are somewhat familiar with XAM QL syntax, the next question to be answered is how do we tell a XAM Storage System to execute a query and return the results to us. Well it turns out that a query is executed as a particular type of XAM job called a query job. The input to a query job is an XSet which you must create. This XSet must contain two items - the field org.snia.xam.job.command which is always set to 'xam.job.query' and a XStream xam.job.query.command which is a UTF-8 text stream (MIME type 'text/plain; charset=utf-8’) containing the actual query expression string.

A query job may optionally be committed before or during the execution of the job, but this commit is not required. The ability to commit a running job is optional and implementation dependant. An application can determine if job commit is supported by checking the XSystem boolean property .xsystem.job.commit.supported. Failure to commit a query job before closing the XSet means the results of the query job are not persistently stored.

A query job is executed by invoking submitJob. If either of the two required items in the query job XSet is not correctly filled in, the standard job error field .xam.job.error is added to the query job XSet and it's value is set to either org.snia.xam::not_a_job or org.snia.xam::unspecified_command and the query job is aborted. As shown in the sample application below, the state of a query job may be determined by examining the value of .xam.job.status.

When a query job is successful, the results are stored in the query job XSet in the form of a new XStream named xam.job.query.results. This XStream has a MIME type of application/vnd.snia.xam.query.xuid_list and contains the XUIDs for the set of XSets resulting from the evaluation of the query. Each XUID is stored in a binary format as part of an 80-byte record. If an XUID is shorter than 80 bytes, the record is zero padded to 80 bytes.

A number of other properties relating to the results XStream are also set up. The xam_int property xam.job.query.results.count contains the current count of the XUID records in the results XStream. This property is updated as results are entered into the results XStream during query processing. An application can use this to provide status information to users as the query job executes. The xam_string property xam.job.query.level indicates the query level which matches the results. Its value is either org.snia.xam.job.query.level.1 or org.snia.xam.job.query.level.2.

It is important to remember that a query job operates within the roles and permissions granted to the connection. That means the results XStream only includes those XSets that are visible and accessible to an application, at least from a read perspective, according to the role under which the query job is executed.

The query job may set the following error codes in .xam.job.error:
[table id=17/]
When a query job is halted for whatever reason, the standard job health and status fields, i.e .xam.job.errorhealth and .xam.job.status, are set to appropriate values and the XAM Storage System may place zero or more additional XUIDs in the results XStream. Thus an application should always see complete XUID values in the results XStream. Resuming a halted query job is not supported.

Note that, unlike a traditional RDBMS, there is no locking of an XAM Storage System during the execution of a query job. Thus a query result is not an instantaneous snapshot of a XAM Storage System. The contents of a XAM Storage System may or may not change while a query job is executed. The general rule is that any XSet that has been stored before the start of the query job is included in the results XStream if it meets the query criteria, and any XSet that has been stored after the query job has completed is not be included in the results XStream. XSets that are stored during the query job execution may or may not be included. The same criteria applies to the results XStream. It may include XUIDs of XSets that are no longer in the XAM Storage System and XUIDs of XSets that no longer meet the search criteria due to a change in the XSet after the particular XSet was queried.

Here is a simple Java application that queries an XAM Storage System to determine it's level of support for XAM queries.


import org.snia.xam.XAMException;
import org.snia.xam.XAMLibrary;
import org.snia.xam.XSystem;
import org.snia.xam.toolkit.XAMXUID;
import org.snia.xam.util.XAMLibraryFactory;

public class CheckQuerySupport
{
    private static XAMLibrary xamLib;
    private static XSystem xSystem;

    public static void main(String[] args)
    {   
        String xri = "snia-xam://centera_vim!XXX.XXXX.XXX.XXX?/home/fpm/xam/xamconnect.pea";

        long exitCode = 0;
        boolean isSupported = false;

        try     
        {       
            xamLib = XAMLibraryFactory.newXAMLibrary();
            System.out.println("Connecting to XSystem: " + xri + "\n");

            xSystem = xamLib.connect(xri);

            // support job commit?
            isSupported = xSystem.getBoolean(XSystem.XAM_XSYSTEM_JOB_COMMIT_SUPPORTED);
            System.out.println("Support job commit? " + (isSupported ? "Yes" : "No"));

            // support continuance of job?
            isSupported = xSystem.getBoolean(XSystem.XAM_XSYSTEM_JOB_QUERY_CONTINUANCE_SUPPORTED);
            System.out.println("Support job continuance? " + (isSupported ? "Yes" : "No"));

            // support level 1 query?
            isSupported = xSystem.getBoolean(XSystem.XAM_XSYSTEM_JOB_QUERY_LEVEL1_SUPPORTED);
            System.out.println("Support level 1 query? " + (isSupported ? "Yes" : "No"));

            // support level 2 query?
            isSupported = xSystem.getBoolean(XSystem.XAM_XSYSTEM_JOB_QUERY_LEVEL2_SUPPORTED);
            System.out.println("Support level 2 query? " + (isSupported ? "Yes" : "No"));

            xSystem.close();
            System.out.println("\nClosed connection to XSystem");

        } catch (XAMException xe) {
            exitCode = xe.getStatusCode();
            System.err.println("XAM Error occured: " + xe.getMessage());
        } catch (IllegalArgumentException e) {
            System.out.println(e.getMessage());
            e.printStackTrace();
            exitCode = 1;
        }

        System.exit((int) exitCode);
    }
}

Here is the output from this application when connected to an EMC Centera XAM Storage System


Connecting to XSystem: snia-xam://centera_vim!XXX.XXX.XXX.XXX?/home/fpm/xam/xamconnect.pea

Support job commit? No
Support job continuance? No
Support level 1 query? Yes
Support level 2 query? No

Closed connection to XSystem

Here is a longer Java application that implements the three sample XSets and the sample queries which are discussed in section 10.8 of the XAM Architecture V1.0 document.


import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import org.snia.xam.XAMException;
import org.snia.xam.XAMLibrary;
import org.snia.xam.XSet;
import org.snia.xam.XStream;
import org.snia.xam.XSystem;
import org.snia.xam.XUID;
import org.snia.xam.toolkit.XAMXUID;
import org.snia.xam.util.XAMLibraryFactory;

public class QueryExample
{
    private static XAMLibrary xamLib;
    private static XSystem xSystem;

    private static void CreateXset(int parm1, String parm2, double parm3)
             throws XAMException
    {
        System.out.print("Create XSet 1 ....  ");

        XSet xSet = xSystem.createXSet(XSet.MODE_UNRESTRICTED);
        xSet.createProperty("com.example.rhc", false, true);
        xSet.createProperty("com,example.foo", false, parm1);
        xSet.createProperty("com.example.bar", false, parm2);
        xSet.createProperty("com.example.num", false, parm3);

        XUID xuID = xSet.commit();
        System.out.println("XUID: " + xuID.toString());

        xSet.close();
    }

   private static void CreateXset(int parm1, int parm2, int parm3)
             throws XAMException
    {
        System.out.print("Create XSet 2 ....  ");

        XSet xSet = xSystem.createXSet(XSet.MODE_UNRESTRICTED);
        xSet.createProperty("com.example.rhc", false, true);
        xSet.createProperty("com,example.foo", false, parm1);
        xSet.createProperty("com.example.bar", false, parm2);
        xSet.createProperty("com.example.num", false, parm3);

        XUID xuID = xSet.commit();
        System.out.println("XUID: " + xuID.toString());

        xSet.close();
    }

    private static void CreateXset(int parm1, int parm2)
             throws XAMException
    {
        System.out.print("Create XSet 3 ....  ");

        XSet xSet = xSystem.createXSet(XSet.MODE_UNRESTRICTED);
        xSet.createProperty("com.example.rhc", false, true);
        xSet.createProperty("com,example.foo", false, parm1);
        xSet.createProperty("com.example.num", false, parm2);

        XUID xuID = xSet.commit();
        System.out.println("XUID: " + xuID.toString());

        xSet.close();
    }

    private static void QueryXsystem(String queryString)
            throws XAMException
    {
        boolean finished = false;
        final int XAM_MAX_XUID = 80;
        int resultCount = 0;
        String status;

        System.out.println("Create query XSet. Query string: " + queryString);

        XSet query = xSystem.createXSet(XSet.MODE_UNRESTRICTED);
        query.createProperty(XSet.XAM_JOB_COMMAND, true, XSet.XAM_JOB_QUERY);

        byte[] buffer = queryString.getBytes();
        XStream queryStream = query.createXStream(XSet.XAM_JOB_QUERY_COMMAND, true, XAMLibrary.TEXT_PLAIN_MIME_TYPE);
        queryStream.write(buffer);
        queryStream.close();

        System.out.println("Submit query job ....");
        query.submitJob();

        System.out.println("Wait for query job to finish ....");

        while (!finished)
        {
            // check  status of query job
            status = query.getString(XSet.XAM_JOB_ERRORHEALTH);
            if (status.equals(XSet.XAM_JOB_ERRORHEALTH_ERROR))
            {
                System.out.println("ERROR: Errorhealth - " + query.getString(XSet.XAM_JOB_ERROR));
                query.haltJob();
                System.out.println("ERROR: Job halted");
                break;
            }

            status = query.getString(XSet.XAM_JOB_STATUS);
            resultCount = (int) query.getLong(XSet.XAM_JOB_QUERY_RESULTS_COUNT);

            // uncomment if you want continuous job status 
            // System.out.println("Job Status: " + status + "  Result count: " + resultCount);

            // exit loop if job complete 
            if (status.equals(XSet.XAM_JOB_STATUS_COMPLETE))
            {
                finished = true;
            }
       }

        System.out.println("Query job completed ....");
        System.out.println("Result set final count is: " + resultCount);

        if (resultCount > 0)
        {
             // open query job result stream
             XStream results = query.openXStream(XSet.XAM_JOB_QUERY_RESULTS, XStream.MODE_READ_ONLY);

             byte rawXUID[] = new byte[XAM_MAX_XUID];
             long bytesRead = 0;

             // output XUIDs in result set
             while( (bytesRead = results.read(rawXUID, XAM_MAX_XUID)) >= 0 )
             {
                 XAMXUID resultXUID = new XAMXUID(rawXUID);
                 System.out.println("XUID: " + resultXUID.toString());
             }

             results.close();
        }

        query.close();
    }

    public static void main(String[] args)
    {
        String xri = "snia-xam://centera_vim!XXX.XXX.XXX.XXX?/home/fpm/xam/xamconnect.pea";
        long exitCode = 0;

        try
        {
            xamLib = XAMLibraryFactory.newXAMLibrary();

            // uncomment next 2 lines if you want extensive XAM Library logging
            // xamLib.setProperty(XAMLibrary.XAM_LOG_VERBOSITY, 1);
            // xamLib.setProperty(XAMLibrary.XAM_LOG_LEVEL, 5);

            System.out.println("Connecting to XSystem " +  xri + "\n");

            xSystem = xamLib.connect(xri);
            xSystem.setProperty(".xsystem.log.verbosity", 1);
            xSystem.setProperty(".xsystem.log.level", 5);

            CreateXset(1, "string", 123.55);
            CreateXset(77, 42, 100);
            CreateXset(6, 200);

            xSystem.close();

            System.out.println("\nClose/reopen XSystem\n");

            xSystem = xamLib.connect(xri);

            // uncomment one or more of the following queries
            // QueryXsystem("select \".xset.xuid\"");
            QueryXsystem("select \".xset.xuid\" where (\"com.example.foo\" > 0) and (\"com.example.foo\" < 50)");
            // QueryXsystem("select \".xset.xuid\" where (\"com.example.bar\" > 0) and (\"com.example.bar\" < 100)");
            // QueryXsystem("select \".xset.xuid\" where exists (\"com.example.bar\")");
            // QueryXsystem("select \".xset.xuid\" where \"com.example.bar\" like '%ing%'");
            // QueryXsystem("select \".xset.xuid\" where \"com.example.num\" >= 124");
            // QueryXsystem("select \".xset.xuid\" where \"com.example.num\" >= 124.6 ");
            // QueryXsystem("select \".xset.xuid\" where (\"com.example.num\" >= 123) and typeof(\"com.example.num\") = 'application/vnd.snia.xam.int'");

            xSystem.close();
            System.out.println("\nClosed connection to XSystem");

        } catch (XAMException xe) {
            exitCode = xe.getStatusCode();
            System.err.println("XAM Error occured: " + xe.getMessage() + "("
                    + exitCode + ")");
        } catch (IllegalArgumentException e) {
            System.out.println(e.getMessage());
            e.printStackTrace();
            exitCode = -1;
        }

        System.exit((int) exitCode);
    }
}

Note that a results XStream produced by a query job may be consumed by an application before the XAM Storage System has finished storing all of the XUIDs in the results XStream if XStream.asyncRead is used instead of XStream.read.

Well I think I have provided enough information and examples to enable you to go play with the XAM query language by yourself. It you need help the I am sure the XAM Developer Group on Google Groups will be more than happy to help you.

In parting I would point out that since XAM Storage System providers are currently only required to implement support for queries over the content metadata, I suspect that support for actual full content-based searches using XAM may not occur for some time.

Linux HPET Support

2009-07-06T00:02:00.000-04:00

IA-PC HPET (High Precision Event Timer) is a specification which was jointly developed by Intel and Microsoft in the early part of this decade.. The latest version is dated October 2004. It's stated purpose is to

initially supplement and eventually replace the legacy 8254 Programmable Interval Timer and the Real Time Clock Periodic Interrupt generation functions that are currently used as the ‘de-facto’ timer hardware for IA-PCs.

The HPET architecture defines a set of timers that can be used by the operating system. A timer block is a combination of a single counter and up to 32 comparators and match registers. The comparator compares the contents of the match register against the value of a free running monotonic up-counter. When the output of the up-counter equals the value in the match register an interrupt is generated. Each of the comparators can output an interrupt. A maximum of 8 timer blocks are supported for a total of 256 timers. Each timer block can have different clocking attributes. Specific implementations may include only a subset of these timers. A minimum of three timers is required.

The specification contains the following block diagram of the HPET architecture.

Some of the timers may be enabled to generate a periodic interrupt. If a timer is set to be periodic, its period is added to the match register each time a match occurs, thus computing the next time for this timer to generate an interrupt.. An up-counter is usually 64 bits wide but 32-bit implementations are permitted by the specification and 64-bit up-counters can also be driven in 32-bit mode. Up-counters run at a minimum of 10 MHz. which is much faster than the older RTC (Real Time Clock) and can thus produce periodic interrupts at a much higher resolution. The registers associated with these timers are mapped to memory space.

The BIOS uses ACPI ( Advanced Configuration and Power Interface) functionality to inform the operating system of the location of the HPET memory-mapped register space. Here is an example of a disassembled ACPI HPET table from an Intel DX48BT2 (AKA BoneTrail) motherboard.


$ cat /sys/firmware/acpi/tables/HPET > /var/tmp/hpet.out
$ iasl -d /var/tmp/hpet.out
$ cat /var/tmp/hpet.dsl
/*
 * Intel ACPI Component Architecture
 * AML Disassembler version 20090123
 *
 * Disassembly of /var/tmp/hpet.out, Sun Jul  5 19:34:47 2009
 *
 * ACPI Data Table [HPET]
 *
 * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
 */

[000h 000  4]                    Signature : "HPET"    /* High Precision Event Timer table */
[004h 004  4]                 Table Length : 00000038
[008h 008  1]                     Revision : 01
[009h 009  1]                     Checksum : CE
[00Ah 010  6]                       Oem ID : "INTEL "
[010h 016  8]                 Oem Table ID : "DX48BT2 "
[018h 024  4]                 Oem Revision : 0000076E
[01Ch 028  4]              Asl Compiler ID : "MSFT"
[020h 032  4]        Asl Compiler Revision : 01000013

[024h 036  4]            Hardware Block ID : 8086A301

[028h 040 12]         Timer Block Register : 
[028h 040  1]                     Space ID : 00 (SystemMemory)
[029h 041  1]                    Bit Width : 00
[02Ah 042  1]                   Bit Offset : 00
[02Bh 043  1]                 Access Width : 00
[02Ch 044  8]                      Address : 00000000FED00000

[034h 052  1]              Sequence Number : 00
[035h 053  2]          Minimum Clock Ticks : 0001
[037h 055  1]        Flags (decoded below) : 00
                              Page Protect : 0
                           4K Page Protect : 0
                          64K Page Protect : 0
Raw Table Data

  0000: 48 50 45 54 38 00 00 00 01 CE 49 4E 54 45 4C 20  HPET8.....INTEL
  0010: 44 58 34 38 42 54 32 20 6E 07 00 00 4D 53 46 54  DX48BT2 n...MSFT
  0020: 13 00 00 01 01 A3 86 80 00 00 00 00 00 00 D0 FE  ................
  0030: 00 00 00 00 00 01 00 00                          ........
$

See page 30 of the HPET v1.0a specification for a detailed breakdown of the individual bits in the Event Time Block (called Hardware Block by the AML disassember). Note that only one Event Timer Block need be described in the HPET table in order to bootstrap an operating system. This is the case here. For non-legacy platforms, the Event Timer Block described in the HPET is the one that provides functionality to replace the 8254/RTC Periodic Interrupt Logic.

Other Event Time Blocks are described in the ACPI namespace. Here is the relevant section from the disassembled ACPI DSDT table.


            Device (HPET)
            {
                Name (_HID, EisaId ("PNP0103"))
                Name (_CRS, ResourceTemplate ()
                {
                    Memory32Fixed (ReadOnly,
                        0xFED00000,         // Address Base
                        0x00004000,         // Address Length
                        )
                })
                Method (_STA, 0, NotSerialized)
                {
                    If (HPEE)
                    {
                        Return (0x0F)
                    }
                    Else
                    {
                        Return (Zero)
                    }
                }
            }

Note the assigned PNPID (PNP0103) for the HPET. Because no _UID is specified it means that there are no other HPET timer blocks.

Here is a list of the HPET-related messages outputted when this particular motherboard is booted up under Fedora 11.


$ dmesg | grep -i HPET
ACPI: HPET CFBF2000, 0038 (r1 INTEL  DX48BT2       76E MSFT  1000013)
ACPI: HPET id: 0x8086a301 base: 0xfed00000
hpet clockevent registered
HPET: 4 timers in total, 0 timers will be used for per-cpu timer
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
hpet0: 4 comparators, 64-bit 14.318180 MHz counter
rtc0: alarms up to one month, 114 bytes nvram, hpet irqs
$

The first line is outputted when the ACPI HPET table is read. The second line is outputted when the ACPI HPET table is mapped into memory by .../arch/x86/kernel/acpi/boot.c. The next line is outputted when the HPET legacy interrupts are started and HPET is registered as the global clock. The following line is outputted when the kernel checks to ensure that at least one timer is reserved for userspace (/dev/hpet.) The next two lines of output comes from the HPET device driver (.../drivers/char/hpet.c.) It shows that 2 timers have allocated interrupts and two do not..

Here is the relevant part of the output from /proc/time_list as it relates to HPET:


Tick Device: mode:     1
Broadcast device
Clock Event Device: hpet
 max_delta_ns:   149983005959
 min_delta_ns:   5000
 mult:           61496114
 shift:          32
 mode:           3
 next_event:     9223372036854775807 nsecs
 set_next_event: hpet_legacy_next_event
 set_mode:       hpet_legacy_set_mode
 event_handler:  tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000000
tick_broadcast_oneshot_mask: 00000000

Here is the output from /proc/sys/dev/hpet and /proc/driver/rtc:


$ cat /proc/sys/dev/hpet/max-user-freq
64
$ cat /proc/driver/rtc
rtc_time : 06:34:31
rtc_date : 2009-07-06
alrm_time : **:24:40
alrm_date : ****-**-**
alarm_IRQ : no
alrm_pending : no
24hr  : yes
periodic_IRQ : no
update_IRQ : no
HPET_emulated : yes
DST_enable : no
periodic_freq : 1024
batt_status : okay

The HPET driver (/dev/hpet) has a similar API to the Real Time Clock driver. It is a character device which can support any number of HPET devices. The kernel API has three interfaces exported from the driver:


hpet_register( struct hpet_task *tp, int periodic )
hpet_unregister( struct hpet_task *tp )
hpet_control( struct hpet_task *tp, unsigned int cmd, unsigned long arg )

The userspace interface to HPET is defined in the header /usr/include/linux/hpet.h. The current set of supported operations is:


#define HPET_IE_ON      _IO('h', 0x01)                        /* interrupt on */
#define HPET_IE_OFF     _IO('h', 0x02)                        /* interrupt off */
#define HPET_INFO       _IOR('h', 0x03, struct hpet_info)     /* get information */
#define HPET_EPI        _IO('h', 0x04)                        /* enable periodic */
#define HPET_DPI        _IO('h', 0x05)                        /* disable periodic */
#define HPET_IRQFREQ    _IOW('h', 0x6, unsigned long)         /* set frequency */

The following example shows how to use the published interface to access a HPET and call a simple periodic signal handler hpet_alarm between 2 and 99 times a second.

#include <stdio.h>
#include <stdlib.h;>
#include <fcntl.h>
#include <time.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <signal.h>
#include <fcntl.h>
#include <sys/time.h>
#include <linux/hpet.h>
#include <stdint.h>
#include <sys/ioctl.h>
#include <signal.h>

static uint16_t hpet_sigio_count;
static uint64_t secs;

static void
hpet_alarm(int val)
{
   struct timespec t;
   clock_gettime(CLOCK_REALTIME, &t);

   if (!secs)  secs = t.tv_sec;

   fprintf(stderr, "hpet_alarm called. iteration: %2d  secs: %ld  nsecs: %ld \n",
                    hpet_sigio_count, (t.tv_sec - secs) , t.tv_sec * 100000 + t.tv_nsec );

   hpet_sigio_count++;
}

int
main(int argc, const char **argv)
{
    struct sigaction old, new;
    struct hpet_info info;
    int              frequency;
    int              iterations;
    int              retval = 0;
    int              fd;
    int              r, i, value;

    if (argc != 3) {
        fprintf(stderr, "Usage: %s frequency(1-64) iterations(10-99)\n", argv[0]);
        return -1;
    }

    frequency = atoi(argv[1]);
    iterations = atoi(argv[2]);

    if (frequency > 64 || frequency < 1 ) {
        fprintf(stderr, "ERROR: Invalid value for frequency\n");
        return -1;
    }

    if (iterations < 10 || iterations > 99 ) {
        fprintf(stderr, "ERROR: Invalid value for iterations\n");
        return -1;
    }

    hpet_sigio_count = 0;

    sigemptyset(&new.sa_mask);
    new.sa_flags = 0;
    new.sa_handler = hpet_alarm;

    sigaction(SIGIO, NULL, &old);
    sigaction(SIGIO, &new, NULL);

    fd = open("/dev/hpet", O_RDONLY);
    if (fd < 0) {
        fprintf(stderr, "ERROR: Failed to open /dev/hpet\n");
        return -1;
    }

    if ((fcntl(fd, F_SETOWN, getpid()) == 1) ||
        ((value = fcntl(fd, F_GETFL)) == 1) ||
        (fcntl(fd, F_SETFL, value | O_ASYNC) == 1)) {
        fprintf(stderr, "ERROR: fcntl failed\n");
        retval = 1;
        goto fail;
    }

    if (ioctl(fd, HPET_IRQFREQ, frequency) < 0) {
        fprintf(stderr, "ERROR: Could not set /dev/hpet to have a %2dHz timer\n", frequency);
        retval = 2;
        goto fail;
    }

    if (ioctl(fd, HPET_INFO, &info) < 0) {
        fprintf(stderr, "ERROR: failed to get info\n");
        retval = 3;
        goto fail;
    }

    fprintf(stdout, "\nhi_ireqfreq: 0x%lx  hi_flags: %0x%lx  hi_hpet: 0x%x  hi_timer: 0x%x\n\n",
            info.hi_ireqfreq,  info.hi_flags, info.hi_hpet, info.hi_timer);

    r = ioctl(fd, HPET_EPI, 0);
    if (info.hi_flags && (r < 0)) {
        fprintf(stderr, "ERROR:  HPET_EPI failed\n");
        retval = 4;
        goto fail;
    }

    if (ioctl(fd, HPET_IE_ON, 0) < 0) {
        fprintf(stderr, "ERROR: HPET_IE_ON failed\n");
        retval = 5;
        goto fail;
    }

    /* wait for specified number of signal interrupts */
    for (i = 0; i < iterations; i++) {
        (void) pause();
    }

    if (ioctl(fd, HPET_IE_OFF, 0) < 0) {
        fprintf(stderr, "ERROR: HPET_IE_OFF failed\n");
        retval = 6;
    }

fail:
    sigaction(SIGIO, &old, NULL);

    if (fd > 0)
        close(fd);

    return retval;
}

Here is the output from this example when it is invoked with a frequency of 32 and an iteration count of 64.


$ sudo ./hpet_example 32 64

hi_ireqfreq: 0x20  hi_flags: 00  hi_hpet: 0x2  hi_timer: 0x4a1cb9c8

hpet_alarm called. iteration:  0  secs: 0  nsecs: 124683205055050 
hpet_alarm called. iteration:  1  secs: 0  nsecs: 124683236313149 
hpet_alarm called. iteration:  2  secs: 0  nsecs: 124683267566342 
hpet_alarm called. iteration:  3  secs: 0  nsecs: 124683298821905 
hpet_alarm called. iteration:  4  secs: 0  nsecs: 124683330077493 
hpet_alarm called. iteration:  5  secs: 0  nsecs: 124683361341893 
hpet_alarm called. iteration:  6  secs: 0  nsecs: 124683392590764 
hpet_alarm called. iteration:  7  secs: 0  nsecs: 124683423849157 
hpet_alarm called. iteration:  8  secs: 0  nsecs: 124683455101917 
hpet_alarm called. iteration:  9  secs: 0  nsecs: 124683486357683 
hpet_alarm called. iteration: 10  secs: 0  nsecs: 124683517617931 
hpet_alarm called. iteration: 11  secs: 0  nsecs: 124683548872198 
hpet_alarm called. iteration: 12  secs: 1  nsecs: 124682580229541 
hpet_alarm called. iteration: 13  secs: 1  nsecs: 124682611481235 
hpet_alarm called. iteration: 14  secs: 1  nsecs: 124682642740016 
hpet_alarm called. iteration: 15  secs: 1  nsecs: 124682673992697 
hpet_alarm called. iteration: 16  secs: 1  nsecs: 124682705247479 
hpet_alarm called. iteration: 17  secs: 1  nsecs: 124682736504664 
hpet_alarm called. iteration: 18  secs: 1  nsecs: 124682767758840 
hpet_alarm called. iteration: 19  secs: 1  nsecs: 124682799014280 
hpet_alarm called. iteration: 20  secs: 1  nsecs: 124682830270129 
hpet_alarm called. iteration: 21  secs: 1  nsecs: 124682861530334 
hpet_alarm called. iteration: 22  secs: 1  nsecs: 124682892784577 
hpet_alarm called. iteration: 23  secs: 1  nsecs: 124682924038220 
hpet_alarm called. iteration: 24  secs: 1  nsecs: 124682955294110 
hpet_alarm called. iteration: 25  secs: 1  nsecs: 124682986550572 
hpet_alarm called. iteration: 26  secs: 1  nsecs: 124683017805756 
hpet_alarm called. iteration: 27  secs: 1  nsecs: 124683049061117 
hpet_alarm called. iteration: 28  secs: 1  nsecs: 124683080318331 
hpet_alarm called. iteration: 29  secs: 1  nsecs: 124683111576954 
hpet_alarm called. iteration: 30  secs: 1  nsecs: 124683142828988 
hpet_alarm called. iteration: 31  secs: 1  nsecs: 124683174083954 
hpet_alarm called. iteration: 32  secs: 1  nsecs: 124683205337967 
hpet_alarm called. iteration: 33  secs: 1  nsecs: 124683236593144 
hpet_alarm called. iteration: 34  secs: 1  nsecs: 124683267851530 
hpet_alarm called. iteration: 35  secs: 1  nsecs: 124683299104054 
hpet_alarm called. iteration: 36  secs: 1  nsecs: 124683330358748 
hpet_alarm called. iteration: 37  secs: 1  nsecs: 124683361617445 
hpet_alarm called. iteration: 38  secs: 1  nsecs: 124683392870249 
hpet_alarm called. iteration: 39  secs: 1  nsecs: 124683424124489 
hpet_alarm called. iteration: 40  secs: 1  nsecs: 124683455379717 
hpet_alarm called. iteration: 41  secs: 1  nsecs: 124683486634424 
hpet_alarm called. iteration: 42  secs: 1  nsecs: 124683517889149 
hpet_alarm called. iteration: 43  secs: 1  nsecs: 124683549144315 
hpet_alarm called. iteration: 44  secs: 2  nsecs: 124682580500695 
hpet_alarm called. iteration: 45  secs: 2  nsecs: 124682611761325 
hpet_alarm called. iteration: 46  secs: 2  nsecs: 124682643011863 
hpet_alarm called. iteration: 47  secs: 2  nsecs: 124682674265864 
hpet_alarm called. iteration: 48  secs: 2  nsecs: 124682705521034 
hpet_alarm called. iteration: 49  secs: 2  nsecs: 124682736776049 
hpet_alarm called. iteration: 50  secs: 2  nsecs: 124682768030654 
hpet_alarm called. iteration: 51  secs: 2  nsecs: 124682799285398 
hpet_alarm called. iteration: 52  secs: 2  nsecs: 124682830544701 
hpet_alarm called. iteration: 53  secs: 2  nsecs: 124682861797319 
hpet_alarm called. iteration: 54  secs: 2  nsecs: 124682893051578 
hpet_alarm called. iteration: 55  secs: 2  nsecs: 124682924306748 
hpet_alarm called. iteration: 56  secs: 2  nsecs: 124682955562132 
hpet_alarm called. iteration: 57  secs: 2  nsecs: 124682986823545 
hpet_alarm called. iteration: 58  secs: 2  nsecs: 124683018073636 
hpet_alarm called. iteration: 59  secs: 2  nsecs: 124683049327560 
hpet_alarm called. iteration: 60  secs: 2  nsecs: 124683080586707 
hpet_alarm called. iteration: 61  secs: 2  nsecs: 124683111841132 
hpet_alarm called. iteration: 62  secs: 2  nsecs: 124683143095147 
hpet_alarm called. iteration: 63  secs: 2  nsecs: 124683174349985 
hpet_alarm called. iteration: 64  secs: 2  nsecs: 124683205607103 
$

Well, I think that I have provided you with enough information so that you should now be able to go away and experiment with the HPET interface yourself.

By the way, not all VMware products support HPET. Currently ESX does not provide a virtual HPET to guest operating systems and in some cases it may be necessary to disable HPET altogether because of timer drift in virtual machines. See VMware TimeKeeping for more information.

P.S. I tested the the above example on an Intel DX48BT2 motherboard running a 2.6.29.5-191 kernel.

XAM Mandated Fields

2009-06-29T13:34:00.000-04:00

In this post I look at what fields are mandated by the SNIA XAM v1.0 specification and write a small Java application to retrieve their default values using the XAM Reference VIM and EMC's Centera XAM VIM.

What is meant by a field in XAM? According to Section 3.1.5 of the XAM specification v1.0, Part 1, a field is

a piece of uniquely identifiable data that can be attached to an XSet, an XSystem, or a XAM Library.

More concretely, a field has a name, a number of attributes that describe how to interact with the object, and a value. Any XAM primary object, i.e. an XSystem, an XSet, or a XAM Library object can contain one or more fields.

Field names are case sentitive UTF-8 encoded strings with a maximum length of 512 bytes and no embedded NULL characters. To avoid namespace clashes, the field namespace is allocated between SNIA, XAM storage system vendors and XAM application vendors. The following table shows the currently reserved namespace for field names:
[table id=7 /]
To avoid field namespace clashes between XAM storage system vendors in the remaining unreserved namespace and aleviate the need for a central XAM field name registry, the first portion of a vendor field name shall be the vendor's domain name in reverse order, followed by the vendor-defined field name, e.g. com.emc.centera.xam.vim.version.

As mentionly previously a field can have attributes. The following four attributes are mandated by the XAM specification:
[table id=8 /]
Two distinct types of fields exist: a property and an XStream. For the purposes of this post, we are only interested in fields which are properties as defined in section 3.1.12 of the XAM specification, i.e

a field whose MIME type attribute is one of the XAM-defined simple types (stypes).

Here is a list of the specified XAM stypes:
[table id=9 /]
Turning now to the two objects that we are interested in, namely the XAM Library object and the XSystem object.

XAM Library properties are always nonbinding i.e. a change in the property value does not trigger the creation of a new XSet with a corresponding new XUID. Some may be readonly and intended to be only inspected by the application. Others such as .xam.log.level may be modifiable by a XAM application to effect a change in the behavior of the XAM Library object. Note however that changes are not persisted by the XAM Library.

Here is the list of the mandated properties for the XAM Library:
[table id=1 /]
An XSystem property is strongly typed to help XAM application interoperability. The stype is checked and the actual MIME type is set based on the specific method that the XAM application uses to create the field. A number of other field consistency checks are also done by an XSystem including checking that field names do not begin with a period, are a valid xam_string and do not have embedded NULLs. See section 6.3.5 of the XAM specification for a complete description of the field consistency checks.

Here is the list of mandated properties for an XSystem object:
[table id=2 /]
Here is a simple Java application which iterates over the list of available fields and output the fields which match the specified field prefix or all fields if no prefix is entered. It uses the XAM FieldIterator class to retrieve the requested fields from either the SNIA-XAM SDK reference VIM or a third party VIM such as the EMC Centera VIM.

import java.io.ByteArrayOutputStream;
import java.io.BufferedOutputStream;
import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import java.util.Calendar;
import java.text.SimpleDateFormat;
import java.util.Properties;

import org.snia.xam.XAMLibrary;
import org.snia.xam.XAMLibraryObj;
import org.snia.xam.XAMException;
import org.snia.xam.XSystem;
import org.snia.xam.XSet;
import org.snia.xam.XIterator;
import org.snia.xam.toolkit.XAMXUID;
import org.snia.xam.util.XAMLibraryFactory;
import org.snia.xam.FieldContainer;

import org.snia.xam.XUID;
import org.snia.xam.base.XAMImplementation;
import org.snia.xam.util.SASLUtils;
import org.snia.xam.vim.reference.ReferenceAuthenticationStatus;
import org.snia.xam.vim.reference.utils.ReferenceSaslUtils;

public class XamFieldIterator {
    private static XAMLibrary xamLib;
    private static XSystem xSystem;
    protected static String xri;

    public static boolean  IS_REFERENCE_VIM = false;
    public static final String TEST_PROP_FILE = "xam.test.props";
    public static final String XRI_PROP  = "xam.test.xri";
    public static final String CONFIG_PROP = "xam.test.vims";
    public static final String USER_PROP = "xam.test.username";
    public static final String PASS_PROP = "xam.test.password";
    protected static final String DEFAULT_USER = "test";
    protected static final String DEFAULT_PASS = "test";
    protected static String s_pass;
    protected static String s_user;

    public static void initLibrary() throws Exception {
        if (xamLib == null) {
            System.out.println("\nInitializing VIM");
            Properties props = new Properties();
            String testPropFile = System.getProperty(TEST_PROP_FILE);
            if (testPropFile == null)
                testPropFile = TEST_PROP_FILE;

            System.out.println("Loading test properties from file: " + testPropFile);
            props.load(new FileInputStream(testPropFile));

            xri = props.getProperty(XRI_PROP);
            s_user = props.getProperty(USER_PROP, DEFAULT_USER);
            s_pass = props.getProperty(PASS_PROP, DEFAULT_PASS);

            // XAM Library loads the VIM name and associated Java implementation class
            System.out.println("Loading the VIM using the Java XAM Library.");
            System.out.println("VIM Configuration contained in file: " +
                    props.getProperty(CONFIG_PROP));
            xamLib = new XAMImplementation(props.getProperty(CONFIG_PROP));
        }
    }

    private static XSystem connectToVIM(String xri) throws Exception {
        XSystem xsystem = null;
        xsystem = xamLib.connect(xri);
        return xsystem;
    }

    private static void authenticate(XSystem system) throws XAMException {
        String defMech = system.getString(XSystem.XAM_XSYSTEM_AUTH_SASL_DEFAULT);
        ByteArrayOutputStream response = new ByteArrayOutputStream(200);
        byte[] inputData = null;
        int retValue = 0;

        if (defMech.equals(ReferenceSaslUtils.SASL_MECHANISM_ANONYMOUS)) {
            retValue = system.authenticate(inputData, response);
        } else if (defMech.equals(SASLUtils.SASL_PLAIN)) {
            byte[] creds = ReferenceSaslUtils.encodeSASLPlain(null,
                    ReferenceAuthenticationStatus.TEST_USERNAME,
                    ReferenceAuthenticationStatus.TEST_PASSWORD);
            retValue = system.authenticate(creds, response);
        } else {
            throw new XAMException("Unknown default SASL mechanism " + defMech);
        }

        // Validate return status value
        if (retValue != XSystem.XAM_SASL_COMPLETE) {
            throw new XAMException("Failed to authenticate.");
        }
    }

    /* a FieldContainer is a XAM superclass for the 3 primary objects (XAMLibrary, XSystem, XSet) */
    private static void iterateFields(FieldContainer fieldContainer, String prefix) throws XAMException {
        String fieldName;
        String fieldType;
        String fieldValue;
        String fieldBinding;
        String fieldReadOnly;

        XIterator xIterator = fieldContainer.openFieldIterator(prefix);
        while (xIterator.hasNext()) {
            fieldName = (String)xIterator.next();
            fieldType = fieldContainer.getFieldType(fieldName);

            fieldBinding = fieldContainer.getFieldBinding(fieldName) != true ? "NB" : "BI";
            fieldReadOnly =  fieldContainer.getFieldReadOnly(fieldName) != true ? "RW" : "RO";

            if (fieldType.equals(XAMLibrary.STYPE_BOOLEAN_MIME_TYPE))
                fieldValue = fieldContainer.getBoolean(fieldName) != true ? "false" : "true";
            else if (fieldType.equals(XAMLibrary.STYPE_INT_MIME_TYPE))
                fieldValue = Long.toString(fieldContainer.getLong(fieldName));
            else if (fieldType.equals(XAMLibrary.STYPE_DOUBLE_MIME_TYPE))
                fieldValue = Double.toString(fieldContainer.getDouble(fieldName));
            else if (fieldType.equals(XAMLibrary.STYPE_XUID_MIME_TYPE))
                fieldValue = fieldContainer.getXUID(fieldName).toString();
            else if (fieldType.equals(XAMLibrary.STYPE_STRING_MIME_TYPE))
                fieldValue = fieldContainer.getString(fieldName);
            else if (fieldType.equals(XAMLibrary.STYPE_DATETIME_MIME_TYPE)) {
                Calendar now = fieldContainer.getDateTime(fieldName);
                int Y = now.get(Calendar.YEAR);
                int M = now.get(Calendar.MONTH);
                int D = now.get(Calendar.DAY_OF_MONTH);
                int h = now.get(Calendar.HOUR_OF_DAY);
                int m = now.get(Calendar.MINUTE);
                int s = now.get(Calendar.SECOND);
                fieldValue = Y + "-" + M + "-" + D + " " + h + ":" + m + ":" + s;
            } else
                fieldValue = fieldContainer.getFieldLength(fieldName) + " bytes";

            System.out.println(String.format("%s   (%s) %s %s   \"%s\"",
                 fieldName, fieldType, fieldBinding, fieldReadOnly, fieldValue));
        }
        xIterator.close();
    }

    public static void main(String[] args) {
        long exitCode = 0;

        InputStreamReader inputReader = new InputStreamReader(System.in);
        BufferedReader stdin = new BufferedReader(inputReader);

        try {
            /* simple check for command line option of "-r" */
            if (args.length == 1 ) {
               if (args[0].equals("-r"))
                  IS_REFERENCE_VIM = true;
            }

            if (IS_REFERENCE_VIM) {
                initLibrary();
                System.out.println("Connecting to XSystem " + xri + "\n");
                xSystem = connectToVIM(xri);
                authenticate(xSystem);
            } else {
                xri = "snia-xam://centera_vim!128.221.200.60?/home/fpm/xam/xamconnect.pea";
                xamLib = XAMLibraryFactory.newXAMLibrary();
                System.out.println("Connecting to XSystem " + xri + "\n");
                xSystem = xamLib.connect(xri);
            }

            System.out.print("Enter prefix to filter results (blank for all): ");
            String prefix = stdin.readLine();

            iterateFields(xSystem, prefix);

            xSystem.close();
            System.out.println( "\nClosed connection to XSystem" );

        } catch (XAMException xe) {
            System.err.println("XAM ERROR: " + xe.getMessage());
            exitCode = 1;
        } catch (IllegalArgumentException e) {
            System.out.println(e.getMessage());
            e.printStackTrace();
            exitCode = 1;
        } catch (IOException e) {
             System.err.println("IO ERROR: " + e.getMessage());
             e.printStackTrace();
             exitCode = 1;
        } catch (Exception ex) {
             ex.printStackTrace();
             exitCode = 1;
        }

        System.exit((int) exitCode);
    }
}

Here is the set of fields, i.e those fields which start with .xam. for the XAMLibrary object when the application is connected to an EMC Centera using EMC's XAM VIM. It is followed by the set of fields, i.e. those fields starting with .xsystem., for an XSystem object which we create using this XAMLibrary object.

$ java XamFieldIterator
Connecting to XSystem snia-xam://centera_vim!XXX.XXX.XXX.XXX?/home/fpm/xam/xamconnect.pea

Enter prefix to filter results (blank for all): .xam

.xam.log.component.filter   (application/vnd.snia.xam.string) NB RW   ""
.xam.log.message.filter   (application/vnd.snia.xam.string) NB RW   ""
.xam.log.max.rollovers   (application/vnd.snia.xam.int) NB RW   "1"
.xam.log.max.size   (application/vnd.snia.xam.int) NB RW   "1048576"
.xam.log.append   (application/vnd.snia.xam.boolean) NB RW   "false"
.xam.log.format   (application/vnd.snia.xam.int) NB RW   "1"
.xam.log.path   (application/vnd.snia.xam.string) NB RW   "xam.log"
.xam.log.verbosity   (application/vnd.snia.xam.int) NB RW   "0"
.xam.log.level   (application/vnd.snia.xam.int) NB RW   "0"

Closed connection to XSystem

$ java XamFieldIterator
Connecting to XSystem snia-xam://centera_vim!XXX.XXX.XXX.XXX?/home/fpm/xam/xamconnect.pea

Enter prefix to filter results (blank for all): .xsystem

.xsystem.auth.identity.authorization   (application/vnd.snia.xam.string) NB RO   "xam_challenge"
.xsystem.auth.identity.authentication   (application/vnd.snia.xam.string) NB RO   "xam_challenge"
.xsystem.limits.maxSizeOfXStream   (application/vnd.snia.xam.int) NB RO   "107374182400"
.xsystem.limits.maxFieldsPerXSet   (application/vnd.snia.xam.int) NB RO   "9223372036854775807"
.xsystem.identity   (application/vnd.snia.xam.string) NB RO   "EMC Centera, ID# 34372862-1dd2-11b2-aea0-f013836b5e75"
.xsystem.auth.SASLmechanism.list.ANONYMOUS   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.SASLmechanism.list.PLAIN   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.SASLmechanism.default   (application/vnd.snia.xam.string) NB RO   "ANONYMOUS"
.xsystem.time   (application/vnd.snia.xam.datetime) NB RO   "2009-5-28 15:10:42"
.xsystem.auth.granule.list.read   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.write-application   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.write-system   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.create   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.delete   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.job   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.job-commit   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.auth.granule.list.hold   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.auth.granule.list.retention-event   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.job.list.xam.job.query   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.expiration   (application/vnd.snia.xam.int) NB RO   "-1"
.xsystem.access   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.job.commit.supported   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.job.xam.job.query.continuance.supported   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.job.xam.job.query.level2.supported   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.job.xam.job.query.level1.supported   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.deletion.autodelete   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.deletion.shred   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.management.policy.default   (application/vnd.snia.xam.string) NB RO   "default"
.xsystem.management.policy.list.default   (application/vnd.snia.xam.string) NB RO   "default"
.xsystem.retention.duration.policy.list.0seconds   (application/vnd.snia.xam.string) NB RO   "0seconds"
.xsystem.retention.duration.policy.list.10min   (application/vnd.snia.xam.string) NB RO   "10min"
.xsystem.retention.duration.policy.list.1day   (application/vnd.snia.xam.string) NB RO   "1day"
.xsystem.retention.duration.policy.list.1hr   (application/vnd.snia.xam.string) NB RO   "1hr"
.xsystem.retention.duration.policy.list.30days   (application/vnd.snia.xam.string) NB RO   "30days"
.xsystem.retention.duration.policy.list.60days   (application/vnd.snia.xam.string) NB RO   "60days"
.xsystem.retention.duration.policy.list.CBR_EMail   (application/vnd.snia.xam.string) NB RO   "CBR_EMail"
.xsystem.retention.duration.policy.list.CBR_Email   (application/vnd.snia.xam.string) NB RO   "CBR_Email"
.xsystem.retention.duration.policy.list.CBR_PACS   (application/vnd.snia.xam.string) NB RO   "CBR_PACS"
.xsystem.log.level   (application/vnd.snia.xam.int) NB RW   "0"
.xsystem.log.verbosity   (application/vnd.snia.xam.int) NB RW   "0"
.xsystem.log.path   (application/vnd.snia.xam.string) NB RW   "centera_vim.log"

Closed connection to XSystem
$

Here are the corresponding outputs when the application instead connects to the SNIA XAM-SDK reference VIM which incidently is written completely in Java; hence the need for two different factory methods in our application.

$ java XamFieldIterator -r
Initializing VIM
Loading test properties from file: xam.test.props
Loading the VIM using the Java XAM Library.
VIM Configuration contained in file: ../config/ReferenceVIM.config
Connecting to XSystem snia-xam://SNIA_Reference_VIM!localhost?dir=/home/fpm/xam/xam_storage

Enter prefix to filter results (blank for all): .xam
.xam.apiLevel   (application/vnd.snia.xam.string) NB RW   "01.00.00"
.xam.identity   (application/vnd.snia.xam.string) NB RW   "SNIA XAM Java Library v1.0"
.xam.log.append   (application/vnd.snia.xam.boolean) NB RW   "false"
.xam.log.level   (application/vnd.snia.xam.int) NB RW   "1"
.xam.log.max.rollovers   (application/vnd.snia.xam.int) NB RW   "1"
.xam.log.max.size   (application/vnd.snia.xam.int) NB RW   "10"
.xam.log.path   (application/vnd.snia.xam.string) NB RW   "SNIA-XAM.log"
.xam.log.verbosity   (application/vnd.snia.xam.int) NB RW   "1"
.xam.vim.list.SNIA_Reference_VIM   (application/vnd.snia.xam.string) NB RW   "SNIA_Reference_VIM"

Closed connection to XSystem

$ java XamFieldIterator -r

Initializing VIM
Loading test properties from file: xam.test.props
Loading the VIM using the Java XAM Library.
VIM Configuration contained in file: ../config/ReferenceVIM.config
Connecting to XSystem snia-xam://SNIA_Reference_VIM!localhost?dir=/home/fpm/xam/xam_storage

Enter prefix to filter results (blank for all): .xsystem
.xsystem.access   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.access.policy.list.read   (application/vnd.snia.xam.string) NB RO   "Read"
.xsystem.access.policy.list.write-application   (application/vnd.snia.xam.string) NB RO   "Write-application"
.xsystem.access.policy.list.write-system   (application/vnd.snia.xam.string) NB RO   "Write-system"
.xsystem.auth.SASLmechanism.default   (application/vnd.snia.xam.string) NB RO   "PLAIN"
.xsystem.auth.SASLmechanism.list.ANONYMOUS   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.SASLmechanism.list.PLAIN   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.expiration   (application/vnd.snia.xam.int) NB RO   "-1"
.xsystem.auth.granule.list.create   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.delete   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.hold   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.job   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.job-commit   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.read   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.retention-event   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.write-application   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.granule.list.write-system   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.auth.identity.authentication   (application/vnd.snia.xam.string) NB RO   ""
.xsystem.auth.identity.authorization   (application/vnd.snia.xam.string) NB RO   ""
.xsystem.deletion.autodelete   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.deletion.shred   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.identity   (application/vnd.snia.xam.string) NB RO   "ultra.localdomain File System"
.xsystem.job.commit.supported   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.job.list.xam.job.query   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.job.xam.job.query.continuance.supported   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.job.xam.job.query.level1.supported   (application/vnd.snia.xam.boolean) NB RO   "true"
.xsystem.job.xam.job.query.level2.supported   (application/vnd.snia.xam.boolean) NB RO   "false"
.xsystem.limits.maxFieldsPerXSet   (application/vnd.snia.xam.int) NB RO   "10000"
.xsystem.limits.maxSizeOfXStream   (application/vnd.snia.xam.int) NB RO   "9223372036854775807"
.xsystem.management.policy.default   (application/vnd.snia.xam.string) NB RO   ".org.snia.refvim.default.mgmt.policy"
.xsystem.management.policy.list..org.snia.refvim.default.mgmt.policy   (application/vnd.snia.xam.string) NB RO   ".org.snia.refvim.default.mgmt.policy"
.xsystem.time   (application/vnd.snia.xam.datetime) NB RO   "2009-5-29 10:24:0"

Closed connection to XSystem
$

As you can see there are lots of interesting and potentially useful information available in the fields associated with a XAMLIbrary or XSystem object. Hopefully this post has improved your knowledge of the mandated fields and the field namespace.

Happy Independence Day!

2009-06-24T16:47:00.003-04:00

I have finally taken the plunge and moved my blog to Wordpress on my own website. I had intended doing this many months ago but work kept getting in the way.

The straw that broke the proverbial camel's back occurred yesterday when I wanted to change the theme of my blog to a more modern looking theme. I followed Blogger's recommendations and exported all my posts and data to my desktop. However it turns out that Blogger does not handle switching themes very well when you have a number of custom widgets. Given that I was going to have to spend significant time fixing up things after a simple theme change, I decided that that time would be better spent switching to WordPress.

The URL for my blog is blog.fpmurphy.com.

It is a work in process at present I have imported most posts from Blogger and will do the minor cleanup required as a result of this import starting at the newest post and working back to the oldest post.

I will continue to post a copy of my blog posts here because search engines seem to index this location more than my website.

XAM Canonical Format

2009-06-20T18:15:00.059-04:00

One of the key requirements for achieving long term data persistence is the ability to move data between archiving systems or, in the language of the SNIA XAM (eXtensible Access Method) specification, moving XSets between XSystems.

The XAM v1.0 specification supports this requirement by providing support for exporting and importing Xsets. It specifies the methods used to export an XSet from an XSystem, the resultant XSet canonical data interchange format (package) and the methods used to import an Xset into an Xsystem.

This post assumes that you are somewhat familiar with XAM and how to program to that specification using Java. It focuses on the format and content of the XSet canonical format package which consists of two main parts: an XML document which describes the policies, properties and streams of one or more XSets followed by the binary representation of the stream(s).

The package format conforms to the 2005 W3C XML-binary Optimized Packaging (XOP) recommendation. To quote from the recommendation:

XOP define a general purpose serialization mechanism for the XML Infoset with binary content that is not only applicable to SOAP and MIME packaging, but to any XML Infoset and any packaging mechanism.

If you are unfamiliar with XOP, and most people are, an article by Andrey Butov in the December 2005 issue of Doctor Dobb's Journal contained a good introduction.

More than one XSet can be contained in a package. However the current XAM SDK reference implementation only supports one XSet. The XML document (AKA the XSet manifest) is a valid and well-formed XML document whose root element is xsets. It can be parsed and manipulated using XSLT and other XML tools. Annex B of the XAM Architecture document contains an XML Schema Definition (XSD) for the XSet manifest.

In order to study the package format in more detail, I wrote a small Java application called StoreHelloWorld which creates a new XSet containing two XStreams. The first Xstream contains the source code for the ubiquitous HelloWorld.java program. The second XStream contains the binary object HelloWorld.class encoded to base64 and with a MIME type of application/base64. Normally you should not encode an XStream but displaying binary files in a blog is problematic and hence the workaround.

Here is the source code for StoreHelloWorld.

import java.io.BufferedOutputStream;
import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.InputStream;
import java.util.Calendar;

import org.snia.xam.XAMException;
import org.snia.xam.XAMLibrary;
import org.snia.xam.XSet;
import org.snia.xam.XStream;
import org.snia.xam.XSystem;
import org.snia.xam.XUID;
import org.snia.xam.util.XAMLibraryFactory;

public class StoreHelloWorld {
   private static XAMLibrary xamLib;
   private static XSystem xSystem;
   private static final int BUFFER_SIZE = 1024 * 128;

    private static XUID storeExportFile(String filename) throws XAMException {
        XUID xuid = null;

        System.out.println("Create XSet ....");
        XSet xSet = xSystem.createXSet(XSet.MODE_UNRESTRICTED);
        xSet.createProperty("app.filename", true, "HelloWorld.");

        try {
            System.out.println("Create XStream 1 ....");
            BufferedInputStream inputStream = new BufferedInputStream(new FileInputStream("HelloWorld.java"));
            XStream xStream = xSet.createXStream("com.rhc.file.source", true, XAMLibrary.DEFAULT_MIME_TYPE);

            byte[] buffer = new byte[BUFFER_SIZE];
            long bytesRead = 0;

            while((bytesRead = inputStream.read(buffer)) > 0) {
                long bytesPos = 0;
                while(bytesPos < bytesRead) {
                    long bytesWrite = xStream.write(buffer, bytesPos, bytesRead - bytesPos);
                    bytesPos += bytesWrite;
                }
            }
            inputStream.close();
            xStream.close();
        } catch (FileNotFoundException fe) {
            throw new IllegalArgumentException("Could not open HelloWorld.java for reading");
        } catch (IOException ioe) {
            System.err.println("Error reading from HelloWorld.java");
            ioe.printStackTrace();
        }

        try {
            System.out.println("Create XStream 2 ....");

            InputStream inputStream = new Base64.InputStream(new FileInputStream("HelloWorld.class"), Base64.ENCODE | Base64.DO_BREAK_LINES);
            XStream xStream = xSet.createXStream("com.rhc.file.binary", true, "application/base64");

            byte[] buffer = new byte[BUFFER_SIZE];
            long bytesRead = 0;

            while((bytesRead = inputStream.read(buffer)) > 0) {
                long bytesPos = 0;
                while(bytesPos < bytesRead) {
                    long bytesWrite = xStream.write(buffer, bytesPos, bytesRead - bytesPos);
                    bytesPos += bytesWrite;
                }
            }
            inputStream.close();
            xStream.close();
        } catch (FileNotFoundException fe) {
            throw new IllegalArgumentException("Could not open HelloWorld.class for reading");
        } catch (IOException ioe) {
            System.err.println("Error reading from HelloWorld.class");
            ioe.printStackTrace();
        }

       System.out.println("Commit XSet .... ");
       xuid = xSet.commit();
       System.out.println("Assigned XUID: " + xuid.toString());

       System.out.println("Close XSet .... ");
       xSet.close();

       File saveFile = new File(filename);
       try {
           System.out.println("\nOpen XSet .... ");
           xSet = xSystem.openXSet(xuid, XSet.MODE_READ_ONLY);

           BufferedOutputStream outputStream = new BufferedOutputStream(new FileOutputStream(saveFile));

           System.out.println("Open XStream for export ....");
           XStream xStream = xSet.openExportXStream();

           byte[] buffer = new byte[BUFFER_SIZE];
           long bytesRead = 0;

           System.out.println("Retrieve data .... ");
           while((bytesRead = xStream.read(buffer)) > 0) {
               outputStream.write(buffer, 0, (int)bytesRead);
           }
           xStream.close();
           outputStream.close();

           System.out.println("Close XSet .... ");
           xSet.close();
        } catch (IOException ioe) {
            System.err.println("Error writing to " + saveFile);
            ioe.printStackTrace();
        }

        return (xuid);
   }
 
   public static void main(String[] args) {
       String xri = XRIstring.get();
       long exitCode = 0;

       InputStreamReader inputReader = new InputStreamReader(System.in);
       BufferedReader stdin = new BufferedReader(inputReader);

       try {
           xamLib = XAMLibraryFactory.newXAMLibrary();

           if (xri == null ) {
               System.out.print("Enter Xsystem address: ");
               String answer = stdin.readLine();
               if (!answer.equals(""))
                  xri = answer;
               else
                  System.exit(0);
           }

           System.out.println("Connecting to XSystem: " + xri );
           xSystem = xamLib.connect(xri);

           System.out.print("Name of export file: ");
           String filename = stdin.readLine();
           if (filename.equals(""))
               System.exit(0);

           XUID xuid = storeExportFile(filename);

           xSystem.close();
           System.out.println("\nClosed connection to XSystem");

           inputReader.close();
           stdin.close();

       } catch (XAMException xe) {
           exitCode = xe.getStatusCode();
           System.err.println(
               "XAM Error occured: " + xe.getMessage() + "(" + exitCode + ")");
       } catch (IOException e) {
           System.out.println("I/O Error occurred: " + e.getMessage());
           e.printStackTrace();
           exitCode = 1;
       } catch (IllegalArgumentException e) {
           System.out.println(e.getMessage());
           e.printStackTrace();
           exitCode = 1;
       }

       System.exit((int) exitCode);
   }
}

Rather than hard code an XRI (XAM Resource Identifier) in each XAM application, I use a separate Java class (XRIstring) to read in the XRI from the standard XAM properties file xam.properties.

I know that this is not kosher according to the current XAM specification but, in my humble opinion, this is the natural location for storing this string as most people will not be connecting simultaneously to multiple XSystems.

Here is the source for that Java class.

import java.io.FileInputStream;
import java.io.IOException;
import java.io.FileNotFoundException;
import java.util.Properties;

public class XRIstring
{
   private static Properties props;
   private static String XRI = ".xam.config.xri";
   private static String xri = null;

   static {

       props = new Properties();
       try {
           props.load(new FileInputStream("xam.properties"));

           xri = props.getProperty(XRI);
           if (xri != null)
               System.out.println("XSystem Address: " + xri );

       } catch (FileNotFoundException e) {
          System.err.println("Could not find properties file. XSystem address must be manually entered");
       } catch (IOException e) {
          System.err.println("Could not access properties file. XSystem address must be manually entered");
       }
   }

   public static String get()
   {
        return(xri);
   }
}

This application also uses Robert Harder's popular Base64 class which is available here. Assuming your CLASSPATH is set up correctly to find the XAM libraries, here is the output from StoreHelloWorld when you build and run it.

$ javac StoreHelloWorld.java XRIstring.java Base64.java

$ java StoreHelloWorld
Connecting to XSystem: snia-xam://centera_vim!XXX.XXX.XXX.XXX?/home/fpm/xam/xamconnect.pea
Name of export file: sample
Create XSet ....
Create XStream 1 ....
Create XStream 2 ....
Commit XSet ....
Assigned XUID: AAAEcwA9S8c0UTVHVDdRSDRSUU9VZUVWMkEyTTJGUUw2UUlHNDE0N1ZTMzFKTzFURkxWUjVKNkFDVEo2Ng==
Close XSet ....

Open XSet for export ....
Retrieve data ....
Close XSet ....

Closed connection to XSystem
$

Here is the contents of the export file (sample) which was created.

Content-Type: Multipart/Related;boundary="___EMC_XOP_BOUNDARY_START___1245537694___EMC_XOP_BOUNDARY_END___";
type="application/xop+xml";
start="<canonicalxset.xml@snia.org>";
start-info="text/xml"
Content-Description: EMC XSet Export Package

--___EMC_XOP_BOUNDARY_START___1245537694___EMC_XOP_BOUNDARY_END___
Content-Type: application/xop+xml; charset=UTF-8; type="text/xml"
Content-Transfer-Encoding: 8bit
Content-ID: <canonicalxset.xml@snia.org>

<xsets xmlns="http://www.snia.org/2007/xam/export"
xmlns:xop="http://www.w3.org/2004/08/xop/include"
xmlns:emcannotations="http://www.emc.com/2008/01/centera/emcannotations"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.snia.org/2007/xam/export XAMCanonicalXSetDefinition.xsd
http://www.w3.org/2004/08/xop/include http://www.w3.org/2004/08/xop/include
http://www.emc.com/2008/01/centera/emcannotations emcannotations.xsd ">
<version>1.0.0</version>

<policies>
  <policy name=".xsystem.management.policy.list.default" type="application/vnd.snia.xam.string" readOnly="true" binding="false" length="7" >
    <value>default</value>
  </policy>
</policies>

<xset>
<properties>
    <property name="app.filename" type="application/vnd.snia.xam.string" readOnly="false" binding="true" length="11"  emcannotations:seqno="0"  >
      <value>HelloWorld.</value>
    </property>
    <property name=".xset.time.creation" type="application/vnd.snia.xam.datetime" readOnly="true" binding="true" length="24"  emcannotations:seqno="3"  >
      <value>2009-06-20T22:20:19.000Z</value>
    </property>
    <property name=".xset.time.access" type="application/vnd.snia.xam.datetime" readOnly="true" binding="false" length="24"  emcannotations:seqno="4"  >
      <value>2009-06-20T22:20:21.000Z</value>
    </property>
    <property name=".xset.time.commit" type="application/vnd.snia.xam.datetime" readOnly="true" binding="false" length="24"  emcannotations:seqno="5"  >
      <value>2009-06-20T22:20:21.000Z</value>
    </property>
    <property name=".xset.time.residency" type="application/vnd.snia.xam.datetime" readOnly="true" binding="false" length="24"  emcannotations:seqno="6"  >
      <value>2009-06-20T22:20:21.000Z</value>
    </property>
    <property name=".xset.xuid" type="application/vnd.snia.xam.xuid" readOnly="true" binding="false" length="61"  emcannotations:seqno="7"  >
      <value>AAAEcwA9S8c0UTVHVDdRSDRSUU9VZUVWMkEyTTJGUUw2UUlHNDE0N1ZTMzFKTzFURkxWUjVKNkFDVEo2Ng==</value>
    </property>
    <property name=".xset.time.xuid" type="application/vnd.snia.xam.datetime" readOnly="true" binding="true" length="24"  emcannotations:seqno="8"  >
      <value>2009-06-20T22:20:21.000Z</value>
    </property>
    <property name=".xset.retention.list.base" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="4"  emcannotations:seqno="9"  >
      <value>base</value>
    </property>
    <property name=".xset.retention.base.enabled" type="application/vnd.snia.xam.boolean" readOnly="true" binding="true" length="1"  emcannotations:seqno="10"  >
      <value>true</value>
    </property>
    <property name=".xset.retention.base.duration" type="application/vnd.snia.xam.int" readOnly="true" binding="true" length="8"  emcannotations:seqno="11"  >
      <value>0</value>
    </property>
    <property name=".xset.retention.base.starttime" type="application/vnd.snia.xam.datetime" readOnly="true" binding="true" length="24"  emcannotations:seqno="12"  >
      <value>2009-06-20T22:20:19.000Z</value>
    </property>
    <property name=".xset.hold" type="application/vnd.snia.xam.boolean" readOnly="true" binding="false" length="1"  emcannotations:seqno="13"  >
      <value>false</value>
    </property>
    <property name=".xset.management.policy" type="application/vnd.snia.xam.string" readOnly="true" binding="false" length="7"  emcannotations:seqno="14"  >
      <value>default</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.creation.poolid" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="39"  emcannotations:seqno="15"  >
      <value>34372862-1dd2-11b2-aea0-f013836b5e75-10</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.retention.base.enabled" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="4"  emcannotations:seqno="16"  >
      <value>true</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.sdk.version" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="7"  emcannotations:seqno="17"  >
      <value>4.0.672</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.modification.poolid" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="39"  emcannotations:seqno="18"  >
      <value>34372862-1dd2-11b2-aea0-f013836b5e75-10</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.retention.period" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="1"  emcannotations:seqno="19"  >
      <value>0</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.type" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="11"  emcannotations:seqno="20"  >
      <value>xam.binding</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.name" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="29"  emcannotations:seqno="21"  >
      <value>EMC Centera Binding XSet Clip</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.creation.date" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="23"  emcannotations:seqno="22"  >
      <value>2009.06.20 22:20:19 GMT</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.modification.date" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="23"  emcannotations:seqno="23"  >
      <value>2009.06.20 22:20:21 GMT</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.creation.profile" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="13"  emcannotations:seqno="24"  >
      <value>xam_challenge</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.modification.profile" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="13"  emcannotations:seqno="25"  >
      <value>xam_challenge</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.numfiles" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="1"  emcannotations:seqno="26"  >
      <value>2</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.totalsize" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="3"  emcannotations:seqno="27"  >
      <value>731</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.refid" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="26"  emcannotations:seqno="28"  >
      <value>6T90EGBR9LE5TFLVR5J6ACTJ66</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.clusterid" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="36"  emcannotations:seqno="29"  >
      <value>34372862-1dd2-11b2-aea0-f013836b5e75</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.prev.clip" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="0"  emcannotations:seqno="30"  >
      <value></value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.clip.naming.scheme" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="3"  emcannotations:seqno="31"  >
      <value>MD5</value>
    </property>
    <property name=".vnd.com.emc.centera.meta.standard.binding.numtags" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="1"  emcannotations:seqno="32"  >
      <value>3</value>
    </property>
    <property name=".vnd.com.emc.centera.hidden.nonbinding.clipid" type="application/vnd.snia.xam.string" readOnly="true" binding="false" length="4"  emcannotations:seqno="33"  >
      <value>NULL</value>
    </property>
    <property name=".vnd.com.emc.centera.hidden.binding.clipid" type="application/vnd.snia.xam.string" readOnly="true" binding="true" length="53"  emcannotations:seqno="34"  >
      <value>4Q5GT7QH4RQOUeEV2A2M2FQL6QIG4147VS31JO1TFLVR5J6ACTJ66</value>
    </property>
</properties>
<xstreams>
  <xstream name="com.rhc.file.source" type="application/octet-stream" readOnly="false" binding="true" length="156"  emcannotations:seqno="1"  >
    <xop:Include href='cid:com.rhc.file.source' />
    <emcannotations:emcannotation emcannotations:embedded="false" >
      <emcannotations:emcblobannotation emcannotations:blobguid="1OROBIQ8CLBN3x8HRFRPNS7UV61G4147VS30JN0TFLVR5J6ACTJ66" emcannotations:offset="0" emcannotations:length="156" />
    </emcannotations:emcannotation>
  </xstream>
  <xstream name="com.rhc.file.binary" type="application/base64" readOnly="false" binding="true" length="575"  emcannotations:seqno="2"  >
    <xop:Include href='cid:com.rhc.file.binary' />
    <emcannotations:emcannotation emcannotations:embedded="false" >
      <emcannotations:emcblobannotation emcannotations:blobguid="09D943LSJ49S2xFF94NIQ5B10MGG4147VS31HG0TFLVR5J6ACTJ66" emcannotations:offset="0" emcannotations:length="575" />
    </emcannotations:emcannotation>
  </xstream>
</xstreams>
</xset>
</xsets>

--___EMC_XOP_BOUNDARY_START___1245537694___EMC_XOP_BOUNDARY_END___
Content-Type: text/text
Content-Transfer-Encoding: text
Content-ID: <TOC>

Offset of AAAEcwA9S8c0UTVHVDdRSDRSUU9VZUVWMkEyTTJGUUw2UUlHNDE0N1ZTMzFKTzFURkxWUjVKNkFDVEo2Ng==:com.rhc.file.source: 10955                                                                                                                                                                                                                                                  
Offset of AAAEcwA9S8c0UTVHVDdRSDRSUU9VZUVWMkEyTTJGUUw2UUlHNDE0N1ZTMzFKTzFURkxWUjVKNkFDVEo2Ng==:com.rhc.file.binary: 11285                                                                                                                                                                                                                                                  

--___EMC_XOP_BOUNDARY_START___1245537694___EMC_XOP_BOUNDARY_END___
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-ID: com.rhc.file.source

/*
 * HelloWorld.java
 *
 */

public class HelloWorld {
    public static void main(String[] args){
       System.out.println("Hello World!");
   }
}

--___EMC_XOP_BOUNDARY_START___1245537694___EMC_XOP_BOUNDARY_END___
Content-Type: application/base64
Content-Transfer-Encoding: binary
Content-ID: com.rhc.file.binary

yv66vgAAADIAHQoABgAPCQAQABEIABIKABMAFAcAFQcAFgEABjxpbml0PgEAAygpVgEABENvZGUB
AA9MaW5lTnVtYmVyVGFibGUBAARtYWluAQAWKFtMamF2YS9sYW5nL1N0cmluZzspVgEAClNvdXJj
ZUZpbGUBAA9IZWxsb1dvcmxkLmphdmEMAAcACAcAFwwAGAAZAQAMSGVsbG8gV29ybGQhBwAaDAAb
ABwBAApIZWxsb1dvcmxkAQAQamF2YS9sYW5nL09iamVjdAEAEGphdmEvbGFuZy9TeXN0ZW0BAANv
dXQBABVMamF2YS9pby9QcmludFN0cmVhbTsBABNqYXZhL2lvL1ByaW50U3RyZWFtAQAHcHJpbnRs
bgEAFShMamF2YS9sYW5nL1N0cmluZzspVgAhAAUABgAAAAAAAgABAAcACAABAAkAAAAdAAEAAQAA
AAUqtwABsQAAAAEACgAAAAYAAQAAAAYACQALAAwAAQAJAAAAJQACAAEAAAAJsgACEgO2AASxAAAA
AQAKAAAACgACAAAACAAIAAkAAQANAAAAAgAO
--___EMC_XOP_BOUNDARY_START___1245537694___EMC_XOP_BOUNDARY_END___--

As you can see this export package is fairly large even though I am only storing two small XStreams in the XSet which I created.

The XSet manifest is everything between the <xsets> and </xsets> tags. It appears to be a valid and well formed XML document although I have not formally checked it.

The first section, i.e. the <policies> section, lists the XSystem policies that are in effect for the included XSet(s). Here the policy is set to the default policy for the Xsystem from which I exported the Xset.

It is followed by a section whose top element is <xset> which details the properties and XStream(s) for a single XSet. Each XSet property element contains the following attributes: type, readonly, binding and length. Each XStream is mapped to an <xstream> element which contains a child element <xop:include> which contains the content-id to the corresponding MIME part containing the actual contents of the XStream.

As you can see the actual contents of an XStream is included as a MIME attachment after the XSet manifest. As an aside, note that there is no requirement for an XStream to verify that its contents actually match the declared MIME type.

The first MIME attachment shown above (AKA the root MIME) lists the offsets of each of the two XStreams. If an XSet does not contain an XStream, then there are no MIME attachments. Other than the root MIME attachment the order of the MIME attachments is not significant. The MIME attachments conform to the requirements of the MIME Multipart/Related Content-type specification (RFC 2387).

Here is the export package generated when StoreHelloWolld was targeted at the SNIA XAM reference VIM (Vendor Interface Module).

Content-Type: Multipart/Related;boundary="--SNIA_REFERENCE_VIM_MIME_BOUNDARY_2009-06-22T21:13:44.330-05:00.XZY_+";type="application/xop+xml";start="<canonicalxset.xml@snia.org>";start-info="text/xml"
Content-Description:Export of XSet: AAA6AwAePKwxMjQ1NzIzMjI0Mjc4ATTLJDRoe2iJ

----SNIA_REFERENCE_VIM_MIME_BOUNDARY_2009-06-22T21:13:44.330-05:00.XZY_+

Content-Type:application/xop+xml;    charset="UTF-8";    type="text/xml"
Content-Transfer-Encoding:8bit
Content-ID:<canonicalxset.xml@snia.org>

<?xml version="1.0" encoding="UTF-8"?>
<xsets xsi:schemaLocation="http://www.snia.org/2007/xam/export/XAMCanonicalXSetDefinition.xsd" xmlns="http://www.snia.org/2007/xam/export" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xop="http://www.w3.org/2004/08/xop/include">
<version>1.0.0</version>
<xset>
<properties>
<property name=".xset.hold" type="application/vnd.snia.xam.boolean" binding="false" readOnly="true" length="1">
        <boolean>false</boolean>
</property>
<property name=".xset.management.policy" type="application/vnd.snia.xam.string" binding="true" readOnly="true" length="36">
        <string>.org.snia.refvim.default.mgmt.policy</string>
</property>
<property name=".xset.retention.base.enabled" type="application/vnd.snia.xam.boolean" binding="true" readOnly="true" length="1">
        <boolean>true</boolean>
</property>
<property name=".xset.retention.base.starttime" type="application/vnd.snia.xam.datetime" binding="true" readOnly="true" length="29">
        <string>2009-06-22T21:13:44.276-05:00</string>
</property>
<property name=".xset.retention.list.base" type="application/vnd.snia.xam.string" binding="true" readOnly="true" length="4">
        <string>base</string>
</property>
<property name=".xset.retention.list.event" type="application/vnd.snia.xam.string" binding="true" readOnly="true" length="5">
        <string>event</string>
</property>
<property name=".xset.time.access" type="application/vnd.snia.xam.datetime" binding="false" readOnly="true" length="29">
        <string>2009-06-22T21:13:44.276-05:00</string>
</property>
<property name=".xset.time.commit" type="application/vnd.snia.xam.datetime" binding="false" readOnly="true" length="29">
        <string>2009-06-22T21:13:44.276-05:00</string>
</property>
<property name=".xset.time.creation" type="application/vnd.snia.xam.datetime" binding="true" readOnly="true" length="29">
        <string>2009-06-22T21:13:44.225-05:00</string>
</property>
<property name=".xset.time.xuid" type="application/vnd.snia.xam.datetime" binding="false" readOnly="true" length="29">
        <string>2009-06-22T21:13:44.276-05:00</string>
</property>
<property name=".xset.xuid" type="application/vnd.snia.xam.xuid" binding="true" readOnly="true" length="30">
        <xuid>AAA6AwAePKwxMjQ1NzIzMjI0Mjc4ATTLJDRoe2iJ</xuid>
</property>
<property name="app.filename" type="application/vnd.snia.xam.string" binding="true" readOnly="false" length="11">
        <string>HelloWorld.</string>
</property>
</properties>
<xstreams>
<xstream name="com.rhc.file.binary" type="application/base64" binding="true" readOnly="false" length="575">
        <org.snia.refvim.xstream_data>XStream_8508062132085140723.data</org.snia.refvim.xstream_data>
<xop:Include href="cid:com.rhc.file.binary@snia.org" xmlns:xop="http://www.w3.org/2004/08/xop/include"/>
</xstream>
<xstream name="com.rhc.file.source" type="application/octet-stream" binding="true" readOnly="false" length="156">
        <org.snia.refvim.xstream_data>XStream_3869980320333904649.data</org.snia.refvim.xstream_data>
<xop:Include href="cid:com.rhc.file.source@snia.org" xmlns:xop="http://www.w3.org/2004/08/xop/include"/>
</xstream>
</xstreams>
</xset>
</xsets>
----SNIA_REFERENCE_VIM_MIME_BOUNDARY_2009-06-22T21:13:44.330-05:00.XZY_+

Content-Type:text/text
Content-Transfer-Encoding:text
Content-ID:<TOC>
Offset of AAA6AwAePKwxMjQ1NzIzMjI0Mjc4ATTLJDRoe2iJ:com.rhc.file.binary@snia.org:3948
Offset of AAA6AwAePKwxMjQ1NzIzMjI0Mjc4ATTLJDRoe2iJ:com.rhc.file.source@snia.org:4706

----SNIA_REFERENCE_VIM_MIME_BOUNDARY_2009-06-22T21:13:44.330-05:00.XZY_+

Content-Type:application/base64
Content-Transfer-Encoding:binary
Content-ID:<com.rhc.file.binary@snia.org>

yv66vgAAADIAHQoABgAPCQAQABEIABIKABMAFAcAFQcAFgEABjxpbml0PgEAAygpVgEABENvZGUB
AA9MaW5lTnVtYmVyVGFibGUBAARtYWluAQAWKFtMamF2YS9sYW5nL1N0cmluZzspVgEAClNvdXJj
ZUZpbGUBAA9IZWxsb1dvcmxkLmphdmEMAAcACAcAFwwAGAAZAQAMSGVsbG8gV29ybGQhBwAaDAAb
ABwBAApIZWxsb1dvcmxkAQAQamF2YS9sYW5nL09iamVjdAEAEGphdmEvbGFuZy9TeXN0ZW0BAANv
dXQBABVMamF2YS9pby9QcmludFN0cmVhbTsBABNqYXZhL2lvL1ByaW50U3RyZWFtAQAHcHJpbnRs
bgEAFShMamF2YS9sYW5nL1N0cmluZzspVgAhAAUABgAAAAAAAgABAAcACAABAAkAAAAdAAEAAQAA
AAUqtwABsQAAAAEACgAAAAYAAQAAAAYACQALAAwAAQAJAAAAJQACAAEAAAAJsgACEgO2AASxAAAA
AQAKAAAACgACAAAACAAIAAkAAQANAAAAAgAO

----SNIA_REFERENCE_VIM_MIME_BOUNDARY_2009-06-22T21:13:44.330-05:00.XZY_+

Content-Type:application/octet-stream
Content-Transfer-Encoding:binary
Content-ID:<com.rhc.file.source@snia.org>

/*
 * HelloWorld.java
 *
 */

public class HelloWorld {
    public static void main(String[] args){
       System.out.println("Hello World!");
   }
}

----SNIA_REFERENCE_VIM_MIME_BOUNDARY_2009-06-22T21:13:44.330-05:00.XZY_+--

As you can see this export package is considerably more compact then the previous export package generated when I used the EMC Centera XAM VIM.

Well, that is about all for now. Hopefully this post was useful to you if you are just coming up to speed on XAM and wondering how to import and export XSets.

BTW, Happy Father's Day to all dads who may be reading this post.

Fedora 11 New Extended File Attributes Namespace

2009-06-15T10:31:00.032-04:00

I am experimenting with XAM (eXtensible Access Method), which is a storage standard developed by SNIA (Storage Networking Industry Association and have developed a first prototype of a VIM (Vendor Interface Module) for the ext4 file system based on adding another namespace to the current extended file attributes support. Since other Linux developers might be interested in how to add an extended attributes namespace to a file system, I decided to publish this post as a guide.

Extended file attributes (EA) are extensions to the normal attributes which are associated with inodes in a file system. They are simply name:value pairs associated with files and directories and whose purpose is to provide additional functionality which is either defined by the operating system or a user application. An EA may be defined or undefined. If an EA is defined, its value may be empty or non-empty. Most of the initial work to support EAs in Linux was done by Andreas Gruenbacher in the 2001 to 2004 timeframe and is based on work done in the SGI XFS file system.

EAs are stored directly in inodes and on additional disk blocks if necessary. Currently all EAs associated with a file must fit in an inode and one additional block. Blocks that contain the identical set of EAs may be shared among several inodes. EAs in inodes and on blocks have a different header followed by multiple entry descriptors. Entry descriptors are sorted in disk blocks but are left unsorted in inodes.

EA names are zero-terminated strings and are always specified using a fully qualified namespace.attribute e.g. system.posix_acl_access or security.selinux. Namespaces are used to define different classes of EAs. Different EA classes are required because the permissions and capabilities required for manipulating EAs in one namespace may differ from one to another. Currently Linux supports EA namespaces for SELinux (security.), system (system.), trusted (trusted.) and user (user.).

Fedora 11 comes with EA support in the Linux kernel and hence it is the kernel which must be modified in order to add another EA namespace. In my case I wanted to add an EA namespace called snia in order to support XAM XSet metadata (see below).

Here is the patch file for the Fedora 11 2.6.29.4-167.fc11 kernel which implements the snia namespace. It should work for both 32-bit and 64-bit kernels but I have only tested it on a 64-bit kernel.

diff -uNrp kernel-2.6.29.orig/fs/ext4/Makefile kernel-2.6.29.new/fs/ext4/Makefile
--- kernel-2.6.29.orig/fs/ext4/Makefile 2009-06-14 22:07:12.191464615 -0400
+++ kernel-2.6.29.new/fs/ext4/Makefile  2009-06-14 22:06:20.009399924 -0400
@@ -8,6 +8,6 @@ ext4-y  := balloc.o bitmap.o dir.o file.o
                   ioctl.o namei.o super.o symlink.o hash.o resize.o extents.o \
                   ext4_jbd2.o migrate.o mballoc.o

-ext4-$(CONFIG_EXT4_FS_XATTR)           += xattr.o xattr_user.o xattr_trusted.o
+ext4-$(CONFIG_EXT4_FS_XATTR)           += xattr.o xattr_user.o xattr_trusted.o xattr_snia.o
 ext4-$(CONFIG_EXT4_FS_POSIX_ACL)       += acl.o
 ext4-$(CONFIG_EXT4_FS_SECURITY)                += xattr_security.o
diff -uNrp kernel-2.6.29.orig/fs/ext4/xattr.c kernel-2.6.29.new/fs/ext4/xattr.c
--- kernel-2.6.29.orig/fs/ext4/xattr.c  2009-06-14 22:11:20.699475221 -0400
+++ kernel-2.6.29.new/fs/ext4/xattr.c   2009-06-14 22:09:48.242839367 -0400
@@ -107,6 +107,7 @@ static struct xattr_handler *ext4_xattr_
 #ifdef CONFIG_EXT4_FS_SECURITY
        [EXT4_XATTR_INDEX_SECURITY]          = &ext4_xattr_security_handler,
 #endif
+       [EXT4_XATTR_INDEX_SNIA]              = &ext4_xattr_snia_handler,
 };

 struct xattr_handler *ext4_xattr_handlers[] = {
@@ -119,6 +120,7 @@ struct xattr_handler *ext4_xattr_handler
 #ifdef CONFIG_EXT4_FS_SECURITY
        &ext4_xattr_security_handler,
 #endif
+       &ext4_xattr_snia_handler,
        NULL
 };
diff -uNrp kernel-2.6.29.orig/fs/ext4/xattr.h kernel-2.6.29.new/fs/ext4/xattr.h
--- kernel-2.6.29.orig/fs/ext4/xattr.h  2009-06-14 22:09:27.016464602 -0400
+++ kernel-2.6.29.new/fs/ext4/xattr.h   2009-06-14 22:07:44.887839381 -0400
@@ -21,6 +21,7 @@
 #define EXT4_XATTR_INDEX_TRUSTED               4
 #define        EXT4_XATTR_INDEX_LUSTRE                 5
 #define EXT4_XATTR_INDEX_SECURITY              6
+#define EXT4_XATTR_INDEX_SNIA                  7

 struct ext4_xattr_header {
        __le32  h_magic;        /* magic number for identification */
@@ -70,6 +71,7 @@ extern struct xattr_handler ext4_xattr_t
 extern struct xattr_handler ext4_xattr_acl_access_handler;
 extern struct xattr_handler ext4_xattr_acl_default_handler;
 extern struct xattr_handler ext4_xattr_security_handler;
+extern struct xattr_handler ext4_xattr_snia_handler;

 extern ssize_t ext4_listxattr(struct dentry *, char *, size_t);
diff -uNrp kernel-2.6.29.orig/fs/ext4/xattr_snia.c kernel-2.6.29.new/fs/ext4/xattr_snia.c
--- kernel-2.6.29.orig/fs/ext4/xattr_snia.c     1969-12-31 19:00:00.000000000 -0500
+++ kernel-2.6.29.new/fs/ext4/xattr_snia.c      2009-06-14 13:37:02.551476954 -0400
@@ -0,0 +1,68 @@
+/*
+ * linux/fs/ext4/xattr_snia.c
+ * Handler for extended user attributes.
+ *
+ * Copyright (C) 2001 by Andreas Gruenbacher, 
+ * Copyright (C) 2009 by Finnbarr P. Murphy, 
+ */
+
+#include 
+#include 
+#include 
+#include "ext4_jbd2.h"
+#include "ext4.h"
+#include "xattr.h"
+
+static size_t
+ext4_xattr_snia_list(struct inode *inode, char *list, size_t list_size,
+                    const char *name, size_t name_len)
+{
+       const size_t prefix_len = XATTR_SNIA_PREFIX_LEN;
+       const size_t total_len = prefix_len + name_len + 1;
+
+#if FPM
+       if (!test_opt(inode->i_sb, XATTR_USER))
+               return 0;
+#endif
+
+       if (list && total_len <= list_size) {
+               memcpy(list, XATTR_SNIA_PREFIX, prefix_len);
+               memcpy(list+prefix_len, name, name_len);
+               list[prefix_len + name_len] = '\0';
+       }
+       return total_len;
+}
+
+static int
+ext4_xattr_snia_get(struct inode *inode, const char *name,
+                   void *buffer, size_t size)
+{
+       if (strcmp(name, "") == 0)
+               return -EINVAL;
+#if FPM
+       if (!test_opt(inode->i_sb, XATTR_USER))
+               return -EOPNOTSUPP;
+#endif
+       return ext4_xattr_get(inode, EXT4_XATTR_INDEX_SNIA, name, buffer, size);
+}
+
+static int
+ext4_xattr_snia_set(struct inode *inode, const char *name,
+                   const void *value, size_t size, int flags)
+{
+       if (strcmp(name, "") == 0)
+               return -EINVAL;
+#if FPM
+       if (!test_opt(inode->i_sb, XATTR_USER))
+               return -EOPNOTSUPP;
+#endif
+       return ext4_xattr_set(inode, EXT4_XATTR_INDEX_SNIA, name,
+                             value, size, flags);
+}
+
+struct xattr_handler ext4_xattr_snia_handler = {
+       .prefix = XATTR_SNIA_PREFIX,
+       .list   = ext4_xattr_snia_list,
+       .get    = ext4_xattr_snia_get,
+       .set    = ext4_xattr_snia_set,
+};
diff -uNrp kernel-2.6.29.orig/fs/xattr.c kernel-2.6.29.new/fs/xattr.c
--- kernel-2.6.29.orig/fs/xattr.c       2009-06-14 22:04:37.472464234 -0400
+++ kernel-2.6.29.new/fs/xattr.c        2009-06-14 22:05:56.759530990 -0400
@@ -42,6 +42,7 @@ xattr_permission(struct inode *inode, co
         * on these is left to the underlying filesystem / security module.
         */
        if (!strncmp(name, XATTR_SECURITY_PREFIX, XATTR_SECURITY_PREFIX_LEN) ||
+           !strncmp(name, XATTR_SNIA_PREFIX, XATTR_SNIA_PREFIX_LEN) ||
            !strncmp(name, XATTR_SYSTEM_PREFIX, XATTR_SYSTEM_PREFIX_LEN))
                return 0;

diff -uNrp kernel-2.6.29.orig/include/linux/xattr.h kernel-2.6.29.new/include/linux/xattr.h
--- kernel-2.6.29.orig/include/linux/xattr.h    2009-06-14 14:53:02.350464255 -0400
+++ kernel-2.6.29.new/include/linux/xattr.h     2009-06-14 14:51:07.585839446 -0400
@@ -33,6 +33,9 @@
 #define XATTR_USER_PREFIX "user."
 #define XATTR_USER_PREFIX_LEN (sizeof (XATTR_USER_PREFIX) - 1)

+#define XATTR_SNIA_PREFIX "snia."
+#define XATTR_SNIA_PREFIX_LEN (sizeof (XATTR_SNIA_PREFIX) - 1)
+
 struct inode;
 struct dentry;

I am going to assume that if you are still reading this post you know how to build, package and install a Linux kernel in Fedora. If not, here is a good tutorial. The only caveat I would make is to ignore the comment that "most text editors know how to break the hardlink correctly to avoid problems." That is just plain wrong. You need to have separate standalone copies of any source file that you intend modifying in both the original and new kernel source trees - not a single hard linked copy - before you start making changes in the new kernel source tree.

Once you are running on the new kernel, you can set and get EAs in the snia namespace as shown in this simple example.

$ touch testfile
$ setfattr -n snia.test -v "hello world" testfile
$ getfattr -n snia.test testfile
# file: testfile
snia.test="hello world"

$

In the next example I set and get the metadata (called fields in XAM terminology) for a sample XAM XSet. A field has a name and a value, and the following standard attributes: type, length, binding and readonly. The attributes and values roughly correspond to the fields outputted as an XML document to a persisted XSet by the ReferenceXSetClient example included with the XAM SDK which is available on the SNIA website. For an introduction to the concepts of an XSystem and XSets a good starting place is Mark Carlson's blog.

$ cat xamtest
# file: testfile
snia.xset.hold.type="application/vnd.snia.xam.boolean"
snia.xset.hold.binding="false"
snia.xset.hold.readOnly="true"
snia.xset.hold.length="1"
snia.xset.hold="0"
snia.xset.management.policy.type="application/vnd.snia.xam.string"
snia.xset.management.policy.binding="true"
snia.xset.management.policy.readOnly="true"
snia.xset.management.policy.length="36"
snia.xset.management.policy="org.snia.refvim.default.mgmt.policy"
snia.xset.retention.base.enabled.type="application/vnd.snia.xam.boolean"
snia.xset.retention.base.enabled.binding="true"
snia.xset.retention.base.enabled.readOnly="true"
snia.xset.retention.base.enabled.length="1"
snia.xset.retention.base="1"
snia.xset.retention.base.starttime.type="application/vnd.snia.xam.datetime"
snia.xset.retention.base.starttime.binding="true"
snia.xset.retention.base.starttime.readOnly="true"
snia.xset.retention.base.starttime.length="29"
snia.xset.retention.base.starttime="2009-06-12T22:34:35.350-05:00"
snia.xset.retention.list.base.type="application/vnd.snia.xam.string"
snia.xset.retention.list.base.binding="true"
snia.xset.retention.list.base.readOnly="true"
snia.xset.retention.list.base.length="4"
snia.xset.retention.list.base="base"
snia.xset.retention.list.event.type="application/vnd.snia.xam.string"
snia.xset.retention.list.event.binding="true"
snia.xset.retention.list.event.readOnly="true"
snia.xset.retention.list.event.length="5"
snia.xset.retention.list.event="event"
snia.xset.time.access.type="application/vnd.snia.xam.datetime"
snia.xset.time.access.binding="false"
snia.xset.time.access.readOnly="true"
snia.xset.time.access.length="29"
snia.xset.time.access="2009-06-12T22:34:35.350-05:00"
snia.xset.time.commit.type="application/vnd.snia.xam.datetime"
snia.xset.time.commit.binding="false"
snia.xset.time.commit.readOnly="true"
snia.xset.time.commit.length="29"
snia.xset.time.commit="2009-06-12T22:34:35.350-05:00"
snia.xset.time.access.type="application/vnd.snia.xam.datetime"
snia.xset.time.access.binding="false"
snia.xset.time.access.readOnly="true"
snia.xset.time.access.length="29"
snia.xset.time.access="2009-06-12T22:34:35.350-05:00"
snia.xset.time.commit.type="application/vnd.snia.xam.datetime"
snia.xset.time.commit.binding="false"
snia.xset.time.commit.readOnly="true"
snia.xset.time.commit.length="29"
snia.xset.time.commit="2009-06-12T22:34:35.350-05:00"
snia.xset.time.creation.type="application/vnd.snia.xam.datetime"
snia.xset.time.creation.binding="true"
snia.xset.time.creation.readOnly="true"
snia.xset.time.creation.length="29"
snia.xset.time.creation="2009-06-12T22:34:35.331-05:00"
snia.xset.time.xuid.type="application/vnd.snia.xam.datetime"
snia.xset.time.xuid.binding="false"
snia.xset.time.xuid.readOnly="true"
snia.xset.time.xuid.length="29"
snia.xset.time.xuid="2009-06-12T22:34:35.331-05:00"
snia.xset.xuid.type="application/vnd.snia.xam.xuid"
snia.xset.xuid.binding="true"
snia.xset.xuid.readOnly="true"
snia.xset.xuid.length="30"
snia.xset.xuid="AAA6AwAeVG4xMjQ0ODY0MDc1MzU4AQWDedILNL7Q"

$ setfattr --restore=xamtest

$ getfattr -m snia. -d testfile
# file: testfile
snia.test="hello world"
snia.xset.hold="0"
snia.xset.hold.binding="false"
snia.xset.hold.length="1"
snia.xset.hold.readOnly="true"
snia.xset.hold.type="\"application/vnd.snia.xam.boolean\" "
snia.xset.management.policy="org.snia.refvim.default.mgmt.policy"
snia.xset.management.policy.binding="true"
snia.xset.management.policy.length="36"
snia.xset.management.policy.readOnly="true"
snia.xset.management.policy.type="application/vnd.snia.xam.string"
snia.xset.retention.base="1"
snia.xset.retention.base.enabled.binding="\"true\" "
snia.xset.retention.base.enabled.length="1"
snia.xset.retention.base.enabled.readOnly="true"
snia.xset.retention.base.enabled.type="application/vnd.snia.xam.boolean"
snia.xset.retention.base.starttime="2009-06-12T22:34:35.350-05:00"
snia.xset.retention.base.starttime.binding="true"
snia.xset.retention.base.starttime.length="29"
snia.xset.retention.base.starttime.readOnly="true"
snia.xset.retention.base.starttime.type="\"application/vnd.snia.xam.datetime\" "
snia.xset.retention.list.base="base"
snia.xset.retention.list.base.binding="true"
snia.xset.retention.list.base.length="4"
snia.xset.retention.list.base.readOnly="true"
snia.xset.retention.list.base.type="application/vnd.snia.xam.string"
snia.xset.retention.list.event="event"
snia.xset.retention.list.event.binding="true"
snia.xset.retention.list.event.length="5"
snia.xset.retention.list.event.readOnly="true"
snia.xset.retention.list.event.type="application/vnd.snia.xam.string"
snia.xset.time.access="2009-06-12T22:34:35.350-05:00"
snia.xset.time.access.binding="false"
snia.xset.time.access.length="29"
snia.xset.time.access.readOnly="true"
snia.xset.time.access.type="application/vnd.snia.xam.datetime"
snia.xset.time.commit="2009-06-12T22:34:35.350-05:00"
snia.xset.time.commit.binding="false"
snia.xset.time.commit.length="29"
snia.xset.time.commit.readOnly="true"
snia.xset.time.commit.type="application/vnd.snia.xam.datetime"
snia.xset.time.creation="2009-06-12T22:34:35.331-05:00"
snia.xset.time.creation.binding="true"
snia.xset.time.creation.length="29"
snia.xset.time.creation.readOnly="true"
snia.xset.time.creation.type="application/vnd.snia.xam.datetime"
snia.xset.time.xuid="2009-06-12T22:34:35.331-05:00"
snia.xset.time.xuid.binding="false"
snia.xset.time.xuid.length="29"
snia.xset.time.xuid.readOnly="true"
snia.xset.time.xuid.type="application/vnd.snia.xam.datetime"
snia.xset.xuid="AAA6AwAeVG4xMjQ0ODY0MDc1MzU4AQWDedILNL7Q"
snia.xset.xuid.binding="true"
snia.xset.xuid.length="30"
snia.xset.xuid.readOnly="true"
snia.xset.xuid.type="application/vnd.snia.xam.xuid"
$

Obviously this work is a proof-of-concept only. The actual XAM namespace is org.snia.xam.* and not the shortened version, i.e. snia.*, which I used in the proof-of-concept. I have not tested it sufficiently to discover boundary conditions or size limitations yet. I suspect that it might be better to store the snia namespace attributes in their own block(s) rather than sharing an inode or block with the other four namespaces and this is what I intend to prototype next.

If anybody else is interested in working with me in this space I would welcome their involvement.

Fedora 11 nVidia Twinview Support

2009-06-11T13:07:00.040-04:00

Fedora 11 (Leonidas) ships with the nouveau nVidia graphics driver preloaded by default if a nVidia graphics card is detected at install time. Previous versions of Fedora used the older X.Org nv driver.

The nouveau project aims at producing Open Source 3D drivers for nVidia graphics cards. According to the nouveau project Wiki

2D-support is in fairly good shape with EXA acceleration, Xv and Randr12 (think of dual-head, rotations, etc.). Randr12 should work for all cards up to, and including, Geforce 9000 series, although some issues with Geforce 8/9 laptops may still exist, for such issues bug reports should be submitted. Randr12 is now the default. Any 3D functionality that might exist is still unsupported, do not ask for instructions to try it. Also, VT switching while X is running is considered lucky."

Well, I certainly quickly ran into the VT switching issue! It worked but not consistently.

Unfortunately the nouveau driver currently does not support nVidia TwinView functionality and I suspect that it will be a long time before it does if ever!

To use TwinView with Fedora 11, you have to load the correct nVidia drivers from rpmfusion.org. I described how to do this in detail in a previous post so I will not repeat that information here.

You also need to modify your grub.conf file to include the nopat kernel boot option as shown below.

title Fedora (2.6.29.4-167.fc11.x86_64)
       root (hd0,1)
       kernel /vmlinuz-2.6.29.4-167.fc11.x86_64 ro root=/dev/mapper/vg_ultra-lv_root rhgb quiet nopat
       initrd /initrd-2.6.29.4-167.fc11.x86_64.img

The nopat option is needed for this particular kernel (2.6.29.4) as it appears to still have broken PAT functionality.

For those readers who are unaware of what PAT is, here is a brief explanation. Traditionally page caching was controlled by a CPU feature called Memory Type Range Registers (MTRR). A CPU has a finite and limited set of MTRRs each of which control part of the physical address space. To overcome this limitation and provide a more flexible architecture, Intel and other x86 CPU vendors added a set of bits to page table entries to control how a CPU does page caching. These bits are called the Page Attribute Table (PAT). Incidentally, the 2.6.26 kernel was the first Linux kernel to support PATs.

Unless you rebuild your initial ramdisk (initrd), the nouveau driver will remain loaded in the kernel. I prefer not to have the nouveau driver loaded in my kernel if I am not using it so I added nouveau to the list of blacklisted drivers in /etc/modprobe.d/blacklist.conf and rebuild initrd.

# mv /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img.backup
# mkinitrd -v /boot/initrd-`uname -r`.img `uname -r`
Creating initramfs
Looking for driver for /dev/mapper/vg_ultra-lv_root in /sys/block/dm-0
Found DeviceMapper component dm-0
Looking for deps of module scsi:t-0x00
Looking for deps of module pci:v00008086d00002922sv00008086sd00005442bc01sc06i01
Looking for driver for /dev/mapper/vg_ultra-lv_swap in /sys/block/dm-1
Found DeviceMapper component dm-1
Using modules:
Building initrd in /tmp/initrd.txR0Kd
/sbin/nash -> /tmp/initrd.txR0Kd/bin/nash
  /usr/lib64/libnash.so.6.0.86 -> /tmp/initrd.txR0Kd/usr/lib64/libnash.so.6.0.86
  /usr/lib64/libbdevid.so.6.0.86 -> /tmp/initrd.txR0Kd/usr/lib64/libbdevid.so.6.0.86
  /lib64/libdevmapper.so.1.02 -> /tmp/initrd.txR0Kd/lib64/libdevmapper.so.1.02
  /lib64/libparted-1.8.so.8 -> /tmp/initrd.txR0Kd/lib64/libparted-1.8.so.8
    /lib64//libparted-1.8.so.8.0.0 -> /tmp/initrd.txR0Kd/lib64//libparted-1.8.so.8.0.0
  /lib64/libblkid.so.1 -> /tmp/initrd.txR0Kd/lib64/libblkid.so.1
    /lib64//libblkid.so.1.0 -> /tmp/initrd.txR0Kd/lib64//libblkid.so.1.0
  /lib64/libselinux.so.1 -> /tmp/initrd.txR0Kd/lib64/libselinux.so.1
  /lib64/libsepol.so.1 -> /tmp/initrd.txR0Kd/lib64/libsepol.so.1
  /lib64/libuuid.so.1 -> /tmp/initrd.txR0Kd/lib64/libuuid.so.1
    /lib64//libuuid.so.1.2 -> /tmp/initrd.txR0Kd/lib64//libuuid.so.1.2
  /lib64/libpopt.so.0 -> /tmp/initrd.txR0Kd/lib64/libpopt.so.0
    /lib64//libpopt.so.0.0.0 -> /tmp/initrd.txR0Kd/lib64//libpopt.so.0.0.0
  /lib64/libresolv.so.2 -> /tmp/initrd.txR0Kd/lib64/libresolv.so.2
    /lib64//libresolv-2.10.1.so -> /tmp/initrd.txR0Kd/lib64//libresolv-2.10.1.so
      /lib64/libc.so.6 -> /tmp/initrd.txR0Kd/lib64/libc.so.6
        /lib64//libc-2.10.1.so -> /tmp/initrd.txR0Kd/lib64//libc-2.10.1.so
          /lib64/ld-linux-x86-64.so.2 -> /tmp/initrd.txR0Kd/lib64/ld-linux-x86-64.so.2
           /lib64//ld-2.10.1.so -> /tmp/initrd.txR0Kd/lib64//ld-2.10.1.so
  /lib64/libdl.so.2 -> /tmp/initrd.txR0Kd/lib64/libdl.so.2
    /lib64//libdl-2.10.1.so -> /tmp/initrd.txR0Kd/lib64//libdl-2.10.1.so
  /usr/lib64/libelf.so.1 -> /tmp/initrd.txR0Kd/usr/lib64/libelf.so.1
    /usr/lib64//libelf-0.141.so -> /tmp/initrd.txR0Kd/usr/lib64//libelf-0.141.so
  /usr/lib64/libnl.so.1 -> /tmp/initrd.txR0Kd/usr/lib64/libnl.so.1
    /usr/lib64//libnl.so.1.1 -> /tmp/initrd.txR0Kd/usr/lib64//libnl.so.1.1
  /lib64/libm.so.6 -> /tmp/initrd.txR0Kd/lib64/libm.so.6
    /lib64//libm-2.10.1.so -> /tmp/initrd.txR0Kd/lib64//libm-2.10.1.so
  /lib64/libgcc_s.so.1 -> /tmp/initrd.txR0Kd/lib64/libgcc_s.so.1
    /lib64//libgcc_s-4.4.0-20090506.so.1 -> /tmp/initrd.txR0Kd/lib64//libgcc_s-4.4.0-20090506.so.1
  /lib64/libreadline.so.5 -> /tmp/initrd.txR0Kd/lib64/libreadline.so.5
    /lib64//libreadline.so.5.2 -> /tmp/initrd.txR0Kd/lib64//libreadline.so.5.2
  /lib64/librt.so.1 -> /tmp/initrd.txR0Kd/lib64/librt.so.1
    /lib64//librt-2.10.1.so -> /tmp/initrd.txR0Kd/lib64//librt-2.10.1.so
      /lib64/libpthread.so.0 -> /tmp/initrd.txR0Kd/lib64/libpthread.so.0
        /lib64//libpthread-2.10.1.so -> /tmp/initrd.txR0Kd/lib64//libpthread-2.10.1.so
  /lib64/libtinfo.so.5 -> /tmp/initrd.txR0Kd/lib64/libtinfo.so.5
    /lib64//libtinfo.so.5.7 -> /tmp/initrd.txR0Kd/lib64//libtinfo.so.5.7
/sbin/modprobe -> /tmp/initrd.txR0Kd/bin/modprobe
/sbin/rmmod -> /tmp/initrd.txR0Kd/bin/rmmod
resolving for MODULES
and that has items of
resolving for availmodules
and that has items of
/sbin/lvm -> /tmp/initrd.txR0Kd/bin/lvm
/etc/lvm -> /tmp/initrd.txR0Kd/etc/lvm
`/etc/lvm/lvm.conf' -> `/tmp/initrd.txR0Kd/etc/lvm/lvm.conf'
/etc/sysconfig/keyboard -> /tmp/initrd.txR0Kd/etc/sysconfig/keyboard
/bin/loadkeys -> /tmp/initrd.txR0Kd/bin/loadkeys
/lib/kbd/keymaps/i386/qwerty/us.map.gz -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/i386/qwerty/us.map.gz
/lib/kbd/keymaps/i386/include/qwerty-layout.inc -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/i386/include/qwerty-layout.inc
/lib/kbd/keymaps/i386/include/compose.inc -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/i386/include/compose.inc
/lib/kbd/keymaps/include/compose.latin4 -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/include/compose.latin4
/lib/kbd/keymaps/include/compose.8859_8 -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/include/compose.8859_8
/lib/kbd/keymaps/include/compose.latin1 -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/include/compose.latin1
/lib/kbd/keymaps/include/compose.latin3 -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/include/compose.latin3
/lib/kbd/keymaps/include/compose.8859_7 -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/include/compose.8859_7
/lib/kbd/keymaps/include/compose.latin2 -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/include/compose.latin2
/lib/kbd/keymaps/include/compose.latin -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/include/compose.latin
/lib/kbd/keymaps/i386/include/linux-with-alt-and-altgr.inc -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/i386/include/linux-with-alt-and-altgr.inc
/lib/kbd/keymaps/i386/include/linux-keys-bare.inc -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/i386/include/linux-keys-bare.inc
/lib/kbd/keymaps/i386/include/euro1.map.gz -> /tmp/initrd.txR0Kd/lib/kbd/keymaps/i386/include/euro1.map.gz
/etc/sysconfig/i18n -> /tmp/initrd.txR0Kd/etc/sysconfig/i18n
/bin/setfont -> /tmp/initrd.txR0Kd/bin/setfont
/lib/kbd/consolefonts/latarcyrheb-sun16.psfu.gz -> /tmp/initrd.txR0Kd/lib/kbd/consolefonts/latarcyrheb-sun16.psfu.gz
/lib/udev/console_init -> /tmp/initrd.txR0Kd/lib/udev/console_init
  /lib64/libglib-2.0.so.0 -> /tmp/initrd.txR0Kd/lib64/libglib-2.0.so.0
    /lib64//libglib-2.0.so.0.2000.1 -> /tmp/initrd.txR0Kd/lib64//libglib-2.0.so.0.2000.1
probing for modules for drm device card0
Adding graphics device card0
Looking for deps of module pci:v000010DEd00000640sv00003842sd0000C959bc03sc00i00: i2c-core nvidia
Adding module i2c-core
Adding module nvidia
resolving for GRAPHICSMODS
and that has items of i2c-core nvidia
Looking for deps of module i2c-core
Looking for deps of module nvidia: i2c-core
copy from `/lib/modules/2.6.29.4-167.fc11.x86_64/kernel/drivers/i2c/i2c-core.ko' [elf64-x86-64] to `/tmp/initrd.txR0Kd/lib/modules/2.6.29.4-167.fc11.x86_64/i2c-core.ko' [elf64-x86-64]
copy from `/lib/modules/2.6.29.4-167.fc11.x86_64/extra/nvidia-173xx/nvidia.ko' [elf64-x86-64] to `/tmp/initrd.txR0Kd/lib/modules/2.6.29.4-167.fc11.x86_64/nvidia.ko' [elf64-x86-64]
/sbin/plymouthd -> /tmp/initrd.txR0Kd/bin/plymouthd
........
........
Adding module scsi_wait_scan
copy from `/lib/modules/2.6.29.4-167.fc11.x86_64/kernel/drivers/scsi/scsi_wait_scan.ko' [elf64-x86-64] to `/tmp/initrd.txR0Kd/lib/modules/2.6.29.4-167.fc11.x86_64/scsi_wait_scan.ko' [elf64-x86-64]
This initrd uses dynamic shared objects.
Adding dynamic linker configuration files.
/etc/ld.so.conf -> /tmp/initrd.txR0Kd/etc/ld.so.conf
/etc/ld.so.conf.d/kernel-2.6.29.4-167.fc11.x86_64.conf -> /tmp/initrd.txR0Kd/etc/ld.so.conf.d/kernel-2.6.29.4-167.fc11.x86_64.conf
/etc/ld.so.conf.d/mysql-x86_64.conf -> /tmp/initrd.txR0Kd/etc/ld.so.conf.d/mysql-x86_64.conf
/etc/ld.so.conf.d/nvidia-lib64.conf -> /tmp/initrd.txR0Kd/etc/ld.so.conf.d/nvidia-lib64.conf
/etc/ld.so.conf.d/xulrunner-64.conf -> /tmp/initrd.txR0Kd/etc/ld.so.conf.d/xulrunner-64.conf
/etc/ld.so.conf.d/qt-x86_64.conf -> /tmp/initrd.txR0Kd/etc/ld.so.conf.d/qt-x86_64.conf
Running ldconfig
#

After rebooting your system, if you use dmesg or lsmod, you will see that the nvidia driver was loaded instead of the nouveau driver.

You will also see that for some reason Plymouth no longer runs with a graphical splash screen if it previously did so. Plymouth is the replacement for the old RedHat Graphical Boot (RHGB). It was written by Ray Strode, Kristian Hogsberg and Peter Jones of Redhat and first shipped in Fedora 10.

Finally, you do not need to modify your xorg.conf file for Fedora 11. It should just work.

Linux Security Capabilities

2009-05-28T11:14:00.082-04:00

In earlier times, the standard security model for GNU/Linux and Unix operating systems gave general users a minimal set of privileges, while granting full privileges to a single user account, i.e. root, that was used to administer the system and users, install software, mount and unmount filesystems, loading kernel modules, bind a process to a privileged port and run many services.

This dependence upon the root account to perform all actions requiring privilege was recognized to be somewhat dangerous in that it was all or nothing and not suited to compartmentalization of roles. Furthermore, it increased the risk of vulnerabilities within a setuid application which may only require root privileges for a very small fraction of its activity such as opening a system file or binding to a privileged port.

This risk was well understood within the open systems community. As a result, IEEE Std.1003.1e (aka POSIX.1e or POSIX.6) was a major effect started in 1995 to develop a standardized set of security interfaces for conforming systems which included access control lists (ACL), audit, separation of privilege (capabilities), mandatory access control (MAC) and information labels.

The work was terminated by IEEE's RevCon in 1998 at draft 17 of the document due to lack of consensus (mostly because of conflicting existing practice.) While the formal standards effort failed, since then much of the draft standard has made its way in the Linux kernel including capabilities which this post explores.

First, what do we mean by Linux capabilities? It is basically an extended verion of the capabilities model described in the draft POSIX.1e standard. Readers familiar with VMS or versions of Unix which include Trusted Computing Base (TCB) will recognize it as being somewhat analogous to as privileges. These capabilities partition the set of root privlileges into a set of distinct logical privileges which may be granted or assigned to processes, users, filesystems and more. As an aside, the term capability originated in a 1966 paper by Jack Dennis and Earl Van Horn (CACM vol 9, #3, pp 143-155, March 1966.) Capabilities can be implemented in many ways including via hardware tags, cryptography, within a programming language (e.g.Java) or using protected address space. For a introduction to capability-based mechanisms go here. Linux uses protected address space and extended file attributes to implement capabilities.

A capability flag is an attribute of a capability. There are three capability flags, named permitted (p), effective (e) and inherited (i), associated with each of the capabilities which, by the way, are documented in the <security/capability.h> header.

A capability set consists of a 64-bit bitmap (prior to libcap 2.03 a bitmap was 32-bits.) A process has a capability state consisting of three capability sets, i.e. inheritable (I), permitted (P) and effective (E), with each capability flag implemented as a bit in each of the three bitmaps.

Whenever a process attempts a privileged operation, the kernel checks that the appropriate bit in the effective set is set. For example, when a process tries to set the monotomic clock, the kernel first checks that the CAP_SYS_TIME bit is set in the process effective set.

The permitted set indicates what capabilities a process can use. A process can have capabilities set in the permitted set that are not in the effective set. This indicates that the process has temporarily disabled this capability. A process is allowed to set its effective set bit only if it is available in the permitted set. The distinction between permitted and effective exists so that a process can bracket operations that need privilege.

The inheritable capabilities are the capabilities of the current process that should be inherited by a program executed by the current process. The permitted set of a process is masked against the inheritable set during exec, while child processes and threads are given an exact copy of the capabilities of the parent process.

Rather than going into a detailed explaination of forced sets and capability calculations, I refer you to Chris Friedhoff's excellent explanation. However, I have summarized the rules in the following diagram.

For our first example of modifying capabilities consider the ping utility. Many people do not realise that it is a setuid executable on most GNU/Linux systems. It needs to be so because it needs privilege to write the type of packages it uses to probe the network.

$ ls -al /bin/ping
-rwsr-xr-x 1 root root 41784 2008-09-26 02:02 /bin/ping

If you copy ping, it loses its setuid bit and fails to work

$ cp /bin/ping .
$ ls -al ping
-rwxr-xr-x 1 fpm fpm 41784 2009-05-29 20:26 ping
$ ./ping localhost
ping: icmp open socket: Operation not permitted

but works if you become root

# ./ping -c1 localhost
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=1 ttl=64 time=0.026 ms

--- localhost.localdomain ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.026/0.026/0.026/0.000 ms

as root add CAP_NET_RAW to the permitted capability and set the legacy effective bit

# /usr/sbin/setcap cap_net_raw=ep ./ping

check what capabilities ping now has

# /usr/sbin/getcap ./ping
./ping = cap_net_raw+ep

and this time ping works without being setuid

$ ls -al ./ping
-rwxr-xr-x 1 fpm fpm 41784 2009-05-29 20:26 ./ping
$ ./ping -c1 localhost
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=1 ttl=64 time=0.026 ms

--- localhost.localdomain ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.026/0.026/0.026/0.000 ms

Another scenario might be that we want to remove the setuid bit from ping but enable a non-root user to use it. We can do this using the PAM pam_cap module which is which is part of the libcap package. The default configuration file for this PAM module is /etc/security/capability.conf but that can be overwritten using a config=filename argument. Another argument which you can pass to the module is debug. Any errors are written to the error log file for this module which is pam_cap.

The following example shows how to configure pam_cap to allow the user test to use ping

cat /security/capability.conf
#
# /etc/security/capability.conf
# last edit FPM 05/29/2009
#

## user 'test' can use ping via inheritance
cap_net_raw test

## everyone else gets no inheritable capabilities
none *


next insure that pam_cap.so is required by su

cat /etc/pam.d/su
#%PAM-1.0
auth            sufficient      pam_rootok.so
# Uncomment the following line to implicitly trust users in the "wheel" group.
#auth           sufficient      pam_wheel.so trust use_uid
# Uncomment the following line to require a user to be in the "wheel" group.
#auth           required        pam_wheel.so use_uid
# FPM added pam_cap.so  5/29/2009
auth            required        pam_cap.so  debug
auth            include         system-auth
account         sufficient      pam_succeed_if.so uid = 0 use_uid quiet
account         include         system-auth
password        include         system-auth
session         include         system-auth
session         optional        pam_xauth.so

now set up the capabilities for the  ping that we are going to use
no legacy effective bit (e), no enabled privilege

# /usr/sbin/setcap -r ./ping
# /usr/sbin/setcap cap_net_raw=p ./ping
# /usr/sbin/getcap ./ping
./ping = cap_net_raw+p

can I ping as fpm? no.

$ id -un
fpm
$ cd ~test
$ ./ping -q -c1 localhost
ping: icmp open socket: Operation not permitted

can I ping as test? yes.

$ su - test
Password:
$ id -un
test
$ ./ping -q -c1 localhost
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.

--- localhost.localdomain ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.024/0.024/0.024/0.000 ms

The undocumented /usr/sbin/capsh utility can be used to manage capabilities and reduce the potential for vulnerabilities in shell scripts. It is a simple wrapper around the bash shell which can be used to raise and lower both the bset and pI capabilities before invoking /bin/bash.

using the previous ping example which was given CAP_NET_RAW

$ id -nu
root
$ /usr/sbin/getcap ./ping
./ping = cap_net_raw+ep

now use capsh to drop CAP_NET_RAW and change to uid 500 (fpm)
the operation is no longer permitted

$ /usr/sbin/capsh --drop=cap_net_raw --uid=500 --
$ id -nu
fpm
$ ./ping -q -c1 localhost
bash: ./ping: Operation not permitted

permanently drop CAP_NET_RAW

$ /usr/sbin/setcap -r ./ping

and invoke ping using capsh --caps

$ capsh --caps="cap_net_raw-ep" -- -c "./ping -c1 -q localhost"
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.

--- localhost.localdomain ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.056/0.056/0.056/0.000 ms
 
use capsh to print out capabilities

$ id -un
fpm
$ /usr/sbin/capsh --print
Current: =
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,
cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,
cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,
cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,
cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,
cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin
Securebits: 00/0x0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=500

There are many other options to capsh and you probably should spend some time experimenting with them. Invoke capsh with the --help option to see the full list. Note that setuid shell scripts do have capabilities.

All the examples shown above work on Fedora 10 which uses libcap 2.10. If you wish to experiment on other GNU/Linux distributions, it must have a kernel >= 2.6.24 built with CONFIG_FILE_CAPABILITIES=y, a version of libcap >=2.08 and a filesystem that supports extended attributes e.g. ext3 or ext4. Note that CONFIG_CAPABILITIES=y should no longer be required.

Why are not more people aware of and using capabilities? I believe that poor documentation is the primary reason. For example, Fedora 10 is missing the man pages for getpcaps, capsh and pam_cap, and the Fedora Security Guide does not even mention capabilities (or for that matter ACLs!) I cannot but help thinking that capabilities are treated as the poor cousin to SELinux by the Fedora Project.

Well this blog post is getting a bit long so I am going to stop. I have barely scratched the surface of capabilities. After Fedora 11 ships, I will cover capabilities from a programming perspective. Meanwhile, if you want to keep current with what is happening with Linux capabilities, Andrew Morgan's Fully Capable is the one to bookmark.

Windows Parallel Filesystems

2009-05-26T13:27:00.031-04:00

I recently was involved in some development work for a quasi-parallel filesystem for Microsoft Windows. As a result of that involvement my interest was piqued and I decided to do so research on what the state of research and development is in the field of parallel filesystems designed specifically for Microsoft Windows.

First a quick review of what I mean by a parallel file system. There are any number of different types of parallel file systems available. Some allow multiple systems and applications to share common pools of storage as in a clusered filesystem. Some split the data across two or more nodes to improve access time and redundancy. Other variants split files into lots of small chunks, stores these chunks on different disks in a round-robin fashion, and re-combine them upon reading to get back the original file.

The earliest instance of Microsoft Windows-specific parallel fileystem that I have found to date is the parallel filesystem developed by the ARGOS group at Universidad Carlos 111 de Madrid, Madrid, ES. This research group developed a prototype of a parallel file system for a network of Microsft Windows nodes which they called WinPFS. They presented their work at COSET 2004 and a number of other workshops. WinPFS was implemented as a new fileyystem type fully integrated within the Microsoft Windows kernel. This has the advantage that no modification or recompilation of user applications is needed to take advantage of the parallel filesystem.

The goal of this research group was to build a parallel file system for networks of Microsoft Windows computers using Microsoft Windows shared folders to access remote data in parallel. The implementation is based on file system redirectors which redirect requests to remote nodes using UNC (Universal Naming Convention) and the SMB and/or CIFS protocols. WinPFS is registered as a virtual remote file system and access to remote data is through a new shared folder \\PFS. The basic file operation primatives are: create, read, write, and create directory.

The prototype was developed on the Windows XP platform, and has been tested with a cluster of seven Windows XP nodes and a Windows 2003 Server node in various configurations. Maximum throughput for write operations were 250 Mbit/s and 1200 Mbit/s for read operations. The research team reported that the bottleneck for writes was the disks and for reads was the network. As far as I can tell this project is no longer under active development.

Another interesting experimental parallel file system for Microsoft Windows was developed by Lungpin Yeh, Juei-Ting Sun, Sheng-Kai Hung & Yarsun Hsu of the National Tsing-Hua University, ROC. They presented a paper on their initial implementation at the 2007 High Performance Computation Conference.

Their implementation consists of three main components: A metadata server, I/O node daemons (IOD) and an API library (libwpvfs) to enable users to develop their own applications on top of the parallel file system. libwpvfs, which uses the .NET framework, handles all communications with the metadata server and the IODs and supports six basic file operation primatives: open, create, read, write, seek, and close.

The metadata server maps each filename to a unique 64-bit file ID and maintains other information about the file such as striping size, node location and count. While one metadata server is obviously a possible single point of failure, mirroring and redundancy can be used to improve relaibility.

When an application wants to access a file, the library first connects to the metadata server to acquire the metadata for the file. The library then connects to the appropriate I/O node daemons listed in the metadata, and these node I/O daemons then access the correct file and send the appropriate stripes back to the library for handoff to the calling application.

Under test conditions with 5 nodes, a maximum of 109 MB/s for writes and 85 MB/s for reads was measured.

If one of the nodes fails the parallel file system can still work, i.e. the library can make use of the remaining healthy nodes, but the data within the failed node is not available anymore. To overcome this limitation, the research team plan to add node fault tolerance in a future version so that the parallel filesystem will fully work even if some of the I/O nodes fail.

I am certain that there are other examples of parallel filesystems specifically targeted at networks of Microsoft Windows computers which have been developed by the academic research community and probably by Microsoft itself. As I come across such work, I plan to add details to this post.

In conclusion, a parallel filesystem can not only provide a larger and/or global storage space for applications by combining storage resources on different nodes but also increase the performance of an application because the application can access files in parallel.

Traditionally this kind of solution was only available for Unix and GNU/Linux systems. Examples include PVFS, GPFS, ParFiSys and Vesta. However, because of the advantages that parallel fileystems can offer, and because of the ubiquity of Microsoft Windows computers I expect to see a number of commercial parallel filesystems targeted at networks of Microsoft Windows computers emerge over the next 5 to 10 years.

P.S. The graphics in this post were copied from published papers of the respective research teams.

KSH93 Message Localization

2009-05-23T15:56:00.046-04:00

The current version of ksh93 (93t+ 2009-05-01) supports localization of internal error messages and getopts messages but localization of user messages in shell scripts is flawed.

For a project I am working on, I needed to be able to supply localized messages for a small number of shell scripts and thus found myself in the bowels of ksh93 figuring out how to make it so.

First a word of warning. For the unwary, modifying the source code for ksh93 and libast is not for the faint-hearted. It is a fairly complex code base with many levels of abstraction in places. This post assumes that you are reasonably familiar with libast and ksh93 sources and know how to rebuild ksh93 from these sources.

Before getting down to the source code modifications, a discussion of the design decisions and trade offs is in order. The major design decision was to use the POSIX message catalog (catopen(), catgets(), catclose()) localization model rather than the GNU gettext() localization model because (1) libast already provided the three cat* APIs as wrappers around the equivalent underlying libc APIs and more importantly (2) I did not want to contaminate the ksh93 sources which have an Common Public Licence V1.0 license with GNU gettext() sources which have a GPL license.

The second major design decision I made was that the message catalogs would have the same name as the shell script and and would reside in the system default message catalog location rather than locally (although this can be overwritten by the NLSPATH environmental variable.) My current development system is Fedora 11 (Leonidas) x86_64. There the default location for localized messages is /usr/share/locale/%l/LC_MESSAGES/. Thus, for example, the french message catalog for a shell script named fpm would be /usr/share/locale/fr/LC_MESSAGES/fpm assuming LC_MESSAGES was set to fr_FR.UTF-8 or something equivalent.

The final major design decision I made was not to require the use of set and message numbers for identifying message strings as is normal in the POSIX message catalog model but rather to follow the gettext() model whereby a message string conveys sufficient information to find it's equivalent localized string.

The way this works is that you must provide a C locale message catalog with a single set of messages (set 1) which ksh93 reads to determine the message number. It then uses that message number to retrieve the same message from set 1 of the localized message catalog. This means that the C locale message catalog and the various other localized message catalogs must be kept synchronized for correct localized message retrieval. Note that ksh93 defaults to the original message string if it cannot find that message string in the C locale message catalog or in the localized message catalog.

Modifications are only necessary to two source files. Add the following function to .../lib/libast/translate.c.

/* 
 * Support internationalized messages in $"..." format
 * Assumes that catalog name is exactly the same as the script name 
 */
char*
fpm_translate(const char* cmd, const char* msg)
{
    register char*  r;
    register char*  u;
    char *          t;
    nl_catd         d;
    int             n;
    int             msgno;
    char *          j;
    char *          s;

    r = (char*)msg;

    if (!msg) return NiL;

    if (cmd && (t = strrchr(cmd, '/')))
        cmd = (const char*)(t + 1);
    
    /* first locate the message in the C locale and determine message number */
    u = setlocale(LC_MESSAGES, NiL);
    j = strdup(u);
    setlocale(LC_MESSAGES, "C");
    if (( d = find("C", cmd)) != NOCAT )
    {
        n = 0;
        msgno = 0;
        for (;;)
        {
            n++;
            if ((s = catgets(d, 1, n, NULL)) == NULL)
                break;
            if (!strcmp(msg, s))
            {
                msgno = n;
                break;
            }
        }
        catclose(d);
    }
    setlocale(LC_MESSAGES, j);
    free(j);

    /* then use that message number to retrieve equivalent L10N message */
    if (( d = find(setlocale(LC_MESSAGES,NiL), cmd)) != NOCAT )
    {
        s = catgets(d, 1,  msgno, msg);
        r = strdup(s);
        catclose(d);
    }

    return r;
}

and modify sh_endword() in .../cmd/ksh93/sh/lex.c as follows:

#if NO_LOCALIZED_MESSAGES 
#if ERROR_VERSION >= 20000317L
                     msg = ERROR_translate(0,error_info.id,0,ep);
#else
#   if ERROR_VERSION >= 20000101L
                     msg = ERROR_translate(error_info.id,ep);
#   else
                     msg = ERROR_translate(ep,2);
#   endif
#endif
#else
                     msg = fpm_translate(shp->st.cmdname, ep);
#endif

Then rebuild ksh93 using .../bin/package make or with your own custom build script.

Here is a simple shell script (demo) which has a localized French message catalog. Use the system gencat to compile the two message catalogs, not UWIN's gencat.

$ cat demo
#!/bin/ksh
name="Finnbarr"
echo "ksh93 message translation example"
echo  $"Hello"
print $"Goodbye"
printf $"Welcome %s\n" $name
print $"No Translation available"

$ cat demo.msg.c
$quote "
$set 1  This is the C locale message set
1 "Hello"
2 "Goodbye"
3 "Welcome %s\\n"

$ cat demo.msg.fr
$quote "
$set 1  This is the French locale message set
1 "Bonjour"
2 "Au Revoir"
3 "Bienvenu %s\\n"

Note the requirement to guard newlines with an backslash.

Here is an example of the output from demo under various locale settings.

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
 
$ ./demo
A simple ksh93 message translation example
Hello
Goodbye
Welcome Finnbarr
No translated message

$ unset LANG
$ locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
 
$ ./demo
A simple ksh93 message translation example
Hello
Goodbye
Welcome Finnbarr
No translated message
 
$ export LC_MESSAGES=fr_FR.UTF-8
$ locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES=fr_FR.UTF-8
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

$ ./demo
A simple ksh93 message translation example
Bonjour
Au Revoir
Bienvenu Finnbarr
No translated message
$

Well, that's about all for now. I have provided sufficient information for you to try out localized message strings with ksh93 for yourself if this is something you need.

Note that this is still a work in progress. I have not yet tested support for multibyte message catalogs (I am not sure if ksh93 even works in multibyte locales!). Since I have hijacked the existing hook for error message catalogs, I probably should figure out how to get localization of ksh93 error messages working again. There may be memory leaks or boundary conditions that I am unaware of. Therefore if you are going to use this modification in a production environment, test and test again!

A useful future enhancement would be to decouple the name of the message catalog from the name of the shell script and also allow message set numbers other than 1. This could be achieved by using a pair of environmental variables or by means of a msgcat compound variable where msgcat.name is the name of the message catalog to be used and msgcat.set is the actual message set to be used.

Have a great Memorial Day, and take a few moments to remember those who have sacrificed so that we can be free.

Comments and feedback are welcome as always.

[2009-09-05] Recently I spent some more time in the bowels of the AST library (libast) and made an interesting discovery regarding message catalog support for shell scripts. I discovered that ../lib/libast/misc/translate.c is where the real problem is located. The translate() function contains a call to catget() but only for message set number 3 (three). This is hardcoded into this call via the AST_MESSAGE_SET define in ../libast/include/ast_std.h. So unless you define $set 3 in your message catalog, internationalization or localization of a ksh93 shell script simply will not work.

Charles Merton Richmond RIP

2009-05-06T11:14:00.002-04:00

Yesterday my old friend and collegue Charles Merton Richmond (Charlie) died at home in Cebu PH of a massive heart attack.

Charlie and I go back a long time. We first met when Lotus Development set up a UNIX porting group in Dublin, Ireland. Charlie was Digital Equipment Corporation's on-site go-to person for the port of Lotus 1-2-3 to DEC Ultrix. At that time Charlie was single-handedly raising his son, Keith, and brought him with him to Dublin for the 6 month assignment.

I was a principal software engineer responsible for the Ultrix, HP and SCO UNIX ports of 1-2-3 so we quickly came to know each other pretty well. The UNIX porting group was a lively and interesting group with many characters and we often had a good evening at the Grave Diggers in Glasnevin. Charlie thoroughly enjoyed Irish pubs and restaurants. The older and more oddball the pub or restaurant the happier he was. He could not get over the fact that the Irish (and English) would put hot curry sauce over french fies or deep fry Mars bars as a dessert.

When the Ultrix port was finished Charles and his son moved back to their high rise apartment in downtown Boston. Since I used visit the Lotus headquarters in Cambridge on a regular basis, I would frequently meet up with Charlie for sushi and drinks. Charles was a true sushi aficionado and I learned a lot about the finer points of sushi and other Japanese foods from him.

Charlie was the person who recruited me to go work fulltime for Digital Equipment Corporation. We ended up working together on localization and internationalization issues for nearly 18 months in Manalapan, NJ and Nashua, NH. He was an amazing driver. Many times we left Manapan NJ at 3.30 in the afternoon and would make downtown Boston by 7.30 that evening.

Charlie also loved motorbikes but was badly injured by a careless driver on the Southeast Expressway in early 2000 while on his way to work. During a followup visit to a specialist, he was again badly injured by a vehicle whose driver failed to see him on the crosswalk!

Over the years Charlie and I worked together on various projects. In 2004 he moved to Cebu where he met and married a wonderful Filipana woman named Ashley. Last year I worked with Charlie in Cebu on a couple of projects.

Charlie and I had a tentative schedule to again work together on a project in Cebu later this year and I was really looking forward to spending time with him again. Alas, that will not now happen.

I shall miss you, Charlie.

Rest in peace.

Porting WaitForSingleObject to Linux - Part 2

2009-04-30T09:38:00.000-04:00

In my last post I discussed the use of WaitForSingleObject in relation to mutexes and possible ways to implement equivalent functionality when porting such code to GNU/Linux. In this post I will describe the use of this API with event objects in Microsoft Windows and suggest possible ways of posting such code to GNU/Linux or Unix.

First, some background on event objects. An event object is just another type of Windows kernel dispatcher object. From a coding prespective, an event object is a synchronization object which encapsulates one or more kernel dispatcher objects and whose synchronization semantics are accessable via WaitForSingleObject and its cousins. At any given time a synchronization object is either nonsignaled or signaled, i.e. the object can only be in one of two possible states.

All of the WaitFor family of APIs including WaitForSingleobject wait on an object handle or handles until some specified criteria is met. The two basic criteria for all these APIs are the signaled state of the object on whose handle it is waiting and a time-out value. Thus a thread which calls this API waits till the specified object enters the signaled state or the specified time-out has expired. Little or no CPU time is used when such a thread is in the wait state.

In the case of events, a CreateEvent or OpenEvent returns a handle to an event object. When an event is in the signaled state it means that that the event has the capacity to release one or more threads waiting for this particular event to be signaled. When an event is in the nonsignaled state it will not release any waiting thread. Initially the state of an event is nonsignaled. An event object's state is set explicitly to signaled by SetEvent or PulseEvent. Event objects are also used in overlapped operations such as reading from a socket, in which case the event object state is set to signaled by the kernel rather than by an application.

Events also come in two reset types. If an event is a manual-reset event, then all WaitForSingleObjects return that wait for that event if so configured. In other words a manual-reset event can trigger action by one or more WaitForSingleObject or its cousins. A manual-reset event object's state must be reset explicitly to nonsignaled by ResetEvent.

For an auto-reset event object, WaitForSingleObject and it's relations reset the state of the event object to nonsignaled before returning. While an auto-reset event is guaranteed to set the event to nonsignaled and release a single thread that is waiting on the event to occu, if more than one thread is waiting for this particular event to occur then which particular thread is released is random.

Consider the following skeleton Microsoft Windows application.

HANDLE hEvent;
HANDLE hMutex;
HANLE  hThread;
....
BOOLEAN  Run = TRUE;
....

main()
{

    hMutex = CreateMutex(0, FALSE, NULL);
    hEvent = CreateEvent( NULL, FALSE, FALSE, NULL);
    .....
    hThread = CreateThread( NULL, 0 (LPTHREAD_START_ROUTINE)SendThread, 0, 0, 0);
    .....

    /* produce and post messages */
    .....

    /* clean up and exit */
    CloseHandle(hEvent);
    CloseHandle(hMutex);
}


DWORD SendThread (LPVOID Parm)
{
    do {
          /* wait for the event */
          WaitForSingleObject( hEvent, INFINITE );

          /* send the message */
          ....
          ....
          /* lock message queue and delete message */
          WaitForSingleObject( hMutex, INFINITE );
          .....
          .....
          ReleaseMutex( hMutex );

    } while ( Run };

}

PostMessage( PMESSAGE Message )
{
    /* lock message queue and add message */
    WaitForSingleObject( hMutex, INFINITE );
    .....
    .....
    ReleaseMutex( hMutex );

    /* signal SendThread that there is a message in the queue */
    SetEvent( hEvent );
}

Assume that the purpose of this application, for whatever reason, is to broadcast UDP messages. A separate thread of execution, i.e SendThread, is created to handle the actual broadcasting. It simple takes whatever messages are in the send message queue amd broadcasts them. If there are no messages in the message queue, the thread waits (yields) until it is signalled that there is one or more messages waiting to be broadcast.

Here is the same application code after it was ported to GNU/Linux using POSIX theads.

pthread_cond_t   hEvent;
pthread_mutex_t  hMutex;
pthread_t        hThread;
....
BOOLEAN  Run = TRUE;
....

main()
{
    pthread_mutexattr_t  mattr;
    pthread_attr_t       attr;

    /* initialize condition variable */
    pthread_cond_init( &hEvent, NULL);

    /* initialize mutex as a recursive type */
    pthread_mutexattr_init( &mattr );
    pthread_mutexattr_settype( &mattr, PTHREAD_MUTEX_RECURSIVE );
    pthread_mutex_init( &hMutex, &mattr );

    /* create thread as detached and process scope */
    pthread_attr_init( &attr);
    pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
    pthread_attr_setscope(&attr, PTHREAD_SCOPE_PROCESS);
    pthread_create( &hThread, &attr, SendThread, 0);

    .....
    /* produce and post messages */
    .....

    /* clean up and exit */
    pthread_attr_destroy( &attr );
    pthread_cond_destroy( &hEvent );
    pthread_mutex_destroy( &hMutex );
}

void *
SendThread ( void *Parm )
{
    do {
          /* wait for the event */
          pthread_mutex_lock( &hMutex );
          pthread_cond_wait( &hEvent, &hMutex );
          pthread_mutex_unlock( &hMutex );

          /* send the message */
          ....
          ....

          /* lock message queue and delete message */
          pthread_mutex_lock( &hMutex );
          .....
          .....
          pthread_mutex_unlock( &hMutex );

    } while ( Run };

}

PostMessage( PMESSAGE Message )
{
    /* lock message queue and add message */
    pthread_mutex_lock( &hMutex );
    .....
    .....
    pthread_cond_signal( &hEvent );    /* signal SendThread */
    pthread_mutex_unlock( &hMutex );

}

*** UNDER CONSTRUCTION ***

Porting WaitForSingleObject to Linux - Part 1

2009-04-29T16:01:00.000-04:00

Recently I was involved in porting a 32-bit application which was initially written for Microsoft Windows NT to GNU/Linux. This application contained a large number of calls to NtWaitForSingleObject and a smaller number of calls to NtWaitForMultipleObject.

Now anybody who has had to port code containing more than a few instances of these particular Win32 APIs, or their close cousins WaitForSingleObjectEx, MsgWaitForMultipleObjects, MsgWaitForMultipleObjectsEx, etc. to Unix or GNU/Linux is probably already shuddering with the recollection of long arduous days and nights of trial and error coding to try and correctly mimic the semantics and functionality of these particular Microsoft Windows specific APIs, but for the reader who has not yet had to attempt to port such an application, this post and my next post may help you save your sanity (and possibly your hair!) sometime in the future.

By the way, both of these APIs are marked deprecated in MSDN by Microsoft but still work as expected in Windows NT and Windows XP. I am not sure about Windows Vista or Windows 7 as I have not tested them on these operating systems. The two deprecated APIs have been replaced by the equivalant APIs WaitForSingleObject and WaitForMultipleObject respectively. For the remainder of this post I shall just discuss the replacement APIs but most of what I say will be valid for either the deprecated or the replacement API.

On first examination WaitForSingleObject seems fairly benign. The description in MSDN states that "This function returns when the specified object is in the signaled state or when the time-out interval elapses". Sounds like a fairly simple and innocuous API, right? Maybe something similar to the POSIX.1 API pthread_cond_timedwait. Well, you are dead wrong and this post and the following will explain why.

WaitForSingleObject and its cousins can wait for a signal from any or all of the following "objects": change notification, console input, event, job, memory resource notification, mutex, process, semaphore, thread and waitable timer and in limited circumstances on files and file I/O. When appropriately signaled, a thread is unblocked and continues. No published standardized API in the GNU/Linux or Unix world comes come to handling this range of objects in a single API.

This is probably the one single area where a Win32 API is better designed than the GNU/Linux or Unix API set. In GNU/Linux and Unix there are specific APIs to wait for different kinds of events: select waits for I/O events such as sockets, semop waits for semaphores, wait waits for child processes, etc.

Unfortunately the semantics of WaitForSingleObject and its cousins varies depending on the different object types. For example, it destroys a thread, but only unlocks a mutex. For this reason understanding Win32 source code from the viewpoint of porting to GNU/Linux can be much more difficult.

Our first example, i.e. locking a mutex (or mutants as they are sometimes called by Windows programmers), is probably the simplest use of WaitForSingleObject and is relatively easy to port to GNU/Linux. Consider the following Windows code which creates 2 threads, and a mutex to control access to an integer variable x

INTEGER x;
HANDLE  hMutex;
...

void function1() 
{
   ...
   WaitForSingleObject(hMutex,INFINATE);
   x= 5;
   ReleaseMutex(hMutex);
   ...
}

void funtion2() 
{
   ...
   WaitForSingleObject(hMutex,INFINATE);
   x = 12;
   ReleaseMutex(hMutex);
   ...
}

int WINAPI WinMain(...) 
{
   hMutex = CreateMutex(NULL,FALSE,NULL);
   ....
   HANDLE hThread1 = CreateThread(..., (LPTHREAD_START_ROUTINE)function1, ...);
   ....
   HANDLE hThread2 = CreateThread(..., (LPTHREAD_START_ROUTINE)function2, ...);
   ....
}

Here is the same code ported to GNU/Linux.

int x;
pthread_mutex_t  hMutex;
...

void *function1( void *Context) 
{
   ...
   pthread_mutex_lock(&hMutex);
   x= 5;
   pthread_mutex_unlock(&hMutex);
   ...
}

void *funtion2( void *Context) 
{
   ...
   pthread_mutex_lock(&hMutex);
   x = 12;
   pthread_mutex_unlock(&hMutex);
   ...
}

int WINAPI WinMain(...) 
{
   pthread_mutexattr_t mAttr;
   pthread_t hThread1;
   pthread_t hThread2;

   pthread:mutexattr_init(&mAttr);
   pthread_mutexattr_settype(&mAttr, PTHREAD_MUTEX_RECURSIVE);
   pthread_mutex_init(&hMutex, &mAttr);
   ....
   pthread_create(&hThread1, NULL, function1, 0);
   ....
   pthread_create(&hThread2, NULL, function2, 0);
   ....

}

I am going to assume that you are familar with POSIX threads and thus the above code snippet should be fairly self-explanatory. When dealing with a mutex, WaitForSingleObject with no timeout is functionally equivalant to pthread_mutex_lock. The only thing of note is that you should probably use a recursive mutex since a Window thread can relock a mutex without first unlocking it.

If the second parameter in WaitForSingleObject is 0 and not INFINITE, you should probably use pthread_mutex_trylock instead of pthread_mutex_lock since WaitForSingleObject does not enter a wait state if the mutex is not, to use Microsoft's term, signaled; it always returns immediately.

If the second parameter in WaitForSingleObject is neither 0 or INFINITE, you should probably use pthread_mutex_timedlock instead of pthread_mutex_lock. Under Windows the time-out interval is in milliseconds. Under GNU/Linux you must pass a populated timespec structure to pthread_mutex_timedlock. Another wrinkle is that the interval clock and thus the clock resolution may or may not be based on the CLOCK_REALTIME clock depending on whether the Timers option is supported or not. Therefore care needs to be taken to ensure that the time interval is correctly converted.

The second wrinkle is that pthread_mutex_timedlock and pthread_mutex_trylock do not work with recursive mutexes which are locked by the same thread as shown by the following example.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <time.h>
#include <pthread.h>

int
main(int  argc,
     char *argv)
{
    pthread_mutex_t hMutex1;
    pthread_mutex_t hMutex2 = PTHREAD_MUTEX_INITIALIZER;
    pthread_mutexattr_t mAttr;
    struct timespec abstime;
    int rc;

    pthread_mutexattr_init(&mAttr);
    pthread_mutexattr_settype(&mAttr, PTHREAD_MUTEX_RECURSIVE_NP);
    pthread_mutex_init(&hMutex1, &mAttr);

    /* lock the mutexes */
    pthread_mutex_lock(&hMutex1);
    pthread_mutex_lock(&hMutex2);

    /* set absolute time to now + 5 seconds */
    clock_gettime(CLOCK_REALTIME, &abstime);
    abstime.tv_sec += 5;

    /* try to relock the first mutex */
    if ((rc = pthread_mutex_timedlock(&hMutex1, &abstime)) > 0 ) {
        fprintf(stderr, "ERROR first pthread_mutex_timedlock. %s\n", strerror(rc));
    }

    /* set absolute time to now + 5 seconds */
    clock_gettime(CLOCK_REALTIME, &abstime);
    abstime.tv_sec += 5;

    /* try to relock the second mutex */
    if ((rc = pthread_mutex_timedlock(&hMutex2, &abstime)) > 0 ) {
        fprintf(stderr, "ERROR second pthread_mutex_timedlock. %s\n", strerror(rc));
    }

    exit(EXIT_SUCCESS);
}

The only way around this problem is to either use non-recursive mutexes or ensure that threads do not attempt to relock recursive mutexes. In my experience Windows programmers tend to be rather loose and free in this respect so careful code inspection is usually required.

Well that is all for now about the use of WaitForSingleObject with mutexes. See my next post for ideas on how to port code which uses this API to handle events.

Fedora 10 Dual Head Graphics Card

2009-03-28T13:50:00.060-04:00

This post details how to install and configure two monitors on a single EVGA GeForce 9500 GT graphics (video) card under Fedora 10.

When I was researching dual head graphics cards for use with Fedora, I came across a lot of forum and blog entries on the subject but most of them contained obsolete or incorrect information - especially about TwinView, Xinerama and the X Window System configuration file. Hopefully this post will provide readers with some useful up-to-date information on the subject.

This particular graphics card (EVGA P/N 01G-P3-N959-TR) was of interest to me for a number of reasons but primarly because the card contains two heads each with DVI-I connectors which would allow me to move away from VGA cabling altogether since my two HP w1907 19" LCD monitors also have DVI-I connectors. In addition two heads on one graphics card enabled me to free up a PCI slot by eliminating the need for a second graphics card in my workstation.

Using TwinView was also of interest to me because it uses a single X screen, i.e. the driver conceals all information about the two separate monitors from the X server, and both monitors share the same frame buffer. This means that I can continue to use functionality such as accelerated OpenGL without a problem.

There are many graphics cards available which contain the nVidia 9500GT GPU but, to date, few of these cards come with dual DVI-I heads. In addition nVidia fully supports their graphics cards on GNU/Linux and Solaris platforms. Another feature of modern nVidia GPUs which is of great interest to me is their support for the CUDA (Compute Unified Device Architecture) architecture which I wish to experiment with a part of a project which I am currently working on for a customer.

You can use a single monitor with this card out-of-the-box using the default nv driver which comes standard with Fedora. However, to use two monitors with this card, you need to download and install the correct package for your specific kernel, i.e.
kmod-nvidia-2.6.27.19-170.2.35.fc10.x86_64-180.29-1.fc10.1.x86_64
in my particular case as my present kernel is 2.6.27.19-170.2.35. Selecting this package will cause three other packages to be also downloaded and installed on your system unless they already exist. These packages are livna-config-display-*, xorg-x11-drv-nvidia-*, and xorg-x11-drv-nvidia-libs-*.

These packages not not available in the regular Fedora repositories. You need to add the rpmfusion repositories to /etc/yum.conf or to the /etc/yum.repos.d subdirectory. Here is one way of installed the repository configuration files into /etc/yum.repos.d.

rpm -ivh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm
rpm -ivh http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-rpmfusion-*

Before you actually download these packages, make a copy of your X Server configuration file i.e. /etc/X11/xorg.conf if one exists. On Fedora 10, it may not if you have a very simple setup. Don't say that you were not warned!

You then should reboot your system and use the nVidia configuration utility either from the command line (/usr/bin/nvidia-settings) or via the /Applications/System Tools/nVidia Display Settings menu option to configure the dual monitor setup.

The following is a screenshot of the nVidia configuration utility after invocation.
and here is a screenshot of the utility showing the X Server display Configuration panel.

There are limits to how you can configure your setup. You can have only one virtual screen per graphics card. Thus zaphod mode, i.e. the ability to assign each monitor on a device to a different screen, is not supported nor can you combine monitors from multiple graphic cards into a single screen.

Unless you modify your X Server configuration file all settings will be lost upon the next reboot or restart of the X Server. For this reason the utility has options to create a new X Server configuration file (in my case /etc/X11/xorg.conf) or to merge with your existing configuration file.

For completeness, here is a copy of the /etc/X11/xorg.conf which I currently use with this configuration.

#
# FPMurphy 3/28/2009 Twinview EVGA 9500 GT
#

Section "ServerLayout"
Identifier    "Layout0"
Screen    0   "Screen0" 0 0
InputDevice   "Keyboard0" "CoreKeyboard"
InputDevice   "Mouse0" "CorePointer"
EndSection

Section "Files"
FontPath      "/usr/share/fonts/default/Type1"
ModulePath    "/usr/lib64/xorg/modules/extensions/nvidia"
ModulePath    "/usr/lib64/xorg/modules"

EndSection

Section "Module"
Load          "dbe"
Load          "extmod"
Load          "type1"
Load          "freetype"
Load          "glx"
EndSection

Section "ServerFlags"
Option        "Xinerama" "0"
EndSection

Section "InputDevice"
Identifier     "Mouse0"
Driver         "mouse"
Option         "Protocol" "auto"
Option         "Device" "/dev/input/mice"
Option         "ZAxisMapping" "4 5 6 7"
EndSection

Section "InputDevice"
Identifier     "Keyboard0"
Driver         "kbd"
Option         "XkbLayout" "us"
Option         "XkbModel" "pc105"
EndSection

Section "Monitor"
Identifier     "Monitor0"
VendorName     "HP"
ModelName      "w1907"
HorizSync       24.0 - 83.0
VertRefresh     50.0 - 76.0
Option         "DPMS"
EndSection

Section "Device"
Identifier     "Device0"
Driver         "nvidia"
VendorName     "NVIDIA Corporation"
BoardName      "GeForce 9500 GT"
EndSection

Section "Screen"
Identifier     "Screen0"
Device         "Device0"
Monitor        "Monitor0"
DefaultDepth    24
Option         "TwinView" "1"
Option         "TwinViewXineramaInfoOrder" "DFP-1"
Option         "metamodes" "DFP-0: nvidia-auto-select +1440+0, DFP-1: nvidia-auto-select +0+0"
SubSection     "Display"
   Depth       24
EndSubSection
EndSection

If the nVidia GLX module loaded correctly, you should able to run /usr/bin/glxgears successfully. If it fails to run, check your /var/log/Xorg.O.log file to see if the nVidia libglx library is being loaded or not. You should see something like the following

(II) Loading /usr/lib64/xorg/modules/extensions/nvidia//libglx.so
(II) Module glx: vendor="NVIDIA Corporation"
      compiled for 4.0.2, module version = 1.0.0
      Module class: X.Org Server Extension
(II) NVIDIA GLX Module  173.14.18  Mon Mar  2 15:55:02 PST 2009
(II) Loading extension GLX

If you do not see that specific path for the nVidia libglx.so you need to go back and fix up the ModulePath lines in your xorg.conf. Note that the order of the ModulePath lines is important as each path is searched in turn from first to last.

With this configuration the output from the xrandr utility should be something to this:

#xrandr -q
Screen 0: minimum 2880 x 900, current 2880 x 900, maximum 2880 x 900
default connected 2880x900+0+0 0mm x 0mm
 2880x900       50.0*
#

If you are unfamiliar with xrandr which is the command line interface to the X Server RandR extension, I recommend that you become familar with it as it is one of the more useful extensions in the X Window System when you have more than one head on a graphics card.

The following is a list of the nvidia-related kernel modules.

# modprobe -l | grep nvidia
/lib/modules/2.6.27.19-170.2.35.fc10.x86_64/extra/nvidia/nvidia.ko
/lib/modules/2.6.27.19-170.2.35.fc10.x86_64/kernel/drivers/video/nvidia/nvidiafb.ko
/lib/modules/2.6.27.19-170.2.35.fc10.x86_64/kernel/drivers/video/backlight/mbp_nvidia_bl.ko
# lsmod | grep nv
nvidia               8117992  20
i2c_core               29216  2 nvidia,i2c_i801
#

One thing to watch out for is the nvidia system service can change your /etc/X11/xorg.conf file without informing you. To eliminate this possibility, I strongly recommend that you turn the service off.

# chkconfig --list  nvidia
nvidia          0:off 1:off 2:on 3:on 4:on 5:on 6:off
# chkconfig nvidia off
#

Note that these instructions may not apply to Fedora 11 (aka Leonidas) as it is getting a new default driver for nVidia grahics chipsets called nouveau. The nv driver will still be installed by default and will be used as a fall-back where nouveau fails to work for whatever reason.

Because the nVidia kernel module is built specifically for each kernel release, you need to download and install a new nVidia kernel module each time your kernel is updated. If you suddenly are confronted with a non-responsive blank screen after a reboot, this is probably what has happened. The easiest fix in this case is to log in to one of the virtual consoles, downgrade the system to runlevel 2, remove the old kmod-nvidia-173* package, and install the new kmod-nvidia-173* package followed by a reboot.

[ Updated Apr 28 2009] Alternatively you can let your system always automatically download and install the latest nVidia kernel module whenever a new kernel is downloaded by simply installing a meta-package called kmod-nvidia-173xx-173.14.18-1.fc10.1.x86_64 instead of the kernel specific package.

Here is the relevant portion of /var/log/yum.log which shows the packages which were installed as a result of installing this meta-package

Apr 23 15:36:59 Installed: system-config-display-1.1.1-1.fc10.noarch
Apr 23 15:36:59 Installed: livna-config-display-0.0.22-1.fc10.noarch
Apr 23 15:37:00 Installed: xorg-x11-drv-nvidia-173xx-libs-173.14.18-1.fc10.x86_64
Apr 23 15:37:06 Installed: xorg-x11-drv-nvidia-173xx-173.14.18-1.fc10.x86_64
Apr 23 15:37:09 Installed: kmod-nvidia-173xx-2.6.27.21-170.2.56.fc10.x86_64-173.14.18-1.fc10.1.x86_64
Apr 23 15:37:09 Installed: kmod-nvidia-173xx-173.14.18-1.fc10.1.x86_64

As usual, I hope that this information has been of help to you. If you are a software developer, I strongly encourage you to upgrade to a dual head system as studies have consistently shown that your productivity can increase by as much as 20% with such a configuration.

Bootable DVD/CDROM

2009-03-14T21:00:00.009-04:00

Have you ever wanted to determine whether a DVD or CDROM is a bootable CDROM or DVD from within a shell script? If you use GNU/Linux you could parse the output from the isoinfo utility which is part of the CDRecord tools suite.

What if the isoinfo utility is not available on your operating system? In this case you have to resort to rolling you own compiled langauge utility or shell script. To do this you need to understand where to find the specific bytes on a CDROM/DVD which indicate whether it is a bootable CDROM/DVD or not. The relevant specification covering bootable CDROMs (and by extension bootable DVDs) is the El Torito specification which is an extension to the ISO 9660 specification.

Our first shell script uses the dd utility to read in the required bytes which are then sliced and diced using the xxd and cut utilities.. Sector 17 (hexadecimal 11, the Boot Record Volume) contains two bytes at a specific offset which are an absolute pointer to the Booting Catalog which in turn contains a specific byte which indicates whether a CDROM/DVD is bootable or not.

#!/bin/bash
#
# use dd to read in the required bytes
#

DEVICE="/dev/sr0"
VERBOSE=1

# get 2 bytes at absolute offset 34887
bchex=`dd if=${DEVICE} skip=$(( 0x800 * 0x11 + 0x47))  bs=1 count=2 conv=swab 2>/dev/null | xxd | cut -d" " -f2`

# get one byte
bcdec=`printf "%d" 0x${bchex}`
bootable=`dd if=${DEVICE}  bs=1 skip=$(( 0x800 * bcdec + 0x20))  count=1  2>/dev/null | xxd | cut -d" " -f2`

if [[ $bootable = 88 ]]; then
   [ $VERBOSE ] && printf "Bootable CDROM\n"
   exit 0
else
   [ $VERBOSE ] && printf "Not a bootable CDROM\n"
   exit 1
fi

The previous shell script is classical in its approach to the problem in that it uses a number of small specialized utilities to achieve the desired result.

Another way of achieving the same result is to use the extended I/O features of ksh93 as shown in the following shell script.

#!/bin/ksh93
#
# Cannot open DVD/CDROM for read/write - hence cannot seek 
# forward on file descriptor. Workaround is to close and 
# reopen the file descriptor.
#

DEVICE="/dev/sr0"
VERBOSE=1

# read 2 bytes
redirect 3< $DEVICE || exit 3
3<#(( 0x800 * 0x11 + 0x47 ))
read -u3 -N 1 bchex1
read -u3 -N 1 bchex2
redirect 3<&-

# calculate absolute offset
integer bcdec=$(printf "%d" 0x$(printf "%x%x\n"  "'$bchex2" "'$bchex1"))

# read 1 byte
redirect 3< $DEVICE || exit 3
3<#(( 0x800 * bcdec + 0x20 ))
read -u3 -N 1 bootable
redirect 3<&-

if [[ $(printf "%x" "'$bootable") = 88 ]]; then
   [ $VERBOSE ] && printf "Bootable CDROM\n"
   exit 0
else
   [ $VERBOSE ] && printf "Not a bootable CDROM\n"
   exit 1
fi

The advantage of this approach is that all operations are done within the existing shell process without the overhead of fork/execs to invoke external utilities such as dd etc.

Note that both of the above shell scripts assume that the CDROM or DVD being checked is either non-bootable or contains a single boot-image configuration. They can easily be extended to check for a multiple boot-image configuration. That is left as an exercise for you the reader.

For the record, these two shell scripts were tested on Fedora 10 using version M of ksh93.

KSH93 Bit Manipulation

2009-02-20T19:54:00.050-05:00

When programmers think about bitwise manipulation, they usually think about using C or C++ to solve their problem since both programming languages provide a rich set of features which make it easy to perform bitwise manipulation.

However it is possible to just as easily perform such operations using the ksh93 shell. This post will explain what bitwise manipulation and number conversion facilities are available in ksh93 and provide some hopefully useful utilities and examples.

Many programmers are unaware that ksh93 has builtin support for different numeral radices (AKA arithmetic bases) as shown in the following example.

$ print $(( 10 + 20 ))
30
$ print $(( 2#10 + 20 ))
22
$ print $(( 2#10 + 4#20 ))
10
$

where base 2 is denoted by 2#, base 4 by #4 and so on. The general case syntax is [base#]n where base is a decimal number between 2 and 36 representing the arithmetic base and n is a number in that base. You can mix and match bases within expressions. As you would expect, the default base, i.e. if [base#] is omitted, is decimal(10).

Here is an example of how to convert decimal 255 to base 2, base 8 and base 16 respectively using the typeset command.

$ typeset -i2 a=255
$ echo $a
2#11111111
$ typeset -i8 a=255
$ echo $a
8#377
$ typeset -i16 a=255
$ echo $a
16#ff
$

Note that ksh93 displays the base prefix before each number unless it is a decimal number, i.e. base 10.

The bitwise manipulation operators in ksh93 are AND, OR, XOR, SHIFT LEFT and SHIFT RIGHT. There are no SHIFT ROTATE or COMPLEMENT operators. The following is the list of supported bitwise operators in decreasing order of precedence which, by the way, is the same as in the C programming language.

<< >>	Bitwise shift left, shift right
&	Bitwise AND
^	Bitwise XOR
\|	Bitwise OR

The following shell script includes both logical and bitwise operators together with the expected output for each statement. Notice the difference in output between a logical and bitwise operator.

#!/bin/ksh93

integer -i2 a=5
integer -i2 b=6

# expected output 1
print $(( a < b ))
# expected output 0
print $(( a > b ))
# expected output 3
print $(( a ^ b ))
# expected output 4
print $(( a & b ))
# expected output 1
print $(( a && b ))
# expected output 7
print $(( a | b ))
# expected output 1
print $(( a || b ))

A useful feature of bitwise operators is that by using the shift left and shift right bitwise operators you can mimic multiplication or division by 2 or a power of 2.

$ x=4
$ print $((x << 1))
8
$ print $(( x << 2 ))
16
$ print $(( x >> 1 ))
2
$ print $(( x >> 2 ))
1

Here is a fairly simple shell script which uses numeric base conversion and bitwise operators to calculate the network and broadcast addresses for a given IP address and subnet mask.

#!/bin/ksh93

typeset -i2 mask=255

[[ $# != 2 ]] && {
   echo "Usage: $0 ipaddress subnetmask"
   exit 1
}

SaveIFS=$IFS
IFS=.
typeset -a IParr=($1)
typeset -a NMarr=($2)
IFS=$SaveIFS

typeset -i2 ip1=${IParr[0]}
typeset -i2 ip2=${IParr[1]}
typeset -i2 ip3=${IParr[2]}
typeset -i2 ip4=${IParr[3]}

typeset -i2 nm1=${NMarr[0]}
typeset -i2 nm2=${NMarr[1]}
typeset -i2 nm3=${NMarr[2]}
typeset -i2 nm4=${NMarr[3]}

echo
echo "       IP Address: $1"
echo "      Subnet Mask: $2"
echo "  Network Address: $((ip1 & nm1)).$((ip2 & nm2)).$((ip3 & nm3)).$((ip4 & nm4))"
echo "Broadcast Address: $((ip1 | (mask ^ nm1))).$((ip2 | (mask ^ nm2))).$((ip3 | (mask ^ nm3))).$(( ip4 | (mask ^ nm4)))"
echo

exit 0

Some sample output for this script.

$ ./calculate-address 10.150.12.1 255.255.255.0

       IP Address: 10.150.12.1
      Subnet Mask: 255.255.255.0
  Network Address: 10.150.12.0
Broadcast Address: 10.150.12.255

$ ./calculate-address 10.150.12.1 255.255.254.0

       IP Address: 10.150.12.1
      Subnet Mask: 255.255.254.0
  Network Address: 10.150.12.0
Broadcast Address: 10.150.13.255

The next example calculates the network and broadcast addresses for a Classless Inter-Domain Routing (CIDR) compliant IP address. Note the use of the shift right operator in calculating the network mask.

#!/bin/ksh93

[[ $# != 1 ]] && {
   echo "Usage: $0 ipaddress/netmask"
   exit 1
}

SaveIFS=$IFS
IFS="./"
typeset -a IParr=($1)
IFS=$SaveIFS

typeset -i2 ip1=${IParr[0]}
typeset -i2 ip2=${IParr[1]}
typeset -i2 ip3=${IParr[2]}
typeset -i2 ip4=${IParr[3]}
typeset -i2 cidr=${IParr[4]}

typeset -i2 nm1=0 nm2=0 nm3=0 nm4=0

typeset -i quad=$(( cidr / 8 ))
sigbits=$(( cidr % 8 ))
if (( sigbits != 0 )); then
   slot=$(( 256 - ( 256 >> $sigbits ) ))
fi

for (( i=1; i < 5; i++ ))
do
   nameref nm=nm${i}
   if (( quad != 0 )); then
      nm=255
      (( --quad ))
   elif (( quad == 0 )); then
      nm=slot
      break
   fi
done

typeset -i2 mask=255

print
print "           IP Address: $((ip1)).$((ip2)).$((ip3)).$((ip4))"
print "    CIDR Netmask Mask: $((nm1)).$((nm2)).$((nm3)).$((nm4))"
print " CIDR Network (Route): $((ip1 & nm1)).$((ip2 & nm2)).$((ip3 & nm3)).$((ip4 & nm4))"
print "    Broadcast Address: $((ip1 | (mask ^ nm1))).$((ip2 | (mask ^ nm2))).$((ip3 | (mask ^ nm3))).$((ip4 | (mask ^ n
m4)))"
print

exit 0

Sample output:

$ ./calculate-address "192.168.21.12/18"

           IP Address: 192.168.21.12
    CIDR Network Mask: 255.255.192.0
 CIDR Network (Route): 192.168.0.0
    Broadcast Address: 192.168.63.255

While both the bash and zsh shells also provide a full set of bitwise operators, only ksh93 and zsh with their builtin support for numeric base conversions makes such bitwise operators truly useful in shell scripts.

Monitoring D-Bus

2009-02-15T11:05:00.032-05:00

Because D-Bus operates for the most part under the hood invisible to the user, means of monitoring D-Bus messages is important for both activation and debugging purposes. In this post I examine how to monitor and act on such messages. I assume that you are relatively familiar with D-Bus. If not, you should look at my previous posts on D-Bus to gain an understanding of the basic concepts.

Probably the most commonly used tool is the command line utility dbus-monitor which was written by Philip Blundell as part of the reference D-Bus implementation and which is standard on most D-Bus enabled platforms. It is a very powerful utility but unfortunately it's man page is terse and fails to explain the most powerful feature of dbus-monitor i.e. watch expressions.

Suppose, for example, I want to monitor the creation, saving and deletion of Tomboy notes using D-Bus signals. To get the list of the supported signals I could introspect Tomboy using qdbus i.e.

$ qdbus org.gnome.Tomboy /org/gnome/Tomboy/RemoteControl | grep signal
signal void org.gnome.Tomboy.RemoteControl.NoteAdded(QString uri)
signal void org.gnome.Tomboy.RemoteControl.NoteDeleted(QString uri, QString title)
signal void org.gnome.Tomboy.RemoteControl.NoteSaved(QString uri)
$

or I could use dbus-monitor to look at the signals generated when Tomboy creates, saves and deletes a note.

$ dbus-monitor type=signal interface="org.gnome.Tomboy.RemoteControl"
signal sender=org.freedesktop.DBus -> dest=:1.102 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameAcquired
   string ":1.102"
signal sender=:1.63 -> dest=(null destination) path=/org/gnome/Tomboy/RemoteControl; interface=org.gnome.Tomboy.RemoteControl; member=NoteAdded
   string "note://tomboy/0ae6b0b3-9394-44e9-ad4a-46726e68cebc"
signal sender=:1.63 -> dest=(null destination) path=/org/gnome/Tomboy/RemoteControl; interface=org.gnome.Tomboy.RemoteControl; member=NoteSaved
   string "note://tomboy/0ae6b0b3-9394-44e9-ad4a-46726e68cebc"
signal sender=:1.63 -> dest=(null destination) path=/org/gnome/Tomboy/RemoteControl; interface=org.gnome.Tomboy.RemoteControl; member=NoteDeleted
   string "note://tomboy/0ae6b0b3-9394-44e9-ad4a-46726e68cebc"
   string "New Note 1"
$

As you can see Tomboy issues three signals - NoteAdded when a note is added, NoteSaved when a note is saved, and NoteDeleted when a note is deleted. Interestingly, for some reason NoteDeleted provides both the uri and the title of the note whereas the other two signals only provide the uri.

Suppose I just want to to monitor when a Tomboy note is saved by a user. I can use a dbus-monitor watch expression for this purpose as shown in the following example.

dbus-monitor "type='signal',sender='org.gnome.Tomboy',interface='org.gnome.Tomboy.RemoteControl',path='/org/gnome/Tomboy/RemoteControl',member='NoteSaved'"

A watch expression is essentially a message filter. Note the syntax! Everything between the two double quotes is part of the watch expression.

Unfortunately the syntax of a watch expression is not documented in the dbus-monitor man page. However from looking at the source code for dbus-monitor, it appears that a watch expression consists of one or more name-value pairs, e.g. sender='org.gnome.Tomboy' separated by commas. The following are valid names: type, sender, destination, path, interface, member and error_name. The value of type must be one of the following: signal, method_call, method_return or error.

Probably the most powerful feature of dbus-monitor is the fact that you are not limited to using just one watch expression at a time. The following example simultaneously monitors all 3 Tomboy signals and uses awk to parse the output from dbus-monitor and display a meaningful message.

#!/bin/bash

OJECT="'org.gnome.Tomboy'"
IFACE="'org.gnome.Tomboy.RemoteControl'"
DPATH="'/org/gnome/Tomboy/RemoteControl'"

WATCH1="type='signal', sender=${OJECT}, interface=${IFACE}, path=${DPATH}, member='NoteAdded'"
WATCH2="type='signal', sender=${OJECT}, interface=${IFACE}, path=${DPATH}, member='NoteSaved'"
WATCH3="type='signal', sender=${OJECT}, interface=${IFACE}, path=${DPATH}, member='NoteDeleted'"

dbus-monitor "${WATCH1}" "${WATCH2}" "${WATCH3}" | \
awk '
/member=NoteAdded/ { getline; print "Created note " substr($2,7) }
/member=NoteSaved/ { getline; print "Added note " substr($2,7) }
/member=NoteDeleted/ { getline; print "Deleted note " substr($2,7) }
'

Here is the output generated when I clicked on the Tomboy icon to create a new note, waited for the automatic save and then selected the delete option to delete the note.

$ ./test
Created note //tomboy/3da026dc-f6ee-4637-8a94-bec6e2844824"
Added note //tomboy/3da026dc-f6ee-4637-8a94-bec6e2844824"
Deleted note //tomboy/3da026dc-f6ee-4637-8a94-bec6e2844824"

Well that is all for this post. Enjoy!

P.S. I tested these examples using Fedora 10. I have no idea whether they will work on previous versions of Fedora or other Linux distributions.