Angry Unix Programmer

CL-MW 0.3 Released!

psilord — Tue, 27 Mar 2012 09:59:53 CDT

I had given a talk at the TC Lispers group in Minneapolis, MN on March 5th, 2012 about the reasoning behind and some of the internal design of CL-MW. For those interested, here are the slides of the presentation.

Out of the wonderful conversation with the attendees, I gathered a small list of improvements to CL-MW which I would be implementing in the future. These improvements include such things as: a macro (gotten from the expansion of mw-task-algorithm) which names a function available in its body which submits tasks with a specific task policy, adding a control connection to the master process so you can asynchronously add in new work or retrieve results with another process, what the master algorithm checkpoint API might look like, separating out the asynchronous I/O packet buffer layer I built on top of IOLib into its own library, etc.

I've released CL-MW version 0.3 which implements the first of these improvements. Now you can say:

;; First, define a simple task algorithm
(define-mw-task-algorithm (foo)
  (+ foo 10))

;; later in the code:

...
(mw-with-task-policy-for-foo (add-foo :retry nil)
  ;; add a bunch of tasks, but don't retry them if they fail.
  (mapc #'add-foo '(1 2 3 4 5)))
...

;; and then you just get the results back asynchronously in the master's loop.

CL-MW is quicklisp installable so try it out and see what you think!

End of Line.

Random Garbage

psilord — Wed, 16 Nov 2011 10:30:02 CST

A friend of mine was asking about how to write a small interpreter so he can define new AI functions that his project can use at runtime. After his face blanked over when I started explaining how to do it in Common Lisp (really, there isn't that much explanation, it is like a fundamental property of Common Lisp to do such things), he hastily mentioned he wanted it in C.

Oh.

So, I hacked together a trivial demonstration program. This program runs a very small interpreter which allows one to compile a C file into a shared object, then load the shared object, and bind the functions inside of the shared object to a structure full of function pointers that you can then invoke manually. It is intended that one writes their varied functions as different C files that they can load and swap out at runtime.

Here is the lame makefile which compiles a program called stuff.c.

# Makefile

stuff: stuff.c
    gcc -Wall -g stuff.c -o stuff -ldl

clean:
    rm -f stuff *.o *.so

Here is stuff.c. This program is set up using a traditional interpreter design. However, it is totally barebones and I don't deal with the interpreter environment in any meaningful way (other than its reification and global nature) since you can't define new variables or functions in the interpreter. Also, the lexical and parsing analysis of the interpreted forms are horriffic at best. This is because doing such things in C is a pain in the ass unless you use flex and bison or are prepared to write a helluva lot more code. However, if I did that, this wouldn't be the simple demonstration that it is.

Note that I chose to perform the linking to the loaded library functions via an explicit indirection with the f structure in the Env structure. I could have just taken the func_name variable in eval_invoke() and simply performed a dlsym() call upon it and called the resulting pointer with the arguments. If I had done that, I could have called ANY function in the loaded library (well, with the same protoype at any rate). It is generally more general (in some respects) to do such a thing. However, I chose the method I did because through the indirection I can associate functions of different C linkage names to the symbols I use to identify them--such as the different names of the default functions in relation to the functions names as defined in the foo/bar.c codes.

A real world example of why the method I chose is useful would be if I wanted to have multiple implementations of C functions with the exact same name loaded at the same time where I could pick and choose between them. In the method I chose, I could additionally associate a namespace (or package name) with a shared object (meaning I'd pair the f and lib_name fields into a 'Package' structure and have a hash table of them in the Env keyed by package name that is specifed when loading the shared object) and use another syntax in the interpreter to state which function I want to call out of which namespace/package. This would be an exercise for the reader to implement.

/* This is stuff.c */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <dlfcn.h>

#define MATCH 0

#define TRUE 1
#define FALSE 0

#define DONE 0
#define NOT_DONE 1

#define BSIZE 1024

/* the type of the functions we care about in the compiled code */
typedef int (*FUNC)(int a, int b);

/* The extension of how to map function fromthe shared library to an API
    to them have very obvious extensions that what I did here. I didn't
    do any of them.
*/
typedef struct Funcs_t
{
    FUNC fun1;
    FUNC fun2;
    FUNC fun3;
} Funcs;

/* The global environmental structure */
typedef struct Env_t
{
    Funcs f;
    void *lib_handle;
} Env;

/* The invocable functions in the global environment are defaulted to these
    functions.
*/
int stub1(int a, int b)
{
    printf("Default stub1(%d, %d): called.\n", a, b);
    return a + b;
}

int stub2(int a, int b)
{
    printf("Default stub2(%d, %d): called.\n", a, b);
    return a + b;
}

int stub3(int a, int b)
{
    printf("Default stub3(%d, %d): called.\n", a, b);
    return a + b;
}

char *prompt_input(char *buf, int size, FILE *fin)
{
    printf("> ");
    fflush(NULL);
    return fgets(buf, size, fin);
}

int eval_help(char *args, Env *e)
{
    printf(
"Help:\n"
"  help               This help message\n"
"  quit               Quits the program\n"
"  compile <file>     Produces shared library of C file, don't provide the .c extension\n"
"  load <file>        Loads library named NAME by loading libNAME.so\n"
);

    return NOT_DONE;
}

int eval_quit(char *args, Env *e)
{
    printf("Quitting!\n");
    return DONE;
}

/* compile a source file (without the .c extension) and create a shared
    object we can load later. The error checking and reporting in this function
    is criminally bad.
*/
int eval_compile(char *args, Env *e)
{
    char cmd[BSIZE], file[BSIZE], buf[BSIZE];
    int ret;

    if (sscanf(args, "%s %s", cmd, file) != 2) {
        printf("eval_compile: bad arity!\n");
        return NOT_DONE;
    }

    /* construct and execute the compilaiton command. I hope everything
        is in your path.
    */

    sprintf(buf, "gcc -Wall -DPIC -fpic -c %s.c", file);
    ret = system(buf);
    if (ret != 0) {
        printf("Sorry, an error happened during compilation.\n");
        return NOT_DONE;
    } else {
        printf("Compile [%s.c]: OK\n", file);
    }

    /* Now produce the shared object */

    sprintf(buf, "gcc -shared -Wl,-soname,lib%s.so.1 %s.o -lc -o lib%s.so", 
        file, file, file);
    ret = system(buf);
    if (ret != 0) {
        printf("Sorry, an error happened during shared library generation.\n");
    } else {
        printf("Library generation [lib%s.so]: OK\n", file);
    }

    return NOT_DONE;
}

/* We only allow you to invoke the functions in the Env structure. You
    denote the names by "fun1" "fun2" and "fun3". This is a bare skeleton
    of how to do such things since I don't even create a symbol table for
    the mapping of the interpreter function symbol to actual C functions.
*/

int eval_invoke(char *cmd, Env *e)
{
    char buf[BSIZE], func_name[BSIZE];
    int arg0, arg1;
    int ret;

    if (sscanf(cmd, "%s %s %d %d", buf, func_name, &arg0, &arg1) != 4) {
        printf("eval_invoke: bad arity!\n");
        return NOT_DONE;
    }

    /* now execute the function we wanted to run with the arguments. */ 

    if (strncmp("fun1", func_name, 4) == MATCH) {
        printf("[Invoking function fun1...]\n");
        ret = (e->f.fun1)(arg0, arg1);
        printf("[Result] %d\n", ret);
    } else if (strncmp("fun2", func_name, 4) == MATCH) {
        printf("[Invoking function fun2...]\n");
        ret = (e->f.fun2)(arg0, arg1);
        printf("[Result] %d\n", ret);
    } else if (strncmp("fun3", func_name, 4) == MATCH) {
        printf("[Invoking function fun3...]\n");
        ret = (e->f.fun3)(arg0, arg1);
        printf("[Result] %d\n", ret);
    } else {
        printf("I'm sorry, there is no function to invoke by that name.\n");
    }

    return NOT_DONE;
}

int eval_load(char *cmd, Env *e)
{
    void *new_lib = NULL;
    char buf[BSIZE], lib_name[BSIZE];
    char name[BSIZE];

    if (sscanf(cmd, "%s %s", buf, lib_name) != 2) {
        printf("eval_load: bad arity!\n");
        return NOT_DONE;
    }

    sprintf(name, "./lib%s.so", lib_name);
    new_lib = dlopen(name, RTLD_NOW | RTLD_LOCAL);

    if (new_lib == NULL) {
        printf("Failed to load library: %s\n", name);
        return NOT_DONE;
    }

    /* close any previous one */
    if (e->lib_handle != NULL) {
        dlclose(e->lib_handle);
    }

    /* keep a reference to the new one */
    e->lib_handle = new_lib;

    /* "link" the functions in the Env to the ones we just loaded */
    e->f.fun1 = dlsym(e->lib_handle, "fun1");
    if (e->f.fun1 == NULL) {
        printf("Warning, unable to resolve fun1() from library %s, "
            "assuming initial stub1().\n", name);
        e->f.fun1 = stub1;
    }

    e->f.fun2 = dlsym(e->lib_handle, "fun2");
    if (e->f.fun2 == NULL) {
        printf("Warning, unable to resolve fun2() from library %s, "
            "assuming initial stub2().\n", name);
        e->f.fun2 = stub2;
    }

    e->f.fun3 = dlsym(e->lib_handle, "fun3");
    if (e->f.fun3 == NULL) {
        printf("Warning, unable to resolve fun3() from library %s, "
            "assuming initial stub3().\n", name);
        e->f.fun3 = stub3;
    }

    printf("Functions Linked!\n");

    return NOT_DONE;
}

/* The basic structure of the interpreter */
int eval_command(char *cmd, Env *e)
{
    printf("Evaluating command: '%s'\n", cmd);

    /* check to see what I have and run the appropriate handler */

    if (strncmp("help", cmd, 4) == MATCH) {
        return eval_help(cmd, e);
    }

    if (strncmp("quit", cmd, 4) == MATCH) {
        return eval_quit(cmd, e);
    }

    if (strncmp("compile", cmd, 7) == MATCH) {
        return eval_compile(cmd, e);
    }

    if (strncmp("invoke", cmd, 6) == MATCH) {
        return eval_invoke(cmd, e);
    }

    if (strncmp("load", cmd, 4) == MATCH) {
        return eval_load(cmd, e);
    }

    printf("Sorry, I don't know how to do that command.\n");
    return NOT_DONE;
}


int main(void)
{
    char buf[BSIZE];
    int done = NOT_DONE;
    char *ret = NULL;
    char *nl = NULL;
    Env e;

    /* set up defaults */
    e.f.fun1 = stub1;
    e.f.fun2 = stub2;
    e.f.fun3 = stub3;
    e.lib_handle = NULL;
    
    /* run the read/eval/print loop until done */

    printf("Welcome to a simple demonstration interpreter.\n");

    eval_help(NULL, &e);

    ret = prompt_input(buf, BSIZE, stdin);
    while(ret != NULL && done == NOT_DONE)
    {
        /* I'm not doing any real whitespace trimming, so be VERY careful */

        /* get rid of newline */
        nl = strstr(buf, "\n");
        if (nl != NULL) {
            *nl = '\0';
        }

        done = eval_command(buf, &e);
        if (done == NOT_DONE) {
            ret = prompt_input(buf, BSIZE, stdin);
        }
    }

    /* Clean up, if any */
    if (e.lib_handle != NULL) {
        dlclose(e.lib_handle);
        e.lib_handle = NULL;
    }
    
    return 0;
}

Now, here is the first file that we'll be using as a replacement for the stub functions. This file (and bar.c below) must be in the current working directory when you start the stuff program.

/* This is foo.c */

#include <stdio.h>
#include <stdlib.h>

int fun1(int a, int b)
{
    printf("This is foo.c:fun1()\n");
    fflush(NULL);

    return a + b;
}

int fun2(int a, int b)
{
    printf("This is foo.c:fun2()\n");
    fflush(NULL);

    return a + b;
}

int fun3(int a, int b)
{
    printf("This is foo.c:fun3()\n");
    fflush(NULL);

    return a + b;
}

And here is bar.c, another definition of the above functions.

/* This is bar.c */

#include <stdio.h>
#include <stdlib.h>

int fun1(int a, int b)
{
    printf("This is bar.c:fun1()\n");
    fflush(NULL);

    return a + b;
}

int fun2(int a, int b)
{
    printf("This is bar.c:fun2()\n");
    fflush(NULL);

    return a + b;
}

int fun3(int a, int b)
{
    printf("This is bar.c:fun3()\n");
    fflush(NULL);

    return a + b;
}

Now that we have everything defined, here is an interaction with the program. Notice the compilation of the above C files happens by us asking to compile them in the interpreter. Also notice how the output of the functions "fun1", "fun2", and "fun3" change away from the default to what is defined in each separate C file.

Linux black > ./stuff
Welcome to a simple demonstration interpreter.
Help:
  help               This help message
  quit               Quits the program
  compile <file>     Produces shared library of C file, don't provide .c
  load <file>        Loads library named NAME by loading libNAME.so\n"
> invoke fun1 10 10
Evaluating command: 'invoke fun1 10 10'
[Invoking function fun1...]
Default stub1(10, 10): called.
[Result] 20
> invoke fun2 10 10
Evaluating command: 'invoke fun2 10 10'
[Invoking function fun2...]
Default stub2(10, 10): called.
[Result] 20
> invoke fun3 10 10
Evaluating command: 'invoke fun3 10 10'
[Invoking function fun3...]
Default stub3(10, 10): called.
[Result] 20
> compile foo
Evaluating command: 'compile foo'
Compile [foo.c]: OK
Library generation [libfoo.so]: OK
> compile bar
Evaluating command: 'compile bar'
Compile [bar.c]: OK
Library generation [libbar.so]: OK
> load foo
Evaluating command: 'load foo'
Functions Linked!
> invoke fun1 10 10
Evaluating command: 'invoke fun1 10 10'
[Invoking function fun1...]
This is foo.c:fun1()
[Result] 20
> invoke fun2 10 10
Evaluating command: 'invoke fun2 10 10'
[Invoking function fun2...]
This is foo.c:fun2()
[Result] 20
> invoke fun3 10 10
Evaluating command: 'invoke fun3 10 10'
[Invoking function fun3...]
This is foo.c:fun3()
[Result] 20
> load bar
Evaluating command: 'load bar'
Functions Linked!
> invoke fun1 10 10
Evaluating command: 'invoke fun1 10 10'
[Invoking function fun1...]
This is bar.c:fun1()
[Result] 20
> invoke fun2 10 10
Evaluating command: 'invoke fun2 10 10'
[Invoking function fun2...]
This is bar.c:fun2()
[Result] 20
> invoke fun3 10 10
Evaluating command: 'invoke fun3 10 10'
[Invoking function fun3...]
This is bar.c:fun3()
[Result] 20
> quit
Evaluating command: 'quit'
Quitting!

Enhancement of the interpreter would go in the direction of allowing all of the functions in the shared object to be discovered and shoved into a symbol table stored in the Env environment so they can be called. In addition, the arguments of the functions would be more flexibly defined so you can pass other data types to them or define them to have different arities. There is definitely more that can be done.

End of Line.

Tasks Whipped Stiff, Not Dry.

psilord — Sun, 13 Mar 2011 01:26:06 CST

CL-MW Version 0.2 is now available!

CL-MW is a Master/Slave library to help author pleasantly parallel Common Lisp applications. It is batch scheduler friendly and has examples and documentation in its complete manual for interfacing with the Condor batch scheduler. CL-MW is designed to be very easy to use and will readily generate executables that do not require an installed lisp environment for easy executable distribution across clusters.

The major feature enhancement for version 0.2 is that basic support for the lambda list keywords &optional, &key, and &rest have been made available for task algorithms. A new example in the sources demonstrates the use.

CL-MW is currently only supported on SBCL at this time. I will accept patches for other CL environments.

CL-MW is a part of Quicklisp and can be installed thusly:

* (ql:quickload "cl-mw")

You can find CL-MW and see an example hello world application here.

May this software serve your needs.

End of Line.

It's 1980 Baby!

psilord — Tue, 01 Feb 2011 04:28:43 CST

Once again I've made extensive changes to Option-9. I've released version 0.7, which implements hit points, damage points, mines, new powerups, almost completely reorganizes the code. Most importantly it adds a visually cool new weapon called the Tesla Field Weapon. The one trick with that weapon is you need to collect 5 of the asterisk style powerups to see it in the full glory.

Take a look at the screen shot.

I've also once again updated the theory of operations document. At this point, I'm likely not going to hack on the game anymore, since I think I've extracted what I wanted to learn out of it. Also, the document is starting to get a little unwieldy, so I'm unlikely to update it for future revisions of the game, if any.

Check it out.

End of Line.

The Best of 1979 Redux

psilord — Sun, 02 Jan 2011 02:32:37 CST

After the public release of my Option 9 Theory of Operation document, people that actually know lisp read it and suggested a pile of changes to better my code and CLOS style. Most of those changes were in how I named my accessor functions and other generic functions. I definitely appreciate their comments.

I've made the changes, which were pretty extensive, released version 0.2, which has only a better repl as a new feature, and updated the theory of operations document to describe the new code and new object protocol. The changes definitely simplified my code and I'm closer to how CLOS wants me to think about OO problems.

Check it out.

End of Line.

The Best of 1979

psilord — Fri, 31 Dec 2010 01:58:31 CST

In my quest to become fluent in Lisp, I wrote a small space shoot'em up video game called " Option 9 ". There were two versions of the game. The first one only took about 10 hours to write start to finish and that included learning how to use lispbuilder-sdl and cl-opengl. However, it was such a complete piece of unmaintainable trash that it will die a bitter death in the bit bucket. The second version took about the same amount of time to write and had many more features. It is this one that I will describe.

In the second version, I wanted to learn the Common Lisp Object System, or CLOS for short. My main source of learning CLOS is a book by Sonja E. Keene called "Object Oriented Programming in Common Lisp: A Programmer's Guide to CLOS". This book has VERY minor differences from the ANSI CLOS standard, which I may detail in a future blog post, and was written in 1989. However, it is still extremely useful and I found it to be a good and relevant read. It has excellent and clear explanations and does a great job of presenting the material.

I will say that CLOS is pretty awesome. Whenever I finish writing OOP code in Java or C++, I feel like I have to take a shower and speak in hushed tones about something in a garbage bag smelling of cheap perfume and meth that just got dumped into a ravine at 3am outside of Billings, Montana.

The thing I found with CLOS was that it just worked and I didn't have to write a lot of extraneous garbage to use it very effectively. In C++, there are so many mines in the minefield whose locations you have to remember that you don't have time or energy to dance in the grass. Even with my beginning understanding and use of CLOS, I easily saw that one just wrote less code while using it to represent an OO idea.

The biggest surprise was just how plainly useful multi-methods were. Multi-methods are generic methods which can specialize on more than one parameter. This is usually implemented with the Visitor Pattern in other languages which only support single dispatch. Having multiple dispatch readily in the language (and integrated with the rest of Lisp) prevented a lot of other code from being written that would obfuscate the intent. I appreciated this enormously. I find that I truly enjoy looking at code whose signal to noise ratio is very high.

Here is a screen shot of Option 9 in all of its vector art glory:

I wrote up a Theory of Operations document which is also a postmortem so you can both understand the code and my thought process in writing it. I provide the source so you can see it work for yourself. I don't just include my write up in this blog post itself since it is like 30 pages with images and that's a bit much to shove down the throat of RSS readers.

Check it out.

End of Line.

Slaves Get Stuff Done

psilord — Wed, 03 Nov 2010 12:59:18 CDT

After a serious amount of time learning Lisp and hacking on my various projects, one of my projects has been released.

Check out CL-MW!

It is a distributed master/slave library written in pure Common Lisp designed to be easily integrated into batch processing systems like Condor. It currently works on SBCL and uses IOLib for its networking. I welcome patches to make it better and have a roadmap of features that need implementation.

I've written it to handle around 10,000 concurrent slaves and billions of fine to medium grain tasks. It has been tested in the low thousands of slaves and low billions of tasks with Condor and it ran very smoothly with moderate slave churn. Of course, once it gets into the wild I'm sure more bugs will be found and features desired.

Included are some example programs which illustrate the use of the library and a manual explaining how it all works. I would like to specifically thank Alan De Smet for his application of his excellent editing skills to the manual.

I'll soon make the rounds on comp.lang.lisp and cliki.net and whatnot announcing the library. It would be nice if it ended up in quicklisp too, so I have to figure that out as well. Hopefully other people besides me find CL-MW useful.

End of Line.

Observation 1

psilord — Thu, 21 Oct 2010 05:50:03 CDT

Intelligence is like a vaccine. If a lot of people get vaccinated the stupid is kept at bay. But I fear too many people are relying on herd immunity of late. When the first big stupid plague hits it'll be lights out for humanity.

End of Line.

A Thief of Time

psilord — Mon, 18 Oct 2010 02:33:57 CDT

I absolutely hate waiting for computers. One of the places I found myself waiting was emacs trying as hard as it can to suck Firefox or Konqueror up into my memory when ever I want to look up a Common Lisp symbol in SLIME in the Hyperspec. I could keep a browser window open to to the Hyperspec, but cycling to the window out of the many I have, or unminimizing it with the mouse and groveling around in it also became damn annoying after a while.

So I wrote this tiny piece of code. It uses the hyperspec package and fires up w3m in an xterm with the documentation in question. The interface to this code is the macro clhs. It is a macro because it'll autoquote the symbol given to it so you don't have to do it yourself. I love it a lot because I type a lisp command, the window pops up, I can navigate, and then make the window go away all without leaving the keyboard because the window pops up with keyboard focus. I've bracketed the code with #+sbcl because this is very specific to that implementation. It also needs the hyperspec-lookup library, but that is easy to get and configure.

I love it when the license is longer than the code.

;; Copyright (c) 2010 Peter Keller (psilord@cs.wisc.edu)
;; 
;; Permission is hereby granted, free of charge, to any person obtaining a copy
;; of this software and associated documentation files (the "Software"), to deal
;; in the Software without restriction, including without limitation the rights
;; to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
;; copies of the Software, and to permit persons to whom the Software is
;; furnished to do so, subject to the following conditions:
;; 
;; The above copyright notice and this permission notice shall be included in
;; all copies or substantial portions of the Software.
;; 
;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
;; IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
;; FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
;; AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
;; LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
;; OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
;; THE SOFTWARE.

#+sbcl (require :hyperspec-lookup)

#+sbcl (defun clhs-impl (sym)
         (let* ((sym-name (symbol-name sym))
                (url (hyperspec:lookup sym-name)))
           (sb-ext:process-close
            (sb-ext:run-program
             "/bin/sh"
             (list "-c"
                   (concatenate 'string "COLUMNS=80 /usr/bin/xterm "
                                "-geom 80x70 -e /usr/bin/w3m " url " &"))
             :output t))))

#+sbcl (defmacro clhs (sym)
         `(clhs-impl ',sym))

You'd use it like this:

* (clhs read)

[control returns to repl and an xterm window pops up at the right web page]

I'm sure it won't be too hard to rebind SLIME's clhs lookup function to something like the above. Maybe when it isn't 2:30am, I'll make it work.

End of Line.

Card Catalog

psilord — Tue, 28 Sep 2010 01:12:02 CDT

I ran into a problem with SBCL where not only did I need to save the lisp image into an executable, but I also had to save the shared libraries loaded, via dlopen(), with the lisp image into the current working directory next to the executable. The shared libraries I needed to save were located specifically in the sb-sys:*shared-objects* variable. The reason for this is so I could execute the application on another machine which very likely didn't have all of the shared libraries. The shared libraries were for interfaces from lisp to C generated by CFFI-GROVEL or other specialty libraries not often found on random machines.

There is a chunk of code which implements this mess, but the part which I'm going to show in this post is the portion which calls /sbin/ldconfig -p, parses the output, and returns a hash table that one can query against to convert a bare library to an absolute pathname. The code really isn't spectacular in any way and I'll probably iterate over it some more removing dumb crap I did, but it is very useful, which is good enough for me.

Of course, dlopen() uses other methods to find libraries, like RPATH and crap. This code only implements a library lookup via the /etc/ld.so.cache file.

;; This code snippet is under the Apache Version 2 license and is Copyright
;; 2010 Peter Keller (psilord@cs.wisc.edu). It requires CL-PPCRE and 
;; is SBCL specific.

;; Perform the body, which is assumed to open the stream strm in question
;; and write to it. Ugly, make better...
(defmacro stream->string-list ((strm) &body body)
  (let ((g (gensym))
        (h (gensym)))
    `(let ((,h '()))
       (with-input-from-string
           (,g (with-output-to-string (,strm)
                 (progn
                   ,@body)))
         (loop while (let ((line (read-line ,g nil)))
                       (when line
                         (push line ,h))
                       line))
         (nreverse ,h)))))

;; Convert the machine type to a keyword.
(defun get-machine-type ()
  (let ((mt (machine-type)))
    (cond
      ((equalp "X86" mt)
       :x86)
      ((equalp "X86-64" mt)
       :x86-64)
      (t
       (error "Unknown machine type ~A, please port!~%" mt)))))

;; Given a list of flags associated with a line from ldconfig -p, find me
;; the library type the library is.
(defun find-lib-type (split-flags)
  (if (find "libc6" split-flags :test #'equalp)
      "libc6"
      (if (find "ELF" split-flags :test #'equalp)
          "ELF"
          (assert "Unknown lib type. Please port!~%"))))

;; Given a list of flags associated with a line from the ldconfig -p, find
;; me the specific architecture associated with the library.
(defun find-lib-arch (split-flags)
  (if (find "x86-64" split-flags :test #'equalp)
      :x86-64
      :x86))

;; Take a precut line from the ldconfig -p output and merge it with the rest
;; of the lines in the hash table ht.
(defun merge-ld.so.cache-line (bare-lib split-flags absolute-lib ht)
  ;; Ensure the bare-lib has a hash table entry in the master table.
  (when (null (gethash bare-lib ht))
    (setf (gethash bare-lib ht) (make-hash-table :test #'equalp)))

  ;; The type of the library is either libc6 or ELF, but not both. So
  ;; find out which one it is and set the type in the hash table value
  ;; for the library in question. Ensure the type didn't change!
  (let ((lib-type (find-lib-type split-flags))
        (vht (gethash bare-lib ht)))
    (let ((prev-lib-type (gethash :type vht)))
      (if (null prev-lib-type)
          (setf (gethash :type vht) lib-type)
          (when (not (equal prev-lib-type lib-type))
            (error "~A changed library type!" bare-lib)))))

  ;; For each arch, if the value list doesn't exist, make one and
  ;; insert it, otherwise insert the entry at the end of the list. We
  ;; do it at the end because we're following the search order as
  ;; found in the file.
  (let ((lib-arch (find-lib-arch split-flags))
        (vht (gethash bare-lib ht)))
    (let ((prev-lib-list (gethash lib-arch vht)))
      (if (null prev-lib-list)
          (setf (gethash lib-arch vht) (list absolute-lib))
          (rplacd (last (gethash lib-arch vht)) (list absolute-lib))))))

(defun parse-ld.so.cache (&key (program "/sbin/ldconfig") (args '("-p")))
  ;; Read all of the output of the program as lines.
  (let ((ht (make-hash-table :test #'equalp))
        (lines (stream->string-list
                   (out-stream)
                 (sb-ext:process-close
                  (sb-ext:run-program program args :output out-stream)))))

    ;; Pop the first line off, it is a count of libs and other junk
    (pop lines)

    ;; Assemble the master hash table which condenses the ldconfig -p info
    ;; into a meaningful object upon which I can query.
    (dolist (line lines)
      (register-groups-bind
          (bare-lib flags absolute-lib)
          ("\\s*(.*)\\s+\\((.*)\\)\\s+=>\\s+(.*)\\s*" line)

        (let ((split-flags
               (mapcar #'(lambda (str)
                           (setf str (regex-replace "^\\s+" str ""))
                           (regex-replace "\\s+$" str ""))
                       (split "," flags))))
          (merge-ld.so.cache-line bare-lib split-flags absolute-lib ht))))
    ht))

;; Convert a bare-lib into an absolute path depending upon
;; architecture and whatnot. Either returns an absolute path, or nil.
;;
;; Can specify an ordering of :all, :first (the default), or :last.
;; The ordering of :all will present the libraries in the order found
;; out of the output for ldconfig -p.
(defun query-ld.so.cache (bare-lib ht &key (ordering :first))
  (let ((vht (gethash bare-lib ht)))
    (if (null vht)
        nil
        (let ((all-absolute-libs (gethash (get-machine-type) vht)))
          (ecase ordering
            (:all
             all-absolute-libs)
            (:first
             (car all-absolute-libs))
            (:last
             (last all-absolute-libs)))))))

You'd use it like (suppose in SLIME's REPL):

CL-MW> (setf cache (parse-ld.so.cache))
#<HASH-TABLE :TEST EQUALP :COUNT 1275 {B546831}>
CL-MW> (query-ld.so.cache "libGL.so" cache)
"/usr/lib/mesa/libGL.so"
CL-MW> (query-ld.so.cache "libGL.so" cache :ordering :all)
("/usr/lib/mesa/libGL.so" "/usr/lib/libGL.so")
CL-MW> (query-ld.so.cache "libGL.so" cache :ordering :first)
"/usr/lib/mesa/libGL.so"
CL-MW> (query-ld.so.cache "libGL.so" cache :ordering :last)
("/usr/lib/libGL.so")

End of Line.

Hooked on Phonics

psilord — Sat, 28 Aug 2010 09:44:18 CDT

I've been learning about reader-macros (along with set-macro-character and get-macro-character) in Lisp recently. Reader macros are functions associated with characters--known as dispatching macro characters, in the readtable where the association may happen at load or runtime.

When invoked, the reader macro function is passed the stream and the character which invoked it. At this point the reader-macro can consume however many characters it wants (which could be enormous!) and implement a full lexical analysis and parsing algorithm for a totally different language embedded into Lisp. However, it must return either no value or a single lisp object--which could a list of other objects. Also, note that a reader macro may not have side effects because it is implementation defined if it'll get called a few times on the same portion of the stream due to things like backspace being used for standard input.

Examples of reader macro use would be to write a reader macro to convert a pure SQL statement into a lisp form, or convert Perl-like regular expressions into PPCRE calls (such as described in the book Let over Lambda). I'm curious about them because I want to write a heavily annotating lisp tokenizer which would have much of the behavior of a lisp reader, but keeps track of everything it does and where the tokens came from.

I found while writing the code for this post that knowing the above somehow wasn't good enough. I screwed up the understanding of reader macros pretty badly for a while. It could be because I'm likely stupid, but I'm going to believe that it is because of the Absinthe.

I've included the small examples I wrote so one can see how the reader macro works. After this, I'll describe the situation which really tripped me up for a while.

;;;; You are free to use, modify, and redistribute this
;;;; code. Attribution to me, Peter Keller (psilord@cs.wisc.edu), is
;;;; appreciated, but not required.  There is no warranty with this
;;;; code.

(defun reader-iota-0 (str ch)
  "(reader-iota-0 str ch)

  A reader-macro for forms like <ch><integer> which will return 
  a list of integers from 0 to (1- <integer>). If not quoted this list
  will be evaluated."

  (declare (ignorable ch))
  (let ((int (read str)))
    (loop for x from 0 to (1- int) collect x)))

(defun reader-iota-1 (str ch)
  "(reader-iota-1 str ch)

  A reader-macro for forms like <ch><integer> which will return 
  a list of integers from 1 to <integer>. If not quoted this list
  will be evaluated."

  (declare (ignorable ch))
  (let ((int (read str)))
    (loop for x from 1 to int collect x)))

(defmacro with-macro-character ((ch func) &body body)
  "(with-macro-character (ch func) body)

  Bind the reader macro func to ch and execute the body in this
  environment.  Restore the original reader-macros when this form is
  done."

  (let ((c (gensym))
        (f (gensym))
        (o (gensym)))
    `(let ((,c ,ch)
           (,f ,func))
       (let ((,o (get-macro-character ,c)))
         (set-macro-character ,c ,f)
         (unwind-protect
              (progn ,@body)
           (set-macro-character ,c ,o))))))

(defmacro with-macro-characters (pairs &body body)
  "(with-macro-characters ((ch1 func1) (ch1 func2) ...) body)

  Bind the reader macro func1 to ch1, and so on, and execute the body
  in this environment. Restore the original reader-macros when this
  form is done."

  (if (null pairs)
      `(progn ,@body)
      `(if (oddp (length ',(car pairs)))
           (error "with-macro-characters: ~A must be a pair of a character and a reader-macro-function" ',(car pairs))
           (with-macro-character ,(car pairs)
             (with-macro-characters ,(cdr pairs)
               ,@body)))))

(defun try-it ()
  (with-macro-characters
      ((#\! #'reader-iota-0)
       (#\@ #'reader-iota-1))
    (concatenate 'list
                 '(first)
                 (read-from-string "!10")
                 '(second)
                 (read-from-string "@10"))))

;; This next modification will take affect for the rest of the source
;; file including anything loaded after this since I'm doing it at the
;; toplevel. We use eval-when to ensure that the compiler's readtable
;; is modified while reading the source file.
;;
;; Note reversed assignment as opposed to the function try-it!
(eval-when (:compile-toplevel :load-toplevel :execute)
  (set-macro-character #\@ #'reader-iota-0)
  (set-macro-character #\! #'reader-iota-1))

(defun foo ()
  (let ((a '!10))
    (let ((b '@10))
      (concatenate 'list '(first) a '(second) b))))

The main thing I really learned was that a reader macro only is in effect when the lisp reader is active. That sounds completely obvious right--I mean, it is even in the name of the damn thing isn't it? However, there is one place where it is easy to misunderstand it. Lemme explain where it works and why first.

When you load a lisp file, load executes each toplevel form as it encounters it in order. If there happens to be a toplevel call to set-macro-character, it will be executed. The rest of the code in the lisp file, including anything loaded after that form, will have the reader macro available for use since the toplevel set-macro-character form adjusted the currently executing lisp reader which is in the process of loading the code. The function foo is a good example of this load time behavior since it occurs after the toplevel association of the reader macro functions.

Now, suppose you use the reader macro in a function, like try-it above. At this point, the readtable is only modified at runtime. The lisp reader is only affected when it is reading, suppose with READ, READ-CHAR, READ-SEQUENCE, or READ-FROM-STRING. We invoke the reader explicitly to convert the special form into lisp.

Here is the place where it doesn't intuitively work. Suppose I have this function which utilizes the reader macros defined above. It looks similar to the function foo in that you use the dispatch macro characters after you adjust the readtable, but it behaves very differently:

(defun wont-work ()
    (set-macro-character #\! #'reader-iota-0)
    (set-macro-character #\@ #'reader-iota-1)

    (let ((ret (concatenate 'list '(first) '!10 '(second) '@10)))

      (set-macro-character #\! nil)
      (set-macro-character #\@ nil)
      ret))

This won't work at all with the implication lexically suggested in the code. In fact, it won't even compile because you'll get an error similar to:

The value !10 is not of type SEQUENCE.

The reason why is simply that by the time this function is executing, it has already been completely processed by a lisp reader and '!10 and '@10 are symbols at execution time. No uses of the lisp reader occur in the function wont-work so changing the readtable at runtime here is meaningless. This took me longer than it should have to fully understand. There is much more to reader macros than I have mentioned here and I'm still reading the docs some more...

I will caution that messing with read macros, especially in a REPL, is often dangerous. If you mess it up or pick a bad character to be a dispatch macro character, say #\- or something along those lines, you'll get a pile of esoteric errors out of the corrupted lisp reader and it basically becomes useless. In these scenarios--unlike life, just exit the image, restart it, and try again.

End of Line.