A proposal to simplify calling L# from C#

Note: this has also been posted to its own thread in the LSharp Google Group. If you want to discuss the merits of this proposal, please consolidate all comments there. Thanks.

Calling L# from C#

Right now, it's much harder than it needs to be to call L# code directly from C#. Since one obvious use of L# is as a scripting language for .Net applications, it would be nice if it was really simple. Of course, one good way to make this happen would be with an L# compiler that produces a .Net assembly that can be called from any .Net language. However, as useful as this would be, it still might not be ideal for some application scripting scenarios. In some situations, it may be more appropriate to use the interpreter (which already exists), and simplify the interface to make it easier to call from C# and other languages.

Suppose you have a lisp environment already running, and you want to call a simple function you have defined. The easiest way to call it right now is probably to create an L# string and evaluate it, like so:

object result = Runtime.EvalString("(my-function)",
    new LSharp.Environment());

This isn't a very clean way to work with code, but it should work in simple cases. In more elaborate cases, building correct strings gets to be an increasingly elaborate problem. And really, even if it weren't that hard, marshalling data to and from strings for interoperation isn't an especially good practice.

The other way to call L# code from C# is to work with the actual L# objects: symbols, functions, closures, and so on. Here is some simple example code for calling the system function map and the user-defined function (technically, a Closure object) concat from C#:

LSharp.Environment env = new LSharp.Environment();
Console.WriteLine(Functions.Load(new Cons("mylib.ls"),
    env));

// build the arguments for map
ArrayList myarraylist = new ArrayList();
myarraylist.Add(env.GetValue(Symbol.FromName("double")));
Cons intCons = Cons.FromArray(new object[]
    { 1, 2, 3, 4, 5 });
myarraylist.Add(intCons);

// call map
Cons args = Cons.FromICollection(myarraylist).Reverse();
Function map = (Function)env.GetValue(
    Symbol.FromName("map"));
Console.WriteLine(map(args, env));

// call concat
object[] myarray = { "foo", "bar", 42, "baz" };
args = Cons.FromArray(myarray);
Closure concat = (Closure)env.GetValue(
    Symbol.FromName("concat"));
Console.WriteLine(concat.Invoke(args));

The library that is being loaded, mylib.ls is simply:

; mylib.ls -- simple sample library
;
; Copyright (c) 2006, Andrew Norris
; Simple license: reuse this however you like.

(= double (fn (x) (* x 2)))

(= concat (fn (&rest items)
    (let sb (new system.text.stringbuilder)
      (foreach elem items
        (call append sb elem))
      (tostring sb ))))

When the above C# code executes, it will output:

(2 4 6 8 10)
foobar42baz

Using the L# interpreter objects directly is a better idiom in general, because it doesn't require data structures to be marshalled as strings, and because it allows for some things to be checked at compile-time. However, it's obvious that this code is pretty cumbersome, if all you want to do is call a couple of functions.

A side note: the ability to call a couple of trivial functions like this generally isn't worth the trouble of embedding the L# interpreter in your C# application. However, there are some things that may be much easier to write in L#, such as list manipulation functions or code that uses macros effectively.

1. Add a string indexer to Environment

It would be nice if it were easier to access symbol values in the L# Environment. C# doesn't have symbols, so you have to create one from a string. The GetValue call adds some an additional operation as well. This seems like a good place to start in making it simpler to call L#.

The current method of dereferencing a symbol isn't that complex an operation if it's called occasionally, but working with symbols in the environment is one of the two most basic operations in L#, along with calling a function. To be usable, it needs to be concise. A string indexer makes this code much simpler:

LSharp.Environment env = new LSharp.Environment();
Console.WriteLine(Functions.Load(new Cons("mylib.ls"), env));

// build the arguments for map
ArrayList myarraylist = new ArrayList();
myarraylist.Add(env["double"]);
Cons intCons = Cons.FromArray(new object[] { 1, 2, 3, 4, 5 });
myarraylist.Add(intCons);

// call map
Cons args = Cons.FromICollection(myarraylist).Reverse();
Function map = (Function)env["map"];
Console.WriteLine(map(args, env));

// call concat
object[] myarray = { "foo", "bar", 42, "baz" };
args = Cons.FromArray(myarray);
Closure concat = (Closure)env["concat"];
Console.WriteLine(concat.Invoke(args));

As you can see, it doesn't affect the rest of the code, but it makes recovering the functions from the environment much simpler.

Adding a basic string indexer to Environment is simple. The code looks like this:

public object this[string s]
{
    get { return GetValue(Symbol.FromName(s)); }
    set { Assign(Symbol.FromName(s), value); }
}

Of course, this only gets things started. There are a lot of other things we can improve as well.

2. Simplify the interface for converting a .Net data structure to a Cons

The interface for converting a collection of items to a Cons can usefully be simplified. Cons contains some useful conversion functions, and SpecialForms.The() does a good job of wiring them up. But, really, anything that can be enumerated can be easily converted to a Cons, if we add a new method. Cons.FromIEnumerable() is simple, and virtually identical to Cons.FromICollection():

public static Cons FromIEnumerable(IEnumerable enumerable)
{
    object list = null;
    foreach (object o in enumerable)
    {
        list = new Cons(o, list);
    }
    return (Cons)list.Reverse();
}

This doesn't have a significant impact on the code right away. But it will enable us to do some more important steps later on.

LSharp.Environment env = new LSharp.Environment();
Console.WriteLine(Functions.Load(new Cons("mylib.ls"), env));

// build the arguments for map
ArrayList myarraylist = new ArrayList();
myarraylist.Add(env["double"]);
Cons intCons = Cons.FromIEnumerable(new object[] { 1, 2, 3, 4, 5 });
myarraylist.Add(intCons);

// call map
Function map = (Function)env["map"];
Cons args = Cons.FromIEnumerable(myarraylist);
Console.WriteLine(map(args, env));

// call concat
object[] myarray = { "foo", "bar", 42, "baz" };
args = Cons.FromIEnumerable(myarray);
Closure concat = (Closure)env["concat"];
Console.WriteLine(concat.Invoke(args));

It also enables you to convert any class that implements IEnumerable:

Cons cons = Cons.FromIEnumerable(someRandomThingWithAnEnumeration);

The advantage of this is that whenever something new comes along that needs to be turned into a Cons, it can already be done if there is any standard way to reference all the items.

3. Make it easy to convert nested .Net data structures to Conses

After creating the code to consistently convert enumerable data structures to Conses, one obvious problem is that it won't automatically handle nested data structures. In the following code, the array and the ArrayList have to be converted separately:

ArrayList myarraylist = new ArrayList();
myarraylist.Add(env["double"]);
Cons intCons = Cons.FromArray(new object[] { 1, 2, 3, 4, 5 });
myarraylist.Add(intCons);

Cons args = Cons.FromICollection(myarraylist).Reverse();

Since nested lists -- often, deeply nested ones -- are one of the most basic building blocks of Lisp code, it would be nice if we could convert the whole data structure at once. Forutunately, if we extend the FromIEnumerable method we just built, there's a straightforward solution:

public static Cons FromIEnumerable(IEnumerable enumerable, bool isRecursive)
{
    object list = null;
    foreach (object o in enumerable)
    {
        if (isRecursive && o is IEnumerable) {
            o = FromIEnumerable(o as IEnumerable, true);
        }
        list = new Cons(o, list);
    }
    return (Cons)list;
}

Now we can easily convert the data structure to a nice, Lispy nested list in one step:

ArrayList myarraylist = new ArrayList();
myarraylist.Add(env["double"]);
int[] intArray = new object[] { 1, 2, 3, 4, 5 });
myarraylist.Add(intArray);

Cons args = Cons.FromIEnumerable(myarraylist, true);

This doesn't have a significant impact on our sample code here, but for cases where there is data that is already in elaborate .Net data structures and needs to be passed to L#, it will make things significantly easier, and avoid the need to walk the data structure tree.

4. Simplify building Conses in C#

Passing nested data to an L# function from C# is still fairly cumbersome. While building arrays and ArrayLists is often more natural for working with C# data, it's cumbersome for packaging up arguments, as we saw in the last section when we built the arguments to pass to map. Consider the code we've just been looking at:

ArrayList myarraylist = new ArrayList();
myarraylist.Add(env["double"]);
int[] intArray = new object[] { 1, 2, 3, 4, 5 });
myarraylist.Add(intArray);

Cons args = Cons.FromIEnumerable(myarraylist, true);

This could be done differently by building a Cons directly, of course:

Cons intCons = Cons.FromArray(new object[] { 1, 2, 3, 4, 5 });
Cons args = new Cons(intCons);
Cons args = new Cons(env["double"], args);

This is simpler, but still harder than it needs to be. Also, it either involves performing the steps out of the usual order -- this is normal in Lisp, but unusual in C# -- or adding a Reverse() operation each time.

By contrast, in L#, the equivalent code is simply:

(map double '(1 2 3 4 5))

Obviously, if we could simplify the C# version, it would be a lot easier to package arguments and call L# functions. For example, this code would be much closer to ideal:

Cons args = Cons.Build(env["double"], Cons.Build(1, 2, 3, 4, 5));

That can be implemented relatively straightforwardly by creating a Build method that can take any number of arguments and simply build a list out of them:

public static Cons Build(params object[] items)
{
    Object cons = null;
    for (int i = items.Length - 1; i >= 0; i--)
    {
        cons = new Cons(items[i], cons);
    }
    return (Cons)cons;
}

This simplifies the example code to:

LSharp.Environment env = new LSharp.Environment();
Console.WriteLine(Functions.Load(new Cons("mylib.ls"), env));

// call map
Cons args = Cons.Build(env["double"], Cons.Build(1, 2, 3, 4, 5));
Function map = (Function)env["map"];
Console.WriteLine((Cons)map(args, env));

// call concat
object[] myarray = { "foo", "bar", 42, "baz" };
args = Cons.FromIEnumerable(myarray, false);
Closure concat = (Closure)env["concat"];
Console.WriteLine(concat.Invoke(args));

5. Provide an identical interface for functions or closures

Wouldn't it be nice if you could have one simple way to call an L# operation, regardless of whether it was a system function, a user-defined function (closure), or a special form? And wouldn't it be nice if you could pass the operation whatever data structure you had without converting it?

For example, here's what the code to call map and concat might look like if you could do that:

Cons result = (Cons)map(args);

string sresult = (string)concat(myarray);

That's a lot simpler than the earlier example, right? Well, with delegates, it's easy to produce that interface:

public delegate Object fn(IEnumerable arguments);

There's really only one problem with this delegate: it doesn't match the signatures of Closure or Function, so you can't use it, at least directly. Fortunately, there's a way around that problem:

6. Build an implementation class that can make function calls simple

Although the interface to Closure and Function doesn't match the fn delegate signature, it's straightforward to make an adapter class that makes them match. Basically, the idea is to automatically translate data structures to Conses, and automatically bind functions to the current environment. The class then contains a method with the proper signature for the fn delegate, and a property that will return it.

public class FunctionBinding
{
    private Environment _env;
    private Function _f;
    private Closure _c;
    private bool _isClosure;

    public FunctionBinding(Environment env, Function f)
    {
        _f = f;
        _env = env;
        _c = null;
        _isClosure = false;
    }

    public FunctionBinding(Closure c)
    {
        _f = null;
        _env = null;
        _c = c;
        _isClosure = true;
    }

    private Object Impl(IEnumerable arguments)
    {
        // special handling for closures that take no
        // arguments and functions with empty lists
        if (arguments == null)
        {
            if (_isClosure)
                return _c.Invoke();
            else
                return _f(new Cons(null), _env);
        }

        // build a cons out of the argument
        Cons cons = null;
        if (arguments is Cons)
            cons = arguments as Cons;
        else
            cons = Cons.FromIEnumerable(arguments, true);

        // invoke the closure or function
        if (_isClosure)
            return _c.Invoke(cons);
        else
            return _f(cons, _env);
    }

    public fn BoundFunction
    {
        get { return Impl; }
    }

    public static fn Bind(Environment env, Function f)
    {
        FunctionBinding fb = new FunctionBinding(env, f);
        return fb.BoundFunction;
    }
}

This makes it really simple to call L# functions, but complicates retrieving them. The problem is that we have to build a FunctionBinding object around the return value we get from the environment. At this point, our code would work like this:

LSharp.Environment env = new LSharp.Environment();
Console.WriteLine(Functions.Load(new Cons("mylib.ls"), env));

// call map
Cons args = Cons.Build(env["double"], Cons.Build(1, 2, 3, 4, 5));
fn map = (new FunctionBinding(env, (Function)env["map"])).BoundFunction;
Console.WriteLine(map(args));

// call concat
object[] myarray = { "foo", "bar", 42, "baz" };
fn concat = (new FunctionBinding((Closure)env["concat"])).BoundFunction;
Console.WriteLine(concat(myarray));

7. Wire things up so FunctionBindings are created automagically

Fortunately, we can simplify things. By modifying the string indexer we built earlier, we can automatically bind functions and closures, and return the easy-to-call bound functions.

public object this[string s]
{
    get
    {
        object o = GetValue(Symbol.FromName(s));
        if (o is Function)
            o = (new FunctionBinding(this, (Function)o)).BoundFunction;
        else if (o is Closure)
            o = (new FunctionBinding((Closure)o)).BoundFunction;
        return o;
    }
    set { Assign(Symbol.FromName(s), value); }
}

This will also convert our call to the double function as well, which means the interpreter needs to be able to handle fn delegates:

public static object Apply (object function, object arguments, Environment environment) 
{
    if (function.GetType() == typeof(Function)) 
    {
        return ((Function) function) ((Cons)arguments,environment);
    }

    // If function is an LSharp Closure, then invoke it
    if (function.GetType() == typeof(Closure)) 
    {
        if (arguments == null)
            return ((Closure)function).Invoke();
        else
            return ((Closure)function).Invoke((Cons)arguments);
    }

    if (function is fn)
    {
        return ((fn)function)((IEnumerable)arguments);
    }
    else 
    {
        // It must be a .net method call
        return Call(function.ToString(),(Cons)arguments);
    }
}

After this final change, our code simplifies to this:

LSharp.Environment env = new LSharp.Environment();
Console.WriteLine(Functions.Load(new Cons("mylib.ls"), env));

// call map
Cons args = Cons.Build(env["double"], Cons.Build(1, 2, 3, 4, 5));
fn map = (fn)env["map"];
Console.WriteLine(map(args));

// call concat
object[] myarray = { "foo", "bar", 42, "baz" };
fn concat = (fn)env["concat"];
Console.WriteLine(concat(myarray));

And if you want, you can simplify it even further, though it makes the code a bit denser:

LSharp.Environment env = new LSharp.Environment();
Console.WriteLine(Functions.Load(new Cons("mylib.ls"), env));

// call map
Cons args = Cons.Build(env["double"], Cons.Build(1, 2, 3, 4, 5));
Console.WriteLine(((fn)env["map"])(args));

// call concat
object[] myarray = { "foo", "bar", 42, "baz" };
Console.WriteLine(((fn)env["concat"])(myarray));

Conclusion

Implementing these changes should make it much easier to call L# code from inside C#, and without resorting to building evaluation strings. Hopefully, this will make it more practical to take sections of code that can be implemented much more easily in L# than in C# and build a multiple-language implementation. For example, in a program that has a section that requires elabortate list manipulation, but other sections that need to be in C#.

With these changes, we were able to simplify our sample code from

LSharp.Environment env = new LSharp.Environment();
Console.WriteLine(Functions.Load(new Cons("mylib.ls"), env));

// build the arguments for map
ArrayList myarraylist = new ArrayList();
myarraylist.Add(env.GetValue(Symbol.FromName("double")));
Cons intCons = Cons.FromArray(new object[] { 1, 2, 3, 4, 5 });
myarraylist.Add(intCons);

// call map
Function map = (Function)env.GetValue(Symbol.FromName("map"));
Cons args = Cons.FromICollection(myarraylist).Reverse();
Console.WriteLine(map(args, env));

// call concat
object[] myarray = { "foo", "bar", 42, "baz" };
args = Cons.FromArray(myarray);
Closure concat = (Closure)env.GetValue(Symbol.FromName("concat"));
Console.WriteLine(concat.Invoke(args));

to

LSharp.Environment env = new LSharp.Environment();
Console.WriteLine(Functions.Load(new Cons("mylib.ls"), env));

// call map
Cons args = Cons.Build(env["double"], Cons.Build(1, 2, 3, 4, 5));
Console.WriteLine(((fn)env["map"])(args));

// call concat
object[] myarray = { "foo", "bar", 42, "baz" };
Console.WriteLine(((fn)env["concat"])(myarray));

Of course, to do the equivalent in L#, the code is

(load "mylib.ls")

(prl (map double '(1 2 3 4 5)))

(prl (concat "foo" "bar" 42 "baz"))

So L# functions still aren't nearly as easy to work with from C# as they are natively. Even keeping this in mind, these changes should make it a lot more practical to embed direct calls to L# code in C#.

The main thing this proposal hasn't touched on, of course, is macros. If you have macros embedded in L# functions, this won't make any difference -- the macros get expanded at read time, and your function can be called from C# just like any other function. But what if you want to call a macro directly? Or what if you want to embed a macro call inside the arguments to the function you call?

The techniques discussed in this article won't handle macros correctly, because by the time a function is being called, L# assumes all macros have been expanded. In a future post, I'll discuss how to use similar C# techniques with macros.

Finally, in the future, adding compiled code will be an important way to make L# code more accessible from C#. That will allow L# libraries to be attached at compile time and statically bound, rather than loaded and called at runtime. While that will be a powerful technique for some situations, the ability to easily call interpreted L# at runtime will still have its place in many other scenarios.

Tags: .Net, L#, LSharp, C#, Lisp