Hobbes – A Language and Embedded JIT Compiler

Building

To build hobbes, you will need LLVM 3.3 or later, cmake 3.4 or later, GNU gcc 4.8 or later, and a version 2.5 or later Linux kernel. With LLVM, cmake, and g++ installed, after downloading this code you should be able to build and install hobbes just by running:

$ cmake .
$ make
$ make install

Depending on where you've installed LLVM, this might not work until you give cmake the path to LLVM's cmake modules. In particular, you want the path to LLVM's LLVMConfig.cmake file. If you set the environment variable LLVM_DIR to point to that directory, then the previous steps should complete successfully.

The build process will produce a static library, libhobbes.a, which can be linked into a C++ executable (if you want to use hobbes in a .so instead, the build produces a different static library to use, libhobbes-pic.a). In addition, the build process will produce two utility programs, hi and hog. The hi program is a basic interactive interpreter for hobbes expressions and the hog program will efficiently record structured data produced by applications into data files (these files can be loaded and queried at the same time by hi). The source code for these programs can be informative as well, because they demonstrate many different aspects of the hobbes API.

Embedding

Let's consider how to embed hobbes in a simple C++ program. This code implements a very basic shell, similar to hi:

#include <hobbes/hobbes.H>
#include <iostream>
#include <string>

int main() {
  hobbes::cc c;
  while (std::cin) {
    std::cout << "> " << std::flush;
    std::string line;
    std::getline(std::cin, line);
    try {
      std::cout << c.compile("print(" + line + ")")() << std::endl;
    } catch (std::exception& ex) {
      std::cout << ex.what() << std::endl;
    }
  }
  return 0;
}

To compile this program, we need to link against hobbes and LLVM:

$ g++ -std=c++11 -I <hobbes include path> -I test.cpp -o test \
    -L <hobbes lib path> -lhobbes -ldl -lrt -ltinfo -lz \
    -L `llvm-config --libs x86asmparser x86codegen x86 mcjit passes`

The explicit path statements may not be necessary depending on where/how LLVM and hobbes have been installed on your system. The inline invocation of the llvm-config program is typical with users of LLVM, to avoid explicitly listing several libraries. If everything worked correctly, we should now have a simple shell where we can evaluate hobbes expressions.

Evaluation

Now that we have a working shell, we can experiment with expressions to get a sense of how the hobbes language works. To start with, we have some typical constant values:

> 'c'
'c'
> 42
42
> 3.14159
3.14159

In total, these are the set of supported primitive types/constants:

These primitives can be combined with arrays:

> [1, 2, 3]
[1, 2, 3]
> "foobar"
"foobar"
> 0xdeadbeef
0xdeadbeef

They can also be combined with records/tuples:

> {name="Jimmy", age=45, job="programmer"}
{name="Jimmy", age=45, job="programmer"}
> ("Jimmy", 45, "programmer")
("Jimmy", 45, "programmer")

Or with arrays of records/tuples, where they will print conveniently as a table:

> [{name="Jimmy", age=45, job="programmer"}, {name="Chicken", age=40, job="programmer"}]
name    age job
------- --- ----------
Jimmy   45  programmer
Chicken 40  programmer

> [("Jimmy", 45, "programmer"),("Chicken", 40, "programmer")]
Jimmy   45  programmer
Chicken 40  programmer

We can combine types with variants or sums (the "nameless" form of variants, as tuples are to records). Because "naked" variant introduction is incomplete by definition, we may sometimes need to introduce explicit type annotations:

> |food="pizza"|::|vehicle:int,food:[char]|
|food="pizza"|
> |1="pizza"|::(int+[char])
|1="pizza"|

Also, variants can be combined with isorecursive types to make linked lists. These have a convenient form when printed, and built-in functions to avoid the recursive and variant type boilerplate:

> roll(|1=("chicken", roll(|1=("hot dog", roll(|1=("pizza", roll(|0=()|))|))|))|) :: ^x.(()+([char]*x))
"chicken":"hot dog":"pizza":[]
> cons("chicken", cons("hot dog", cons("pizza", nil())))
"chicken":"hot dog":"pizza":[]

These types and type constructors together make up the "algebraic data types" common in the ML family of programming languages (SML, ocaml, Haskell, ...). The syntax and names for these types also come from ML and common academic programming language textbooks as in TaPL and PFPL.

As in Haskell, hobbes supports a form of qualified types with type classes and even user-defined constraint resolvers (which we will see in more detail later). Among many other uses, this allows both mixed-type arithmetic and type inference to coexist:

> 0X01+2+3.0+4L+5S
15
> (\x y z u v.x+y+z+u+v)(0X01,2,3S,4L,5.0)
15

We can also use the hi program (a slightly more complex interpreter than the example earlier, distributed with hobbes) to evaluate expressions like this and also inspect their types. For example, the types for the primitive expressions we considered earlier can be queried:

$ hi
hi : an interactive shell for hobbes
type ':h' for help on commands

> :t ()
()
> :t false
bool
> :t 'c'
char
> :t 0Xff
byte
> :t 42S
short
> :t 42
int
> :t 42L
long
> :t 42.0f
float
> :t 42.0
double

And the types for record, array, variant and recursive values:

> :t [1..10]
[int]
> :t "foobar"
[char]
> :t 0xdeadbeef
[byte]
> :t {name="Jimmy", age=45, job="programmer"}
{ name:[char], age:int, job:[char] }
> :t ("Jimmy", 45, "programmer")
([char] * int * [char])
> :t cons("foo",nil())
^x.(() + ([char] * x))

Also for qualified types, we can use this feature to view the set of open/unresolved type constraints:

> :t \x.x.foo
(a.foo::b) => (a) -> b
> :t .foo
(a.foo::b) => (a) -> b

Here we see that the type of a simple function which just returns the value of the "foo" field in its input (both in the fully explicit \x.x.foo form as well as the equivalent .foo shorthand) has a constraint (the part to the left of the => in its type -- the same as in Haskell). In this case, the entire type states that if a type a has a field named foo with type b, the function has type a -> b. The additional complexity of this constraint allows this function to work with any type a meeting that condition. This particular constraint allows a kind of duck typing for functions.

Storage

If we're using hobbes inside of a C++ process (as opposed to just running scripts with hi), at some point we will need to bind C++ values (typically functions) to a hobbes::cc instance so that they can be used in dynamic expressions at a later stage. Much of this work is automated, often allowing us to avoid thinking about binding logic.

For example, consider a C++ function that we might have already defined:

int applyDiscreteWeighting(int x, int y) {
  return x * y;
}

Now we can import this definition into our previous program, and bind it to our hobbes::cc instance by adding this line:

c.bind("applyDiscreteWeighting", &applyDiscreteWeighting);

Then when we run our program, its shell will allow use of this function and will reject any expression using it in a way inconsistent with its type structure:

> applyDiscreteWeighting(3, 4)
12
> applyDiscreteWeighting(3, "pizza")
*** stdin:1,1-22: Cannot unify types: int != [char]

In other cases, it may be more convenient to use a type understood natively by hobbes and with dynamic memory allocation:

typedef std::pair<size_t, const hobbes::array<char>*> MyRecord;

const hobbes::array<MyRecord>* loadRecords(int key) {
  static const char* names[] = { "chicken", "hot dog", "foobar", "jimmy" };
  auto rs = hobbes::makeArray<MyRecord>(key);
  for (size_t i = 0; i < rs->size; ++i) {
    rs->data[i].first = i;
    rs->data[i].second = hobbes::makeString(names[i % 4]);
  }
  return rs;
}

// [...]
c.bind("loadRecords", &loadRecords);
// [...]

These calls to hobbes::makeArray and hobbes::makeString will allocate out of the calling thread's memory region and will produce data that hobbes knows how to generically destructure and print in a convenient form, as we can see by running this in our test program again:

> loadRecords(10)
0 chicken
1 hot dog
2 foobar
3 jimmy
4 chicken
5 hot dog
6 foobar
7 jimmy
8 chicken
9 hot dog

If we'd prefer to represent records by structs with significant field names rather than as tuples, we can instead define our type with the DEFINE_STRUCT macro (this is purely a syntactic difference -- the final runtime representation of tuples and structs is identical in hobbes). For example, if we'd defined the MyRecord type in the previous example this way:

DEFINE_STRUCT(MyRecord,
  (size_t, index),
  (const hobbes::array<char>*, name)
);

And change the initialization code in loadRecords to refer to these fields by name:

// [...]
rs->data[i].index = i;
rs->data[i].name = hobbes::makeString(names[i % 4]);
// [...]

Then we can access those fields by name from hobbes and e.g. tables of such records will be printed with a header showing these field names:

> loadRecords(10)
index name
----- -------
0     chicken
1     hot dog
2     foobar
3     jimmy
4     chicken
5     hot dog
6     foobar
7     jimmy
8     chicken
9     hot dog

We can also bind "in reverse" using C function types, so that rather than making our C++ code appear to be valid hobbes symbols, we can make hobbes expressions appear to be valid C++ symbols. For example, consider a C++ binding like:

int iter2(int (*pf)(int,int), int x) {
  return pf(pf(x, x), pf(x, x));
}

// [...]
c.bind("iter2", &iter2);
// [...]

Here we've bound a function that itself takes a function as input, and we can use the hobbes syntax for anonymous functions to decide logic that this C++ function will use:

> iter2(\x y.x*x+y, 2)
42

Variants can be passed between C++ and hobbes as the hobbes::variant type:

typedef hobbes::variant<int, const hobbes::array<char>*> classification;

const classification* classifyParameter(int x) {
  if (x < 100) {
    return hobbes::make<classification>(42 - x);
  } else {
    return hobbes::make<classification>(hobbes::makeString("achievement unlocked"));
  }
}

// [...]
c.bind("classifyParameter", &classifyParameter);
// [...]

These should also print in a standard way (showing the constructor name "0" for the first case and "1" for the second case, since the constructors for this variant have no names), assuming that the payload types of your variant can be printed:

> classifyParameter(16)
|0=26|
> classifyParameter(8675309)
|1="achievement unlocked"|

In the process of binding application variables and functions to a hobbes::cc instance, hobbes will decide how to "lift" every C++ type to an equivalent hobbes type. Many common cases are already handled transparently, as above where types like size_t and std::pair and hobbes::array are "lifted" without any special work required on our part.

If hobbes doesn't know how to lift a type, it will be lifted as an "opaque C++ type" such that hobbes just checks for consistent usage (and will respect class hierarchies and derived/base conversions as necessary) but will otherwise know nothing about the structure of the type. For example, consider a binding case like this (meant to represent a complex type hierarchy):

class Top {
public:
  Top(int x, double y, int z) : x(x), y(y), z(z) { }
  int getX() const { return this->x; }
  double getY() const { return this->y; }
  int getZ() const { return this->z; }
private:
  int x;
  double y;
  int z;
};

class Left : public virtual Top {
public:
  Left(int x, double y, int z) : Top(x, y, z) { }
};

class Right : public virtual Top {
public:
  Right(int x, double y, int z) : Top(x, y, z) { }
};

class Bottom : public Left, public Right {
public:
  Bottom(int x, double y, int z) : Left(x*z, y*y, z*x), Right(x*x, y*y, z*z), Top(x, y, z) { }
};

// [...]
c.bind("getX", memberfn(&Top::getX));
c.bind("getY", memberfn(&Top::getY));
c.bind("getZ", memberfn(&Top::getZ));

static Bottom b(1, 3.14159, 42);
c.bind("b", &b);
// [...]

Now back in our shell, we can invoke the bound Top methods on our bound Bottom instance (hobbes will coordinate with gcc to apply the correct pointer adjustments):

> b.getX()
1
> b.getY()
3.14159
> b.getZ()
42

If the default logic for lifting your C++ type is insufficient, the hobbes::lift type can be specialized to determine a more useful description (the code in hobbes/lang/tylift.H has many examples of this; this is where the default set of lift definitions are made).

Networking

Rather than using hobbes to "pull" application data with ad-hoc dynamic queries, we can also use hobbes to "push" large volumes of application data to local or remote storage for real-time or historical analysis. In this case, we don't need a compiler instance but instead just need to define some basic quality-of-service parameters for the storage process.

For example, let's consider a simple mock weather monitor:

#include <hobbes/hobbes.H>

DEFINE_STORAGE_GROUP(
  WeatherMonitor,
  /* an arbitrary name for */