Monday, February 25, 2013

Moving away from XText

When I started with PSHDL, I used XText for developing the language. It seemed very beneficial for me as I had plenty of experience with it (some of which was bad experience), but others were very good. If you think about it, the idea of XText is pretty nice. You create a quite straight forward language definition and by some magic you will get a complete Eclipse Editor with syntax highlighting, outline, rename refactoring, linking etc. This is really awesome. But as PSHDL evolved it become harder to justify the usage of XText. After all it comes with some baggage.

If you don't know XText, go check it out. One of the first things that annoyed me however is the fact that the generated model is based on EMF and Eclipse. So everything is fine as long as you want is an Eclipse editor. However I discovered that PSHDL has to be more than just an Eclipse editor. While it is possible to create a command line version of it, it comes with some nasty side-effects. The first one is that it depends on none less than 45 external jars to run. This makes the compiler executable a 25MBytes biggy that takes 850ms just to fire up a completely useless OSGI and plugin management system. Compiling a bigger piece of code takes 5s.

As "replacement" I used ANTLR4. This of course does not give me a full blown Eclipse IDE, but it gives me the freedom to implement other things. For example I can now access the comments and bring most of them over to the VHDL code. This helps to bring some documentation to VHDL. I can also add documenting Comments JavaDoc style. This allows me to annotate the port declaration. Another thing is that I can give better error messages and provide better guidance towards a correct source in case of syntax errors. Also the compiler can now become much smaller. It is now 3.8MBytes and take 3.8s to compile the same source.

But what is much more important than anything else: I can now easily embed it into any IDE that I like, not just Eclipse. People are more willing to integrate it in their workflow if it is small and fast. The downside is: I have to develop my own Eclipse tooling. But I am fairly familiar with Eclipse and so that is no big problem for me.

Thursday, February 7, 2013

Using XTend

In the previous post I explained how I moved away from AspectJ towards XTend, so I want to share my experience in migrating. Let's start with a quick list of things I like:

Things that are cool

  • Polymorphic runtime dispatch
    • This was the main reason to switch to XTend
  • Template Strings
    • They can come in very handy when you want to create stuff like for example a HTML file
  • Extension Methods
    • You can "extend" any type and make things look more OOP like instead of the regular procedural code where you don't see the what the operation is actually performed on
  • The whole accessing getters as fields
    • I just have so many getters that it makes a lot of sense
  • The reduced noise by leaving ; away

A cool example:

def dispatch String toString(HDLConcat concat, SyntaxHighlighter highlight) 
  '''«FOR HDLExpression cat : concat.cats SEPARATOR highlight.operator("#")» «highlight.operator(cat.toString(highlight))» «ENDFOR»'''

Instead of:

public String HDLConcat.toString(SyntaxHighlighter highlight) {
 StringBuilder sb = new StringBuilder();
 String spacer = "";
 for (HDLExpression cat : getCats()) {
  spacer = highlight.operator("#");
 return sb.toString();

Things I am not passionate about:

  • val and var it's quite alright that you can leave the type away or that you can declare something as final, but I like my types visible and so I declare them anyway in most cases
  • The tooling. It is descent, but not yet something to brag about
  • The "everything is an expression" idea. Ternary operators are nice, no need to replace them with if then else constructs. So far I always wrote a return statement instead of relying on the "the last statement is the return value" thingy.
  • The all imports should be explicit (vs. the usage of wildcards)

Things that are not optimal:

  • In order to use nested classes, you will have to use the $ sign to refer to them. Knowing that this is the JVM type of referring to them is not a good excuse.

Things that are plain stupid:

  • As of now you can not allocate Arrays (because the [] brackets are used for closures). This however is already a planned change
  • There are no character literals. Because of stupid auto boxing I had to write a little helper method instead

So, instead of simply writing:

Character.toString((char) (i + 'I'))

I know had to write:

def String asIndex(Integer integer) {
    val int i='I'.charAt(0)
    return Character::toString((i + integer) as char);

Because this can be seen as an extension method, I was however able to write this:

  • Working with arrays yields horrible performance because they are wrapped to a list
  • Working with enums, especially in switch cases is awkward to say the least

Here is a simple example:

switch (obj.presentation){
case HDLLiteral$HDLLiteralPresentation::STR:
    return null
case HDLLiteral$HDLLiteralPresentation::BOOL:
    return null

Where is your the type inference now? To be fair, you can create a static import for those enums, but switches are strange beasts in XTend. They are not real switch cases translated 1:1 to Java, instead they are cascaded if statements. Also fall throughs and empty cases are not supported.

switch (type: obj.type) {
case type==OR || type==XOR:{
    return Ranges::closed(ZERO, ONE.shiftLeft( leftRange.upperEndpoint.bitLength ).subtract( ONE ))
case AND: {
    return Ranges::closed(ZERO, leftRange.upperEndpoint.min( ONE.shiftLeft( rightRange.upperEndpoint.bitLength ).subtract( ONE )))
case type==LOGI_AND || type==LOGI_OR: {
    return Ranges::closed(ZERO, ONE)

The resulting Java code for the last example is:

boolean _matched = false;
if (!_matched) {
 boolean _or = false;
 boolean _equals = Objects.equal(type, HDLBitOpType.OR);
 if (_equals) {
   _or = true;
 } else {
   boolean _equals_1 = Objects.equal(type, HDLBitOpType.XOR);
   _or = (_equals || _equals_1);
 if (_or) {
   BigInteger _upperEndpoint = leftRange.upperEndpoint();
   int _bitLength = _upperEndpoint.bitLength();
   BigInteger _shiftLeft = BigInteger.ONE.shiftLeft(_bitLength);
   BigInteger _subtract = _shiftLeft.subtract(BigInteger.ONE);
   return Ranges.<BigInteger>closed(BigInteger.ZERO, _subtract);
if (!_matched) {
 if (Objects.equal(type,HDLBitOpType.AND)) {
   BigInteger _upperEndpoint_1 = leftRange.upperEndpoint();
   BigInteger _upperEndpoint_2 = rightRange.upperEndpoint();
   int _bitLength_1 = _upperEndpoint_2.bitLength();
   BigInteger _shiftLeft_1 = BigInteger.ONE.shiftLeft(_bitLength_1);
   BigInteger _subtract_1 = _shiftLeft_1.subtract(BigInteger.ONE);
   BigInteger _min = _upperEndpoint_1.min(_subtract_1);
   return Ranges.<BigInteger>closed(BigInteger.ZERO, _min);
if (!_matched) {
 boolean _or_1 = false;
 boolean _equals_2 = Objects.equal(type, HDLBitOpType.LOGI_AND);
 if (_equals_2) {
   _or_1 = true;
 } else {
   boolean _equals_3 = Objects.equal(type, HDLBitOpType.LOGI_OR);
   _or_1 = (_equals_2 || _equals_3);
 if (_or_1) {
   return Ranges.<BigInteger>closed(BigInteger.ZERO, BigInteger.ONE);

My summary

XTend is a nice language, but in some places strange decisions have been made. Regarding the language, as well as regarding the realization of the generated code. Here a last example that is just plain horrible:

def static listTest() {
    val list=new ArrayList<String>
    for (i:0..list.size-1)
        System::out.println("Item:"+i+" is "+list.get(i))

If the list is empty (as it is because I didn't add anything) it will crash because the index 0 is out of range. And it is not like you could use the plain java style for loop. You have to create a range...

Switching away from AspectJ

Up until now I used AspectJ to insert functions into the AST. This allowed me to conveniently call something like .constantEvaluate() on any node and get back the value if applicable. Also all functionality for creating a constant from the AST was encapsulated into one file. For example for Literals this would look like this:
public BigInteger HDLLiteral.constantEvaluate(HDLEvaluationContext context) {
 switch (getPresentation()){
  case STR:
  case BOOL:
   return null;
   return getValueAsBigInt();

This would insert the function constantEvaluate into the type HDLLiteral. On a byte code level this function would really been copied into it. But the downside of using AspectJ is that this really only happens at the byte code level. And so quite a few tools, including GWT, could not use the result. I really like GWT however and want to be able to do fancy stuff with it on So I needed to find a replacement. Just inserting the functions into the respective classes was not an option to me, as I really want to have a good seperation of concerns. So I decided to use XTend.

The nice thing about XTend is that is has a polymorphic dispatch. You essentially just declare a bunch of overloaded functions and during runtime XTend will dispatch the call to the most specific function. One would assume that Java does this automatically, but the called method is determined at compile time. Take this example:
protected CharSequence _print(final Integer i) {
 StringConcatenation _builder = new StringConcatenation();
 _builder.append(i, "");
 return _builder;

protected CharSequence _print(final String s) {
 StringConcatenation _builder = new StringConcatenation();
 _builder.append(s, "");
 return _builder;

protected CharSequence _print(final Object o) {
 StringConcatenation _builder = new StringConcatenation();
 _builder.append(o, "");
 return _builder;

public CharSequence print(final Object i) {
 if (i instanceof Integer) {
  return _print((Integer) i);
 } else if (i instanceof String) {
  return _print((String) i);
 } else if (i != null) {
  return _print(i);
 } else {
  throw new IllegalArgumentException("Unhandled parameter types");

public static void main(String[] args) {
 Bla b = new Bla();
 Integer i = 5;
 String s = "Hallo";
 Object o = "Moin";

The output will be:


What you can see is that the called method depends on the declared type, not the acutal type. This is fixed with the call of print vs _print. To generated the print method, all you have to do is write the following in XTend:
class Bla {

 def dispatch print(Integer i)'''Integer:«i»'''
 def dispatch print(String s)'''String:«s»'''
 def dispatch print(Object o)'''Object:«o»'''
With that in place, you can create a very nice collection of constantEvaluate functions that automatically dispatch to the type determined at runtime. This is very neat. This way you can create a single XTend class which has all constantEvaluates with a function for each type. While playing around with XTend I however discovered a few downsides that I will explain in the next post..