Monday, February 25, 2013

Moving away from XText

When I started with PSHDL, I used XText for developing the language. It seemed very beneficial for me as I had plenty of experience with it (some of which was bad experience), but others were very good. If you think about it, the idea of XText is pretty nice. You create a quite straight forward language definition and by some magic you will get a complete Eclipse Editor with syntax highlighting, outline, rename refactoring, linking etc. This is really awesome. But as PSHDL evolved it become harder to justify the usage of XText. After all it comes with some baggage.

If you don't know XText, go check it out. One of the first things that annoyed me however is the fact that the generated model is based on EMF and Eclipse. So everything is fine as long as you want is an Eclipse editor. However I discovered that PSHDL has to be more than just an Eclipse editor. While it is possible to create a command line version of it, it comes with some nasty side-effects. The first one is that it depends on none less than 45 external jars to run. This makes the compiler executable a 25MBytes biggy that takes 850ms just to fire up a completely useless OSGI and plugin management system. Compiling a bigger piece of code takes 5s.

As "replacement" I used ANTLR4. This of course does not give me a full blown Eclipse IDE, but it gives me the freedom to implement other things. For example I can now access the comments and bring most of them over to the VHDL code. This helps to bring some documentation to VHDL. I can also add documenting Comments JavaDoc style. This allows me to annotate the port declaration. Another thing is that I can give better error messages and provide better guidance towards a correct source in case of syntax errors. Also the compiler can now become much smaller. It is now 3.8MBytes and take 3.8s to compile the same source.

But what is much more important than anything else: I can now easily embed it into any IDE that I like, not just Eclipse. People are more willing to integrate it in their workflow if it is small and fast. The downside is: I have to develop my own Eclipse tooling. But I am fairly familiar with Eclipse and so that is no big problem for me.


  1. Some corrections:
    - Xtext does not depend on Eclipse nor OSGi. It's runtime can be used in any vanilla Java environment (which includes Netbeans and IntelliJ).
    - It's trivial to get access to the comments and people do it all the time.
    - The needed runtime jars are much smaller than you described. For instance Xtend's standalone compiler is 10MBytes big and that hasn't even been tree-shaken.

    1. You are right in that there are many things that you can customize about how XText does one thing vs. another. And now that I have used ANTLR 4 on my own, I might even be able to do it in XText. But the thing is: If you want something else than what is described in the official documentation, it gets very quickly very complicated. There for example was one scoping issue that I was not able to resolve in XText:
      Enum E={A,B,C}
      E a;
      if (a==C) //C==a
      Even after spending quite some time in walking through various examples I was not able to infer the right hand side from the left hand side (and the other way around). But just because I wasn't able to, doesn't mean it is not possible. With my self written API this is trivial.

      So for my stand alone compiler I created the following:
      I was calling the XStandaloneSetup.doSetup then I got myself the IGenerator and run it. Then I used "create runnable jar" from that launch, which left me with those 50MB of dependencies. I can not tell whether I truly need all of them, but this was the easiest way for me to get something that I could run on the command line. This also includes the core of Eclipse:

      By depending on Eclipse I also mean that this is the environment where you actually get some benefits. I am not sure how much you would have to work to make it work in Netbeans as I have no experience there. But running standalone was important to me.

      Don't get me wrong, I really like XText and am still using and recommending it for many use cases. Actually the backbone data structures of PSHDL are generated by it. But when it comes to full blown languages, you pay the price of being easy in the beginning, which is something that happens to all frameworks. If you look at XTend you can also see where the problems of using XText for a full blown language are. My XTend files are several hundreds of lines and the performance for editing has severely degraded. But maybe that is fixed in the next version, or is just an artifact of my setup. Seeing however that Sigasi provides a leightweight editor as well for bigger files, there might be something to it.

    2. The long empty block meant to list:

      It appears they really don't like < and > in their comment ;)

  2. Cool post. Speaking about online technologies. I suggest to keep your paperwork stored online. Read about data rooms and everything will be clear.