Ruby Shootout Status Update
Antonio Cangiano July 7th, 2008
I want to provide those who are waiting for the Ruby shootout with a heads up. The benchmark suite needs some substantial changes in order to ensure accuracy and fairness for all the VMs involved.
This will delay the execution (and reporting) of the shootout further, but it will be worth it. I definitely prefer a shootout that’s published later in July (or heck even August) that is realistic, fair and provides interesting metrics (e.g. CPU time and memory) over an inaccurate one that was put together in a rush just for the sake of publishing it tomorrow.
For those interested in the technical details, we are trying to separate the parsing and “compilation” of definitions from the actual execution of the code (which needs to be timed). I accomplished this by creating a Proc for each benchmark, and then tested the time spent executing its call method. The problem with this approach is that it penalizes VMs that don’t JIT procs, like JRuby for example.
We also thought about defining a method instead of a Proc, but eval won’t accept class definitions or constants within methods. The workaround would be using MyClass.class_eval instead of class MyClass in the benchmarks and Module#const_set for the constants (or changing them to instance variables, for example). But we’re shooting for a cleaner solution in which we divide definitions from their actual execution in separate files, and only time the latter.
And of course, we also need to add cross-platform memory measurement into the picture. It may take a while, but stay tuned. 
