• 

• Sponsors

  • 

  • 

  • 

• Now available

  • 

• Follow me

  Follow me on Twitter

• Download DB2

  • 
• Search for:

• Recent Posts

  • Heads up: IBM is looking for top notch student hackers
  • DB2 support for Ruby/Rails turns 2.0
  • Grab DB2 Express-C 9.7.1 now and join the DB2Night Show
  • Setup Ruby Enterprise Edition, nginx and Passenger (aka mod_rails) on Ubuntu
  • What is DB2 pureScale?
  • Add code highlighting to your Google Waves
  • Getting MacRuby’s compiler to work
  • The CRTC tries to eliminate ISP competition in Canada
  • Benchmarking Tornado vs. Twisted Web vs. Tornado on Twisted
  • Installing and configuring DB2 Express-C 9.7 on Ubuntu 9.04 Server

• Recent Comments

  • Robert on Why MacRuby Matters (Present & Future)
  • Robert on Why MacRuby Matters (Present & Future)
  • Guest Post: What’s IronRuby, and how do I put it on Rails? « IUpdateable from Eric Nelson on Comparing the performance of IronRuby, Ruby 1.8 and Ruby 1.9 on Windows
  • Andy on How to install Django with MySQL on Mac OS X
  • Daniel on Sony Vaio FW Series: A Developer’s Review

• Categories

  • .NET (15)
  • Books (32)
  • Canada (3)
  • Cocoa and Objective-C (6)
  • DB2 (57)
  • Django (26)
  • Erlang (4)
  • General (68)
  • Hardware (3)
  • Haskell (6)
  • Industry News (1)
  • Mac (29)
  • Mathematics (3)
  • Merb (11)
  • Photography (1)
  • Python (34)
  • Quick Tips (20)
  • Reviews (4)
  • Ruby (98)
  • Ruby Benchmark Suite (13)
  • Ruby on Rails (89)
  • Scala (1)
  • Screencasts (3)
  • Startup (10)
  • This Week in Ruby (15)
  • Zenbits (3)

• Blogroll

  • Anant Jhingran
  • ChannelDB2
  • DB2 Express-C
  • DB2 on Rails Blog
  • Django
  • FreeDB2
  • Giovanni Intini
  • Haskell
  • Irshad Raihan
  • My Math Blog
  • Passiomatic
  • Planet DB2
  • Programming Screencasts
  • Python
  • Ruby
  • Ruby on Rails
  • Stacktrace.it (ITA)
  • Vintage Blog

In a previous article I compared the performance of Ruby on Windows, built through Microsoft Visual C++ and GCC. The numbers for the MinGW version were very impressive. So the question now becomes, how does its performance compare to that of Ruby on Linux? To quote one person (Alex) who commented on the aforementioned post:

With the new mingw32 substantial speed improvements, think it makes sense now to also test at least the baseline (MRI) on Mac/Linux on the same battery of tests, so we Windows folks could get a better idea of how far behind are we yet and what the different Windows interpreters speed target shall be.

Any sort of performance improvement for something that is notoriously slow on Windows is more than welcome, but would this be enough to fill the gap between Ruby’s performance on Windows and on Linux? How much faster is Ruby on Linux? Let’s find out.

Setup

• As a reminder, the operating systems used were Windows XP SP3 32bit and Ubuntu 9.04 32 bit.
• The Ruby implementations tested were ruby 1.8.6 (2009-03-31 patchlevel 368) [i386-mingw32], ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-mingw32], Ruby 1.8.6 (built from source on Linux) and Ruby 1.9.1 (built from source on Linux as well). The same identical versions of Ruby were used for both operating systems. I’m aware that these are not the latest available versions for Linux, but we are trying to compare apples to apples.
• I used the Ruby Benchmark Suite; the times reported are the best out of five runs, with a timeout of 300 seconds per each iteration.

Benchmark results

The following table/image compares the performance of Ruby 1.8.6 on Windows and Linux. A light green background indicates which of the two was faster. The total times exclude tests that raised an error or were not available (N/A) for any of the four implementations, but includes timeouts (they are counted as 300 seconds to provide a lower bound). The ratio column indicates how many times faster Ruby on Linux was:

The second table/image below compares Ruby 1.9.1 on Windows and on Linux, using the same criteria as above.

Note: The totals shown are different from the ones seen in other posts since the subset of benchmarks included in the totals is different.

Conclusion

According to the geometric mean of the ratios for these tests, it appears that on average Ruby 1.8.6 on Linux is about twice as fast as Ruby 1.8.6 on Windows. Conversely, Ruby 1.9.1 on Linux is about 70% faster than the Windows version.

The Windows implementations use GCC 3.4.5 (a four year old compiler) at the moment, while I built the implementations on Ubuntu with GCC 4.3.3 (which is available by default). This helps, at least in part, to justify the performance gap. Luis Lavena, leader of the Windows port, confirmed to me that a switch to GCC 4.4.x is planned for the future. This should significantly increase performance on Windows yet again, and bump Ruby’s speed on Windows a bit closer to the speed that’s obtainable on Linux.

For the time being, switching to Ruby 1.9.1 on Windows will give you a performance that is better than what’s obtained by those who are still using Ruby 1.8.x on Linux. If it’s possible, switch.

In yesterday’s post I compared IronRuby 0.9, Ruby 1.8.6 (from the One-Click Installer) and Ruby 1.9.1 (downloaded from the official site) against one another. IronRuby did great, but the discussion in the comment section quickly veered towards what version of the One-Click Ruby Installer should have been used.

I justified my choice of using the “old” One-Click Installer, by the fact that I wasn’t aware of official releases of the new installer. As well as that the old One-Click Installer is the most widely downloaded version. Very few people are familiar with the upcoming version of the project. This point is about to change.

Luis Lavena took over the One-Click Installer project and has been working on the next version (RubyInstaller from now on), the aim of which is to replace the One-Click Installer by building Ruby 1.8 and 1.9 with MinGW and GCC. In theory, this brings performance gains on Windows to the table, and gets rid of having to use Visual C++ 6 (a “ten year old compiler”) to build Ruby and other native gems. The project also strives to be lighter by bundling fewer (unnecessary) gems for Windows users.

There’s no doubt that in the long run, this new project will become the de facto standard for Windows, but the questions on everyone’s mind are, should I bother with it now? How much of a performance boost are we talking about here? 10%? 20%? Let’s find out.

In this follow up article I’m going to compare the performance of Ruby 1.8.6 from the One-Click Installer (mswin32), Ruby 1.8.6 from RubyInstaller (mingw32), Ruby 1.9 (mswin32) downloaded from the Ruby-Lang.com site, and Ruby 1.9 from the RubyInstaller project (mingw32) against one another. I’ll copy and paste part of the setup and disclaimer from yesterday’s post, for those who haven’t read it. Feel free to skip this part if you wish.

Setup

• The benchmarks were run within a virtual machine with 2 GB of DDR3 RAM and a 2.66 GHz Intel Core 2 Duo processor. The operating system adopted was Windows XP SP3 (32 bit) with the .NET Framework 3.5 SP1 installed.
• Here I employed a large subset of the current Ruby Benchmark Suite project. The source code for all of the benchmarks is available within the repository.
• The best time out of five runs is reported for each benchmark. When the results report a Timeout, it means that more than 300 seconds were required for a single iteration and was therefore interrupted. Conversely, N/A means that a test that was compatible for the given implementation was not available or the setup on my machine was lacking the required libraries to execute it (1 test affected).

Disclaimer

• An attempt has been made to improve the quality of the tests. Some of them may be more representative of realistic workloads, but most of them remain micro-benchmarks. They are indicative of how these implementations compare, but cannot be viewed as a guarantee of how they will actually affect your own programs.
• There are lies, damned lies, and statistics.

Benchmark results

The table/image below shows the times for each benchmark, for Ruby 1.8.6 (mswin32), Ruby 1.8.6 (mingw32), Ruby 1.9.1 (mswin32), and Ruby 1.9.1 (mingw32). In the table I used (RI) as shorthand for RubyInstaller to indicate mingw32 versions.

Red values are errors, timeouts and inapplicable tests. Green, bold values indicate better times than what Ruby 1.8.6 (mswin32) delivered. A pale yellow background indicates the best time for a given benchmark. Total time is the run-time for the subset of benchmarks that were successfully executed by all four implementations. Timeouts have been included this time around. Each timeout has been counted as an additional 300 seconds.

The total runtime (including timeouts) is summarized in the chart below:

Conclusion

Wow! Ruby 1.8.6 (mingw32) improves from 3% to 664% (depending on the test), over the current One-Click Installer. The geometric mean of the ratios (read “on average”) tells us that it was about 283% faster. The Ruby 1.9.1 version provided by the RubyInstaller was slower than the mswin32 version in a couple of tests, but faster everywhere else. How much faster? Up to 342% faster, with an average (again calculated through the geometric mean of the ratios) of a 77% increase in speed.

These finds prompt me to ask, what are we waiting for? You know how unresponsive Ruby is on Windows, and how tests take forever to execute? These mingw32-based releases may very well solve this. And incidentally the bar has been raised for IronRuby as well. I formally invite the Ruby on Windows community to embrace these two projects.

In my latest article I discussed the importance of JRuby as a means of introducing Ruby to the Enterprise world. Most of the companies that belong to this ecosystem are Java based, but we cannot forget that a sizable portion of them are Microsoft-centric. Within these companies, Ruby will be far more welcome if a .NET implementation is available. The answer to this need is sufficiently fulfilled by IronRuby (version 0.9 was just released).

IronRuby has been progressing fast lately. First came support for Rails and then, with this release, a great deal of effort has been placed on improving performance. In the past, IronRuby was all but fine-tuned. In fact, it was several times slower than Ruby MRI, as the team worked on improving compatibility with Ruby 1.8 and mostly ignored performance.

In this article I’m going to provide some performance results for IronRuby 0.9 on Windows, which I’m sure will interest readers of this blog as well as of my book. Before revealing all the details, let’s start with the setup and a disclaimer. Please read through it carefully, because the old, trite comments about how “micro-benchmarks are useless” won’t be published. We’ve already been there, folks. Thank you all for your understanding.

Setup

• The benchmarks were run within a virtual machine with 2 GB of DDR3 RAM and a 2.66 GHz Intel Core 2 Duo processor. The operating system adopted was Windows XP SP3 (32 bit) with the .NET Framework 3.5 SP1 installed.
• Here I employed a large subset of the current Ruby Benchmark Suite project. The source code for all of the benchmarks is available within the repository.
• The best time out of five runs is reported for each benchmark. When the results report a Timeout, it means that more than 300 seconds were required for a single iteration and was therefore interrupted. Conversely, N/A means that a test that was compatible for the given implementation was not available (2 tests) or the setup on my machine was lacking the required libraries to execute it (1 test).
• I realize that there is a MinGW One-Click Installer effort that may improve the performance of Ruby on Windows. Here however, I used the regular One-Click Installer as it is the most common one (it’s been downloaded almost 3.5 million times so far). For Ruby 1.9 I used the binary provided by the download page on the official Ruby site.

Disclaimer

• An attempt has been made to improve the quality of the tests. Some of them may be more representative of realistic workloads, but most of them remain micro-benchmarks. They are indicative of how IronRuby compares to MRI (Ruby 1.8) and KRI (Ruby 1.9) performance-wise, but cannot be viewed as a guarantee of how this will actually affect your own programs.
• I didn’t calculate a RubySpec completeness score for IronRuby. That said, IronRuby 0.9 should be a fairly complete implementation of Ruby 1.8.
• There are lies, damned lies, and statistics.

Benchmark results

The table below shows the times for each benchmark, for IronRuby 0.9, Ruby 1.8.6 (2008-08-11 patchlevel 287) and Ruby 1.9.1p0 (2009-01-30 revision 21907):

Red values are errors, timeouts, inapplicable tests and times that were worse than Ruby 1.8.6. Green, bold values are better times than what Ruby 1.8.6 delivered. A pale yellow background indicates the best time for a given benchmark. Total time is the runtime for the subset of benchmarks that were successfully executed by all three implementations (whose cardinality is 54).

The total runtime is summarized by the chart below:

And let’s compare each of the “macro-benchmarks” on an individual basis:

Conclusions

IronRuby went from being much slower than Ruby MRI to considerably faster across nearly all the tests. That’s major progress for sure, and the team behind the project deserves mad props for it.

One final warning before we get too excited here. IronRuby is not faster than Ruby 1.9.1 at this stage. Don’t let that first chart mislead you. While it’s faster in certain tests, it’s also slower is many others. Currently, it’s situated between Ruby 1.8.6 and Ruby 1.9.1, but much closer to the latter. The reason why this chart is misleading is that it doesn’t take into account any tests that timed out, and several of such timeouts were caused by IronRuby (more than those caused by Ruby 1.9.1). If you were to add, say, 300 seconds to the total, for each timeout for the two implementations, you’d quickly see that Ruby 1.9.1 still has the edge. The second chart that compares macro-benchmarks does a better job at realistically showing how IronRuby sits between Ruby 1.8.6 and Ruby 1.9.1 from a performance standpoint. If you were to plot every single benchmark on a chart, you’d find a similar outcomes for a large percentage of the tests.

Whether it’s faster than Ruby 1.9 or not, now that good performances are staring to show up, it’s easier to see IronRuby delivering on it’s goal of becoming the main implementation choice for those who both develop and deploy on Windows. This, paired with the .NET and possible Visual Studio integration, the great tools available to .NET developers, and the ability to execute Ruby code in the browser client-side thanks to projects like Silverlight/Moonlight and Gestalt, make the project all the more interesting.

What are your thoughts on IronRuby, and how will this dramatic performance gain affect your projects?

Next Page →