Ruby VM shootout on Rock, Paper, Scissors!
August 11th, 2008
Update: I’ve added benchmark numbers for JRuby 1.1.3
I’ve been messing with RubyRPS (Rock, paper scissors) lately, it’s a ton of fun! I thought it might be cool to benchmark the progression of Rubinius, and that little experiment turned into a full blown ruby interpreter shootout. Sure, this is hardly fair since RPS doesn’t use much in the way of complicated stuff (lots of loops over array, etc), but most benchmarks are bullshit anyways. I just want my RPS bots to run fast!
Here are the versions I tested.
- ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0] (macports)
- ruby 1.8.7 (2008-06-20 patchlevel 22) [i686-darwin9.3.0] (macports)
- ruby 1.9.0 (2008-03-01 revision 15664) [i686-darwin9.4.0] (macports)
- ruby 1.9.0 (2008-07-25 revision 18218) [i686-darwin9.4.0] (snapshot 1.9.0.3)
- Rubinius 0.9.0 (ruby 1.8.6 compatible) (ffb998bf5) (08/07/2008) [i686-apple-darwin9.4.0] (from trunk)
- JRuby 1.8.6 (2008-08-07 rev 6555) [i386-jruby1.1.1] (macports)
- MacRuby version 0.3 (ruby 1.9.0 2008-06-03) [universal-darwin9.4] (downloaded binary)
Benchmarking is rather simple:
# time ruby ./run.rb
# time ruby1.8.7 ./run.rb
# time ruby 1.9 ./run.rb
# time rubinius ./run.rb
# time jruby ./run.rb
# time macruby ./run.rb
| Interpreter | Memory Usage | Real | User | System |
| Ruby 1.8.6 | 3MB | 0m36.183s | 0m35.426s | 0m0.241s |
| Ruby 1.8.7 | 10MB | 0m17.778s | 0m17.086s | 0m0.154s |
| Ruby 1.9.0 | 11MB | 1m17.092s | 0m53.843s | 0m22.234s |
| Ruby 1.9.0.3 | 10MB | 0m10.350s | 0m9.911s | 0m0.111s |
| JRuby 1.1.1 | 28MB | 0m18.023s | 0m16.198s | 0m0.417s |
| JRuby 1.1.3 | 28MB | 0m18.622s | 0m17.482s | 0m0.321s |
| Rubinius | 26MB | 0m42.836s | 0m41.556s | 0m0.479s |
| MacRuby 0.3 | 476MB | 0m42.471s | 0m40.302s | 0m0.987s |
How about a threaded run just for fun?
# time ruby ./run.rb --threaded
# time ruby1.8.7 ./run.rb --threaded
# time ruby 1.9 ./run.rb --threaded
# time rubinius ./run.rb --threaded
# time jruby ./run.rb --threaded
# time macruby ./run.rb --threaded
| Interpreter | Memory Usage | Real | User | System |
| Ruby 1.8.6 | 4MB | 1m12.343s | 1m9.703s | 0m0.695s |
| Ruby 1.8.7 | 13MB | 0m18.019s | 0m17.561s | 0m0.188s |
| Ruby 1.9 | 13MB | 1m18.341s | 0m54.516s | 0m21.947s |
| Ruby 1.9.0.3 | 13MB | 0m10.505s | 0m10.013s | 0m0.126s |
| JRuby 1.1.1 | 34MB | 0m17.048s | 0m22.013s | 0m1.890s |
| JRuby 1.1.3 | 31MB | 0m15.630s | 0m22.580s | 0m1.475s |
| Rubinus | 52MB | 0m43.745s | 0m41.893s | 0m0.591s |
| MacRuby 0.3 | Error! | stack level too deep (SystemStackError) |
Conclusion
Ruby 1.9 runs RPS really fast. You may have to compile it yourself for now if you’re on a Mac, since the version in Macports does have some issues. Also, I’d recommend that anyone who is still running Rails apps on 1.8.6 to move to 1.8.7 to for an increase in speed. We’ve been using 1.8.7 to run Learnhub for a few weeks now and have noticed a decrease in memory leaks as well. Be sure to check for compatibility with your Rails app though, 1.8.7 doesn’t work with some older versions of Rails.
Surprises
I was surprised that Jruby did so well. It performs just as well as MRI 1.8.7 (with about double the memory usage). I was expecting the interpreter startup time to hold Jruby back, but that didn’t turn out to be the case. Rubinius also performed a bit better than expected (they’ve been working on correctness rather than speed thus far). It wasn’t much slower than MRI 1.8.6, although it consumed far more RAM. My understanding is that the VM is currently being rewritten, so I’ll run a few more tests when it’s released.
MacRuby is also an interesting project, with a goal of porting MRI 1.9 to run directly on top of Mac OS X core technologies such as the Objective-C common runtime and garbage collector, and the CoreFoundation framework. It’s very much a work in progress at this point.
9 Responses to “Ruby VM shootout on Rock, Paper, Scissors!”
Sorry, comments are closed for this article.
August 11th, 2008 at 09:52 AM Very interesting article, as usual. I am curious about the differences between MRI 1.8.6 and 1.8.7.
August 11th, 2008 at 10:46 AM I'm curious if JRuby 1.1.3 would do better, or even a trunk build. They've been working on speed a lot lately and I suspect a trunk build would be better that the 1.1.1 version.
August 11th, 2008 at 02:26 PM I love these types of posts Wesley, thanks a lot for taking the time to publish your results and even doing the tests all together.
August 11th, 2008 at 02:42 PM Trying this myself I found jruby 1.1.3 with the -J-server option was much faster.
August 12th, 2008 at 01:53 PM @Kevin: I just added results for 1.1.3, it didn't really make much difference. That doesn't mean that 1.1.3 isn't faster for other applications though.
August 13th, 2008 at 12:36 PM I tried "jruby 1.1.3 with the -J-server option" and its much faster! what is -J-server?
August 13th, 2008 at 10:52 PM FYI, the -J-server option (or --server in 1.1.3+) is a JVM flag that tells it to use the optimizing "server" VM. It will take a bit longer to start up and a bit longer to reach full speed, but it will run a lot faster in the long term than the default "client" VM. When I ran RPS on JRuby trunk on Java 6 server VM, it was only slightly slower than Ruby 1.9. So I'm filing that as a bug...JRuby should be faster.
August 13th, 2008 at 10:53 PM Also, you may want to make sure you're running with Java 6 if possible. It's a good 20-25% faster than Java 5 which is still the default on Mac.
August 31st, 2008 at 11:04 AM Wonder how these numbers compare to JRuby 1.1.4! :)