I’ve spent the past day playing with Ruby and decided to test some basic performance – iterating and string parsing – to get an idea of what the performance is really like.
Ruby is not doing so well in my tests.
Note to Ruby Experts: If anyone can demonstrate what I’m doing wrong in my code or testing, I would love to be corrected.
My approach was to write the exact same code in Java and Ruby that loads up a file, reads each lines, tokenizes it into words using whitespace as the delimiter and counts up the tokens.
This avoids network or database IO and other external resources – except the filesystem which I don’t consider a significant variable in the test.
Further testing I’m planning on doing will test Rails, multi-threading and other common things I do in my apps.
On my laptop, a MacBook Pro 2.53Ghz Core 2 Duo with 4GB memory, the average times are:
- Ruby 1.8.6 app: 8022ms
- Java 5 app: 2986ms
- Java 6 app: 1443ms
Source Code
After downloading, strip the .doc ending off of the files.
Program Output
macbook-pro:src benjc$ /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin/java FileReadParse
Starting to read file…
The number of tokens is: 7764115
It took 1502 ms
macbook-pro:performance benjc$ ruby file_read_parse.rb
Starting to read file …
The number of tokens is: 7764115
It took 7999.955 ms
Filed under: Code, Performance
Ben,
I would say that there is an order of magnitude performance difference between Java and Ruby for pure CPU related tasks. Recent developments in Ruby VMs may have closed the gap a little, but its a decent rule of thumb, even if it is a bit pessimistic.
Note however, that it is rare to use pure ruby in any application, except web frameworks, shell scripts, testing code. The vast majority of real world ruby applications will use native extensions for CPU intensive work. So an order of magnitude performance difference in synthetic benchmarks will be closed down a lot in real world use.
regards,
Richard
Richard,
Thank you for your response on this subject.
Is there a list of native extensions that you recommend as being the most obvious or dominant to accommodate the common performance concerns of real world apps that you’ve been involved in?
Where do you consider a good source of what those would be? For example, I don’t see anything jump out at me on http://rubyforge.org/.
I also don’t see anything in http://www.ruby-doc.org/docs/ProgrammingRuby/ or other documentation I’ve ready talking about “native extensions”.
Ben
Hi Ben,
just to be clear, I am talking about Ruby Gems, the ruby standard for installing 3rd party extensions and libraries into your Ruby installation. There are very few extensions designed to replace anything in the standard library.
Distributing gems with a native component requires that the native code is distributed precompiled (on windows systems) or is compiled on the client systems (UNIX/Mac). Which creates its own problems. In particular some extensions may need binaries pre-installed on the system in order to work. So there is a trade-off when you use native extensions. In many instances it may be preferable to use a slower pure ruby gem than an equivalent gem with native extensions, especially where performance is not critical.
All of the GUI toolkits in Ruby involve using native extensions (with the exception of Tk/Tcl which is part of the standard library): Qt, wxWidgets, FxRuby, Shoes (a special case). In these cases CPU performance is critical, and there is no UI toolkit that I am aware of that is pure ruby. The RMagick extension (for image and graphics processing) relies on native extensions. It is also one of the most difficult to install (especially on windows).
There are also a series of native extensions that wrap an existing open source C extension. For example libxml, libssl, libcurl etc.
Ruby gems contain native extensions more usually because it is expedient to do so rather than for CPU performance. They take advantage of existing C code, rather than rewriting it, and producing these bindings is very easy to do in Ruby. If your application code is performance critical then Ruby is probably not the right choice.
In certain cases you will see a choice of gems that provide the same functionality (pure ruby vs non-pure/native) for performance reasons. The obvious areas were things like XML processing (REXML,libxml2), HTML parsing (Hpricot/Nokogiri), Regular Expressions (standard library vs Oniguruma) and some database adapters come to mind also.
Lastly with the maturity of JRuby we are starting to see a trend of native C extensions being ported to Java. Hpricot and Mongrel were some of the early ones, but lately Rmagick has been ported to JRuby as well (its using Batik under the hood I believe).
Richard,
Thanks for the further explanation. I appreciate the insight and opinions.
I’m going to start looking more at JRuby now and see if that mixes the two worlds nicely or not.
Richard, or anyone else reading …
I often hear that Ruby is “fast enough” or that the performance difference is not important since IO is generally the real source of performance issues.
I understand that in certain cases – though when I see potential performance degradation as multiples (2x, 3x, etc) as opposed to small percentages, it makes me question the decision to use Ruby much more than if we were talking percentages of 10-20% – such as 115ms vs 100ms on a page response.
For example, a java environment my team has built provides SOAP/REST webservices for product catalog search – and responsiveness is a very significant measurement criteria of the success of the product and system. Therefore, our average server side response time is something we watch very closely.
It’s difficult for me to accept the use of a language or platform which means I’m taking a significant hit in performance – similar to the first 5-8 years of Java.
Even Java is still improving … Java 5 to Java 6 was a very noticeable improvement in performance at the JVM level (I’ve had 30-80% performance increases from 5 to 6). Same hardware, same code – noticeable improvement in performance and thus responsiveness on webservices, webapps and shorter data processing times.
In the late 90s it was worth it to me to use Java and take the performance hit – as the benefits were so strong over C for so many things.
However, I’m still struggling to see the strong reasons to adopt Ruby when I’m penalized performance-wise and the answer becomes “it’s good enough” or “network IO is usually the slowest aspect, so it’s negligible what Ruby adds” or “CPU is cheap”.
Yes, “CPU is cheap”, but that applies to Java as well. 6 months ago we upgraded our hardware (systems were only 18 months old when we upgraded) and shaved off another 30% just by taking advantage of new CPU architectures and bus speeds that had changed in the previous year.
Perhaps in a straight-forward CRUD app where all that’s being done is retrieving/storing data in a database where IO truly is the bottleneck that no amount of optimization can improve – then it doesn’t matter and “good enough” rings true where IO takes 100ms and the Ruby/Java portion is only 10-20ms on top of the IO.
Thus, what are the “optimal” types of applications Ruby is best suited for if it is admittedly not just slower, but for various types of tasks significantly so?
Hi Ben,
optimal applications for Ruby are shell scripting, task automation, building desktop applications, building web applications, building games. Any application that you would typically use VB, Perl, PHP, Python, bash etc. can be done in Ruby, with the caveat that there are suitable libraries available in Ruby.
It seems to be a very persistent thought that ‘scripting languages’ ( a very ambiguous term itself) are not suited to real application development due to their low performance. For some reason Ruby seems to get a lot of this flak, disproportionate to other languages that are equivalent in performance, like Python, PHP, Perl etc.
I have seen Ruby applications outperform Java applications. This is of course due to the Ruby applications being written better, not to anything inherent in Ruby. A big part of this is that for many tasks Ruby puts you at the right level of abstraction. In certain tasks Java can fight you to get something done, and you frequently end up with code architectures and frameworks that squander the performance advantage. This isn’t an apples to apples comparison or even very scientific. But it happens in practical software development.
Its probably easier to identify the areas where Ruby is impractical for software development:
* Firmware Development/Embedded systems (but JRuby is blurring this a bit, opening up Ruby to certain classes of devices)
* Real time applications
* Writing device drivers or other operating systems integration software
* Integrating into architectures that require long lived processes
erform
I would very much like to see these benchmarks done on Ruby 1.9, which is known to be much faster than earlier versions. It was released 8 months prior to your post, so I don’t know why you didn’t include it.
I justed tested it with Java 1.6 and Ruby 1.9.2 on Dell 580.
It took 1009 ms on Java
It took 2876.52375 ms on Ruby.