Firefox 3.5 Benchmarking in Linux

So I know there are tons of benchmarks already out there about the just released Firefox 3.5, but most of them are Windows focused. So if like me you run a Linux OS there's still a little more to know perhaps?

I know most benchmarkers prefer to do everything very cleanly with nothing but the browser running and maybe even a fresh reboot for each test...but I'm doing this more real world with other programs running in the background and no reboots. Just for reference though, none of the other programs running were changed during the tests to maintain some semblance of scientific objectivity.

I run Ubuntu 8.04 (LTS) still, and so the only official Firefox available to me thru Canonical's repositories is FF 3.0.11. In theory this one should be optimized specifically for use with Ubuntu, but as others have pointed out in the last few months, FF seems to run a bit slower on Linux than Windows of all things. Most believe this to be a matter of optimization, so I am trying out SwiftFox for the first time today too. The version I'm testing is 3.5rc3 and optimized to my Pentium D (prescott, 32bit) CPU, so supposedly it will be faster than the one Ubuntu has shipped and the vanilla binary from Mozilla's website, which is where I got the copy of 3.5 I'll be using for this test. And then on top of those three I'm also going to run the same tests on Google's new Linux alpha version of Chrome (ver:3.0.190.2).

So first off, just for curiosity's sake, let's see how they all do in the ACID3 test:

Firefox 3.0.11: 72%
Firefox 3.5.0: 93%
SwiftFox 3.5.rc3: 93%
Chrome 3.0.190.2: 99.9%

As expected Firefox 3.5 and SwiftFox get the same score as they're technically the same version of the rendering engine. Chrome actually says 100/100, but then it says "linktest failed" below that and there is a big X in the top-right corner, so I've marked it down to 99.9% as I don't know what that really constitutes a score of.

Next up, we'll be running Google's own V8 Benchmark, and the scores are rather surprising.

FF 3.0: 119
FF 3.5: 194
SF 3.5: 230
Chrome: 2492

So, first off the difference between Chrome's score and the other Mozilla based browsers is almost ridiculous. As Google created this benchmark themself, it almost makes you suspicious if they specifically put in tests they knew V8 would handle better than TraceMonkey? So I'll take that portion with a grain of salt. Also these "points" don't have any real intrinsic definition as to how they are calculated, more being better obviously. As for the other three, FF 3.5 doesn't even score twice as high as 3.0 did. Swiftfox however finally proves that it is indeed much better optimized that the other 2 by scoring almost 40 points more than the vanilla build.

Last we'll go with the tried and true SunSpider benchmark provided by the WebKit team.

FF 3.0.11: 5,583ms
FF 3.5: 2,421ms
SF 3.5: 2,111ms
Chrome: 986ms

Here FF 3.5 shows it is clearly much faster than 3.0 by completing the series of tests in less than half the time it took it's older brother. Once again SwiftFox shows us it is certainly faster than its not-so-well optimized cousins. And here Chrome really shines with completing the tests in less than a second! This is over twice as fast as Swiftfox, so maybe that V8Bench score wasn't as artificially bloated as I thought?

There are obviously many other browsers I could have included here, but decided not to bother with. When Konqueror finally switches to WebKit and Squirelfish it may be worth writing about, but for now is last decades' browser. Projects like Midori are also much too new to worry with at the moment. Safari nor IE have native ports for Linux, so they're obviously right out. And last there's Opera... This is a philosophical question for most, but I refuse to use a proprietary browser when there are open source options just as good if not better, so I personally could care less about Opera at all, that and their javascript performance is still quite far behind the likes of TraceMonkey, SquirellFish & V8 from what I understand.

To conclude, Chrome is still extremely alpha at the moment, with no plugin support so it's not really viable for daily use. With no Flash nor HTML5, half the web quickly becomes unusable, so we'll see what the situation is looking like when they finally make their first stable release for Linux based OSes. For now, I know I'll be using SwiftFox from now on ;)

Upcoming Browser JavaScript Engine Benchmarks

I've seen lots of people recently saying that Firefox's TraceMonkey JavaScript engine blows Google's V8 out of the water...but was a little skeptical so I decided to do some benchmarks of my own. Now with any benchmark, everything here needs to be taken with a grain of salt as performance will certainly vary upon which sites you are viewing. For this test I have used WebKit's SunSpider. Also, since Chrome and Safari do not have native ports available on Linux right now, I had to do the test under Windows XP. The test machine is dual-core so multi-process/threaded apps should show a benefit, but I feel it's totally fair as single-core machines are quickly going the way of the dinosaur and do not accurately represent the future, which is what we're talking about here. Also as Chrome does not have a stable/final release yet, I've compared with many other browsers' development builds.

So let's get to some numbers. On my test machine Chrome completes the SunSpider test with a total time of 2423ms; pretty nice! Ok, now what about Firefox? The current release, 3.0.3 takes 4244ms. Well, what about the new beta for 3.1 that just came out? It scores at 3823ms. So, wait...that's not too terribly much faster than 3.0. Do we have TraceMonkey enabled? Nope. I don't know why, but the default build of FF3.1.beta1 comes with TraceMonkey disabled by default, so after enabling it the results jump down to 1654ms. Wow! That really is faster than Chrome. Ok, so how about "minefield" AKA the nightly trunk build of Firefox (FF-3.1.b2pre-20081023): 1567ms! So yes, yes indeed Firefox 3.1 and the TraceMonkey JavaScript engine are indeed faster than Chrome and it's V8 engine.

So what about the other guys? Well, first off let's try Safari which shares it's webkit rendering engine with Chrome. The current release, 3.1.2 scores at 4894ms. So how about the development build? With webkit-r37604 (and Safari/WebKit's new SquirrelFish JavaScript engine): it goes to 1664ms. That's faster than Chrome/V8 too, but not quite as fast as FF/TM.

Opera's latest release is 9.6, and I didn't manage to find a development build available for it's next version...probably because it's still proprietary unlike the open source competitors we've discussed so far. Opera scores 5979ms which is slower than both Safari and Firefox's current stable releases, but it's still much better off than Internet Explorer. Internet Explorer 7 scores an abysmal 90522ms. That makes it almost 58 times slower than Minefield. Microsoft, that's absolutely pathetic. I didn't get a chance to try out IE8, but from what I understand it's not much better off in the JavaScript department. See update below...

Here's a chart I threw together in OpenOffice of the results. I omitted IE7 because it was such a huge difference that it made it hard to tell the difference between all the others.



Lastly, just for fun I tested my iPhone and Mobile Safari (running firmware 2.0.2), it took 136081ms. But hey, for a device running an ARM @ 412Mhz with 128MB of RAM vs. a full fledged computer with a dual core processor, and a gig of ram...that's still got to be better than Internet Explorer 7 did. I wish I had a Windows Mobile device to test, and a Android/G1 too, but oh well.

So, yes, with TraceMonkey enabled...Firefox reclaims its place as the fastest browser, but Safari and Chrome certainly aren't too far behind either. With this extreme increase in JavaScript performance on it's way...maybe it's time to finally retire the old "web 2.0" buzzphrase and move onto Web 2.1

UPDATE:
Seems I owe Microsoft an apology for not benchmarking IE8 before. They really have made some progress. The following numbers can not be fairly compared with the above because it was ran inside a virtual machine with only access to a single core and half the RAM, but here they are. IE8-beta2 scored at 9542ms. But, for reference, in the same virtual environment Firefox "Minefield" was still able to pull off 1873ms. So, as while still no where near as fast as the other guys, IE8 is lightyears ahead of IE7 in its javascript performance.