Bosses are only pleased when they get numbers and charts in reports on top of their tables. At this round, we (André, Diego, Fernanda and I) had to benchmark Mozilla and Gtk-Webkit on arm, and so we did ... Text below is an informal summary of the official report.
Scenario
- Experiments were performed using Nokia N810 Internet tablets (400MHz CPU and 128Mb of RAM memory) with Chinook installed.
- Applications under test were GTK+ embedding sample browsers from both Mozilla (TestGtkEmbed - available from the mozilla-central repository) and WebKit (GtkLauncher - from the WebKit trunk), cross-compiled in Scratchbox 1.0.8, gcc-2005q3-2.
These were the items measured:
1. Page load speed
The page load test focuses on measuring the absolute and relative time needed by each browser to open a given set of locally stored webpages. The test is driven by a JavaScript script that fires the load of each page in the test set. Both the time speed for each Web page to load (relative time) and the total time taken by loading the whole set (absolute time) are measured. The test set is formed by 37 real-world webpages fetched using the httrack crawler and is accessible over an Ad-hoc wi-fi network via the Apache Web server.
2. Memory and CPU consumption of 1)
The goal of this test is to evaluate how well browsers manage system memory during the page load test (above): virtual and physical memory allocated by each browser were monitored and printed out in a comparing chart. As such, a Bash script was developed to "watch" system memory and CPU numbers. It is basically a timer used to poll at every 1 second the browser's virtual and physical memory values from "/proc", and CPU usage from "top" (in batch mode).
3. Javascript engine performance.
Both browsers were ran against Dromaeo and SunSpider Javascript test suites and memory consumption during these tests was measure (same way as in 2).
4. CSS compliance
Both browsers were ran against Acid3 CSS test suite.
Results and Charts
As there is no such general browser benchmark suite (is this even possible ?), we just developed your own tools (bash and JS) for those item above that do not have renowned public benchmarks available (pageload and resource consumption).
ps: I tried Mozilla's Talos suite, which seems fine to test Mozilla but it is not portable for other browsers.
1. Page load speed
Mozilla's and Webkit's page load speed against the given pageset (see table below).
Individual pageload speeds (in ms).DISCLAIMER
- Original page set was formed by 85 webpages, from Talos Mozilla Test Suite. Although TestGtkEmbed ran well through the entire original testset, GtkLauncher always got OOM-killed after a while running the page load test, due to lack of memory (see memory chart in Memory Consumption section). Then, from the original 85 webpages, 37 were chosen to make GtkLauncher to finish this test.
- 10 out of 37 of the remaining webpages in the pageset contain non UTF-8 characters (Russian, Japanese, Chinese, ...). While WebKit misrendered most these fonts, Mozilla went fine for all. Example showed below:
www.3721.com in TestGtkEmbed
www.3721.com in GtkLauncher
Memory consumption while doing page load test in 1).UPDATE: VIRTUAL AND PHYSICAL MEMORY LABELS ARE CHANGED HERE.
CPU load while doing page load test in 1).
Dromaeo numbers.
SunSpider numbers.
Browsers memory use while doing Dromaeo.
Browsers memory use while doing SunSpider.4. CSS compliance
Mozilla on Acid3.
Webkit on Acid3.Some outstandings from the numbers:
- Mozilla managed better memory while doing page load tests, although Webkit was faster. It might probably been had affected by the fact that Mozilla rendered well all non-western chars, Webkit fails.
- Webkit was faster and used less memory while doing both Dromaeo and SunSpider test.
- I do not jemalloc enabled, but would love to.
- Mozilla guys are doing some amazing job on speeding up their Javascript engine: tracemonkey will probably get things much (5x at least ?) faster.
--Antonio Gomes
tonikitoo at gmail dot com
