A Canadian software company, Blaze Software, released a report today of what it calls the “largest ever research study of smart phone browser performance.” The problem is, the study is flawed.
[ad#Google Adsense 300×250 in story]According to the results of the study, Android was 52 percent faster than the iPhone loading Web pages. The company did 45,000 tests with the mobile operating systems to get their results.
There is a good reason for this. According to Blaze’s own documentation the “measurement itself was done using the custom apps which use the platform’s embedded browser. This means WebView (based on Chrome) for Android, and UIWebView (based on Safari) for iPhone.”
The problem with using UIWebView is that, even though it’s based on Safari, it didn’t receive any of the updates that Safari did in iOS 4.3. Using an embedded browser is not the same as using the official browser.
“Their testing is flawed because they didn’t actually test the Safari web browser on the iPhone,” Apple spokesperson, Natalie Kerris, told The Loop. “Instead they only tested their own proprietary app which uses an embedded web viewer that doesn’t take advantage of Safari’s web performance optimizations. Despite this fundamental testing flaw, they still only found an average of a second difference in loading web pages.”
“Obviously someone is looking to make a mountain out of a molehill,” Gartner analyst, Michael Gartenberg, told The Loop. “It’s not an apples to apples test.”
While it’s not mentioned in Blaze’s press release, they did the time difference between Android and iPhone page loading in the documentation. It comes out to just over 1 second.
According to Blaze, Android’s median load time was 2.144 seconds vs. iPhone’s median load time of 3.254 seconds.
Blaze has now admitted in a comment to CNET that their tests were flawed. It doesn’t seem right for a company to test a product unless they know for sure what technology it’s using.
“This test leveraged the embedded browser which is the only available option for iPhone applications,” reads the Blaze statement. “Blaze was under the assumption that Apple would apply the same updates to their embedded browser as they would their regular browser. If this is not the case and according to Apple’s response, it’s certainly possible the embedded browser might produce different results. If Apple decides to apply their optimizations across their embedded browser as well, then we would be more than willing to create a new report with the new performance results.”
Update 2: Added the statement from Blaze.
Update: Added comment from Apple.