∞ Study comparing Android to iPhone Web browsing speed flawed

A Canadian software company, Blaze Software, released a report today of what it calls the “largest ever research study of smart phone browser performance.” The problem is, the study is flawed.

[ad#Google Adsense 300x250 in story]According to the results of the study, Android was 52 percent faster than the iPhone loading Web pages. The company did 45,000 tests with the mobile operating systems to get their results.

One of the biggest surprises the Blaze team found was that “despite significant JavaScript performance gains in the latest Apple iOS 4.3 release and Google Android 2.3 releases, these improvement made no measurable improvement on the actual page load times of the sites tested.”

There is a good reason for this. According to Blaze’s own documentation the “measurement itself was done using the custom apps which use the platform’s embedded browser. This means WebView (based on Chrome) for Android, and UIWebView (based on Safari) for iPhone.”

The problem with using UIWebView is that, even though it’s based on Safari, it didn’t receive any of the updates that Safari did in iOS 4.3. Using an embedded browser is not the same as using the official browser.

Apple’s Safari Web browser included the Nitro JavaScript engine that Apple said runs JavaScript up to twice as fast as its predecessor. Since UIWebView didn’t include any of those enhancements, it’s kind of disingenuous to say that Android beat Safari.

“Their testing is flawed because they didn’t actually test the Safari web browser on the iPhone,” Apple spokesperson, Natalie Kerris, told The Loop. “Instead they only tested their own proprietary app which uses an embedded web viewer that doesn’t take advantage of Safari’s web performance optimizations. Despite this fundamental testing flaw, they still only found an average of a second difference in loading web pages.”

“Obviously someone is looking to make a mountain out of a molehill,” Gartner analyst, Michael Gartenberg, told The Loop. “It’s not an apples to apples test.”

Guy Podjarny, Blaze CTO and Co-Founder said that they “assumed that it would be a closer race and that the latest JavaScript speed improvements would have a more material impact on performance. The fact that Android beat iPhone by such a large margin was not expected.”

The problem of course, is that the embedded browser used in the tests had no JavaScript speed improvements.

While it’s not mentioned in Blaze’s press release, they did the time difference between Android and iPhone page loading in the documentation. It comes out to just over 1 second.

According to Blaze, Android’s median load time was 2.144 seconds vs. iPhone’s median load time of 3.254 seconds.

Blaze has now admitted in a comment to CNET that their tests were flawed. It doesn’t seem right for a company to test a product unless they know for sure what technology it’s using.

“This test leveraged the embedded browser which is the only available option for iPhone applications,” reads the Blaze statement. “Blaze was under the assumption that Apple would apply the same updates to their embedded browser as they would their regular browser. If this is not the case and according to Apple’s response, it’s certainly possible the embedded browser might produce different results. If Apple decides to apply their optimizations across their embedded browser as well, then we would be more than willing to create a new report with the new performance results.”

Update 2: Added the statement from Blaze.

Update: Added comment from Apple.



  • Anonymous

    The test is flawed.

    The test used 3rd party apps with web browsing features. iOS 4.3 does not yet implement the nitro engine in it’s web browsing API’s. As a result, the test does not accurately compare the speed of built in web browsers between iOS 4.3 and Android.

    Check this link:

    http://www.loopinsight.com/2011/03/17/study-comparing-android-to-iphone-web-browsing-speed-flawed/

  • Moo

    Obviously the fact that UIWebView is inferior isn’t irrelevant, as plenty of apps use it. If the study said “embedded” it would suddenly stop being ‘flawed’.

    • Anonymous

      Obviously, if you told the average user to go to Facebook all the time to look at web pages on their phone, they’d think you were an asshat.

      • jwb

        But every time you click on a link in Facebook, it opens in Facebook’s embedded UIWebView. And lots of people get most of their links from Facebook. So, again, this is obviously relevant.

        • Steven Fisher

          Relevant or not is a secondary question. “Lying” is the first question. Does the study actually mean what it’s pretended to mean? No? Then it’s bullshit.

          Once it’s fixed to actually be truthful we can move on to other matters. :)

  • RichardL

    Ha! Apple gets bitten by their own feature lockdown.

    • http://www.facebook.com/profile.php?id=1410814563 David Schaefer

      How?

      • RichardL

        Either by design, neglect or accident Apple effectively hamstrings the browser inside of native apps compared to in Mobile Safari. The Nitro Javascript acceleration engine does not work in the OS-provided web views inside apps with the exception of Apple’s own Safari app.

        There’s some debate whether Javascript acceleration has much bearing on page load times, but to any extent that it does a native-app-based browser test would be handicapped on iOS.

        • kibbles

          yeah but how is that a “feature lockdown”? unless youre an iOS dev, perhaps there are implementation complications at this point in time. since software is constantly in, you know, a state of development & flux, i hardly find that surprising.

          regardless, this does nothing to change the fact that this study is flawed and mis-labeled. period.

        • Anonymous

          4.3 with nitro was a recent update. Unlike Windows Metro they focused on the browser first and got the update out EARLY. 4.4 will have the UIwebview improvements to be sure.

          I mean. It’s all a give and take. Android does the same thing. Their native browser is slower than their in-app browser. That’s why people flock to Dolphin Mini.

          Come now. It’s all about focus. It’s like blaming a boxer for not knowing how to do tae kwon do high kicks. But I really hate the in browser app interfaces. Especially yahoos and google’s. it’s crazy bad.

    • Anonymous

      How? Because a company did not say what they were testing and screwed the pooch?

  • http://twitter.com/franckhertz franckhertz

    I wonder why UIWebkit doesn’t use the new javascript engine. On a Mac, don’t all webkit-enabled apps get whatever performance boosts come in the new versions of Safari? (Thus why we have to restart to install a new Safari, because the whole framework is getting updated.)

    • RichardL

      “I wonder why UIWebkit doesn’t use the new javascript engine.”

      The new Nitro Javascript engine is based on a JIT (just-in-time) compiler rather than a more traditional interpreter. Thus Nitro requires the special privilege or entitlement to turn data into executable code. The entitlement of marking data as executable is reserved by Apple and is off-limits for non-Apple apps.

      • http://www.facebook.com/profile.php?id=18805795 Chris Sanders

        Security!

  • http://www.prjctmobile.com Chris MacDonald

    Also of note is that they are using the median time of a page loaded as opposed to the mean. They were certainly stretching to find any statistic that will create a large discrepancy for some super page views and attention.

    • Oletros

      Just the contrary, median cuts statistical extremes

    • http://twitter.com/melgart matias elgart

      you got your mean vs median reversed there, buddy. mean == average, median == middle value of a sorted list of values (or mean of 2 middle values if even # of elements in the list)

      • http://www.prjctmobile.com Chris MacDonald

        Not sure what you are referring to. The study used the median number which is what I am disputing as being not credible in this case. The median would be great for large outliers which aren’t really an issue when dealing with differences in seconds and smaller differences are irrelevant.

        • http://twitter.com/melgart matias elgart

          well, if we take you sentence “They were certainly stretching to find any statistic that will create a large discrepancy for some super page views and attention.” and analyze that, you’re saying that by using the median, that they were trying to create large discrepancies between values. that’s one way that two of us reading your words interpreted them. if we got your meaning right, how would they use the median to do that, if the median basically cuts outliers, just like you said?

  • Anonymous

    Unless the old javascript engine takes longer to load up then the nitro engine then why would any javascript engine affect page load times at all?

  • Oletros

    And how exactly the new Nitro JS engine helps with load times?

  • Anonymous

    There are 2 other major flaws also:

    • the iPhone browser loads Web apps CORRECTLY in 3.2 seconds, while the Android browser loads Web aps INCORRECTLY in 2.2 seconds … even in Android v3, developers were very disappointed to find the Android Web platform can’t run their apps right • they did not test using a representative mix of Android devices and operating system version … many Android devices will be slower than an iPhone because of slower hardware or an older version of Android, both of which are very common

    It’s really amazing how far backwards you have to bend to find some way to portray Android as better than iPhone. For example, it is common for user base comparisons to include all Android devices, but exclude iPad and iPod, and it is common to include OMS and Tapas user bases along with Android although they are separate incompatible systems with only the same kernel.

    What’s weird with Android is they set out to replace Windows Mobile on low-end generic devices and succeeded, but they look at themselves as a sister of iPhone, the most high-end device, and they want you to think that is their competitor, AND that they are better! The majority of Android phones are free, given to users who do not even know they are Android, just to get them to sign a carrier agreement. Android should just enjoy its own success.

    • http://twitter.com/Oletros Oletros

      You’re joking, isn’t?

      • Jim

        What about? That is quite a blanket statement..

        • http://twitter.com/Oletros Oletros

          No, it’s a statement full wrong facts

          • Extensor

            Care to elaborate? If not then STFU.

          • http://twitter.com/Oletros Oletros

            “while the Android browser loads Web aps INCORRECTLY in 2.2 seconds … even in Android v3, developers were very disappointed to find the Android Web platform can’t run their apps right”

            False, the only thing Android browser doesn’t render fine is some HTML5 tags, and no the majority of web apps.

            ” they did not test using a representative mix of Android devices and operating system version … many Android devices will be slower than an iPhone because of slower hardware or an older version of Android, both of which are very common”

            False, he can ask the same with iPhones 4, 3GS and 3G and iOS 4.3, 4.2, 4.1, etc.

            “It’s really amazing how far backwards you have to bend to find some way to portray Android as better than iPhone. For example, it is common for user base comparisons to include all Android devices, but exclude iPad and iPod, and it is common to include OMS and Tapas user bases along with Android although they are separate incompatible systems with only the same kernel.”

            False, OMS and Tapas are not Android and they aren’t counted on Android stats, they have been counted only on ONE report ONE time.

            “The majority of Android phones are free, given to users who do not even know they are Android”

            False, the majority of Android handset are not given free, and outside US the Android handset sold are the top of line and pricier than iPhone.

            STFU? You can ask him to prove the facts first

          • iBagwan

            All I want to know is, who won the JD Powers Customer Satisfaction Study?

            http://tinyurl.com/497ofrm

          • kibbles

            …like…

    • EsotericPunk

      While claiming that they need to use a representative mix of Android OS’es (and I don’t disagree), don’t forget that there are multiple versions of iOS out there as well. Some of these older iOS versions don’t have optimizations either.

  • http://www.droid-den.com Rachid

    Doesn’t this just go to show that you use whta Steve Jobs wants you to use?

    You HAVE to use safari to get any sort of speed, with android you have a choice of whatever browser you like

    • Extensor

      Actually what it show is that iOS is made with security in mind while Android is a free for all where you take your chances.

      • http://twitter.com/Oletros Oletros

        And that’s the reason that Safari iPhone wasn’t hacked at Pwn2Own and Chrome Android yes

        • http://www.droid-den.com Rachid

          LMAO, do your own research dude, don’t just relay what other people say as the truth. Even though that is the Apple way, its not the correct way.

          You know why? It makes you look stupid, see below for what actually happened at Pwn2Own

          “In day 2 the iPhone 4 and Blackberry Torch 9800 were both exploited. Security researchers Charlie Miller and Dion Blazakis were able to gain access to the iPhone’s address book through a vulnerability in Mobile Safari by visiting their exploit ridden webpage. The iPhone was running iOS 4.2.1, however the flaw exists in the current 4.3 version of the iOS”

          Android WAS NOT hacked during the event.

        • Steven Fisher

          Actually, Chrome wasn’t hacked because the tester didn’t show up: his exploit had been fixed recently, which meant he was eligible only for the crappy hardware not the cash prize.

          • http://twitter.com/Oletros Oletros

            Ups, both Chrome OS and Android Chrome were tried. You have to check the facts

          • Steven Fisher

            Never said Android Chrome. Talking about Chrome.

            Comparing desktop to mobile as if it proves something is just uncool.

            Or did Mobile Safari fall, too? I sort of gave up on the contest as douchebaggery before then. (Paying people to hide security flaws, really?)

          • http://twitter.com/Oletros Oletros

            I’ll repeat because I think tou didn’t understood me:

            Both Chrome Desktop AND Android Chrome were tested and no one was hacked.

            Both Safari Desktop ADN Safari Mobile were tested and BOTH were hacked.

          • Steven Fisher

            Firstly: “The first contestant was a no-show,” said Aaron Portnoy, manager of HP TippingPoint’s security research team, and Pwn2Own’s organizer. “And the other team wanted to work on their BlackBerry vulnerability. So it doesn’t look like anyone will try Chrome.”

            Secondly: Why the @#$% have you dragged this into a discussion of Pwn2own? Who @#$%ing cares? How’s it related to the article?

            Total douche.

          • http://twitter.com/Oletros Oletros

            Ups, both Chrome OS and Android Chrome were tried. You have to check the facts

      • http://www.prjctmobile.com Chris MacDonald

        The recent Pwn2own competition states otherwise.

        • Steven Fisher

          The recent Pwn2own competition says only that the people who had vulnerabilities ready for Safari had theirs survive Safari’s week before patch, whereas the people who had vulnerabilities ready for Chrome had theirs killed.

          Pwn2own is good for entertainment, but thanks to their methodology it’s good for little else.

          Note: I’m not arguing that Safari is secure, I’m saying using Pwn2own as proof anything else “is secure” is laughable.

      • http://www.prjctmobile.com Chris MacDonald

        The recent Pwn2own competition states otherwise.

      • http://www.droid-den.com Rachid

        If having a choice means taking my chances then so be it (although you make a crazy statement right there). I prefer to choose which browser i use, not be told which is hte best for me, and have no choice.

        EG. I would hate for Microsoft to force me to use IE, and no other browser.

  • http://www.blaze.io David Horne
  • dima

    I find it fun that Apple’s damage control consists of admitting to a far larger flaw in iOS than the original one of a slightly slow browser experience pointed out by blaze. Good work apple.

    • Steven Fisher

      Explain this “larger flaw,” please.

  • Steven Fisher

    What a bunch of assholes. All they need to do is update their claim to reflect that it was not “Safari” that they tested. But no, they’d rather post bullshit about how it’s a fair test.

    That’s not the point, assholes. The point is that you tested something other than you said, and made claims for something you didn’t actually test.

    • http://www.droidsector.com/ Jeff

      Not everyone uses Safari, I’ve used iCabMobile on iPad. And it was noticeable slower than Dolphin browser on NI Adam and Nexus S.

  • lkalliance

    Judging from the comments, there are good reasons why Nitro isn’t in the UIWebKit. Per Daring Fireball, Nitro isn’t the only difference…and I expect there are reasons for those differences as well.

    But that leaves us still with the evident conclusions that iPhone apps render web pages significantly slower than Mobile Sarari (which jives with my experience, too, even before Nitro).

    The bulk of the data that comes into apps from the Internet isn’t in the form of web pages that need rendering, most apps just get data and most of the assets are local. But I know I for one do reasonably frequently render a web page in an app…and the anecdotal evidence (I seem to remember one study or other backing it up, but I’m not sure) suggests that more and more consumption of Web data is happening through apps rather than through mobile browsers: this appears to be becoming the paradigm for consuming information. So as an iPhone user I’m very interested in seeing Apple bridge this performance gap between the browser and the apps.

  • http://bw.org/ Bill Weinman

    Actually, this study doesn’t say that Android is faster than iOS — it says that WebView on a Nexus S loads pages faster than UIWebView on an iPhone 4. It’s a very narrow study. Any other device running Android will get different results.

    //B

  • http://pulse.yahoo.com/_UVPUHNNOUET2INSTA3M2SX45ZI su

    IOS 4.3 UIWebView is slow !!. Hopefully there will be some UIWebView performance improvement in IOS 5.