The state of cheating in Android benchmarks

Anand Lal Shimpi and Brian Klug uncover just how bad companies are cheating in benchmark tests. Personally, I’m shocked that such reputable companies would do this. What’s next, are they going to rip off Apple’s UI?

  • Jack

    I can’t wait for an analyst to ask Tim Cook when Apple is finally going to start cheating in benchmark tests in order to catch up to Android.

    • rattyuk

      It’s pretty clear from the data that Google, Motorola and Apple are not attempting to pull these kind of tricks.

      • BC2009

        There the only ones called out as not doing it, though Sony is not mentioned in the scandal at all and neither is Microsoft / Nokia.

  • Agarun Ilyaguyev

    “Personally, I’m shocked that such reputable companies would do this..” Personally, I am not.

    It has happened before in the PC space, and it is trickling down to mobile because OEMs without their own ecosystems continue to whore for specs – even in single digit deltas (i.e. perceived performance), instead of focusing on software optimizations (i.e. actual performance).

    Alas, those of us who’d hoped that Apple’s disregard for the war on specs, in the interest of focusing on actual overall customer experience, would teach other OEMs a valuable lesson, continue to be gravely disappointed.

  • willo

    I can´t wait for the A8 and A8X next year. It will be a whole different playing field for Apple, they are about to put it in fifth gear.

  • My favourite part is definitely the name of the table. (“I Can’t Believe I Have to Make This Table”)

  • Tom

    “Personally, I’m shocked that such reputable companies would do this.” And don’t even think for a sec that the great Google won’t do this. Not literally THIS but.. iPad sales in Japan? Remember?

    • Moeskido

      “Can you really blame Samsung, though?”

      Yeah, I can. I can blame any company that does this. I’d be angry if it had turned out Apple was doing this, too.

      “Don’t hate the player/hate the game” is a ridiculous dismissal of the issue.

      • lol. I took it as them being tired of this as news. They’ve posted this over and over and over as it has happened so often.

        But…every company inflates. Apple says X hours of battery, people get less. That’s just marketing though.

        Sammy is def’ taking it farther (not comparing the two by any means) but it’s just ‘meh’ for me. Shameful but not a big issue.

  • abdoradus

    I don’t want to defend Samsung but the real issue here is the spec worship by the tech press and its audience. A benchmark measures … how fast that specific benchmark runs. Samsung made the benchmark run faster. That far, no harm done. It’s when reviewers imply that the phone that runs the benchmark faster is the better phone when things go wrong but that’s a very questionable implication anyway.

    • Moeskido

      “That far, no harm done.”

      No. They specifically made the benchmark run faster, knowing it would be touted as evidence of a better product. Lots of harm done.

      • Technically they made their device run faster but the device consumers get do not have the same tweaks.

        So it’s like overclocking the CPU in the lab for benchmark tests but selling it not overclocked.

        • Moeskido

          That compounds the deception.

          • I wouldn’t say compounds it. It’s what happened, nothing more.

            Again, not excusing. Just clarifying what happened. They don’t touch the benchmark software, just their hardware. [from my understanding]

          • Moeskido

            “It’s what happened, nothing more”?

            Like “Mistakes were made” or “Why not, everybody’s doing it”?

            Do you believe this was some sort of randomly-occurring natural phenomenon, like Brownian motion or genetic mutation?Was there no human decision-making involved to participate in deceiving consumers?

            Or is your response more like something a five-year-old says when asked why he knocked over a potted plant?

          • Ahh…to the condescending replies we go. :-/

            I’m simply stating what happened for clarity. I have no dog in this fight.

          • Moeskido

            The more companies that engage in this behavior, and the more consumers make buying decisions based upon it, the bar for relying upon available information gets lowered, assessing this gear gets more difficult for everyone, and the market rewards the wrong players.

            To say you have no dog in this fight is untrue; we all do. I’m trying to find out why you’re dismissing this so casually. You’re certainly not adding “clarity” by saying things like “it’s what happened.”

            As you’re so fond of saying: “lol.”

          • We, the tech informed, may look at these stats but no consumer goes in making decisions based on phone X getting 100 more cycles on a benchmark.

            Again…I don’t care that they did it but agree they shouldn’t.

            I’m not dismissing it. I just don’t care. See my other comments. I called it shameful in response to you previously.

            As for not adding clarity, sure I did. You said “They specifically made the benchmark run faster” but, as I responded with clarity, they overclocked their device(s). Subtle difference but one I felt important to make.

            Yeah, I throw lol’s but I’m known for LMBO. 😉

          • Moeskido

            Bad tech writers use those stats to write reviews that consumers will see. And you’re projecting your use-case as one of the “tech-informed.”

            But it’s what happened.

          • True, they do.

            It’s just my opinion but I never not bought a device because the benchmark didn’t measure up. None of the others I’ve helped with device purchases cared, even when I mentioned it to them.

          • Moeskido

            That you believe this issue doesn’t affect you isn’t the point, even though it does.

            It’s all part of the pool of available information. We should try to make sure fewer people piss in it before we go for a swim.

          • Hehe…true. 🙂

          • no consumer goes in making decisions based on phone X getting 100 more cycles on a benchmark.

            Not true in the strictest sense. We like to pretend that none of the asinine stupidity in the tech press makes it to run-of-the-mill consumers, but this simply isn’t true. The “Apple is Doomed/Apple is falling behind/Apple can’t innovate” narrative show up everywhere, in nearly all forms of media. A conversation about Apple products can’t even occur on Facebook without some carpenter (literally) running his mouth about “closed” Apple, or the friend of a friend in IT telling us now Apple’s products are slower.

            Samsung is cheating because it works.

          • None of what you discussed after “not true…” addressed my point. I, by no means, addressed tech press influencing purchases. I specifically referenced benchmarks.

            The rest of your comment [after the first two sentences] are true.

          • Moeskido

            You’re splitting hairs. I’ve already reminded you how the tech press uses things like benchmark stats to generate reviews that consumers will see, which you acknowledged.