∞ Consumer Reports seems suspicious

Written by

Marco Arment:

I’m looking at their full test results (I’ve been a CR website subscriber for six years), and I’m really not confident in the metrics and priorities that they seem to be using. Even some of the measurements seem suspicious to me.

Me too.


  • http://www.yourmaclifeshow.com/ Shawn King

    I’ve *never* had any faith in CR going back to the days of their “reviews” of the original iMac. They constantly get known facts about Apple and Macs dead wrong.

    • auramac

      True- this is nothing new. They don’t seem to comprehend that there’s software involved, for one thing…

  • http://mangochut.net/ mangochutney

    Not worth it anymore.
    We have a similar institution/company in Germany and they, too, fail to correctly test and subsequently rate modern devices, whether those are phones, laptops, GPS systems or anything else for that matter.
    They continue to apply antiquated systems for determining the value and potential usefulness of a device and its features, which are ill-adapted for these times.

  • Anonymous

    I rely on CR for reviews about cars and appliances.  CR is good for your average consumer in general, and their mission is not to rate things for your typical car/computer geek.

    I once told my father how I don’t always trust CR’s computer reviews.  My father, a furniture upholsterer, replied to me that he himself doesn’t always trust their furniture reviews.  Proof to me that once you know a specific product segment well enough, you have to move on from CR.

    That said, Marco makes some great points.  At this point, CR is just following a formula, the formula doesn’t fit here.  I’m sure plenty of Android fans give CR crap too about certain reviews.

    • http://www.yourmaclifeshow.com/ Shawn King

      “Proof to me that once you know a specific product segment well enough, you have to move on from CR.”

      But, that being said, why would you trust their cars and appliances reviews?

      • Anonymous

        Well first off my anecdotal experience has told me that when they say this is a good car, they are usually spot on.  And if I didn’t buy the appliance that was best reviewed, I got roughly the experience they told me I’d have.

        Second it’s the audience.  CR is meant for an average person with average uses.  Most people who discuss the iPhone in forums like this are equivalent to motortrend readers who shop for sporty tires on tirerack.com and have a completely different set of needs than the average driver.  I bought a Jetta because it was practical, and it fits the functions I need it for.  If I wanted a Mercedes for comfort and fun I would have.

        And third it’s the unique product.  Computing devices have a near infinite number of functions they can perform.  CR boils things down to features, because most consumer devices boil down to features.  Computing devices don’t easily translate because of OS, software, and user interface experience.  As a species humans haven’t figured out how to put a number rating on “experience” that fits everyone on the planet.  There’s no way to do that.  To review it in a limited amount of time, you unfortunately have to pigeon hole it a bit.

        I’ve devoted my life to computer products so I know them well, but I don’t have time to learn how to change my own oil.  Car geeks would call that silly, but I call it practical, my job doesn’t rely on oil changes.

  • Anonymous

    The key thing here is that tech reviewers such as Consumer Reports are going to be biased towards Android because the diversity of Android phones confuses the consumer and sends them running for a reviewer’s help. The Apple devices are effectively outside of the nerd/reviewer ecosystem. You almost always buy your first iPhone either because you already have another Apple device and liked it, or because you saw someone else running a marquee app like GarageBand and had to have it. Then in over 90% of cases, you simply buy another iPhone next time you need a phone. No Consumer Reports is needed. No Engadget, no Gizmodo, etc.

    The other thing is that if you want to know which dishwasher is best, you want a nerd to tell you that. You want them to compare specs and test the decibels of noise it puts out as it runs. But which phone to buy? No, you don’t want a nerd to tell you that, because they will overlook all kinds of flaws in the phone. They will not even realize how many workarounds they are doing as they work with that phone, because they are so used to working around those kinds of flaws on all computerized devices except Apple devices. That’s why Windows upgrades are so unpopular: they make the user relearn all the workarounds they didn’t realize they were doing every day. Also, they will be blinded by geegaws like 3D cameras, as if that makes up for the device crashing, or rendering Web pages like garbage, or slowing down a little every day, and so on.

    So most consumers are better off buying the phone that Rolling Stone recommends, not the one that Consumer Reports recommends.

  • http://www.thegraphicmac.com JimD

    Consumer Reports is about as useful as the Better Business Bureau. It looks good on a web page or brochure, but nobody actually cares.