On Friday, we posted news about Consumer Reports very publicly changing their Microsoft Surface opinion to “Not recommended”. From our post:

Take this change with a grain of salt, give the experts a chance to dig through the methodology. But no matter the ultimate resolution, this certainly gives a black eye to Microsoft’s Surface brand.

Over the weekend, I ran across this take from Ben Bajarin (it’s from a subscription-only newsletter, but Ben graciously gave me permission to repost it here):

I want to briefly comment on a consumer reports article that came out yesterday. CR officially does not recommend Microsoft Surface hardware based from their interviews and survey of 91,741 laptop and tablet owners. They claim 25% of Surface owners reported a form of breakage after a few years.

What bothers me most about this is the flawed methodology. Surface sales are not huge, and the installed base is relatively small. So even though CR covered over 91,000 people, there is no way they had what any respectable statistician would consider a representative sample. The fact CR is not further detailing their methodology or telling us how many Surface owners they had is a red flag. Based on some other source data I’ve come across, I’d bet CR had less than 50 people respond who owned a Surface and it was probably more like 30. So 25% of my estimate means five people had an issue with Surface after two years.

The flawed methodology is disingenuous, and sadly Consumer Reports has become a bit of a click-bait outlet. They do this with Apple and now with Microsoft. Media outlets endorsing the CR report and flawed methodology only further affirm that CR can do this in the future. Which means, for all the Apple fans in my Twitter timeline mocking Microsoft and Surface based on this flawed process by CR, this may come back and bite you when CR needs another click-bait headline and targets Apple with their disingenuous agenda and methodology.

Personally, I’m not a fan of Windows, but I am a fan of fairness. When I saw the Consumer Reports flip, something just felt off. Consumer Reports has an outsized reputation, which gives them clout. And, as we all know, with great power comes great responsibility.

If Ben’s take is correct, I’d love to see Consumer Reports either completely reveal their methodology or take the change back and redo their testing in a way that would stand up to public scrutiny.