Consumer Reports - Not enough data

[gregwils]

I saw the 2007 Annual Consumer Reports Auto issue. They rate all models of used cars. One of the things that bothered me was the way some categories jump around from far better to far worse in a single year, when there were no major changes. So I started to poke around to see if I could reach any of my own conclusions about Consumer Reports. I don't think they're biased, but I don't think that their conclusions are statistically valid for low volume models.

Here is some basic math. There are about 10k Boxsters sold in the US each year, give or take a thousand or two. There are about 16.5 million cars sold in the US each year also. So the Boxster makes up about .06% of the market. T(hat's not six percent, it's six one-thousands.)

There are about 1.3 million cars in the Consumer Reports survey, but that's not 1.3 million from each model year. It's just 1.3 million. Let's cut them some slack and say that they had 200,000 cars from each of the last five model years, then it tapers off.

If we assume that their 1.3 million cars has the same mix of models as the population as a whole, then we can determine how many Boxsters are in their sample. So we need to multiply, 200,000 times .06% and you get 120 cars. So Consumer Reports is coming to the conclusions they are reaching based on about 120 cars of the 10,000 produced each year.

They listed the category of Major Engine problems for the 2002 as 'Much Worse than Average'. I don't know the answer to this, but if a 2% failure rate is average and you get one more with a bad engine, then what happens to your score. My guess is that you end up in 'da crapper'. They do get to track problems over time, but unless they are reporting on a different set of 120 cars from each model year, there is no improvement on the statistical validity of their sample.

I don't want to suggest that Porsches are without issues - they aren't, but I don't think Consumer Reports presents reliable data in their reliability analysis. Sorry, I couldn't help myself. Interesting, JD Powers used car data states 'Not enough data'. Sounds like a more responsible assessment.

Interested in comments...

[kls]

I think your math gives a pretty good estimate of what they might be looking at. I believe that in most, if not all of the previous years they listed "insufficient data" for the Boxsters. I still think that it gives a pretty good indication of the Boxsters reliability, putting it about equal to the M3 and much better than the Mercedes SLK or the Corvette.

I was actually relieved to see the "average" reliability rating because in nearly every other area it beat the competition.

[Brucelee]

"I still think that it gives a pretty good indication of the Boxsters reliability, putting it about equal to the M3 and much better than the Mercedes SLK or the Corvette"

Specific to the Corvette, I will believe that when I see some data. Don't know about the Mercedes.

Personally, I have owned about 15 Boxsters through my dealership and about 5 C5 Corvettes. I have also owned 3 C5s as private cars.

Based on my experience, the C5 is simply much less troublesome than the Boxster. I worried about every Boxster I bought and with good reason, I have the paid bills to prove it.

The C5, nada!

IMO!

[gregwils]

"I worried about every Boxster I bought and with good reason, I have the paid bills to prove it."

Don't tell me that, I haven't pulled the trigger yet on a Boxster and am looking for rationalization that my concerns are over-blown. Seriously, I fully expect a bit of a love/hate relationship with untimely, perplexing problems in search of over-priced parts. I will soften the blow by performing my own maintenance and, hopefully, most repairs.

If I get too scared, I could by an S2000. They're a little high strung and visually boring, but the likely reliability is pretty appealing.

All of the European cars seem to suffer from a high failure rate of component parts on these quite complex machines.

MAF
Air/Oil Seporator
O2s
RMSs
Control Arms
Water Pumps
and the beat goes on...

but then you drive it. The Boxster has to been one of the finest total driving experiences ever to hit the road.

[KevinH1990]

Things definitely do go wrong with Boxsters and some things go very wrong - just read the tales of woe from folks who have to replace their engines. So, I'm sure Consumer Reports has it right when they say that some model years of the Boxster are worse than average in reliability.

The Boxster is the first car I bought that Consumer Reports didn't rank as "much better than average" in reliability. (Although for some reason they did rank the 2000 Boxster as "better than average" - probably a statistical anomaly. They didn't have data on the 2000 in the current auto issue, so I can't say how they have held up in their seventh year of life. )

Although my 1996 Infiniti G20 and my Miatas have been fun to drive and very reliable, most of my other highly reliable cars have been a little dull. None of my other cars come close when you compare them to a Boxster for overall enjoyment.

So, don't buy a Boxster if you can't afford a few repairs or don't want to buy an extended warranty. But if you have a few resources in reserve, the Boxster is not a bad bet.

[wanna986]

Originally Posted by gregwils

If I get too scared, I could by an S2000. They're a little high strung and visually boring, but the likely reliability is pretty appealing.

but then you drive it. The Boxster has to been one of the finest total driving experiences ever to hit the road.

Yes, excellent conclusion. Buy the Boxster and fix it yourself if it needs to be repaired. Much more fun in the long run (and you will take it for some long runs!)

[Adam]

On a side note, CR recammends the boxster, but they don't recammend the 911 because of "reliability issues."