The Hidden Story of Reader Ratings

Posted on March 26th, 2009 in AutoSpeed,Driving Emotion,Opinion by Julian Edgar

At the end of all AutoSpeed articles is a reader rating system – you can give any article a score from 1 (bad) to 5 (excellent). As you’d expect, AutoSpeed publisher Web Publications has internal data analysis and display of these reader ratings – now totalling literally hundreds of thousands of scores.

By importing the data into a spreadsheet, I can rank all our articles in terms of numbers of ratings, averaged ratings for individual articles, and so on. I can also see changes over time in the average reader ratings for specific articles.

The other day a reader proposed that, if an article suggests to people ideas they don’t want to hear, they are more likely to give it a low score. So in other words, instead of rating articles on the basis of the quality of journalism, expression, innovative ideas (etc) that are presented, they just rate it on the basis of whether or not they agree with it.

Of course, that’s fair enough – no specific criteria are given to readers as to how they should evaluate articles.

But it also gives rise to an interesting idea. In terms of new car tests, I think that, over the long-term, most readers of new car tests are those who have bought the car in question – and then are looking for confirmation that their purchase decision was a good one!

I base this on two ideas. Firstly, new car tests typically decline in averaged reader ratings over time. So a test might start off with an average of 3.7, but over time that rating invariably drops.

Secondly, the more negative the test, the more marked is the decrease in rating.

So I think that the initial batch of ratings come from people who don’t care one way or another whether the car is praised or denigrated. Then, subsequently, people doing web searches on the car (ie they are interested in the specific car and that’s why they are looking for reviews) find the stories. 

And because they either own the car (and want that reassurance their purchase was a good one) or are thinking of buying that car (and again want the reassurance that their choice is correct), the more faults found with the car in the review, the less they enjoy the story.

Our test of the Honda HRV was scathing. Reader rating score? 1.81 out of 5.

The Audi A8 3.7 we thought way overpriced and underdone. Reader score? 2.04 out of 5.

The Mitsubishi Outlander was a pretty unimpressive car. Score? 2.15.

The Saab 9.5 was terribly outdated. Score? 2.16.

Our test of the Pug 407 HDi Touring  is probably the most negative ever published. 2.25 out of 5.

That simply awful car, the Alfa 166? 2.74 out of 5. (Probably a score that goes against the trend by actually rising over time as people – including owners – finally realise what a disaster the car was.)

And the most recent – the FG Falcon – with a score of 2.87.

(All scores at the time of writing.)

In contrast, positive car tests rate typically about 3.5 – 3.6.

I won’t pretend that there aren’t exceptions to the above argument – some positive new car tests have rated poorly. But typically, what I’ve written above is the case.

I can also see another fascinating element in the article scores.

It sounds arrogant – it probably is – but I have always thought that we’ve been well ahead of the game in much of our editorial content. For example, perhaps five years ahead of everyone else in our interest in hybrid cars – eg in modifying them.

And confirmation of this is coming when we re-run certain articles. Years after they were first published, these articles are now scoring much higher reader ratings than they did when they were first released!

Why? I think that people now perceive them as more relevant.

Yes, for an editor, reader ratings make for fascinating reading…

4 Responses to 'The Hidden Story of Reader Ratings'

Subscribe to comments with RSS

  1. Dave said,

    on March 26th, 2009 at 10:54 am

    I would have thought potential buyers for the reviewed vehicle would rate a critical article highly.

    Personally I would be grateful for the advice – I’d rather read a negative review than own a “bad” car.

  2. John said,

    on March 26th, 2009 at 7:48 pm

    Hi, this is a bit off the topic but reading your article on light weight frames reminded me of the days when we were modifying chrome moly motor bike frames to increase rear suspension travel. Where we worked at the time had a specialist stainless steel manufacturing plant and we discovered that crome moly could be welded easily with heaps of strength if you used “Fox Eze” welding rods. A DC welder is preferable but an AC welder will suffice. If you do have a DC welder dont forget to reverse the polarity, the same applies for welding stainless steel. A fox eze weld is the strongest weld you will ever gat using a stck welder, just like hardened steel and they can be used on mild steel as well. Dont weld cocky shit and expect to grind it off though.

  3. Fred said,

    on April 15th, 2009 at 12:11 am

    I’ve never intentionally rated an article but on quite a few occasions have got to the end of the article and automatically clicked ‘2’…. for page 2 of the article (common on many other websites).

    I wonder how often this happens – is ‘2’ the most common rating?

  4. matt said,

    on February 22nd, 2010 at 10:30 pm

    yes, ive hit 2 on a number of occasions to get to the next page aswell.
    its a shame as i only seem to do it on article i actually really enjoy 🙁
    ….perhaps the location of the rating numbers could be moved?