Not Saussure

April 23, 2007

Anonymity and reputation

Filed under: Blogroll — notsaussure @ 9:32 pm

Chris Dillow takes issue with Brian Appleyard over the latter’s argument that

All western — not just scientific — wisdom is based on identity. Advocates and their critics can be identified and their ideas formally tested. This is nothing to do with the statistics of crowds, and everything to do with the authority of the person. Take that away and truth and judgement become fictions.

Chris gives the example

Take a simple statement, “all swans are white.” The validity (or not) of this has nothing to do with the identity of who utters it. It’s merely a statement about swans, to be tested by looking for a non-white swan.The key test of an idea is not: whose is it? It’s: does it accord with facts and reasoning?

— to which Brian Appleyard ripostes in the comments,

The question you have to ask yourself is: how do you establish that all swans are white? It is the process.

This puzzled me no end. In fact, I’m still a bit puzzled since I’m not sure I haven’t misunderstood what Mr Appleyard is trying to say in his article, but I think he’s missed the point. The obvious answer to his question is precisely that we rely on the wisdom of crowds, since I may advance the proposition that all swans are white, since I’ve never seen one that isn’t, and Chris may agree with me, since he hasn’t either, and then Brian Appleyard says, ‘Hang on a minute, there’s a black one lives on the river near me and I can take you to show it you.’ It’s not Brian Appleyard’s reputation as an ornithologist that helps us here; it’s the fact he can show us a counter-example.

It may well be that, to save time, we consult an ornithologist with a good professional reputation, on the argument that he’s likely to be better informed about swans than the layman and, since he values his reputation, he’ll not knowingly give us false information, but that’s more a time-saver than anything else. Alternatively, we could consult Wikipedia in the reasonable expectation that if all those pictures of Cygnus atratus were PhotoShop jobs, someone would be quick to point this out.

But this, I think, is not quite what Mr Appleyard means. In his Times article, he writes, referring to the anecdote in James Surowiecki’s The Wisdom of Crowds: Why the Many Are Smarter Than the Few about how

Francis Galton, Darwin’s cousin[, ] went to an agricultural show and watched a competition to guess the weight of an ox. Galton, a statistician, gathered together all the guesses and averaged them. The average was almost exactly right; the guesses of even the most expert farmers were all wrong.

Of this, he comments

Wikipedia fails because, though the crowd guessed the weight of the ox, it didn’t make the ox weigh that much. Its weight was a fact out there in the world. An elite — scale-makers and compilers of measuring systems — were the judges of this, not the masses.

In the black swan analogy, I suppose the elite are those who’ve actually seen a black swan (or who’ve studied the subject) and who can, therefore, with confidence state that not all swans are black.

But that’s something very different from this elite of scale-makers and compilers of measuring systems who, pace Mr Appleyard, had nothing to do with the weight of the ox. What Galton’s story about the ox tells us is that, when he took an average of the crowd’s individual estimates, based in part on their experience in using scales and commonly agreed weights and measures, of what would happen when you put the ox on the scales it was more accurate than the guesses of the experts as to what would happen. All the scale-makers and compilers of measuring systems did was provide a convenient method of expressing these guesses and measuring their accuracy.

Seems to me that the only people on whose authority the weight of the ox depended were the people who certified the actual weights and scales used were accurate — the local weights and measures department or their contemporary equivalent. And even then, presumably that was open to challenge — if enough people thought the results their scales and weights gave were wrong, they could test them against other sets.

Generally, though, authority is but one element that we weigh when we’re assessing an argument. If the person advancing it is known to be an expert in his field, that’s obviously someone one takes into consideration. But it’s by no means conclusive, as the case of Hwang Woo-suk demonstrated so dramatically. To my mind, the Judicial Studies Board’s specimen directions on the defendant’s character, both good and bad, are pretty generally applicable to authority and reputation in other fields — an element to take into consideration, but hardly decisive, and certainly can’t, on its own and without reference to the rest of the evidence, decide the case one way or another.

In point of fact, am I not right in thinking that Mr Appleyard is incorrect in saying

All western — not just scientific — wisdom is based on identity. Advocates and their critics can be identified and their ideas formally tested

, at least when research is up for peer review? I’d always thought that it was the practice to send articles out anonymously and to have them reviewed anonymously, precisely so that they’re judged on their content and so this judgement isn’t skewed by deference to the writer or personal hostility, professional jealousy or whatever. Perhaps I’m mistaken — that’s only my impression and it’s hardly a process with which I’m familiar.

Technorati Tags: , ,

Advertisements

5 Comments »

  1. The problem with the ‘wisdom of crowds’ is that sometimes crowds display wisdom that is greater than individuals, and sometimes less. As far as I know, there’s no way to know which it is. That’s not to say it’s not a useful concept, but it needs to be better understood before we rely on it.

    That said, some of the criticisms of it are pretty poor. As you’ve said, the whole point of peer-review is anonymity, although in practice I think it isn’t always strictly observed. In Appleyard’s column, he mentions the wiki user that pretended to be someone he wasn’t and posted incorrect articles on wikipedia – but this happens in old-fashioned scholarly pursuits as well. Loads of people have faked data and some have got away with it for much longer than whatever-his-name-was on wikipedia.

    Comment by thesamovar — April 24, 2007 @ 12:48 am

  2. …send articles out anonymously and to have them reviewed anonymously…

    Very much the examination system, to eliminate the possibility of bias.

    Comment by jameshigham — April 24, 2007 @ 3:49 pm

  3. But all swans are not white. Sorry, pedant mode off, now. ;)

    Comment by Longrider — April 24, 2007 @ 7:15 pm

  4. Sorry, jumped in too soon. Ah, well, I’ll blame my cluster headache… that’s my excuse and I’m sticking to it.

    Like you, I’m bemused – in this instance, mere observation does the trick. You don’t need an argument… Or am I missing something?

    Comment by Longrider — April 24, 2007 @ 7:17 pm

  5. The only sense I can make of it is that he thinks The Wisdom of Crowds recommends asking the crowd the answer to a factual problem (‘are all swans white?’, ‘how much does that ox weigh?’). But that would suggest he’d never actually read the book — otherwise he’d know it’s about the way crowds tend to be better at decision making and prediction than are small groups of experts.

    But, since he must have read the book, that can’t be the case.

    Comment by notsaussure — April 25, 2007 @ 8:47 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: