Peter Freed wants to you to know that Jon­ah Lehrer is Not a Neu­ro­sci­en­tist. Lehrer does­n’t claim to be, of course. He’s a jour­nal­ist and sci­ence writer who cov­ers devel­op­ments in neu­ro­science, and a good one at that. 

Freed is con­cerned about how Lehrer han­dled a recent study on “the wis­dom of crowds” in a recent op-ed in the Wall Street Jour­nal. The wis­dom of crowds is a long-stand­ing and often-suc­cess­ful idea that you’ll get a bet­ter pre­dic­tion by aggre­gat­ing the respons­es from a bunch of peo­ple posed with the same ques­tion than you’d get by sim­ply ask­ing a giv­en expert, or even aggre­gat­ing the opin­ions of lots of experts. The research Lehrer was point­ing out showed that the wis­dom of the crowd declines when peo­ple have a chance to share infor­ma­tion about what oth­er peo­ple were going to pre­dict. The more peo­ple learn about what oth­er peo­ple think on the issue, the less diver­si­ty of opin­ion remains, and the more con­fi­dent peo­ple become in their increas­ing­ly inac­cu­rate pre­dic­tions. As the researchers observe, this sounds kin­da famil­iar to any­one read­ing the finan­cial pages, or polit­i­cal news, or any of a range of fanat­i­cal sub­pop­u­la­tions (birthers, truthers, cre­ation­ists, to name a few).

Freed’s essay is not so much about Lehrer, actu­al­ly, as about how sci­en­tists read. First, how and why a par­tic­u­lar phrase in Lehrer’s essay stopped him, but then how one reads a sci­en­tif­ic paper:

I opened the paper and did what I always do — skip the intro, go straight to the tables and fig­ures, and then to the meth­ods. If you ever read a sci­ence paper, you should do the same thing your­self. Read­ing intros and con­clu­sions first is for suck­ers — they can say any­thing the author wants, and read­ing them allows the author’s “spin” — as we sci­en­tists call intros and con­clu­sions — to frame your analy­sis of the data. In sci­ence, the only thing that mat­ters is the meth­ods and the data, because it’s where the author can’t hide behind a spun story.

This is exact­ly right. It’s not some­thing you’re taught in a lec­ture in your sci­ence class­es. It’s one of many quirky bits of sci­en­tif­ic cul­ture that’s passed along by oral tra­di­tion, the sort of thing stu­dents learn on their own or fall through the cracks in sci­ence edu­ca­tion. It’s a bizarre way to read any doc­u­ment, but it’s what sci­en­tists do. We pound the struc­ture of sci­en­tif­ic papers into our stu­dents when we teach them to write lab reports, and then years lat­er we let them in on a secret: that they should read the paper out of order.

About halfway through Freed’s essay, he real­izes he’d assumed Lehrer was talk­ing about a medi­an rather than a mean, and thus was mis­read­ing Lehrer and not prop­er­ly match­ing Lehrer’s essay with the paper he was cit­ing. It should be clear­ly stat­ed that Lehrer only referred to the medi­an, so the fault lies with the read­er here, and Freed does­n’t blame Lehrer for it. But then, alas, he just keeps on not under­stand­ing why Lehrer used the medi­an, and along the way mis­rep­re­sents what a medi­an is, what a mean is, and why they are different.

For instance, Freed writes:

It had nev­er occurred to me — as it nev­er occurred tot he study’s authors — that Lehrer would be talk­ing about an actu­al group medi­an.… maybe he did­n’t know what medi­an meant. Because medi­an guess­es are not guess­es by a crowd, as Lehrer states. They are guess­es by a sin­gle per­son. … Which is to say, they have noth­ing to do with the point of the Jour­nal’s arti­cle [the wis­dom of crowds]! …the medi­an per­son is defined as the per­son with the mid­dle val­ue in a group — half the group is above them, and half below. Which means that the medi­an per­son is one guy. Not a crowd! A sin­gle per­son! Which is why the num­bers are so pret­ty — they were cho­sen by indi­vid­u­als. …the authors prob­a­bly just includ­ed it [the medi­an] for kicks, and not as a mea­sure of crowd wis­dom — which made sense, giv­en it was­n’t a crowd answer. … And then I read the meth­ods sec­tion, which con­firmed my read­ing in spades. There I found this sen­tence: “this con­firms that the geo­met­ric mean.… is an accu­rate mea­sure of the wis­dom of crowds for our data.” … The authors explic­it­ly say that it is the geo­met­ric mean that should be used to judge crowd wis­dom. Not the median!

All of this is wrong. Well almost all. It’s true that a medi­an is the val­ue which has as many val­ues above it as below. It’s wrong, though, to jump from the medi­an answer to the medi­an per­son. And to sug­gest that a medi­an val­ue does­n’t rep­re­sent the crowd is absurd. It’s def­i­n­i­tion requires us to con­sid­er the val­ue of every oth­er response, not just one response. 

Nor is the medi­an an inap­pro­pri­ate mea­sure for this pur­pose. As the authors make clear in the sec­ond sen­tence of the paper — in the abstract of the paper no less ‑ “Already Gal­ton (1907) found evi­dence that the medi­an esti­mate of a group can be more accu­rate than esti­mates of experts.” There are all sorts of good rea­sons to pre­fer the medi­an as a mea­sure. Indeed, that ital­i­cized quo­ta­tion above is from a sec­tion of the paper jus­ti­fy­ing their use of the geo­met­ric mean instead of the more wide­ly used medi­an. As they explain in the Meth­ods sec­tion:

a high wis­dom-of-crowd indi­ca­tor implies that the truth is close to the medi­an. Thus, it implic­it­ly defines the medi­an as the appro­pri­ate mea­sure of aggre­ga­tion. In our empir­i­cal case this is not in conflict with the choice of the geo­met­ric mean as can be seen by the sim­i­lar­i­ty of the geo­met­ric mean and the medi­an in Table 1 [the Table Freed is focused on]. A the­o­ret­i­cal rea­son is that the geo­met­ric mean and the medi­an coin­cide for a log-nor­mal distribution.

In oth­er words, the medi­an is the appro­pri­ate mea­sure of the crowd’s choice, but for var­i­ous com­pu­ta­tion­al rea­sons spe­cif­ic to this study, it was bet­ter to use a geo­met­ric mean.

Freed dou­bles down on this mis­han­dling of sta­tis­tics when he writes:

“arith­metric mean,” which is just a fan­cy-pants way of say­ing “the num­ber that the stu­dents actu­al­ly guessed.” That is, this is the num­ber that real peo­ple lit­er­al­ly wrote down when guess­ing the answer to the question.

No. No no no no no. No one in the study actu­al­ly wrote down that there were 26,773 immi­grants to Zurich. Some­one did write down that 10,000 peo­ple immi­grat­ed to Zurich, and if you recall that was why Freed did­n’t want to use the medi­an. Indeed, exact­ly as many peo­ple wrote down a num­ber larg­er than 10,000 as wrote a small­er num­ber, while far more wrote a num­ber small­er than 26,773 than wrote down a larg­er num­ber. The sug­ges­tion that the larg­er num­ber is more rep­re­sen­ta­tive of what peo­ple wrote down bog­gles the mind and defies basic mathematics.

Freed here is try­ing to say that the arith­metic mean is a supe­ri­or mea­sure of the col­lec­tive judg­ment than the geo­met­ric mean, which in turn is supe­ri­or to the medi­an, and he’s crit­i­ciz­ing Lehrer for tak­ing the oppo­site view. This, it must be empha­sized, despite the fact that the authors of the paper under dis­cus­sion agree with Lehrer and dis­agree entire­ly with Freed.

And they should dis­agree with Freed. The arith­metic mean is not the rel­e­vant sta­tis­tic here.

To under­stand why, let’s quick­ly review what these terms mean.

The arith­metic mean is what you cal­cu­late when some­one tells you to cal­cu­late the aver­age. Add the num­bers up and divide by the num­ber of things you added. For var­i­ous math­e­mat­i­cal rea­sons, it is often the case for the sorts of num­bers you and I deal with that the arith­metic mean is a good sum­ma­ry of the loca­tion of a giv­en col­lec­tion of num­bers. It works won­der­ful­ly, for instance, with respect to peo­ple’s heights. It works there because height is dis­trib­uted fair­ly sym­met­ri­cal­ly around its cen­tral point, as can be seen in the pho­to here, which shows the heights of shows genet­ics stu­dents of at the Uni­ver­si­ty of Con­necti­cut in 1996. One has to squint a bit to see past the vari­abil­i­ty in the data, but larg­er sam­ples bear out the obser­va­tion that height is dis­trib­uted in the shape of a stan­dard bell curve, known as the nor­mal distribution.

But this curve, and thus the arith­metic mean, is not nor­mal in all cas­es. It does­n’t work, for instance, for wealth dis­tri­b­u­tion. With an arith­metic mean, if Bill Gates and a home­less per­son walk into a bar, the arith­metic mean of the wealth of each per­son in the bar will sky­rock­et because $56 bil­lion divid­ed by any num­ber of bar patrons is still enor­mous. But that does­n’t tell us as much as we want to know about the actu­al income dis­tri­b­u­tion. Sim­i­lar­ly, sim­ply sum­ming the house­hould income of every US house­hold and divid­ing by the num­ber of house­holds gave you $60,528 in 2006 (accord­ing to Wikipedia). But in the chart shown here, you can see that most Amer­i­cans make a lot less than that. Indeed, about 63% of house­holds earned less than that. To get a bet­ter mea­sure of the actu­al dynam­ics, you can do one of two things.

First, you can com­pute a dif­fer­ent sort of aver­age by mul­ti­ply­ing all the num­bers, and then tak­ing the nth root of the result (where n is the num­ber of quan­ti­ties being aver­aged). Doing this tends to de-empha­size the effect of large num­bers, and is thus going to be small­er (or the same size as) the arith­metic mean. It’s also more com­pli­cat­ed to com­pute on the fly, so is less com­mon­ly used.

The medi­an is eas­i­er to com­pute, because it cor­re­sponds to a val­ue greater than half and less than half of all the oth­er num­bers. The medi­an house­hold income is $44,389, over $17,000 less than the arith­metic mean (and quite close to the geo­met­ric mean). For many pur­pos­es, this is a far prefer­able met­ric, espe­cial­ly when you’re more inter­est­ed in where peo­ple stand than in what the num­bers are (the arith­metic mean is what you’d get if you took away every­one’s mon­ey and divid­ed it into equal piles, the medi­an cor­re­sponds to what most peo­ple actu­al­ly earn). Because income dis­tri­b­u­tion is skewed (as seen in the fig­ure above), an arith­metic mean is less appro­pri­ate. We want a method like a geo­met­ric mean or a medi­an that bet­ter bal­ances the impact of very large numbers. 

There is no per­fect mea­sure to use, and to the extend Freed is treat­ing the arith­metic mean as “the num­ber that the stu­dents actu­al­ly guessed,” he’s mis­lead­ing read­ers. More than half of the stu­dents guessed a small­er num­ber than the arith­metic mean, and treat­ing that num­ber as the gold stan­dard deval­ues those stu­dents in pref­er­ence for the ones who made insane­ly large guesses.

The medi­an has always been the pre­ferred mea­sure for these sorts of “wis­dom of the crowds” mea­sures, and because the geo­met­ric mean best matched the medi­an, that was the sta­tis­tic the researchers chose to work with. And because the medi­an is the num­ber nor­mal­ly used in these sorts of stud­ies, that’s what Lehrer reported.

Peters dis­miss­es all of this by say­ing, essen­tial­ly, that math is scary. The geo­met­ric mean is sim­ply “some­thing con­fus­ing involv­ing log­a­rithms,” so ignore math and the­o­ry and stick with an inap­pro­pri­ate sta­tis­tic. (“Involv­ing log­a­rithms” because the log­a­rithms trans­form the com­pu­ta­tion­al­ly obnox­ious mul­ti­pli­ca­tions and nth roots described above into sim­pler addi­tions and divi­sions fol­lowed by an exponentiation).

So, Freed spends 5500 words to con­vince us “we need not to get spun. Which is what sure­ly hap­pened to Lehrer.” And he wants us to believe that Lehrer got spun into using the medi­an because the authors used the geo­met­ric mean rather than the arith­metic mean. Even though Freed can­not seem to under­stand what a geo­met­ric mean is, let alone why it real­ly is prefer­able here, he thinks it’s Lehrer who must’ve got­ten it wrong. Freed spun himself.

Freed’s broad­er points ‑ about the impor­tance of read­ing papers and news reports crit­i­cal­ly, of think­ing care­ful­ly about the under­ly­ing assump­tions, of check­ing num­bers to make sure they jibe with log­ic, and to ensure that exam­ples aren’t being cher­ry-picked ‑ those are all great. But they’d be that much stronger if the sub­stan­tive issue he’s rais­ing was­n’t a mis­read­ing of Lehrer, the orig­i­nal paper, and basic statistics.