The Lancet has pub­lished a study of mor­tal­i­ty in Iraq, a fol­lowup to a sim­i­lar study from a year ago. In this study, they esti­mat­ed that over 650,000 more peo­ple died in Iraq dur­ing the US occu­pa­tion than would have died otherwise.

The Ques­tion­able Author­i­ty has some objec­tions. I’ll start off by point­ing out that he isn’t dis­put­ing the basic con­clu­sion. Mike writes:

even if I am cor­rect, and all of these errors result in over­es­ti­mates of the total num­ber of deaths, the num­ber is still going to be much high­er than the “offi­cial” totals. The pop­u­la­tion of Iraq is being harmed by this war, and it is being harmed much more than either the Iraqi or US gov­ern­ment are will­ing to admit.

And his cri­tique rais­es some valid points. Empha­siz­ing mor­tal­i­ty caused by coali­tion forces while not empha­siz­ing sec­tar­i­an vio­lence is a mis­take. It’s worth not­ing that the sec­tar­i­an vio­lence is itself a symp­tom of the occu­pa­tion, and giv­en that the mili­tias are civil­ians, it’s hard to sep­a­rate sec­tar­i­an vio­lence from oth­er forms of vio­lent death (the study splits vio­lent death into three cat­e­gories: coali­tion-caused, caused by oth­ers, or unknown). Amer­i­can forces wear uni­forms and dri­ve Amer­i­can mil­i­tary vehi­cles, so peo­ple who report that a vio­lent death was caused by coali­tion forces have some basis for that assessment. 

There are oth­er valid rea­sons to focus on coali­tion-caused deaths. We con­trol the coali­tion, and the coali­tion is sup­posed to be help­ing peo­ple. If we aren’t bet­ter than the ter­ror­ists, why are we even both­er­ing? How­ev­er, those objec­tions are polit­i­cal, not sci­en­tif­ic, and that’s Mike’s point. And he’s right.

His method­olog­i­cal cri­tiques are less accu­rate. He makes four points:

• The base­line pop­u­la­tion esti­mates are dif­fer­ent than the pre­vi­ous study.
• The num­ber of clus­ters sam­pled in an area is pro­por­tion­ate to population.
• The street inter­sec­tion sam­pled is cho­sen randomly.
• Com­put­ing dif­fer­ences between non­vi­o­lent mor­tal­i­ty rates that are not sta­tis­ti­cal­ly different.

The last point is right, but he gets a sta­tis­ti­cal point wrong.

Mike writes:

The num­ber on non-vio­lent excess deaths was obtained by sub­tract­ing the pre-inva­sion non-vio­lent death rate from the post inva­sion non-vio­lent death rate. I’d have no prob­lem with that approach, if they had shown that the post-inva­sion increase was sig­nif­i­cant, but they did not. In fact, when they con­duct­ed a sig­nif­i­cance test, they obtained a p‑value of 0.523, which is not at all sig­nif­i­cant. (If you aren’t famil­iar with p‑values, that more or less means that there’s a 52% chance that the dif­fer­ence between pre- and post-inva­sion rates was just due to the luck of the draw.)

The par­en­thet­i­cal is wrong. A p‑value of 0.523 does not indi­cate the prob­a­bil­i­ty that the two pop­u­la­tions (pre- and post- inva­sion) are dif­fer­ent, it indi­cates how like­ly you’d be to get this result by chance alone if there were no difference. 

That mat­ters because it actu­al­ly strength­ens his point. A p‑value of 0.99 would indi­cate that the two sam­ples are sur­pris­ing­ly sim­i­lar, per­haps even more than you’d expect by chance. A p‑value of 0.5 says that noth­ing at all unusu­al is going on, there’s real­ly no dif­fer­ence. And as Mike observes, remov­ing that effect drops the num­ber of excess deaths by 50,000. “Only” 600,000.

The oth­er three points Mike rais­es seem less valid.

Choos­ing a ran­dom street cor­ner to start with (a ran­dom main street inter­sect­ing a ran­dom res­i­den­tial street) intro­duces no bias. Mike writes that “towns tend to have more main streets per unit of area than rur­al areas do,” but since they are just choos­ing one loca­tion, that does­n’t seem to mat­ter much.

Choos­ing a high­er pop­u­la­tion esti­mate does inflate the mor­tal­i­ty esti­mate to some extent, but unless there’s some basis for think­ing the UNDP esti­mate is worse than oth­er esti­mates, there’s no basis for claim­ing that this is a flaw, or even a bias (except in com­par­isons between this study and the ear­li­er one). 

Sam­pling clus­ters in pro­por­tion to pop­u­la­tion seems entire­ly rea­son­able to me. Sam­pling pop­u­la­tions by choos­ing a ran­dom geo­graph­i­cal point (as done in the 2004 study) seems like it would bias you towards rur­al areas which would tend to deflate your esti­mate of excess mor­tal­i­ty. Sam­pling pro­por­tion­al to pop­u­la­tion will tend to sam­ple house­holds (as opposed to geog­ra­phy) at ran­dom, which seems more accu­rate to me.

This is all borne out by the paper’s own inter­nal checks. The graph here shows that this analy­sis and the pre­vi­ous analy­sis are very close to iden­ti­cal – com­pare the green line and the pur­ple one. The authors point out:

Appli­ca­tion of the mor­tal­i­ty rates report­ed here to the peri­od of the 2004 sur­vey gives an esti­mate of 112,000 (69,000–155,000) excess deaths in Iraq in that peri­od. Thus, the data pre­sent­ed here val­i­dates our 2004 study, which con­ser­v­a­tive­ly esti­mat­ed an excess mor­tal­i­ty of near­ly 10,000 as of Sep­tem­ber, 2004.

They got near­ly iden­ti­cal results for iden­ti­cal time peri­ods, which means that the changes to their tech­nique did not undu­ly alter their analysis.