NHST and Systematic Biases

Sta­tis­ti­cal back­ground on p-value prob­lems: Against nul­l-hy­poth­e­sis sig­nifi­cance test­ing

The ba­sic na­ture of ‘sig­nifi­cance’ be­ing usu­ally de­fined as p < 0.05 means we should ex­pect some­thing like >5% of stud­ies or ex­per­i­ments to be bo­gus (op­ti­misti­cal­ly), but that only con­sid­ers “false pos­i­tives”; re­duc­ing “false neg­a­tives” re­quires sta­tis­ti­cal power (weak­ened by small sam­ples), and the two com­bine with the base rate of true un­der­ly­ing effects into a to­tal er­ror rate. Ioan­ni­dis 2005 points out that con­sid­er­ing the usual p val­ues, the un­der­pow­ered na­ture of many stud­ies, the rar­ity of un­der­ly­ing effects, and a lit­tle bi­as, even large ran­dom­ized tri­als may wind up with only an 85% chance of hav­ing yielded the truth. One sur­vey of re­ported p-val­ues in med­i­cine yield­ing a lower bound of false pos­i­tives of 17%.

Yet, there are too many pos­i­tive re­sults¹ (psy­chi­a­try, neu­ro­bi­ol­ogy bio­med­i­cine, bi­ol­ogy, ecol­ogy & evo­lu­tion, psy­chol­ogy 12 3 4 5, eco­nom­ics’ top jour­nals, so­ci­ol­ogy, gene-dis­ease cor­re­la­tions) given effect sizes (and pos­i­tive re­sults cor­re­late with per capita pub­lish­ing rates in US states & vary by pe­riod & coun­try—ap­par­ently chance is kind to sci­en­tists who must pub­lish a lot and re­cent­ly!); then there come the in­ad­ver­tent er­rors which might cause re­trac­tion, which is rare, but the true re­trac­tion rate may be 0.1–1% (“How many sci­en­tific pa­pers should be re­tract­ed?”), is in­creas­ing & seems to pos­i­tively cor­re­late with jour­nal pres­tige met­rics (mod­ulo the con­found­ing fac­tor that fa­mous papers/journals get more scruti­ny), not that any­one pays any at­ten­tion to such things; then there are ba­sic sta­tis­ti­cal er­rors in >11% of pa­pers (based on the high­-qual­ity pa­pers in Na­ture and the British Med­ical Jour­nal; “In­con­gru­ence be­tween test sta­tis­tics and P val­ues in med­ical pa­pers”, Gar­cía-Ber­thou 2004) or 50% in neu­ro­science.

And only then can we get into repli­cat­ing at all. See for ex­am­ple The At­lantic ar­ti­cle “Lies, Damned Lies, and Med­ical Sci­ence” on John P. A. Ioan­ni­dis’s re­search show­ing 41% of the most cited med­ical re­search failed to be repli­cated—were wrong. For de­tails, you can see Ioan­ni­dis’s “Why Most Pub­lished Re­search Find­ings Are False”², or Be­g­ley’s failed at­tempts to repli­cate 47 of 53 ar­ti­cles on top can­cer jour­nals (lead­ing to Booth’s “Be­g­ley’s Six Rules”; see also the Na­ture Biotech­nol­ogy ed­i­to­r­ial & note that full de­tails have not been pub­lished be­cause the re­searchers of the orig­i­nal stud­ies de­manded se­crecy from Be­g­ley’s team), or Ku­mar & Nash 2011’s “Health Care Myth Busters: Is There a High De­gree of Sci­en­tific Cer­tainty in Mod­ern Med­i­cine?” who write ‘We could ac­cu­rately say, “Half of what physi­cians do is wrong,” or “Less than 20% of what physi­cians do has solid re­search to sup­port it.”’ Nu­tri­tional epi­demi­ol­ogy is some­thing of a fish in a bar­rel; after Ioan­ni­dis, is any­one sur­prised that when Young & Karr 2011 fol­lowed up on 52 cor­re­la­tions tested in 12 RCTs, 0⁄52 repli­cated and the RCTs found the op­po­site of 5?

At­tempts to use an­i­mal mod­els to in­fer any­thing about hu­mans suffer from all the method­olog­i­cal prob­lems pre­vi­ously men­tioned, and add in in­ter­est­ing new forms of er­ror such as mice sim­ply be­ing ir­rel­e­vant to hu­mans, lead­ing to cases like <150 sep­sis clin­i­cal tri­als all fail­ing—be­cause the drugs worked in mice but hu­mans have a com­pletely differ­ent set of ge­netic re­ac­tions to in­flam­ma­tion.

‘Hot’ fields tend to be new fields, which brings prob­lems of its own, see “Large-S­cale As­sess­ment of the Effect of Pop­u­lar­ity on the Re­li­a­bil­ity of Re­search” & dis­cus­sion. (Fail­ure to repli­cate in larger stud­ies seems to be a hall­mark of biological/medical re­search. Ioan­ni­dis per­forms the same trick with bio­mark­ers, find­ing less than half of the most-cited bio­mark­ers were even sta­tis­ti­cal­ly-sig­nifi­cant in the larger stud­ies. 12 of the more promi­nent SNP-IQ cor­re­la­tions failed to repli­cate on a larger da­ta.) As we know now, al­most the en­tire can­di­date-gene lit­er­a­ture, most things re­ported from 2000–2010 be­fore large-s­cale GWASes started to be done (and com­pletely fail­ing to find the can­di­date-ge­nes), is noth­ing but false pos­i­tives! The repli­ca­tion rates of can­di­date-genes for things like in­tel­li­gence, per­son­al­i­ty, gene-en­vi­ron­ment in­ter­ac­tions, psy­chi­atric dis­or­der­s–the whole schmeer—are lit­er­ally ~0%. On the plus side, the par­lous state of affairs means that there are some cheap heuris­tics for de­tect­ing un­re­li­able pa­pers—sim­ply ask­ing for data & be­ing refused/ignored cor­re­lates strongly with the orig­i­nal pa­per hav­ing er­rors in its sta­tis­tics.

This epi­demic of false pos­i­tives is ap­par­ently de­lib­er­ately and know­ing ac­cepted by epi­demi­ol­ogy; Young’s 2008 “Every­thing is Dan­ger­ous” re­marks that 80–90% of epi­demi­ol­o­gy’s claims do not repli­cate (eg. the NIH ran 20 ran­dom­ized-con­trolled-tri­als of claims, and only 1 repli­cat­ed) and that lack of ‘mul­ti­ple com­par­isons’ (ei­ther Bon­fer­roni or Ben­jam­in-Hochberg) is taught: “Roth­man (1990) says no cor­rec­tion for mul­ti­ple test­ing is nec­es­sary and Van­den­broucke, PLoS Med (2008) agrees” (see also Per­neger 1998 who also ex­plic­itly un­der­stands that no cor­rec­tion in­creases type 2 er­rors and re­duces type 1 er­rors). Mul­ti­ple cor­rec­tion is nec­es­sary be­cause its ab­sence does, in fact, re­sult in the over­state­ment of med­ical ben­e­fit (God­frey 1985, Pocock et al 1987, Smith 1987). The av­er­age effect size for find­ings con­firmed meta-an­a­lyt­i­cally in psychology/education is d = 0.5³ (well be­low sev­eral effect sizes from n-back/IQ stud­ies); when mov­ing from lab­o­ra­tory to non-lab­o­ra­tory set­tings, meta-analy­ses repli­cate find­ings cor­re­late ~0.7 of the time, but for so­cial psy­chol­ogy the repli­ca­tion cor­re­la­tion falls to ~0.5 with >14% of find­ings ac­tu­ally turn­ing out to be the op­po­site (see An­der­son et al 1999 and Mitchell 2012; for ex­ag­ger­a­tion due to non-blind­ing or poor ran­dom­iza­tion, Wood et al 2008). (Meta-analy­ses also give us a start­ing point for un­der­stand­ing how un­usual medium or large effects sizes are⁴.) Psy­chol­ogy does have many chal­lenges, but prac­ti­tion­ers also hand­i­cap them­selves; an older overview is the en­ter­tain­ing “What’s Wrong With Psy­chol­o­gy, Any­way?”, which men­tions the ob­vi­ous point that sta­tis­tics & ex­per­i­men­tal de­sign are flex­i­ble enough to reach sig­nifi­cance as de­sired. In an in­ter­est­ing ex­am­ple of how method­olog­i­cal re­forms are no panacea in the pres­ence of con­tin­ued per­verse in­cen­tives, an ear­lier method­olog­i­cal im­prove­ment in psy­chol­ogy (re­port­ing mul­ti­ple ex­per­i­ments in a sin­gle pub­li­ca­tion as a check against re­sults not be­ing gen­er­al­iz­able) has merely demon­strated the wide­spread p-value hack­ing or ma­nip­u­la­tion or pub­li­ca­tion bias when one notes that given the low sta­tis­ti­cal power of each ex­per­i­ment, even if the un­der­ly­ing phe­nom­ena were real it would still be wildly im­prob­a­ble that all n ex­per­i­ments in a pa­per would turn up sta­tis­ti­cal­ly-sig­nifi­cant re­sults, since power is usu­ally ex­tremely low in ex­per­i­ments (eg. in neu­ro­science, “be­tween 20–30%”). These prob­lems are per­va­sive enough that I be­lieve they en­tirely ex­plain any “de­cline effects”⁵.

The fail­ures to repli­cate “sta­tis­ti­cally sig­nifi­cant” re­sults has led one blog­ger to caus­ti­cally re­mark (see also “Para­psy­chol­o­gy: the con­trol group for sci­ence”, “Us­ing de­grees of free­dom to change the past for fun and profit”, “The Con­trol Group is Out Of Con­trol”):

Para­psy­chol­o­gy, the con­trol group for sci­ence, would seem to be a thriv­ing field with “sta­tis­ti­cally sig­nifi­cant” re­sults aplen­ty…­Para­psy­chol­o­gists are con­stantly protest­ing that they are play­ing by all the stan­dard sci­en­tific rules, and yet their re­sults are be­ing ig­nored—that they are un­fairly be­ing held to higher stan­dards than every­one else. I’m will­ing to be­lieve that. It just means that the stan­dard sta­tis­ti­cal meth­ods of sci­ence are so weak and flawed as to per­mit a field of study to sus­tain it­self in the com­plete ab­sence of any sub­ject mat­ter. With two-thirds of med­ical stud­ies in pres­ti­gious jour­nals fail­ing to repli­cate, get­ting rid of the en­tire ac­tual sub­ject mat­ter would shrink the field by only 33%.

Cosma Shal­izi:

…Let me draw the moral [about pub­li­ca­tion bi­as]. Even if the com­mu­nity of in­quiry is both too clue­less to make any con­tact with re­al­ity and too hon­est to nudge bor­der­line find­ings into sig­nifi­cance, so long as they can keep com­ing up with new phe­nom­ena to look for, the mech­a­nism of the file-drawer prob­lem alone will guar­an­tee a steady stream of new re­sults. There is, so far as I know, no Jour­nal of Ev­i­dence-Based Harus­picy filled, is­sue after is­sue, with method­olog­i­cal­ly-fault­less pa­pers re­port­ing the abil­ity of sheep’s liv­ers to pre­dict the win­ners of sumo cham­pi­onships, the out­come of speed dates, or real es­tate trends in se­lected sub­urbs of Chica­go. But the diffi­culty can only be that the ev­i­dence-based harus­pices aren’t try­ing hard enough, and some friendly ri­valry with the plas­tro­mancers is called for. It’s true that none of these find­ings will last forever, but this con­stant over­turn­ing of old ideas by new dis­cov­er­ies is just part of what makes this such a dy­namic time in the field of harus­picy. Many schol­ars will even tell you that their fa­vorite part of be­ing a harus­pex is the fre­quency with which a new sac­ri­fice over-turns every­thing they thought they knew about read­ing the fu­ture from a sheep’s liv­er! We are very ex­cited about the re­newed in­ter­est on the part of pol­i­cy-mak­ers in the rec­om­men­da­tions of the man­tic arts…

And this is when there is enough in­for­ma­tion to repli­cate; open ac­cess to any data for a pa­per is rare (e­co­nom­ics: <10%) the eco­nom­ics jour­nal Jour­nal of Mon­ey, Credit and Bank­ing, which re­quired re­searchers pro­vide the data & soft­ware which could repli­cate their sta­tis­ti­cal analy­ses, dis­cov­ered that <10% of the sub­mit­ted ma­te­ri­als were ad­e­quate for re­peat­ing the pa­per (see “Lessons from the JMCB Archive”). In one cute eco­nom­ics ex­am­ple, repli­ca­tion failed be­cause the dataset had been heav­ily edited to make par­tic­i­pants look bet­ter (for more eco­nom­ic­s-spe­cific cri­tique, see Ioan­ni­dis & Doucou­lia­gos 2013). Avail­abil­ity of data, often low, de­creases with time, and many stud­ies never get pub­lished re­gard­less of whether pub­li­ca­tion is legally man­dated.

Tran­scrip­tion er­rors in pa­pers seem to be com­mon (pos­si­bly due to con­stantly chang­ing analy­ses & p-hack­ing?), and as soft­ware and large datasets be­comes more in­her­ent to re­search, the need and the prob­lem of it be­ing pos­si­ble to repli­cate will get worse be­cause even ma­ture com­mer­cial soft­ware li­braries can dis­agree ma­jorly on their com­puted re­sults to the same math­e­mat­i­cal spec­i­fi­ca­tion (see also Anda et al 2009). And spread­sheets are es­pe­cially bad, with er­ror rates in the 88% range (“What we know about spread­sheet er­rors”, Panko 1998); spread­sheets are used in all ar­eas of sci­ence, in­clud­ing bi­ol­ogy and med­i­cine (see “Er­ror! What bio­med­ical com­put­ing can learn from its mis­takes”; fa­mous ex­am­ples of cod­ing er­rors in­clude Dono­hue-Le­vitt & Rein­hart-Ro­goff), not to men­tion reg­u­lar busi­ness (eg the Lon­don Whale).

Psy­chol­ogy is far from be­ing per­fect ei­ther; look at the ex­am­ples in The New Yorker’s “The Truth Wears Off” ar­ti­cle (or look at some ex­cerpts from that ar­ti­cle). Com­puter sci­en­tist Pe­ter Norvig has writ­ten a must-read es­say on in­ter­pret­ing sta­tis­tics, “Warn­ing Signs in Ex­per­i­men­tal De­sign and In­ter­pre­ta­tion”; a num­ber of warn­ing signs ap­ply to many psy­cho­log­i­cal stud­ies. There may be in­cen­tive prob­lems: a trans­plant re­searcher dis­cov­ered the only way to pub­lish in Na­ture his in­abil­ity to repli­cate his ear­lier Na­ture pa­per was to offi­cially re­tract it; an­other in­ter­est­ing ex­am­ple is when, after Daryl Bem got a pa­per pub­lished in the top jour­nal JPSP demon­strat­ing pre­cog­ni­tion, the jour­nal re­fused to pub­lish any repli­ca­tions (failed or suc­cess­ful) be­cause… “‘We don’t want to be the Jour­nal of Bem Repli­ca­tion’, he says, point­ing out that other high­-pro­file jour­nals have sim­i­lar poli­cies of pub­lish­ing only the best orig­i­nal re­search.” (Quoted in New Sci­en­tist) One does­n’t need to be a ge­nius to un­der­stand why psy­chol­o­gist An­drew D. Wil­son might snark­ily re­mark “…think about the mes­sage JPSP is send­ing to au­thors. That mes­sage is ‘we will pub­lish your crazy story if it’s new, but not your sen­si­ble story if it’s merely a repli­ca­tion’.” (You get what you pay for.) In one large test of the most fa­mous psy­chol­ogy re­sults, 10 of 13 (77%) repli­cat­ed. The repli­ca­tion rate is un­der 1⁄3 in one area of psy­chol­ogy touch­ing on ge­net­ics. This de­spite the sim­ple point that repli­ca­tions re­duce the risk of pub­li­ca­tion bi­as, and in­crease sta­tis­ti­cal pow­er, so that a repli­cated re­sult is more likely to be true. And the small sam­ples of n-back stud­ies and nootropic chem­i­cals are es­pe­cially prob­lem­at­ic. Quot­ing from Nick Bostrom & An­ders Sand­berg’s 2006 “Con­verg­ing Cog­ni­tive En­hance­ments”:

The re­li­a­bil­ity of re­search is also an is­sue. Many of the cog­ni­tion-en­hanc­ing in­ter­ven­tions show small effect sizes, which may ne­ces­si­tate very large epi­demi­o­log­i­cal stud­ies pos­si­bly ex­pos­ing large groups to un­fore­seen risks.

Par­tic­u­larly trou­bling is the slow­down in drug dis­cov­ery & med­ical tech­nol­ogy dur­ing the 2000s, even as ge­net­ics in par­tic­u­lar was ex­pected to pro­duce earth­-shak­ing new treat­ments. One biotech ven­ture cap­i­tal­ist writes:

The com­pany spent $6^$5₂₀₁₁M or so try­ing to val­i­date a plat­form that did­n’t ex­ist. When they tried to di­rectly re­peat the aca­d­e­mic founder’s data, it never worked. Upon re-ex­am­i­na­tion of the lab note­books, it was clear the founder’s lab had at the very least mas­saged the data and shaped it to fit their hy­poth­e­sis. Es­sen­tial­ly, they sys­tem­at­i­cally ig­nored every piece of neg­a­tive da­ta. Sadly this “fail­ure to re­peat” hap­pens more often than we’d like to be­lieve. It has hap­pened to us at At­las [Ven­ture] sev­eral times in the past decade…The un­spo­ken rule is that at least 50% of the stud­ies pub­lished even in top tier aca­d­e­mic jour­nals—Sci­ence, Na­ture, Cell, PNAS, etc…—­can’t be re­peated with the same con­clu­sions by an in­dus­trial lab. In par­tic­u­lar, key an­i­mal mod­els often don’t re­pro­duce. This 50% fail­ure rate is­n’t a data free as­ser­tion: it’s backed up by dozens of ex­pe­ri­enced R&D pro­fes­sion­als who’ve par­tic­i­pated in the (re)test­ing of aca­d­e­mic find­ings. This is a huge prob­lem for trans­la­tional re­search and one that won’t go away un­til we ad­dress it head on.

Half the re­spon­dents to a 2012 sur­vey at one can­cer re­search cen­ter re­ported 1 or more in­ci­dents where they could not re­pro­duce pub­lished re­search; two-thirds of those were un­able to “ever able to ex­plain or re­solve their dis­crepant find­ings”, half had trou­ble pub­lish­ing re­sults con­tra­dict­ing pre­vi­ous pub­li­ca­tions, and two-thirds failed to pub­lish con­tra­dic­tory re­sults. An in­ter­nal Bayer sur­vey of 67 projects (com­men­tary) found that “only in ~20–25% of the projects were the rel­e­vant pub­lished data com­pletely in line with our in­-house find­ings”, and as far as as­sess­ing the projects went:

…de­spite the low num­bers, there was no ap­par­ent differ­ence be­tween the differ­ent re­search fields. Sur­pris­ing­ly, even pub­li­ca­tions in pres­ti­gious jour­nals or from sev­eral in­de­pen­dent groups did not en­sure re­pro­ducibil­i­ty. In­deed, our analy­sis re­vealed that the re­pro­ducibil­ity of pub­lished data did not sig­nifi­cantly cor­re­late with jour­nal im­pact fac­tors, the num­ber of pub­li­ca­tions on the re­spec­tive tar­get or the num­ber of in­de­pen­dent groups that au­thored the pub­li­ca­tions. Our find­ings are mir­rored by ‘gut feel­ings’ ex­pressed in per­sonal com­mu­ni­ca­tions with sci­en­tists from acad­e­mia or other com­pa­nies, as well as pub­lished ob­ser­va­tions. [apro­pos of above] An un­spo­ken rule among ear­ly-stage ven­ture cap­i­tal firms that “at least 50% of pub­lished stud­ies, even those in top-tier aca­d­e­mic jour­nals, can’t be re­peated with the same con­clu­sions by an in­dus­trial lab” has been re­cently re­ported (see Fur­ther in­for­ma­tion) and dis­cussed 4.

Physics has rel­a­tively small sins; “As­sess­ing un­cer­tainty in phys­i­cal con­stants” (Hen­rion & Fischoff 1985); Han­son’s sum­ma­ry:

Look­ing at 306 es­ti­mates for par­ti­cle prop­er­ties, 7% were out­side of a 98% con­fi­dence in­ter­val (where only 2% should be). In seven other cas­es, each with 14 to 40 es­ti­mates, the frac­tion out­side the 98% con­fi­dence in­ter­val ranged from 7% to 57%, with a me­dian of 14%.

Nor is peer re­view re­li­able or ro­bust against even low lev­els of col­lu­sion. Sci­en­tists who win the No­bel Prize find their other work sud­denly be­ing heav­ily cited, sug­gest­ing ei­ther that the com­mu­nity ei­ther badly failed in rec­og­niz­ing the work’s true value or that they are now suck­ing up & at­tempt­ing to look bet­ter by as­so­ci­a­tion. (A math­e­mati­cian once told me that often, to boost a pa­per’s ac­cep­tance chance, they would add ci­ta­tions to pa­pers by the jour­nal’s ed­i­tors—a prac­tice that will sur­prise none fa­mil­iar with Good­hart’s law and the use of ci­ta­tions in tenure & grants.)

The for­mer ed­i­tor Richard Smith amus­ingly re­counts his doubts about the mer­its of peer re­view as prac­ticed, and physi­cist Michael Nielsen points out that peer re­view is his­tor­i­cally rare (just one of Ein­stein’s 300 pa­pers was peer re­viewed; the fa­mous Na­ture did not in­sti­tute peer re­view un­til 1967), has been poorly stud­ied & not shown to be effec­tive, is na­tion­ally bi­ased, er­ro­neously re­jects many his­toric dis­cov­er­ies (one study lists “34 No­bel Lau­re­ates whose awarded work was re­jected by peer re­view”; Hor­robin 1990 lists oth­er), and catches only a small frac­tion of er­rors. And ques­tion­able choices or fraud? For­get about it:

A pooled weighted av­er­age of 1.97% (N = 7, 95%­CI: 0.86–4.45) of sci­en­tists ad­mit­ted to have fab­ri­cat­ed, fal­si­fied or mod­i­fied data or re­sults at least on­ce—a se­ri­ous form of mis­con­duct by any stan­dard­—and up to 33.7% ad­mit­ted other ques­tion­able re­search prac­tices. In sur­veys ask­ing about the be­hav­iour of col­leagues, ad­mis­sion rates were 14.12% (N = 12, 95% CI: 9.91–19.72) for fal­si­fi­ca­tion, and up to 72% for other ques­tion­able re­search prac­tices…When these fac­tors were con­trolled for, mis­con­duct was re­ported more fre­quently by medical/pharmacological re­searchers than oth­ers.

And psy­chol­o­gists:

We sur­veyed over 2,000 psy­chol­o­gists about their in­volve­ment in ques­tion­able re­search prac­tices, us­ing an anony­mous elic­i­ta­tion for­mat sup­ple­mented by in­cen­tives for hon­est re­port­ing. The im­pact of in­cen­tives on ad­mis­sion rates was pos­i­tive, and greater for prac­tices that re­spon­dents judge to be less de­fen­si­ble. Us­ing three differ­ent es­ti­ma­tion meth­ods, we find that the pro­por­tion of re­spon­dents that have en­gaged in these prac­tices is sur­pris­ingly high rel­a­tive to re­spon­dents’ own es­ti­mates of these pro­por­tions. Some ques­tion­able prac­tices may con­sti­tute the pre­vail­ing re­search norm.

In short, the se­cret sauce of sci­ence is not ‘peer re­view’. It is repli­ca­tion!

Systemic Error Doesn’t Go Away

Why is­n’t the so­lu­tion as sim­ple as elim­i­nat­ing dat­a­min­ing by meth­ods like larger n or pre-reg­is­tered analy­ses? Be­cause once we have elim­i­nated the ran­dom er­ror in our analy­sis, we are still left with a (po­ten­tially ar­bi­trar­ily large) sys­tem­atic er­ror, leav­ing us with a large to­tal er­ror.

None of these sys­tem­atic prob­lems should be con­sid­ered mi­nor or method­olog­i­cal quib­bling or fool­ish ide­al­ism: they are sys­tem­atic bi­ases and as such, they force an up­per bound on how ac­cu­rate a cor­pus of stud­ies can be even if there were thou­sands upon thou­sands of stud­ies, be­cause the to­tal er­ror in the re­sults is made up of ran­dom er­ror and sys­tem­atic er­ror, but while ran­dom er­ror shrinks as more stud­ies are done, sys­tem­atic er­ror re­mains the same.

A thou­sand bi­ased stud­ies merely re­sult in an ex­tremely pre­cise es­ti­mate of the wrong num­ber.

This is a point ap­pre­ci­ated by sta­tis­ti­cians and ex­per­i­men­tal physi­cists, but it does­n’t seem to be fre­quently dis­cussed. An­drew Gel­man has a fun demon­stra­tion of se­lec­tion bias in­volv­ing candy, or from pg812–1020 of Chap­ter 8 “Suffi­cien­cy, An­cil­lar­i­ty, And All That” of Prob­a­bil­ity The­o­ry: The Logic of Sci­ence by E.T. Jaynes:

The clas­si­cal ex­am­ple show­ing the er­ror of this kind of rea­son­ing is the fa­ble about the height of the Em­peror of Chi­na. Sup­pos­ing that each per­son in China surely knows the height of the Em­peror to an ac­cu­racy of at least ±1 me­ter, if there are N = 1,000,000,000 in­hab­i­tants, then it seems that we could de­ter­mine his height to an ac­cu­racy at least as good as

$\frac{1}{\sqrt{1,000,000,000}}m=0.003cm$ (8-49)

merely by ask­ing each per­son’s opin­ion and av­er­ag­ing the re­sults.

The ab­sur­dity of the con­clu­sion tells us rather force­fully that the $\sqrt{N}$ rule is not al­ways valid, even when the sep­a­rate data val­ues are causally in­de­pen­dent; it re­quires them to be log­i­cally in­de­pen­dent. In this case, we know that the vast ma­jor­ity of the in­hab­i­tants of China have never seen the Em­per­or; yet they have been dis­cussing the Em­peror among them­selves and some kind of men­tal im­age of him has evolved as folk­lore. Then knowl­edge of the an­swer given by one does tell us some­thing about the an­swer likely to be given by an­oth­er, so they are not log­i­cally in­de­pen­dent. In­deed, folk­lore has al­most surely gen­er­ated a sys­tem­atic er­ror, which sur­vives the av­er­ag­ing; thus the above es­ti­mate would tell us some­thing about the folk­lore, but al­most noth­ing about the Em­per­or.

We could put it roughly as fol­lows:

er­ror in es­ti­mate = $S\pm \frac{R}{\sqrt{N}}$ (8-50)

where S is the com­mon sys­tem­atic er­ror in each da­tum, R is the RMS ‘ran­dom’ er­ror in the in­di­vid­ual data val­ues. Un­in­formed opin­ions, even though they may agree well among them­selves, are nearly worth­less as ev­i­dence. There­fore sound sci­en­tific in­fer­ence de­mands that, when this is a pos­si­bil­i­ty, we use a form of prob­a­bil­ity the­ory (i.e. a prob­a­bilis­tic mod­el) which is so­phis­ti­cated enough to de­tect this sit­u­a­tion and make al­lowances for it.

As a start on this, equa­tion (8-50) gives us a crude but use­ful rule of thumb; it shows that, un­less we know that the sys­tem­atic er­ror is less than about 1⁄3 of the ran­dom er­ror, we can­not be sure that the av­er­age of a mil­lion data val­ues is any more ac­cu­rate or re­li­able than the av­er­age of ten⁶. As Henri Poin­care put it: “The physi­cist is per­suaded that one good mea­sure­ment is worth many bad ones.” This has been well rec­og­nized by ex­per­i­men­tal physi­cists for gen­er­a­tions; but warn­ings about it are con­spic­u­ously miss­ing in the “soft” sci­ences whose prac­ti­tion­ers are ed­u­cated from those text­books.

Or pg1019–1020 Chap­ter 10 “Physics of ‘Ran­dom Ex­per­i­ments’”:

…N­ev­er­the­less, the ex­is­tence of such a strong con­nec­tion is clearly only an ideal lim­it­ing case un­likely to be re­al­ized in any real ap­pli­ca­tion. For this rea­son, the law of large num­bers and limit the­o­rems of prob­a­bil­ity the­ory can be grossly mis­lead­ing to a sci­en­tist or en­gi­neer who naively sup­poses them to be ex­per­i­men­tal facts, and tries to in­ter­pret them lit­er­ally in his prob­lems. Here are two sim­ple ex­am­ples:

1. Sup­pose there is some ran­dom ex­per­i­ment in which you as­sign a prob­a­bil­ity p for some par­tic­u­lar out­come A. It is im­por­tant to es­ti­mate ac­cu­rately the frac­tion f of times A will be true in the next mil­lion tri­als. If you try to use the laws of large num­bers, it will tell you var­i­ous things about f; for ex­am­ple, that it is quite likely to differ from p by less than a tenth of one per­cent, and enor­mously un­likely to differ from p by more than one per­cent. But now, imag­ine that in the first hun­dred tri­als, the ob­served fre­quency of A turned out to be en­tirely differ­ent from p. Would this lead you to sus­pect that some­thing was wrong, and re­vise your prob­a­bil­ity as­sign­ment for the 101’st tri­al? If it would, then your state of knowl­edge is differ­ent from that re­quired for the va­lid­ity of the law of large num­bers. You are not sure of the in­de­pen­dence of differ­ent tri­als, and/or you are not sure of the cor­rect­ness of the nu­mer­i­cal value of p. Your pre­dic­tion of f for a mil­lion tri­als is prob­a­bly no more re­li­able than for a hun­dred.
2. The com­mon sense of a good ex­per­i­men­tal sci­en­tist tells him the same thing with­out any prob­a­bil­ity the­o­ry. Sup­pose some­one is mea­sur­ing the ve­loc­ity of light. After mak­ing al­lowances for the known sys­tem­atic er­rors, he could cal­cu­late a prob­a­bil­ity dis­tri­b­u­tion for the var­i­ous other er­rors, based on the noise level in his elec­tron­ics, vi­bra­tion am­pli­tudes, etc. At this point, a naive ap­pli­ca­tion of the law of large num­bers might lead him to think that he can add three sig­nifi­cant fig­ures to his mea­sure­ment merely by re­peat­ing it a mil­lion times and av­er­ag­ing the re­sults. But, of course, what he would ac­tu­ally do is to re­peat some un­known sys­tem­atic er­ror a mil­lion times. It is idle to re­peat a phys­i­cal mea­sure­ment an enor­mous num­ber of times in the hope that “good sta­tis­tics” will av­er­age out your er­rors, be­cause we can­not know the full sys­tem­atic er­ror. This is the old “Em­peror of China” fal­la­cy…

In­deed, un­less we know that all sources of sys­tem­atic er­ror—rec­og­nized or un­rec­og­nized—­con­tribute less than about one-third the to­tal er­ror, we can­not be sure that the av­er­age of a mil­lion mea­sure­ments is any more re­li­able than the av­er­age of ten. Our time is much bet­ter spent in de­sign­ing a new ex­per­i­ment which will give a lower prob­a­ble er­ror per tri­al. As Poin­care put it, “The physi­cist is per­suaded that one good mea­sure­ment is worth many bad ones.”⁷ In other words, the com­mon sense of a sci­en­tist tells him that the prob­a­bil­i­ties he as­signs to var­i­ous er­rors do not have a strong con­nec­tion with fre­quen­cies, and that meth­ods of in­fer­ence which pre­sup­pose such a con­nec­tion could be dis­as­trously mis­lead­ing in his prob­lems.

Schlaifer much ear­lier made the same point in Prob­a­bil­ity and Sta­tis­tics for Busi­ness De­ci­sions: an In­tro­duc­tion to Man­age­r­ial Eco­nom­ics Un­der Un­cer­tainty, Schlaifer 1959, pg488–489 (see also Meng 2018/Shi­rani-Mehr et al 2018):

31.4.3 Bias and Sam­ple Size

In Sec­tion 31.2.6 we used a hy­po­thet­i­cal ex­am­ple to il­lus­trate the im­pli­ca­tions of the fact that the vari­ance of the mean of a sam­ple in which bias is sus­pected is

$${\sigma }^2\left(x\right)={\sigma }^2\left(\tilde{\beta }\right)+\frac{1}{n}{\sigma }^2\left(\widetilde{ϵ}\right)\frac{N-n}{N-1}$$

so that only the sec­ond term de­creases as the sam­ple size in­creases and the to­tal can never be less than the fixed value of the first term. To em­pha­size the im­por­tance of this point by a real ex­am­ple we re­call the most fa­mous sam­pling fi­asco in his­to­ry, the pres­i­den­tial poll con­ducted by the Lit­er­ary Di­gest in 1936. Over 2 mil­lion reg­is­tered vot­ers filled in and re­turned the straw bal­lots sent out by the Di­gest, so that there was less than one chance in 1 bil­lion of a sam­pling er­ror as large as 2⁄10 of one per­cent­age point⁸, and yet the poll was ac­tu­ally off by nearly 18 per­cent­age points: it pre­dicted that 54.5 per cent of the pop­u­lar vote would go to Lan­don, who in fact re­ceived only 36.7 per cent.^{9 10}

Since sam­pling er­ror can­not ac­count for any ap­pre­cia­ble part of the 18-point dis­crep­an­cy, it is vir­tu­ally all ac­tual bias. A part of this to­tal bias may be mea­sure­ment bias due to the fact that not all peo­ple voted as they said they would vote; the im­pli­ca­tions of this pos­si­bil­ity were dis­cussed in Sec­tion 31.3. The larger part of the to­tal bi­as, how­ev­er, was al­most cer­tainly se­lec­tion bias. The straw bal­lots were mailed to peo­ple whose names were se­lected from lists of own­ers of tele­phones and au­to­mo­biles and the sub­pop­u­la­tion which was effec­tively sam­pled was even more re­stricted than this: it con­sisted only of those own­ers of tele­phones and au­to­mo­biles who were will­ing to fill out and re­turn a straw bal­lot. The true mean of this sub­pop­u­la­tion proved to be en­tirely differ­ent from the true mean of the pop­u­la­tion of all United States cit­i­zens who voted in 1936.

It is true that there was no ev­i­dence at the time this poll was planned which would have sug­gested that the bias would be as great as the 18 per­cent­age points ac­tu­ally re­al­ized, but ex­pe­ri­ence with pre­vi­ous polls had shown bi­ases which would have led any sen­si­ble per­son to as­sign to $\tilde{\beta }$ a dis­tri­b­u­tion with $\sigma \left(\tilde{\beta }\right)$ equal to at least 1 per­cent­age point. A sam­ple of only 23,760 re­turned bal­lots, one 1⁄100th the size ac­tu­ally used, would have given $\sigma \left(ϵ\right)$ a value of only 1⁄3 per­cent­age point, so that the stan­dard de­vi­a­tion of x would have been

$$\sigma \left(x\right)=\sqrt{{\sigma }^2\left(\tilde{\beta }\right)+{\sigma }^2\left(ϵ\right)}=\sqrt{1+.11}=1.05$$

per­cent­age points. Us­ing a sam­ple 100 times this large re­duced $\sigma \left(ϵ\right)$ from 1⁄3 point to vir­tu­ally ze­ro, but it could not affect $\sigma \left(\tilde{\beta }\right)$ and thus on the most fa­vor­able as­sump­tion could re­duce $\sigma \left(x\right)$ only from 1.05 points to 1 point. To col­lect and tab­u­late over 2 mil­lion ad­di­tional bal­lots when this was the great­est gain that could be hoped for was ob­vi­ously ridicu­lous be­fore the fact and not just in the light of hind­sight.

What’s par­tic­u­larly sad is when peo­ple read some­thing like this and de­cide to rely on anec­dotes, per­sonal ex­per­i­ments, and al­ter­na­tive med­i­cine where there are even more sys­tem­atic er­rors and no way of re­duc­ing ran­dom er­ror at all! Sci­ence may be the lens that sees its own flaws, but if other epis­te­molo­gies do not boast such long de­tailed self­-cri­tiques, it’s not be­cause they are flaw­less… It’s like that old Jamie Za­w­in­ski quote: Some peo­ple, when faced with the prob­lem of main­stream med­i­cine & epi­demi­ol­ogy hav­ing se­ri­ous method­olog­i­cal weak­ness­es, say “I know, I’ll turn to non-main­stream med­i­cine & epi­demi­ol­o­gy. After all, if only some med­i­cine is based on real sci­en­tific method and out­per­forms place­bos, why both­er?” (Now they have two prob­lem­s.) Or per­haps Isaac Asi­mov: “John, when peo­ple thought the earth was flat, they were wrong. When peo­ple thought the earth was spher­i­cal, they were wrong. But if you think that think­ing the earth is spher­i­cal is just as wrong as think­ing the earth is flat, then your view is wronger than both of them put to­geth­er.”

See Also

• “The Iron Laws of Eval­u­a­tion”

• How Often Does Cor­re­la­tion=­Causal­i­ty?

• Why Cor­re­la­tion Usu­ally ≠ Cau­sa­tion: Causal Nets Cause Com­mon Con­found­ing

• Lit­tle­wood’s Law and the Global Me­dia

• The Ex­is­ten­tial Risk of Math Er­ror

• Ques­tion­able re­search find­ings:

  • Lep­rechaun hunt­ing
  • “Hy­dro­cephalus and In­tel­li­gence: The Hol­low Men”
  • On the ‘Mouse Utopia’ ex­per­i­ment
  • Dual N-Back Meta-Analy­sis
  • Lu­nar cir­ca­dian rhythms
  • On the Jeanne Cal­ment longevity anom­aly
  • Munks­gaard et al 2016, “A repli­ca­tion and method­olog­i­cal cri­tique of the study ‘Eval­u­at­ing drug traffick­ing on the Tor Net­work’”

Appendix