NHST and Systematic Biases

Sta­tis­ti­cal back­ground on p-value prob­lems: Against nul­l-hy­poth­e­sis sig­nif­i­cance test­ing

The basic nature of ‘sig­nif­i­cance’ being usu­ally defined as p < 0.05 means we should expect some­thing like >5% of stud­ies or exper­i­ments to be bogus (op­ti­misti­cal­ly), but that only con­sid­ers “false pos­i­tives”; reduc­ing “false neg­a­tives” requires sta­tis­ti­cal power (weak­ened by small sam­ples), and the two com­bine with the base rate of true under­ly­ing effects into a total error rate. Ioan­ni­dis 2005 points out that con­sid­er­ing the usual p val­ues, the under­pow­ered nature of many stud­ies, the rar­ity of under­ly­ing effects, and a lit­tle bias, even large ran­dom­ized tri­als may wind up with only an 85% chance of hav­ing yielded the truth. One sur­vey of reported p-val­ues in med­i­cine yield­ing a lower bound of false pos­i­tives of 17%.

Yet, there are too many pos­i­tive results¹ (psy­chi­a­try, neu­ro­bi­ol­ogy bio­med­i­cine, biol­ogy, ecol­ogy & evo­lu­tion, psy­chol­ogy 12 3 4 5, eco­nom­ics’ top jour­nals, soci­ol­ogy, gene-dis­ease cor­re­la­tions) given effect sizes (and pos­i­tive results cor­re­late with per capita pub­lish­ing rates in US states & vary by period & coun­try—ap­par­ently chance is kind to sci­en­tists who must pub­lish a lot and recent­ly!); then there come the inad­ver­tent errors which might cause retrac­tion, which is rare, but the true retrac­tion rate may be 0.1–1% (“How many sci­en­tific papers should be retract­ed?”), is increas­ing & seems to pos­i­tively cor­re­late with jour­nal pres­tige met­rics (mod­ulo the con­found­ing fac­tor that famous papers/journals get more scruti­ny), not that any­one pays any atten­tion to such things; then there are basic sta­tis­ti­cal errors in >11% of papers (based on the high­-qual­ity papers in Nature and the British Med­ical Jour­nal; “Incon­gru­ence between test sta­tis­tics and P val­ues in med­ical papers”, Gar­cía-Ber­thou 2004) or 50% in neu­ro­science.

And only then can we get into repli­cat­ing at all. See for exam­ple The Atlantic arti­cle “Lies, Damned Lies, and Med­ical Sci­ence” on John P. A. Ioan­ni­dis’s research show­ing 41% of the most cited med­ical research failed to be repli­cated—were wrong. For details, you can see Ioan­ni­dis’s “Why Most Pub­lished Research Find­ings Are False”², or Beg­ley’s failed attempts to repli­cate 47 of 53 arti­cles on top can­cer jour­nals (lead­ing to Booth’s “Beg­ley’s Six Rules”; see also the Nature Biotech­nol­ogy edi­to­r­ial & note that full details have not been pub­lished because the researchers of the orig­i­nal stud­ies demanded secrecy from Beg­ley’s team), or Kumar & Nash 2011’s “Health Care Myth Busters: Is There a High Degree of Sci­en­tific Cer­tainty in Mod­ern Med­i­cine?” who write ‘We could accu­rately say, “Half of what physi­cians do is wrong,” or “Less than 20% of what physi­cians do has solid research to sup­port it.”’ Nutri­tional epi­demi­ol­ogy is some­thing of a fish in a bar­rel; after Ioan­ni­dis, is any­one sur­prised that when Young & Karr 2011 fol­lowed up on 52 cor­re­la­tions tested in 12 RCTs, 0⁄52 repli­cated and the RCTs found the oppo­site of 5?

Attempts to use ani­mal mod­els to infer any­thing about humans suf­fer from all the method­olog­i­cal prob­lems pre­vi­ously men­tioned, and add in inter­est­ing new forms of error such as mice sim­ply being irrel­e­vant to humans, lead­ing to cases like <150 sep­sis clin­i­cal tri­als all fail­ing—be­cause the drugs worked in mice but humans have a com­pletely dif­fer­ent set of genetic reac­tions to inflam­ma­tion.

‘Hot’ fields tend to be new fields, which brings prob­lems of its own, see “Large-S­cale Assess­ment of the Effect of Pop­u­lar­ity on the Reli­a­bil­ity of Research” & dis­cus­sion. (Fail­ure to repli­cate in larger stud­ies seems to be a hall­mark of biological/medical research. Ioan­ni­dis per­forms the same trick with bio­mark­ers, find­ing less than half of the most-cited bio­mark­ers were even sta­tis­ti­cal­ly-sig­nif­i­cant in the larger stud­ies. 12 of the more promi­nent SNP-IQ cor­re­la­tions failed to repli­cate on a larger data.) As we know now, almost the entire can­di­date-­gene lit­er­a­ture, most things reported from 2000–2010 before large-s­cale GWASes started to be done (and com­pletely fail­ing to find the can­di­date-­ge­nes), is noth­ing but false pos­i­tives! The repli­ca­tion rates of can­di­date-­genes for things like intel­li­gence, per­son­al­i­ty, gene-en­vi­ron­ment inter­ac­tions, psy­chi­atric dis­or­der­s–the whole schmeer—are lit­er­ally ~0%. On the plus side, the par­lous state of affairs means that there are some cheap heuris­tics for detect­ing unre­li­able papers—sim­ply ask­ing for data & being refused/ignored cor­re­lates strongly with the orig­i­nal paper hav­ing errors in its sta­tis­tics.

This epi­demic of false pos­i­tives is appar­ently delib­er­ately and know­ing accepted by epi­demi­ol­ogy; Young’s 2008 “Every­thing is Dan­ger­ous” remarks that 80–90% of epi­demi­ol­o­gy’s claims do not repli­cate (eg. the NIH ran 20 ran­dom­ized-­con­trolled-­tri­als of claims, and only 1 repli­cat­ed) and that lack of ‘mul­ti­ple com­par­isons’ (ei­ther Bon­fer­roni or Ben­jam­in-Hochberg) is taught: “Roth­man (1990) says no cor­rec­tion for mul­ti­ple test­ing is nec­es­sary and Van­den­broucke, PLoS Med (2008) agrees” (see also Per­neger 1998 who also explic­itly under­stands that no cor­rec­tion increases type 2 errors and reduces type 1 errors). Mul­ti­ple cor­rec­tion is nec­es­sary because its absence does, in fact, result in the over­state­ment of med­ical ben­e­fit (God­frey 1985, Pocock et al 1987, Smith 1987). The aver­age effect size for find­ings con­firmed meta-­an­a­lyt­i­cally in psychology/education is d = 0.5³ (well below sev­eral effect sizes from n-back/IQ stud­ies); when mov­ing from lab­o­ra­tory to non-lab­o­ra­tory set­tings, meta-­analy­ses repli­cate find­ings cor­re­late ~0.7 of the time, but for social psy­chol­ogy the repli­ca­tion cor­re­la­tion falls to ~0.5 with >14% of find­ings actu­ally turn­ing out to be the oppo­site (see Ander­son et al 1999 and Mitchell 2012; for exag­ger­a­tion due to non-blind­ing or poor ran­dom­iza­tion, Wood et al 2008). (Meta-­analy­ses also give us a start­ing point for under­stand­ing how unusual medium or large effects sizes are⁴.) Psy­chol­ogy does have many chal­lenges, but prac­ti­tion­ers also hand­i­cap them­selves; an older overview is the enter­tain­ing “What’s Wrong With Psy­chol­o­gy, Any­way?”, which men­tions the obvi­ous point that sta­tis­tics & exper­i­men­tal design are flex­i­ble enough to reach sig­nif­i­cance as desired. In an inter­est­ing exam­ple of how method­olog­i­cal reforms are no panacea in the pres­ence of con­tin­ued per­verse incen­tives, an ear­lier method­olog­i­cal improve­ment in psy­chol­ogy (re­port­ing mul­ti­ple exper­i­ments in a sin­gle pub­li­ca­tion as a check against results not being gen­er­al­iz­able) has merely demon­strated the wide­spread p-value hack­ing or manip­u­la­tion or pub­li­ca­tion bias when one notes that given the low sta­tis­ti­cal power of each exper­i­ment, even if the under­ly­ing phe­nom­ena were real it would still be wildly improb­a­ble that all n exper­i­ments in a paper would turn up sta­tis­ti­cal­ly-sig­nif­i­cant results, since power is usu­ally extremely low in exper­i­ments (eg. in neu­ro­science, “between 20–30%”). These prob­lems are per­va­sive enough that I believe they entirely explain any “decline effects”⁵.

The fail­ures to repli­cate “sta­tis­ti­cally sig­nif­i­cant” results has led one blog­ger to caus­ti­cally remark (see also “Para­psy­chol­o­gy: the con­trol group for sci­ence”, “Using degrees of free­dom to change the past for fun and profit”, “The Con­trol Group is Out Of Con­trol”):

Para­psy­chol­o­gy, the con­trol group for sci­ence, would seem to be a thriv­ing field with “sta­tis­ti­cally sig­nif­i­cant” results aplen­ty…­Para­psy­chol­o­gists are con­stantly protest­ing that they are play­ing by all the stan­dard sci­en­tific rules, and yet their results are being ignored—that they are unfairly being held to higher stan­dards than every­one else. I’m will­ing to believe that. It just means that the stan­dard sta­tis­ti­cal meth­ods of sci­ence are so weak and flawed as to per­mit a field of study to sus­tain itself in the com­plete absence of any sub­ject mat­ter. With two-thirds of med­ical stud­ies in pres­ti­gious jour­nals fail­ing to repli­cate, get­ting rid of the entire actual sub­ject mat­ter would shrink the field by only 33%.

Cosma Shal­izi:

…Let me draw the moral [about pub­li­ca­tion bias]. Even if the com­mu­nity of inquiry is both too clue­less to make any con­tact with real­ity and too hon­est to nudge bor­der­line find­ings into sig­nif­i­cance, so long as they can keep com­ing up with new phe­nom­ena to look for, the mech­a­nism of the file-­drawer prob­lem alone will guar­an­tee a steady stream of new results. There is, so far as I know, no Jour­nal of Evi­dence-Based Harus­picy filled, issue after issue, with method­olog­i­cal­ly-­fault­less papers report­ing the abil­ity of sheep’s liv­ers to pre­dict the win­ners of sumo cham­pi­onships, the out­come of speed dates, or real estate trends in selected sub­urbs of Chica­go. But the dif­fi­culty can only be that the evi­dence-based harus­pices aren’t try­ing hard enough, and some friendly rivalry with the plas­tro­mancers is called for. It’s true that none of these find­ings will last forever, but this con­stant over­turn­ing of old ideas by new dis­cov­er­ies is just part of what makes this such a dynamic time in the field of harus­picy. Many schol­ars will even tell you that their favorite part of being a harus­pex is the fre­quency with which a new sac­ri­fice over-­turns every­thing they thought they knew about read­ing the future from a sheep’s liv­er! We are very excited about the renewed inter­est on the part of pol­i­cy-­mak­ers in the rec­om­men­da­tions of the man­tic arts…

And this is when there is enough infor­ma­tion to repli­cate; open access to any data for a paper is rare (eco­nom­ics: <10%) the eco­nom­ics jour­nal Jour­nal of Mon­ey, Credit and Bank­ing, which required researchers pro­vide the data & soft­ware which could repli­cate their sta­tis­ti­cal analy­ses, dis­cov­ered that <10% of the sub­mit­ted mate­ri­als were ade­quate for repeat­ing the paper (see “Lessons from the JMCB Archive”). In one cute eco­nom­ics exam­ple, repli­ca­tion failed because the dataset had been heav­ily edited to make par­tic­i­pants look bet­ter (for more eco­nom­ic­s-spe­cific cri­tique, see Ioan­ni­dis & Doucou­lia­gos 2013). Avail­abil­ity of data, often low, decreases with time, and many stud­ies never get pub­lished regard­less of whether pub­li­ca­tion is legally man­dated.

Tran­scrip­tion errors in papers seem to be com­mon (pos­si­bly due to con­stantly chang­ing analy­ses & p-hack­ing?), and as soft­ware and large datasets becomes more inher­ent to research, the need and the prob­lem of it being pos­si­ble to repli­cate will get worse because even mature com­mer­cial soft­ware libraries can dis­agree majorly on their com­puted results to the same math­e­mat­i­cal spec­i­fi­ca­tion (see also Anda et al 2009). And spread­sheets are espe­cially bad, with error rates in the 88% range (“What we know about spread­sheet errors”, Panko 1998); spread­sheets are used in all areas of sci­ence, includ­ing biol­ogy and med­i­cine (see “Error! What bio­med­ical com­put­ing can learn from its mis­takes”; famous exam­ples of cod­ing errors include Dono­hue-Le­vitt & Rein­hart-Ro­goff), not to men­tion reg­u­lar busi­ness (eg the Lon­don Whale).

Psy­chol­ogy is far from being per­fect either; look at the exam­ples in The New Yorker’s “The Truth Wears Off” arti­cle (or look at some excerpts from that arti­cle). Com­puter sci­en­tist Peter Norvig has writ­ten a must-read essay on inter­pret­ing sta­tis­tics, “Warn­ing Signs in Exper­i­men­tal Design and Inter­pre­ta­tion”; a num­ber of warn­ing signs apply to many psy­cho­log­i­cal stud­ies. There may be incen­tive prob­lems: a trans­plant researcher dis­cov­ered the only way to pub­lish in Nature his inabil­ity to repli­cate his ear­lier Nature paper was to offi­cially retract it; another inter­est­ing exam­ple is when, after Daryl Bem got a paper pub­lished in the top jour­nal JPSP demon­strat­ing pre­cog­ni­tion, the jour­nal refused to pub­lish any repli­ca­tions (failed or suc­cess­ful) because… “‘We don’t want to be the Jour­nal of Bem Repli­ca­tion’, he says, point­ing out that other high­-pro­file jour­nals have sim­i­lar poli­cies of pub­lish­ing only the best orig­i­nal research.” (Quoted in New Sci­en­tist) One does­n’t need to be a genius to under­stand why psy­chol­o­gist Andrew D. Wil­son might snark­ily remark “…think about the mes­sage JPSP is send­ing to authors. That mes­sage is ‘we will pub­lish your crazy story if it’s new, but not your sen­si­ble story if it’s merely a repli­ca­tion’.” (You get what you pay for.) In one large test of the most famous psy­chol­ogy results, 10 of 13 (77%) repli­cat­ed. The repli­ca­tion rate is under 1⁄3 in one area of psy­chol­ogy touch­ing on genet­ics. This despite the sim­ple point that repli­ca­tions reduce the risk of pub­li­ca­tion bias, and increase sta­tis­ti­cal pow­er, so that a repli­cated result is more likely to be true. And the small sam­ples of n-back stud­ies and nootropic chem­i­cals are espe­cially prob­lem­at­ic. Quot­ing from Nick Bostrom & Anders Sand­berg’s 2006 “Con­verg­ing Cog­ni­tive Enhance­ments”:

The reli­a­bil­ity of research is also an issue. Many of the cog­ni­tion-en­hanc­ing inter­ven­tions show small effect sizes, which may neces­si­tate very large epi­demi­o­log­i­cal stud­ies pos­si­bly expos­ing large groups to unfore­seen risks.

Par­tic­u­larly trou­bling is the slow­down in drug dis­cov­ery & med­ical tech­nol­ogy dur­ing the 2000s, even as genet­ics in par­tic­u­lar was expected to pro­duce earth­-shak­ing new treat­ments. One biotech ven­ture cap­i­tal­ist writes:

The com­pany spent $6${}_{\text{2011}}^{\text{5}}$M or so try­ing to val­i­date a plat­form that did­n’t exist. When they tried to directly repeat the aca­d­e­mic founder’s data, it never worked. Upon re-ex­am­i­na­tion of the lab note­books, it was clear the founder’s lab had at the very least mas­saged the data and shaped it to fit their hypoth­e­sis. Essen­tial­ly, they sys­tem­at­i­cally ignored every piece of neg­a­tive data. Sadly this “fail­ure to repeat” hap­pens more often than we’d like to believe. It has hap­pened to us at Atlas [Ven­ture] sev­eral times in the past decade…The unspo­ken rule is that at least 50% of the stud­ies pub­lished even in top tier aca­d­e­mic jour­nals—Sci­ence, Nature, Cell, PNAS, etc…—­can’t be repeated with the same con­clu­sions by an indus­trial lab. In par­tic­u­lar, key ani­mal mod­els often don’t repro­duce. This 50% fail­ure rate isn’t a data free asser­tion: it’s backed up by dozens of expe­ri­enced R&D pro­fes­sion­als who’ve par­tic­i­pated in the (re)test­ing of aca­d­e­mic find­ings. This is a huge prob­lem for trans­la­tional research and one that won’t go away until we address it head on.

Half the respon­dents to a 2012 sur­vey at one can­cer research cen­ter reported 1 or more inci­dents where they could not repro­duce pub­lished research; two-thirds of those were unable to “ever able to explain or resolve their dis­crepant find­ings”, half had trou­ble pub­lish­ing results con­tra­dict­ing pre­vi­ous pub­li­ca­tions, and two-thirds failed to pub­lish con­tra­dic­tory results. An inter­nal Bayer sur­vey of 67 projects (com­men­tary) found that “only in ~20–25% of the projects were the rel­e­vant pub­lished data com­pletely in line with our in-­house find­ings”, and as far as assess­ing the projects went:

…de­spite the low num­bers, there was no appar­ent dif­fer­ence between the dif­fer­ent research fields. Sur­pris­ing­ly, even pub­li­ca­tions in pres­ti­gious jour­nals or from sev­eral inde­pen­dent groups did not ensure repro­ducibil­i­ty. Indeed, our analy­sis revealed that the repro­ducibil­ity of pub­lished data did not sig­nif­i­cantly cor­re­late with jour­nal impact fac­tors, the num­ber of pub­li­ca­tions on the respec­tive tar­get or the num­ber of inde­pen­dent groups that authored the pub­li­ca­tions. Our find­ings are mir­rored by ‘gut feel­ings’ expressed in per­sonal com­mu­ni­ca­tions with sci­en­tists from acad­e­mia or other com­pa­nies, as well as pub­lished obser­va­tions. [apro­pos of above] An unspo­ken rule among ear­ly-stage ven­ture cap­i­tal firms that “at least 50% of pub­lished stud­ies, even those in top-tier aca­d­e­mic jour­nals, can’t be repeated with the same con­clu­sions by an indus­trial lab” has been recently reported (see Fur­ther infor­ma­tion) and dis­cussed 4.

Physics has rel­a­tively small sins; “Assess­ing uncer­tainty in phys­i­cal con­stants” (Hen­rion & Fischoff 1985); Han­son’s sum­ma­ry:

Look­ing at 306 esti­mates for par­ti­cle prop­er­ties, 7% were out­side of a 98% con­fi­dence inter­val (where only 2% should be). In seven other cas­es, each with 14 to 40 esti­mates, the frac­tion out­side the 98% con­fi­dence inter­val ranged from 7% to 57%, with a median of 14%.

Nor is peer review reli­able or robust against even low lev­els of col­lu­sion. Sci­en­tists who win the Nobel Prize find their other work sud­denly being heav­ily cited, sug­gest­ing either that the com­mu­nity either badly failed in rec­og­niz­ing the work’s true value or that they are now suck­ing up & attempt­ing to look bet­ter by asso­ci­a­tion. (A math­e­mati­cian once told me that often, to boost a paper’s accep­tance chance, they would add cita­tions to papers by the jour­nal’s edi­tors—a prac­tice that will sur­prise none famil­iar with Good­hart’s law and the use of cita­tions in tenure & grants.)

The for­mer edi­tor Richard Smith amus­ingly recounts his doubts about the mer­its of peer review as prac­ticed, and physi­cist Michael Nielsen points out that peer review is his­tor­i­cally rare (just one of Ein­stein’s 300 papers was peer reviewed; the famous Nature did not insti­tute peer review until 1967), has been poorly stud­ied & not shown to be effec­tive, is nation­ally biased, erro­neously rejects many his­toric dis­cov­er­ies (one study lists “34 Nobel Lau­re­ates whose awarded work was rejected by peer review”; Hor­robin 1990 lists oth­er), and catches only a small frac­tion of errors. And ques­tion­able choices or fraud? For­get about it:

A pooled weighted aver­age of 1.97% (N = 7, 95%­CI: 0.86–4.45) of sci­en­tists admit­ted to have fab­ri­cat­ed, fal­si­fied or mod­i­fied data or results at least once—a seri­ous form of mis­con­duct by any stan­dard­—and up to 33.7% admit­ted other ques­tion­able research prac­tices. In sur­veys ask­ing about the behav­iour of col­leagues, admis­sion rates were 14.12% (N = 12, 95% CI: 9.91–19.72) for fal­si­fi­ca­tion, and up to 72% for other ques­tion­able research prac­tices…When these fac­tors were con­trolled for, mis­con­duct was reported more fre­quently by medical/pharmacological researchers than oth­ers.

And psy­chol­o­gists:

We sur­veyed over 2,000 psy­chol­o­gists about their involve­ment in ques­tion­able research prac­tices, using an anony­mous elic­i­ta­tion for­mat sup­ple­mented by incen­tives for hon­est report­ing. The impact of incen­tives on admis­sion rates was pos­i­tive, and greater for prac­tices that respon­dents judge to be less defen­si­ble. Using three dif­fer­ent esti­ma­tion meth­ods, we find that the pro­por­tion of respon­dents that have engaged in these prac­tices is sur­pris­ingly high rel­a­tive to respon­dents’ own esti­mates of these pro­por­tions. Some ques­tion­able prac­tices may con­sti­tute the pre­vail­ing research norm.

In short, the secret sauce of sci­ence is not ‘peer review’. It is repli­ca­tion!

Systemic Error Doesn’t Go Away

Why isn’t the solu­tion as sim­ple as elim­i­nat­ing dat­a­min­ing by meth­ods like larger n or pre-reg­is­tered analy­ses? Because once we have elim­i­nated the ran­dom error in our analy­sis, we are still left with a (po­ten­tially arbi­trar­ily large) sys­tem­atic error, leav­ing us with a large total error.

None of these sys­tem­atic prob­lems should be con­sid­ered minor or method­olog­i­cal quib­bling or fool­ish ide­al­ism: they are sys­tem­atic biases and as such, they force an upper bound on how accu­rate a cor­pus of stud­ies can be even if there were thou­sands upon thou­sands of stud­ies, because the total error in the results is made up of ran­dom error and sys­tem­atic error, but while ran­dom error shrinks as more stud­ies are done, sys­tem­atic error remains the same.

A thou­sand biased stud­ies merely result in an extremely pre­cise esti­mate of the wrong num­ber.

This is a point appre­ci­ated by sta­tis­ti­cians and exper­i­men­tal physi­cists, but it does­n’t seem to be fre­quently dis­cussed. Andrew Gel­man has a fun demon­stra­tion of selec­tion bias involv­ing candy, or from pg812–1020 of Chap­ter 8 “Suf­fi­cien­cy, Ancil­lar­i­ty, And All That” of Prob­a­bil­ity The­o­ry: The Logic of Sci­ence by E.T. Jaynes:

The clas­si­cal exam­ple show­ing the error of this kind of rea­son­ing is the fable about the height of the Emperor of Chi­na. Sup­pos­ing that each per­son in China surely knows the height of the Emperor to an accu­racy of at least ±1 meter, if there are N = 1,000,000,000 inhab­i­tants, then it seems that we could deter­mine his height to an accu­racy at least as good as

$\frac{1}{\sqrt{1,000,000,000}}m=0.003cm$ (8-49)

merely by ask­ing each per­son’s opin­ion and aver­ag­ing the results.

The absur­dity of the con­clu­sion tells us rather force­fully that the $\sqrt{N}$ rule is not always valid, even when the sep­a­rate data val­ues are causally inde­pen­dent; it requires them to be log­i­cally inde­pen­dent. In this case, we know that the vast major­ity of the inhab­i­tants of China have never seen the Emper­or; yet they have been dis­cussing the Emperor among them­selves and some kind of men­tal image of him has evolved as folk­lore. Then knowl­edge of the answer given by one does tell us some­thing about the answer likely to be given by anoth­er, so they are not log­i­cally inde­pen­dent. Indeed, folk­lore has almost surely gen­er­ated a sys­tem­atic error, which sur­vives the aver­ag­ing; thus the above esti­mate would tell us some­thing about the folk­lore, but almost noth­ing about the Emper­or.

We could put it roughly as fol­lows:

error in esti­mate = $S\pm \frac{R}{\sqrt{N}}$ (8-50)

where S is the com­mon sys­tem­atic error in each datum, R is the RMS ‘ran­dom’ error in the indi­vid­ual data val­ues. Unin­formed opin­ions, even though they may agree well among them­selves, are nearly worth­less as evi­dence. There­fore sound sci­en­tific infer­ence demands that, when this is a pos­si­bil­i­ty, we use a form of prob­a­bil­ity the­ory (i.e. a prob­a­bilis­tic mod­el) which is sophis­ti­cated enough to detect this sit­u­a­tion and make allowances for it.

As a start on this, equa­tion (8-50) gives us a crude but use­ful rule of thumb; it shows that, unless we know that the sys­tem­atic error is less than about 1⁄3 of the ran­dom error, we can­not be sure that the aver­age of a mil­lion data val­ues is any more accu­rate or reli­able than the aver­age of ten⁶. As Henri Poin­care put it: “The physi­cist is per­suaded that one good mea­sure­ment is worth many bad ones.” This has been well rec­og­nized by exper­i­men­tal physi­cists for gen­er­a­tions; but warn­ings about it are con­spic­u­ously miss­ing in the “soft” sci­ences whose prac­ti­tion­ers are edu­cated from those text­books.

Or pg1019–1020 Chap­ter 10 “Physics of ‘Ran­dom Exper­i­ments’”:

…Nev­er­the­less, the exis­tence of such a strong con­nec­tion is clearly only an ideal lim­it­ing case unlikely to be real­ized in any real appli­ca­tion. For this rea­son, the law of large num­bers and limit the­o­rems of prob­a­bil­ity the­ory can be grossly mis­lead­ing to a sci­en­tist or engi­neer who naively sup­poses them to be exper­i­men­tal facts, and tries to inter­pret them lit­er­ally in his prob­lems. Here are two sim­ple exam­ples:

1. Sup­pose there is some ran­dom exper­i­ment in which you assign a prob­a­bil­ity p for some par­tic­u­lar out­come A. It is impor­tant to esti­mate accu­rately the frac­tion f of times A will be true in the next mil­lion tri­als. If you try to use the laws of large num­bers, it will tell you var­i­ous things about f; for exam­ple, that it is quite likely to dif­fer from p by less than a tenth of one per­cent, and enor­mously unlikely to dif­fer from p by more than one per­cent. But now, imag­ine that in the first hun­dred tri­als, the observed fre­quency of A turned out to be entirely dif­fer­ent from p. Would this lead you to sus­pect that some­thing was wrong, and revise your prob­a­bil­ity assign­ment for the 101’st tri­al? If it would, then your state of knowl­edge is dif­fer­ent from that required for the valid­ity of the law of large num­bers. You are not sure of the inde­pen­dence of dif­fer­ent tri­als, and/or you are not sure of the cor­rect­ness of the numer­i­cal value of p. Your pre­dic­tion of f for a mil­lion tri­als is prob­a­bly no more reli­able than for a hun­dred.
2. The com­mon sense of a good exper­i­men­tal sci­en­tist tells him the same thing with­out any prob­a­bil­ity the­o­ry. Sup­pose some­one is mea­sur­ing the veloc­ity of light. After mak­ing allowances for the known sys­tem­atic errors, he could cal­cu­late a prob­a­bil­ity dis­tri­b­u­tion for the var­i­ous other errors, based on the noise level in his elec­tron­ics, vibra­tion ampli­tudes, etc. At this point, a naive appli­ca­tion of the law of large num­bers might lead him to think that he can add three sig­nif­i­cant fig­ures to his mea­sure­ment merely by repeat­ing it a mil­lion times and aver­ag­ing the results. But, of course, what he would actu­ally do is to repeat some unknown sys­tem­atic error a mil­lion times. It is idle to repeat a phys­i­cal mea­sure­ment an enor­mous num­ber of times in the hope that “good sta­tis­tics” will aver­age out your errors, because we can­not know the full sys­tem­atic error. This is the old “Emperor of China” fal­la­cy…

Indeed, unless we know that all sources of sys­tem­atic error—rec­og­nized or unrec­og­nized—­con­tribute less than about one-third the total error, we can­not be sure that the aver­age of a mil­lion mea­sure­ments is any more reli­able than the aver­age of ten. Our time is much bet­ter spent in design­ing a new exper­i­ment which will give a lower prob­a­ble error per tri­al. As Poin­care put it, “The physi­cist is per­suaded that one good mea­sure­ment is worth many bad ones.”⁷ In other words, the com­mon sense of a sci­en­tist tells him that the prob­a­bil­i­ties he assigns to var­i­ous errors do not have a strong con­nec­tion with fre­quen­cies, and that meth­ods of infer­ence which pre­sup­pose such a con­nec­tion could be dis­as­trously mis­lead­ing in his prob­lems.

Schlaifer much ear­lier made the same point in Prob­a­bil­ity and Sta­tis­tics for Busi­ness Deci­sions: an Intro­duc­tion to Man­age­r­ial Eco­nom­ics Under Uncer­tainty, Schlaifer 1959, pg488–489 (see also Meng 2018/Shi­rani-Mehr et al 2018):

31.4.3 Bias and Sam­ple Size

In Sec­tion 31.2.6 we used a hypo­thet­i­cal exam­ple to illus­trate the impli­ca­tions of the fact that the vari­ance of the mean of a sam­ple in which bias is sus­pected is

$${\sigma }^2\left(x\right)={\sigma }^2\left(\tilde{\beta }\right)+\frac{1}{n}{\sigma }^2\left(\widetilde{ϵ}\right)\frac{N-n}{N-1}$$

so that only the sec­ond term decreases as the sam­ple size increases and the total can never be less than the fixed value of the first term. To empha­size the impor­tance of this point by a real exam­ple we recall the most famous sam­pling fiasco in his­to­ry, the pres­i­den­tial poll con­ducted by the Lit­er­ary Digest in 1936. Over 2 mil­lion reg­is­tered vot­ers filled in and returned the straw bal­lots sent out by the Digest, so that there was less than one chance in 1 bil­lion of a sam­pling error as large as 2⁄10 of one per­cent­age point⁸, and yet the poll was actu­ally off by nearly 18 per­cent­age points: it pre­dicted that 54.5 per cent of the pop­u­lar vote would go to Lan­don, who in fact received only 36.7 per cent.^{9 10}

Since sam­pling error can­not account for any appre­cia­ble part of the 18-­point dis­crep­an­cy, it is vir­tu­ally all actual bias. A part of this total bias may be mea­sure­ment bias due to the fact that not all peo­ple voted as they said they would vote; the impli­ca­tions of this pos­si­bil­ity were dis­cussed in Sec­tion 31.3. The larger part of the total bias, how­ev­er, was almost cer­tainly selec­tion bias. The straw bal­lots were mailed to peo­ple whose names were selected from lists of own­ers of tele­phones and auto­mo­biles and the sub­pop­u­la­tion which was effec­tively sam­pled was even more restricted than this: it con­sisted only of those own­ers of tele­phones and auto­mo­biles who were will­ing to fill out and return a straw bal­lot. The true mean of this sub­pop­u­la­tion proved to be entirely dif­fer­ent from the true mean of the pop­u­la­tion of all United States cit­i­zens who voted in 1936.

It is true that there was no evi­dence at the time this poll was planned which would have sug­gested that the bias would be as great as the 18 per­cent­age points actu­ally real­ized, but expe­ri­ence with pre­vi­ous polls had shown biases which would have led any sen­si­ble per­son to assign to $\tilde{\beta }$ a dis­tri­b­u­tion with $\sigma \left(\tilde{\beta }\right)$ equal to at least 1 per­cent­age point. A sam­ple of only 23,760 returned bal­lots, one 1⁄100th the size actu­ally used, would have given $\sigma \left(ϵ\right)$ a value of only 1⁄3 per­cent­age point, so that the stan­dard devi­a­tion of x would have been

$$\sigma \left(x\right)=\sqrt{{\sigma }^2\left(\tilde{\beta }\right)+{\sigma }^2\left(ϵ\right)}=\sqrt{1+.11}=1.05$$

per­cent­age points. Using a sam­ple 100 times this large reduced $\sigma \left(ϵ\right)$ from 1⁄3 point to vir­tu­ally zero, but it could not affect $\sigma \left(\tilde{\beta }\right)$ and thus on the most favor­able assump­tion could reduce $\sigma \left(x\right)$ only from 1.05 points to 1 point. To col­lect and tab­u­late over 2 mil­lion addi­tional bal­lots when this was the great­est gain that could be hoped for was obvi­ously ridicu­lous before the fact and not just in the light of hind­sight.

What’s par­tic­u­larly sad is when peo­ple read some­thing like this and decide to rely on anec­dotes, per­sonal exper­i­ments, and alter­na­tive med­i­cine where there are even more sys­tem­atic errors and no way of reduc­ing ran­dom error at all! Sci­ence may be the lens that sees its own flaws, but if other epis­te­molo­gies do not boast such long detailed self­-­cri­tiques, it’s not because they are flaw­less… It’s like that old Jamie Zaw­in­ski quote: Some peo­ple, when faced with the prob­lem of main­stream med­i­cine & epi­demi­ol­ogy hav­ing seri­ous method­olog­i­cal weak­ness­es, say “I know, I’ll turn to non-­main­stream med­i­cine & epi­demi­ol­o­gy. After all, if only some med­i­cine is based on real sci­en­tific method and out­per­forms place­bos, why both­er?” (Now they have two prob­lem­s.) Or per­haps Isaac Asi­mov: “John, when peo­ple thought the earth was flat, they were wrong. When peo­ple thought the earth was spher­i­cal, they were wrong. But if you think that think­ing the earth is spher­i­cal is just as wrong as think­ing the earth is flat, then your view is wronger than both of them put togeth­er.”

See Also

• “The Iron Laws of Eval­u­a­tion”

• How Often Does Cor­re­la­tion=­Causal­i­ty?

• Why Cor­re­la­tion Usu­ally ≠ Cau­sa­tion: Causal Nets Cause Com­mon Con­found­ing

• Lit­tle­wood’s Law and the Global Media

• The Exis­ten­tial Risk of Math Error

• Ques­tion­able research find­ings:

  • Lep­rechaun hunt­ing
  • “Hydro­cephalus and Intel­li­gence: The Hol­low Men”
  • On the ‘Mouse Utopia’ exper­i­ment
  • Dual N-Back Meta-­Analy­sis
  • Lunar cir­ca­dian rhythms
  • On the Jeanne Cal­ment longevity anom­aly
  • Munks­gaard et al 2016, “A repli­ca­tion and method­olog­i­cal cri­tique of the study ‘Eval­u­at­ing drug traf­fick­ing on the Tor Net­work’”

Appendix