“One does not learn com­put­ing by using a hand cal­cu­la­tor, but one can for­get arith­metic.” Perlis 1982↩︎

List­ing other neu­ro­pros­thet­ics is hard. It’s an inter­est­ing idea, but as pro­po­nents of exter­nal­ism like Andy Clark have found, it’s eas­ier to feel that exter­nal­ism is mean­ing­ful than to nail down a clear def­i­n­i­tion which sep­a­rates a neu­ro­pros­thetic or part of one’s mind from a ran­dom tool you like or find use­ful. Con­sider whether a pen­cil and paper a neu­ro­pros­thet­ic: clearly it is not for a child learn­ing to write, who must care­fully com­pose the words in his mind and put them down one after anoth­er, but it is not so clear for an adult who has been writ­ing all his life and can doo­dle or write down thoughts with­out think­ing about them and may even be sur­prised at what they hap­pened to write.

I like this def­i­n­i­tion: “a neu­ro­pros­thetic is any­thing whose results you use with­out fur­ther thought”. So in the clas­sic exam­ple, when Otto needs to go some­where, he never thinks “I am an amne­siac who stores loca­tions in my notepad, and I must look up the loca­tion” - he just looks up the loca­tion. A good heuris­tic would be any­thing whose destruc­tion leaves one feel­ing lost, slow, stu­pid, or igno­rant.

By this stan­dard, I can think of only a few tools I use with­out notice­able thought:

• key­bind­ings such as win­dow man­ager short­cuts, in par­tic­u­lar short­cuts for Google search­es; on occa­sion, XMon­ad’s Prompt gets inscrutably wedged, lock­ing it. When this hap­pens, I have to restart X because I Google every­thing and the key­bind­ing is so engrained that not using it is unbear­able. It would be like try­ing to write with your weak hand.
• Google Cal­en­dar and Pre­dic­tion­Book: it is incred­i­ble how many fol­lowups or reminders or reg­u­larly hap­pen­ing tasks I can put into Google Cal­en­dar or PB. I have out­sourced many habits or thoughts to them, and I no longer think of it as any­thing spe­cial. If either were gone, I would feel fright­ened - what events were pass­ing, what beliefs fal­si­fied, what oppor­tu­ni­ties open­ing up (or clos­ing!) that I had sud­denly become igno­rant of?
• Ever­note, for a sim­i­lar rea­son; many of my mem­o­ries have ceased to be things like “octo­puses see too fast to watch TV and so only HDTV or UHDTV works for them; I read this in Orion Mag­a­zine” and become things like “octo­pus TV Ever­note”, and if I want to know what it was about octo­puses & TV, well, I’ll have to look it up in Ever­note. Mnemosyne plays a sim­i­lar role for me, but there the mem­o­ries are much clearer on their own because of the spaced rep­e­ti­tion.
• my web­site gwern.net; I’ve had to say many times that I don’t know what I think about some­thing, but what­ever that is, it’s on my web­site. (A more extreme form of the Evernote/Mnemosyne neu­ro­pros­thet­ic.) A com­menter once wrote that read­ing gwern.net felt like he was crawl­ing around in my head. He was more right than he real­ized.

as quoted in “Retrieval prac­tice and the main­te­nance of knowl­edge”, Bjork 1988↩︎

From “Close the Book. Recall. Write It Down: That old study method still works, researchers say. So why don’t pro­fes­sors preach it?”; The Chron­i­cle of Higher Edu­ca­tion

Two psy­chol­ogy jour­nals have recently pub­lished papers show­ing that this strat­egy works, the lat­est find­ings from a decades-old body of research. When stu­dents study on their own, “active recall” - recita­tion, for instance, or flash­cards and other self­-quizzing - is the most effec­tive way to inscribe some­thing in long-term mem­o­ry. Yet many col­lege instruc­tors are only dimly famil­iar with that research…

From “The Spac­ing Effect: A Case Study in the Fail­ure to Apply the Results of Psy­cho­log­i­cal Research” (Demp­ster 1988), whose title alone sum­ma­rizes the sit­u­a­tion (see also Kel­ley 2007, Mak­ing Minds: What’s Wrong with Edu­ca­tion - and What Should We Do About It?):

Sec­ond, it [the spac­ing effect] is remark­ably robust. In many cas­es, two spaced pre­sen­ta­tions are about twice as effec­tive as two massed pre­sen­ta­tions (e.g., Hintz­man, 1974; Melton, 1970), and the dif­fer­ence between them increases as the fre­quency of rep­e­ti­tion increases (Un­der­wood, 1970)…

The spac­ing effect was known as early as 1885 when Ebbing­haus pub­lished the results of his sem­i­nal work on mem­o­ry. With him­self as the sub­ject, Ebbing­haus found that for a sin­gle 12-­syl­la­ble series, 68 imme­di­ately suc­ces­sive rep­e­ti­tions had the effect of mak­ing pos­si­ble an error­less recital after seven addi­tional rep­e­ti­tions on the fol­low­ing day. How­ev­er, the same effect was achieved by only 38 dis­trib­uted rep­e­ti­tions spread over 3 days. On the basis of this and other related find­ings, Ebbing­haus con­cluded that ‘with any con­sid­er­able num­ber of rep­e­ti­tions a suit­able dis­tri­b­u­tion of them over a space of time is decid­edly more advan­ta­geous than the mass­ing of them at a sin­gle time’ (Eb­bing­haus, 1885/1913. p. 89)

Son & Simon 2012:

Fur­ther­more, even after acknowl­edg­ing the ben­e­fits of spac­ing, chang­ing teach­ing prac­tices proved to be enor­mously dif­fi­cult. Delaney et al (2010) wrote: “Anec­do­tal­ly, high school teach­ers and col­lege pro­fes­sors seem to teach in a lin­ear fash­ion with­out rep­e­ti­tion and give three or four non­cu­mu­la­tive exams.” (p. 130). Focus­ing on the math domain, where one might expect a very easy-­to-re­view-and-­to-­space strat­e­gy, Rohrer (2009) points out that math­e­mat­ics text­books usu­ally present top­ics in a non-­spaced, non-mixed fash­ion. Even much ear­lier, Vash (1989) had writ­ten: “Edu­ca­tion pol­icy set­ters know per­fectly well that [spaced prac­tice] works bet­ter [than massed prac­tice]. They don’t care. It isn’t tidy. It does­n’t let teach­ers teach a unit and dust off their hands quickly with a nice sense of ‘Well, that’s done.’” (p. 1547).

• Rohrer, D. (2009). “The effects of spac­ing and mix­ing prac­tice prob­lems”. Jour­nal for Research in Math­e­mat­ics Edu­ca­tion, 40, 4-17
• Vash, C. L. (1989). “The spac­ing effect: A case study in the fail­ure to apply the results of psy­cho­log­i­cal research”. Amer­i­can Psy­chol­o­gist, 44, 1547 (a com­ment on Demp­ster’s arti­cle?)

From Psy­chol­o­gy: An Intro­duc­tion:

In one prac­ti­cal demon­stra­tion of the spac­ing effect, Bahrick, Bahrick, Bahrick, & Bahrick (1993) showed that reten­tion of for­eign lan­guage vocab­u­lary was greatly enhanced if prac­tice ses­sions were spaced far apart. For exam­ple, “Thir­teen retrain­ing ses­sions spaced at 56 days yielded reten­tion com­pa­ra­ble to 26 ses­sions spaced at 14 days.” In other words, sub­jects could use half as many study ses­sions, if the study ses­sions were spread over a time period four times as long.

“Synap­tic evi­dence for the effi­cacy of spaced learn­ing”, Kra­mar et al 2012 (“Take your time: Neu­ro­bi­ol­ogy sheds light on the supe­ri­or­ity of spaced vs. massed learn­ing”):

The supe­ri­or­ity of spaced vs. massed train­ing is a fun­da­men­tal fea­ture of learn­ing. Here, we describe unan­tic­i­pated tim­ing rules for the pro­duc­tion of long-term poten­ti­a­tion (LTP) in adult rat hip­pocam­pal slices that can account for one tem­po­ral seg­ment of the spaced tri­als phe­nom­e­non. Suc­ces­sive bouts of nat­u­ral­is­tic theta burst stim­u­la­tion of field CA1 affer­ents markedly enhanced pre­vi­ously sat­u­rated LTP if spaced apart by 1 h or longer, but were with­out effect when shorter inter­vals were used. Analy­ses of F-act­in-en­riched spines to iden­tify poten­ti­ated synapses indi­cated that the added LTP obtained with delayed theta trains involved recruit­ment of synapses that were “missed” by the first stim­u­la­tion bout. Sin­gle spine glu­ta­mate-uncaging exper­i­ments con­firmed that less than half of the spines in adult hip­pocam­pus are primed to undergo plas­tic­ity under base­line con­di­tions, sug­gest­ing that intrin­sic vari­abil­ity among indi­vid­ual synapses imposes a repet­i­tive pre­sen­ta­tion require­ment for max­i­miz­ing the per­cent­age of poten­ti­ated con­nec­tions. We pro­pose that a com­bi­na­tion of local dif­fu­sion from ini­tially mod­i­fied spines cou­pled with much later mem­brane inser­tion events dic­tate that the rep­e­ti­tions be widely spaced. Thus, the synap­tic mech­a­nisms described here pro­vide a neu­ro­bi­o­log­i­cal expla­na­tion for one com­po­nent of a poorly under­stood, ubiq­ui­tous aspect of learn­ing.

There are many stud­ies to the effect that active recall is best. Here’s one recent study, “Retrieval Prac­tice Pro­duces More Learn­ing than Elab­o­ra­tive Study­ing with Con­cept Map­ping”, Karpicke 2011 (cov­ered in Sci­ence Daily and the NYT):

Edu­ca­tors rely heav­ily on learn­ing activ­i­ties that encour­age elab­o­ra­tive study­ing, while activ­i­ties that require stu­dents to prac­tice retriev­ing and recon­struct­ing knowl­edge are used less fre­quent­ly. Here, we show that prac­tic­ing retrieval pro­duces greater gains in mean­ing­ful learn­ing than elab­o­ra­tive study­ing with con­cept map­ping. The advan­tage of retrieval prac­tice gen­er­al­ized across texts iden­ti­cal to those com­monly found in sci­ence edu­ca­tion. The advan­tage of retrieval prac­tice was observed with test ques­tions that assessed com­pre­hen­sion and required stu­dents to make infer­ences. The advan­tage of retrieval prac­tice occurred even when the cri­te­r­ial test involved cre­at­ing con­cept maps. Our find­ings sup­port the the­ory that retrieval prac­tice enhances learn­ing by retrieval-spe­cific mech­a­nisms rather than by elab­o­ra­tive study process­es. Retrieval prac­tice is an effec­tive tool to pro­mote con­cep­tual learn­ing about sci­ence.

From “For­get What You Know About Good Study Habits”. New York Times;

Cog­ni­tive sci­en­tists do not deny that hon­est-­to-­good­ness cram­ming can lead to a bet­ter grade on a given exam. But hur­riedly jam-­pack­ing a brain is akin to speed-­pack­ing a cheap suit­case, as most stu­dents quickly learn - it holds its new load for a while, then most every­thing falls out­….When the neural suit­case is packed care­fully and grad­u­al­ly, it holds its con­tents for far, far longer. An hour of study tonight, an hour on the week­end, another ses­sion a week from now: such so-­called spac­ing improves later recall, with­out requir­ing stu­dents to put in more over­all study effort or pay more atten­tion, dozens of stud­ies have found.

“The idea is that for­get­ting is the friend of learn­ing”, said Dr. Kor­nell. “When you for­get some­thing, it allows you to relearn, and do so effec­tive­ly, the next time you see it.”

That’s one rea­son cog­ni­tive sci­en­tists see test­ing itself - or prac­tice tests and quizzes - as a pow­er­ful tool of learn­ing, rather than merely assess­ment. The process of retriev­ing an idea is not like pulling a book from a shelf; it seems to fun­da­men­tally alter the way the infor­ma­tion is sub­se­quently stored, mak­ing it far more acces­si­ble in the future.

In one of his own exper­i­ments, Dr. Roedi­ger and Jef­frey Karpicke, who is now at Pur­due Uni­ver­si­ty, had col­lege stu­dents study sci­ence pas­sages from a read­ing com­pre­hen­sion test, in short study peri­ods. When stu­dents stud­ied the same mate­r­ial twice, in back­-­to-back ses­sions, they did very well on a test given imme­di­ately after­ward, then began to for­get the mate­r­i­al. But if they stud­ied the pas­sage just once and did a prac­tice test in the sec­ond ses­sion, they did very well on one test two days lat­er, and another given a week lat­er.

The Math­e­mat­ics of Gam­bling, Thorp 1984, “Sec­tion Two: The Wheels”, Chap­ter 4, pg43-44:

It was the spring of 1955. I was fin­ish­ing my sec­ond year of grad­u­ate physics at U.C.L.A…I changed my field of study from physics to math­e­mat­ic­s…I attended classes and stud­ied from 50 to 60 hours a week, gen­er­ally includ­ing Sat­ur­days and Sun­days. I had read about the psy­chol­ogy of learn­ing in order to be able to work longer and hard­er. I found that “spaced learn­ing” worked well: study for an hour, then take a break of at least ten min­utes (show­er, meal, tea, errands, etc.). One Sun­day after­noon about 3 p.m., I came to the co-op din­ing room for a tea break…My head was bub­bling with physics equa­tions, and sev­eral of my good friends were sit­ting around chat­ting.

From Final Jeop­ardy: Man Vs. Machine and the Quest to Know Every­thing, by Stephen Bak­er, pg 214:

The pro­gram he put together tested him on cat­e­gories, gauged his strengths (sciences, NFL foot­ball) and weak­nesses (fash­ion, Broad­way shows), and then directed him toward the prepa­ra­tion most likely to pay off in his own match. To patch these holes in his knowl­edge, Craig used a free online tool called Anki, which pro­vides elec­tronic flash cards for hun­dreds of fields of study, from Japan­ese vocab­u­lary to Euro­pean mon­archs. The pro­gram, in Craig’s words, is based on psy­cho­log­i­cal research on ‘the for­get­ting curve’. It helps peo­ple find holes in their knowl­edge and deter­mines how often they need those areas to be reviewed to keep them in mind. In going over world cap­i­tals, for exam­ple, the sys­tem learns quickly that a user like Craig knows Lon­don, Paris, and Rome, so it might spend more time rein­forc­ing the cap­i­tal of, say, Kaza­khstan. (And what would be the Kazakh cap­i­tal? ‘Astana’, Craig said in a flash. ‘It used to be Almaty, but they moved it.’)

“Our Inter­view With Jeop­ardy! Cham­pion Arthur Chu”:

[Chu:] …Jeop­ardy! is aimed at the sort of aver­age TV view­er, so they’re not going to ask things that are point­lessly obscure…So I used a pro­gram called Anki which uses a method called “spaced rep­e­ti­tion.” It keeps track of where you’re doing well or poor­ly, and pushes you to study the flash­cards you don’t know as well, until you develop an even knowl­edge base about a par­tic­u­lar sub­ject, and I just made flash­cards for those spe­cific things. I mem­o­rized all the world cap­i­tals, it was­n’t that hard once I had the flash­cards and was using them every day. I mem­o­rized the US State Nick­names (they’re on Wikipedi­a), mem­o­rized the basic impor­tant facts about the 44 US Pres­i­dents. I really focused on those. But there’s a lot more stuff to know. I went on Jeop­ardy! know­ing that there was stuff I did­n’t know. For instance, every­one laughs about sports - but I also knew that [sports clues] were the least likely to come up in Dou­ble Jeop­ardy and Final Jeop­ardy and be very impor­tant. So I decided I should­n’t sweat it too much, I should just rec­og­nize that I did­n’t know them and let that go, as long as I can get the high value clues. So that was how I pre­pared.

Alan J. Perlis, “Epi­grams in Pro­gram­ming” (1982)↩︎

Web devel­oper Per­sol writes in August 2012:

I actu­ally wrote a site that did this [spaced rep­e­ti­tion] a few months ago. I had about 4000 users who had actu­ally gone through a com­plete ses­sion…As guessed, the prob­lem is that I could­n’t get peo­ple to start form­ing it as a habit. There is no imme­di­ate pay­back. Less than 20 peo­ple out of 4000 did more than one ses­sion…Ad­di­tion­al­ly, there are at least 18 com­peti­tors. Here’s the list I made at the time. Very few seem to be suc­cess­ful. I shut the site down about a month ago. There are numer­ous free com­peti­tors which don’t have any great annoy­ances. I would­n’t sug­gest start­ing another of these sites unless you fig­ured out an effec­tive way to “gam­ify” it.

…~4000 peo­ple fin­ished a ses­sion. Many more ‘tried’ than 4000…I just could­n’t deter­mine which users were bots that reg­is­tered ran­domly vs users that did­n’t fin­ish the first ses­sion.

• Tried: lots (but unknown)
• Fin­ished 1 ses­sion: ~4000
• Fin­ished >1 ses­sion: ~20 [0.5%]

“Play it Again: The Mas­ter Psy­chophar­ma­col­ogy Pro­gram as an Exam­ple of Inter­val Learn­ing in Bite-­Sized Por­tions”, Stahl et al 2010:

Since Ebbing­haus’ time, a volu­mi­nous amount of research has con­firmed this sim­ple but impor­tant fact: the reten­tion of new infor­ma­tion degrades rapidly unless it is reviewed in some man­ner. A mod­ern exam­ple of this loss of knowl­edge with­out rep­e­ti­tion is a study of car­diopul­monary resus­ci­ta­tion (CPR) skills that demon­strated rapid decay in the year fol­low­ing train­ing. By 3 years post-­train­ing only 2.4% were able to per­form CPR suc­cess­ful­ly.⁶ Another recent study of physi­cians tak­ing a tuto­r­ial they rated as very good or excel­lent showed mean knowl­edge scores increas­ing from 50% before the tuto­r­ial to 76% imme­di­ately after­ward.⁷ How­ev­er, score gains were only half as great 3-8 days later and incred­i­bly, there was no [sta­tis­ti­cal­ly-]sig­nif­i­cant knowl­edge reten­tion mea­sur­able at all at 55 days.⁷ Sim­i­lar results have been reported by us in fol­low-up stud­ies of knowl­edge reten­tion from con­tin­u­ing med­ical edu­ca­tion pro­grams.¹ [Stahl SM, Davis RL. Best Prac­tices for Med­ical Edu­ca­tors. Carls­bad, CA: NEI Press; 2009]

…This may be due to the fact that lec­tures with assigned read­ing are the eas­i­est for teach­ers. Also, med­ical learn­ing is rarely mea­sured imme­di­ately after a lec­ture or after read­ing new mate­r­ial for the first time and then mea­sured again a few days or weeks lat­er, so that the low reten­tion rates of this approach may not be widely appre­ci­at­ed.^1,4 No won­der for­mal med­ical edu­ca­tion con­fer­ences with­out enabling or prac­tice-re­in­forc­ing strate­gies appear to have rel­a­tively lit­tle impact on prac­tice and health­care out­comes.^8,9,10

One study look­ing at cram­ming is the 1993 “Cram­ming: A bar­rier to stu­dent suc­cess, a way to beat the sys­tem or an effec­tive learn­ing strat­e­gy?”, Vacha et al 1993, abstract:

Tested the hypoth­e­sis that cram­ming is an inef­fec­tive study strat­egy by exam­in­ing the weekly study diaries of 166 under­grad­u­ates. All Ss also com­pleted an end-of-se­mes­ter ques­tion­naire mea­sur­ing study habits. Ss were clas­si­fied in the fol­low­ing study pat­terns: ide­al, con­fi­dent, zeal­ous, or cram­mer. Con­trary to the hypoth­e­sis, results sug­gest that cram­ming is an effec­tive approach, most wide­spread in courses using take-home essay exam­i­na­tions and major research papers. Cram­mers’ grades were as good as or bet­ter than those of Ss using other strate­gies; the longer Ss were in col­lege, the more likely it was that they crammed. Cram­mers stud­ied more hours than most stu­dents and were as inter­ested in their courses as other stu­dents.

Note that there is no mea­sure of long-term reten­tion, sug­gest­ing that peo­ple who only care about grades are ratio­nally choos­ing to cram.↩︎

Anki has its Cram Mode and Mnemosyne 2.0 has a cram­ming plu­g­in. When a SRS does­n’t have explicit sup­port, it’s always pos­si­ble to ‘game’ the algo­rithm by set­ting one’s scores arti­fi­cially low, so the SR algo­rithm thinks you are stu­pid and need to do a lot of rep­e­ti­tions.↩︎

“Exam­in­ing the exam­in­ers: Why are we so bad at assess­ing stu­dents?”, New­stead 2002:

Con­way, Cohen and Stan­hope (1992) looked at long term mem­ory for the infor­ma­tion pre­sented on a psy­chol­ogy course. They found that some types of infor­ma­tion, espe­cially that relat­ing to research meth­ods, were remem­bered bet­ter than oth­ers. But in a fol­low up analy­sis, they found that the type of assess­ment used had an effect on mem­o­ry. In essence, mate­r­ial assessed by con­tin­u­ous assess­ment was more likely to be remem­bered than infor­ma­tion assessed by exams.

Stahl 2010:

For exam­ple, sim­ple restudy­ing allows the learner to reex­pe­ri­ence all of the mate­r­ial but actu­ally pro­duces poor long-term reten­tion.^25,26,35 Why do stu­dents keep study­ing the orig­i­nal mate­ri­als? Cer­tainly if this is their only choice, then restudy­ing is a nec­es­sary tac­tic. Another answer may be that repeated study­ing falsely inflates stu­dents’ con­fi­dence in their abil­ity to remem­ber in the future because they sense that they under­stand it now, and they and their instruc­tors may be unaware of the many stud­ies that show poor reten­tion on delayed test­ing after this form of rep­e­ti­tion.^25,26,35

From Kor­nell et al 2010:

Con­trary to the mass­ing-aid­s-in­duc­tion hypoth­e­sis, final test per­for­mance was con­sis­tently and con­sid­er­ably supe­rior in the spaced con­di­tion. A large major­ity of par­tic­i­pants, how­ev­er, judged mass­ing to be more effec­tive than spac­ing, despite mak­ing the judg­ment after tak­ing the test.

…Metacog­ni­tive judg­ments-that is, judg­ments about one’s own mem­ory and cog­ni­tion-are often based on feel­ings of flu­en­cy(e.g., see Ben­jam­in, Bjork, & Schwartz, 1998; Rhodes & Castel, 2008). Because mass­ing nat­u­rally leads to feel­ings of flu­ency and increases short­-term task per­for­mance dur­ing learn­ing, learn­ers fre­quently rate spac­ing as less effec­tive than mass­ing, even when their per­for­mance shows the oppo­site pat­tern (Bad­de­ley & Long­man 1978; Kor­nell & Bjork, 2008; Simon & Bjork, 2001; Zech­meis­ter & Shaugh­nessy, 1980). Aver­aged across Kor­nell and Bjork’s (2008) exper­i­ments, for exam­ple, more than 80% of par­tic­i­pants rated mass­ing as equally or more effec­tive than spac­ing,whereas only 15% of par­tic­i­pants actu­ally per­formed bet­ter in the massed con­di­tion than in the spaced con­di­tion.

…Such an illu­sion was appar­ent in the induc­tion con­di­tion. Con­trary to pre­vi­ous research, how­ev­er, par­tic­i­pants gave higher rat­ings for spac­ing than mass­ing dur­ing rep­e­ti­tion learn­ing (see, e.g., Simon & Bjork, 2001; Zech­meis­ter & Shaugh­nessy, 1980). This out­come may have occurred because of a process of a habit­u­a­tion: Six pre­sen­ta­tions and a total of 30 s spent study­ing a sin­gle paint­ing may have come to seem inef­fi­cient and point­less. Thus, there appears to be a turn­ing point in metacog­ni­tive rat­ings based on flu­en­cy: As flu­ency increas­es, metacog­ni­tive rat­ings increase up to a point, but as flu­ency con­tin­ues to increase and encod­ing or retrieval becomes too easy, metacog­ni­tive rat­ings may begin to decrease.

…In advance of their research, Kor­nell and Bjork (2008) were con­vinced that such induc­tive learn­ing would ben­e­fit from mass­ing, yet their results showed the oppo­site. Undaunt­ed, we remained con­vinced that spac­ing would be more ben­e­fi­cial for rep­e­ti­tion learn­ing than for induc­tive learn­ing- espe­cially for older adults, given their over­all declines in episodic mem­o­ry. The cur­rent results dis­con­firmed our expec­ta­tions once again. If our intu­itions are erro­neous, despite our years spent prov­ing and prais­ing the spac­ing effec­t-in­clud­ing roughly 40 years’ worth con­tributed by Robert A. Bjork-those of the aver­age stu­dent are surely mis­taken as well (as the inac­cu­racy of the par­tic­i­pants’ metacog­ni­tive rat­ings sug­gest­s). We have, per­haps, fallen vic­tim to the illu­sion that mak­ing learn­ing easy makes learn­ing effec­tive, rather than rec­og­niz­ing that spac­ing is a desir­able dif­fi­culty (Bjork 1994) that enhances induc­tive learn­ing as well as rep­e­ti­tion learn­ing well into old age.

From Son & Simon 2012:

Thus, while spac­ing may boost learn­ing, it may be thought to be rel­a­tively inef­fi­cient in terms of study time. As we dis­cuss lat­er, this feel­ing of inef­fi­ciency may be one of the rea­sons that spac­ing is not the more pop­u­lar strat­e­gy. Inter­est­ing­ly, in that same study (Bad­de­ley & Long­man 1978; and see also Pirolli & Ander­son 1985 and Wood­worth & Schlos­berg 1954 [Exper­i­men­tal Psy­chol­ogy]), there was evi­dence of such a thing as labor­ing in vain. That is, exceed­ing a cer­tain num­ber of hours of prac­tice a day (more than approx­i­mately 2 h) led to no increases in learn­ing, as might be expect­ed. Related to the defi­cien­t-pro­cess­ing the­ory men­tioned above, these results are cru­cial in under­stand­ing intu­itively how the spac­ing effect works: We sim­ply get burnt out. These data are also anal­o­gous to the cog­ni­tive lit­er­a­ture on over­learn­ing, which shows that while con­tin­u­ous study over long peri­ods of time might seem ben­e­fi­cial (and even feel good) in the short­-term, the ben­e­fits dis­ap­pear soon after­wards (Rohrer et al. 2005; Rohrer and Tay­lor 2006)…In the above-de­scribed Bad­de­ley and Long­man’s (1978) study, for exam­ple, after postal work­ers prac­ticed typ­ing in either massed or spaced study ses­sions, they had to indi­cate how sat­is­fied they were with the train­ing. Results showed that while spac­ing led to the best learn­ing, it was the least liked. Sim­i­lar­ly, Simon & Bjork (2001) found that peo­ple pre­ferred the mass­ing strat­egy on a motor learn­ing task.

• Bad­de­ley, A. D., & Long­man, D. J. A. (1978). “The influ­ence of length and fre­quency of train­ing ses­sion on the rate of learn­ing to type”. Ergonom­ics, 21, 627-635
• Pirol­li, P., & Ander­son, J. R. (1985). “The role of prac­tice in fact retrieval”

“Study strate­gies of col­lege stu­dents: Are self­-test­ing and sched­ul­ing related to achieve­ment?”, Hartwig & Dun­losky 2012:

Pre­vi­ous stud­ies, such as those by Kor­nell and Bjork (Psy­cho­nomic Bul­letin & Review, 14:219-224, 2007) and Karpicke, But­ler, and Roedi­ger (Mem­ory, 17:471-479, 2009), have sur­veyed col­lege stu­dents’ use of var­i­ous study strate­gies, includ­ing self­-test­ing and reread­ing. These stud­ies have doc­u­mented that some stu­dents do use self­-test­ing (but largely for mon­i­tor­ing mem­o­ry) and reread­ing, but the researchers did not assess whether indi­vid­ual dif­fer­ences in strat­egy use were related to stu­dent achieve­ment. Thus, we sur­veyed 324 under­grad­u­ates about their study habits as well as their col­lege grade point aver­age (GPA). Impor­tant­ly, the sur­vey included ques­tions about self­-test­ing, sched­ul­ing one’s study, and a check­list of strate­gies com­monly used by stu­dents or rec­om­mended by cog­ni­tive research. Use of self­-test­ing and reread­ing were both pos­i­tively asso­ci­ated with GPA. Sched­ul­ing of study time was also an impor­tant fac­tor: Low per­form­ers were more likely to engage in late-night study­ing than were high per­form­ers; mass­ing (vs. spac­ing) of study was asso­ci­ated with the use of fewer study strate­gies over­all; and all stu­dents-but espe­cially low per­form­er­s-were dri­ven by impend­ing dead­lines. Thus, self­-test­ing, reread­ing, and sched­ul­ing of study play impor­tant roles in real-­world stu­dent achieve­ment.

(See also Dun­losky et al 2013.) Note the self­-test­ing cor­re­la­tion excludes flash­cards, a result that both the authors and me found sur­pris­ing. The sleep con­nec­tion is inter­est­ing, given the hypoth­e­sized link between stronger mem­ory for­ma­tion & study­ing before a good night’s sleep - you can hardly get a good night’s sleep if you are cram­ming late into the night (cor­re­lated with lower grades) but you can if you do so at a rea­son­able time in the evening (in time to get a solid night).

See also Susser & McCabe 2012:

Lab­o­ra­tory stud­ies have demon­strated the long-term mem­ory ben­e­fits of study­ing mate­r­ial in mul­ti­ple dis­trib­uted ses­sions as opposed to one massed ses­sion, given an iden­ti­cal amount of over­all study time (i.e., the spac­ing effect). The cur­rent study goes beyond the lab­o­ra­tory to inves­ti­gate whether under­grad­u­ates know about the advan­tage of spaced study, to what extent they use it in their own study­ing, and what fac­tors might influ­ence its uti­liza­tion. Results from a web-based sur­vey indi­cated that par­tic­i­pants (n = 285) were aware of the ben­e­fits of spaced study and would use a higher level of spac­ing under ideal com­pared to real­is­tic cir­cum­stances. How­ev­er, self­-re­ported use of spac­ing was inter­me­di­ate, sim­i­lar to mass­ing and sev­eral other study strate­gies, and ranked well below com­monly used strate­gies such as reread­ing notes. Sev­eral fac­tors were endorsed as impor­tant in the deci­sion to dis­trib­ute study time, includ­ing the per­ceived dif­fi­culty of an upcom­ing exam, the amount of mate­r­ial to learn, how heav­ily an exam is weighed in the course grade, and the value of the mate­r­i­al. Fur­ther, level of metacog­ni­tive self­-reg­u­la­tion and use of elab­o­ra­tion strate­gies were asso­ci­ated with higher rates of spaced study.

Ana­lytic Cul­ture in the US Intel­li­gence Com­mu­ni­ty: An Ethno­graphic Study, John­ston 2005, pg89:

To inves­ti­gate the inten­sity of instruc­tional inter­ac­tions, Art Graesser and Natalie Per­son 1994 com­pared ques­tion­ing and answer­ing in class­rooms with those in tuto­r­ial set­tings.⁵ They found that class­room groups of stu­dents ask about three ques­tions an hour and that any sin­gle stu­dent in a class­room asks about 0.11 ques­tions per hour. In con­trast, they found that stu­dents in indi­vid­ual tuto­r­ial ses­sions asked 20-30 ques­tions an hour and were required to answer 117-146 ques­tions per hour. Reviews of the inten­sity of inter­ac­tion that occurs in tech­nol­o­gy-based instruc­tion have found even more active stu­dent response lev­els. [J. D. Fletcher, Tech­nol­o­gy, the Colum­bus Effect, and the Third Rev­o­lu­tion in Learn­ing.]

Although Graesser & Per­son 1994 also found that sheer num­ber of ques­tions was not nec­es­sar­ily impor­tant, sug­gest­ing dimin­ish­ing mar­ginal returns or per­haps bad ques­tion ask­ing.↩︎

“Super­Memo is based on the insight that there is an ideal moment to prac­tice what you’ve learned. Prac­tice too soon and you waste your time. Prac­tice too late and you’ve for­got­ten the mate­r­ial and have to relearn it. The right time to prac­tice is just at the moment you’re about to for­get. Unfor­tu­nate­ly, this moment is dif­fer­ent for every per­son and each bit of infor­ma­tion. Imag­ine a pile of thou­sands of flash cards. Some­where in this pile are the ones you should be prac­tic­ing right now. Which are they?” Gary Wolf, “Want to Remem­ber Every­thing You’ll Ever Learn? Sur­ren­der to This Algo­rithm”, Wired Mag­a­zine↩︎

“Make no mis­take about it: Com­put­ers process num­bers - not sym­bols. We mea­sure our under­stand­ing (and con­trol) by the extent to which we can arith­me­tize an activ­i­ty.” Perlis, ibid.↩︎

this expo­nen­tial expan­sion is how a SR pro­gram can han­dle con­tin­ual input of cards: if cards were sched­uled at fixed inter­vals, like every other day, review would soon become quite impos­si­ble - I have >18000 items in Mnemosyne, but I don’t have time to review 9000 ques­tions a day!↩︎

See the 2008 meta-­analy­sis, “Learn­ing Styles: Con­cepts and Evi­dence” (APS press release); from the abstract:

…in order to demon­strate that opti­mal learn­ing requires that stu­dents receive instruc­tion tai­lored to their puta­tive learn­ing style, the exper­i­ment must reveal a spe­cific type of inter­ac­tion between learn­ing style and instruc­tional method: Stu­dents with one learn­ing style achieve the best edu­ca­tional out­come when given an instruc­tional method that dif­fers from the instruc­tional method pro­duc­ing the best out­come for stu­dents with a dif­fer­ent learn­ing style. In other words, the instruc­tional method that proves most effec­tive for stu­dents with one learn­ing style is not the most effec­tive method for stu­dents with a dif­fer­ent learn­ing style.

Our review of the lit­er­a­ture dis­closed ample evi­dence that chil­dren and adults will, if asked, express pref­er­ences about how they pre­fer infor­ma­tion to be pre­sented to them. There is also plen­ti­ful evi­dence argu­ing that peo­ple dif­fer in the degree to which they have some fairly spe­cific apti­tudes for dif­fer­ent kinds of think­ing and for pro­cess­ing dif­fer­ent types of infor­ma­tion. How­ev­er, we found vir­tu­ally no evi­dence for the inter­ac­tion pat­tern men­tioned above, which was judged to be a pre­con­di­tion for val­i­dat­ing the edu­ca­tional appli­ca­tions of learn­ing styles. Although the lit­er­a­ture on learn­ing styles is enor­mous, very few stud­ies have even used an exper­i­men­tal method­ol­ogy capa­ble of test­ing the valid­ity of learn­ing styles applied to edu­ca­tion. More­over, of those that did use an appro­pri­ate method, sev­eral found results that flatly con­tra­dict the pop­u­lar mesh­ing hypoth­e­sis.

We con­clude there­fore, that at pre­sent, there is no ade­quate evi­dence base to jus­tify incor­po­rat­ing learn­ing-styles assess­ments into gen­eral edu­ca­tional prac­tice. Thus, lim­ited edu­ca­tion resources would bet­ter be devoted to adopt­ing other edu­ca­tional prac­tices that have a strong evi­dence base, of which there are an increas­ing num­ber. How­ev­er, given the lack of method­olog­i­cally sound stud­ies of learn­ing styles, it would be an error to con­clude that all pos­si­ble ver­sions of learn­ing styles have been tested and found want­i­ng; many have sim­ply not been tested at all.

Fritz, C. O., Mor­ris, P. E., Acton, M., Etkind, R., & Voelkel, A. R (2007). “Com­par­ing and com­bin­ing expand­ing retrieval prac­tice and the key­word mnemonic for for­eign vocab­u­lary learn­ing”. Applied Cog­ni­tive Psy­chol­ogy, 21, 499-526.↩︎

From Balota et al 2006, describ­ing Spitzer 1939, “Stud­ies in reten­tion”:

Spitzer (1939) incor­po­rated a form of expanded retrieval in a study designed to assess the abil­ity of sixth graders to learn sci­ence facts. Impres­sive­ly, Spitzer tested over 3600 stu­dents in Iowa-the entire six­th-­grade pop­u­la­tion of 91 ele­men­tary schools at the time. The stu­dents read two arti­cles, one on peanuts and the other on bam­boo, and were given a 25-item mul­ti­ple choice test to assess their knowl­edge (such as ‘To which fam­ily of plants does bam­boo belong?’). Spitzer tested a total of nine groups, manip­u­lat­ing both the tim­ing of the test (ad­min­is­tered imme­di­ately or after var­i­ous delays) and the num­ber of iden­ti­cal tests stu­dents received (one to three). Spitzer did not incor­po­rate massed or equal inter­val retrieval con­di­tions, but he had at least two groups that were tested on an expand­ing sched­ule of retrieval, in which the inter­vals between tests were sep­a­rated by the pas­sage of time (in days) rather than by inter­ven­ing to-be-learned infor­ma­tion. For exam­ple, in one of the groups, the first test was given imme­di­ate­ly, the sec­ond test was given seven days after the first test, and the third test was given 63 days after the sec­ond test. Thus, in essence, this group was tested on a 0-7-63 day expand­ing retrieval sched­ule. Spitzer com­pared per­for­mance of the expanded retrieval group to a group given a sin­gle test 63 days after read­ing the orig­i­nal arti­cle. On the first (im­me­di­ate) test, the expanded retrieval group cor­rectly answered 53% of the ques­tions. After 63 days and two pre­vi­ous tests, their score was still an impres­sive 43%. The sin­gle test group cor­rectly answered only 25% of the orig­i­nal items after 63 days, giv­ing the expanded retrieval group an 18% reten­tion advan­tage. This is quite impres­sive, given that this large ben­e­fit remained after a 63-­day reten­tion inter­val. Sim­i­lar ben­e­fi­cial effects were found in a group tested on a 0-1-21 day expanded retrieval sched­ule com­pared to a group given a sin­gle test after 21 days. Of course, this study does not decou­ple the effects of test­ing from spac­ing or expan­sion, but the results do clearly indi­cate con­sid­er­able learn­ing and reten­tion using the expanded repeated test­ing pro­ce­dure. Spitzer con­cluded that ‘…ex­am­i­na­tions are learn­ing devices and should not be con­sid­ered only as tools for mea­sur­ing achieve­ment of pupils’ (p. 656, ital­ics added)

“Dis­trib­ut­ing Learn­ing Over Time: The Spac­ing Effect in Chil­dren’s Acqui­si­tion and Gen­er­al­iza­tion of Sci­ence Con­cepts”, Vlach & Sand­hofer 2012:

The spac­ing effect describes the robust find­ing that long-term learn­ing is pro­moted when learn­ing events are spaced out in time, rather than pre­sented in imme­di­ate suc­ces­sion. Stud­ies of the spac­ing effect have focused on mem­ory processes rather than for other types of learn­ing, such as the acqui­si­tion and gen­er­al­iza­tion of new con­cepts. In this study, early ele­men­tary school chil­dren (5-7 year-olds; N = 36) were pre­sented with sci­ence lessons on one of three sched­ules: massed, clumped, and spaced. The results revealed that spac­ing lessons out in time resulted in higher gen­er­al­iza­tion per­for­mance for both sim­ple and com­plex con­cepts. Spaced learn­ing sched­ules pro­mote sev­eral types of learn­ing, strength­en­ing the impli­ca­tions of the spac­ing effect for edu­ca­tional prac­tices and cur­ricu­lum.

See also Balch 2006, who com­pared spac­ing & massed in an intro­duc­tory psy­chol­ogy course as well.↩︎

Roedi­ger & Karpicke 2006b again.↩︎

Balota et al 2006 review:

No feed­back or cor­rec­tion was given to sub­jects if they made errors or omit­ted answers. Lan­dauer & Bjork 1978 found that the expand­ing-in­ter­val sched­ule pro­duced bet­ter recall than equal-in­ter­val test­ing on a final test at the end of the ses­sion, and equal-in­ter­val test­ing, in turn, pro­duced bet­ter recall than did ini­tial massed test­ing. Thus, despite the fact that massed test­ing pro­duced nearly error­less per­for­mance dur­ing the acqui­si­tion phase, the other two sched­ules pro­duced bet­ter reten­tion on the final test given at the end of the ses­sion. How­ev­er, the dif­fer­ence favor­ing the expand­ing retrieval sched­ule over the equal-in­ter­val sched­ule was fairly small at around 10%. In research fol­low­ing up Lan­dauer and Bjork’s (1978) orig­i­nal exper­i­ments, prac­ti­cally all stud­ies have found that spaced sched­ules of retrieval (whether equal-in­ter­val or expand­ing sched­ules) pro­duce bet­ter reten­tion on a final test given later than do massed retrieval tests given imme­di­ately after pre­sen­ta­tion (e.g., Cull, 2000; Cull, Shaugh­nessy, & Zech­meis­ter, 1996), although excep­tions do exist. For exam­ple, in Exper­i­ments 3 and 4 of Cull et al (1996), massed test­ing pro­duced per­for­mance as good as equal-in­ter­val test­ing on a 5-5-5 sched­ule, but most other exper­i­ments have found that any spaced sched­ule of test­ing (ei­ther equal-in­ter­val or expand­ing) is bet­ter than a massed sched­ule for per­for­mance on a delayed test. How­ev­er, whether expand­ing sched­ules are bet­ter than equal-in­ter­val sched­ules for long-term reten­tion-the other part of Lan­dauer and Bjork’s inter­est­ing find­ings-re­mains an open ques­tion. Balota, Duchek, and Logan (in press) have pro­vided a thor­ough con­sid­er­a­tion of the rel­e­vant evi­dence and have shown that it is mixed at best, and that most researchers have found no dif­fer­ence between the two sched­ules of test­ing. That is, per­for­mance on a final test at the end of a ses­sion often shows no dif­fer­ence in per­for­mance between equal-in­ter­val and expand­ing retrieval sched­ules.

Cull, for those curi­ous (Cull, W. L. (2000). “Untan­gling the ben­e­fits of mul­ti­ple study oppor­tu­ni­ties and repeated test­ing for cued recall”. Applied Cog­ni­tive Psy­chol­ogy, 14, 215-235):

Cull (2000) com­pared expanded retrieval to equal inter­val spaced retrieval in a series of four exper­i­ments designed to mimic typ­i­cal teach­ing or study strate­gies encoun­tered by stu­dents. He exam­ined the role of test­ing ver­sus sim­ply restudy­ing the mate­ri­al, feed­back, and var­i­ous reten­tion inter­vals on final test per­for­mance. Paired asso­ciates (an uncom­mon word paired with a com­mon word, such as bairn-print) were pre­sented in a man­ner sim­i­lar to the flash­card tech­niques stu­dents often use to learn vocab­u­lary words. The inter­vals between retrieval attempts of to-be-learned infor­ma­tion ranged from min­utes in some exper­i­ments to days in oth­ers. Inter­est­ing­ly, across four exper­i­ments, Cull did not find any evi­dence of an advan­tage of an expanded con­di­tion over a uni­form spaced con­di­tion (i.e., no [sub­stan­tial] expanded retrieval effec­t), although both con­di­tions con­sis­tently pro­duced large advan­tages over massed pre­sen­ta­tions. He con­cluded that dis­trib­uted test­ing of any kind, expanded or equal inter­val, can be an effec­tive learn­ing aid for teach­ers to pro­vide for their stu­dents.

The Balota et al 2006 review offers a syn­the­sis of cur­rent the­o­ries on how massed and spaced dif­fer, based on mem­ory encod­ing:

Accord­ing to encod­ing vari­abil­ity the­o­ry, per­for­mance on a mem­ory test is depen­dent upon the over­lap between the con­tex­tual infor­ma­tion avail­able at the time of test and the con­tex­tual infor­ma­tion avail­able dur­ing encod­ing. Dur­ing massed study, there is rel­a­tively lit­tle time for con­tex­tual ele­ments to fluc­tu­ate between pre­sen­ta­tions and so this con­di­tion pro­duces the high­est per­for­mance in an imme­di­ate mem­ory test, when the test con­text strongly over­laps with the same con­tex­tual infor­ma­tion encoded dur­ing both of the massed pre­sen­ta­tions. In con­trast, when there is spac­ing between the items, there is time for fluc­tu­a­tion to take place between the pre­sen­ta­tions dur­ing study, and hence there is an increased like­li­hood of hav­ing mul­ti­ple unique con­texts encod­ed. Because a delayed test will also allow fluc­tu­a­tion of con­text, it is bet­ter to have mul­ti­ple unique con­texts encod­ed, as in the spaced pre­sen­ta­tion for­mat, as opposed to a sin­gle encoded con­text, as in the massed pre­sen­ta­tion for­mat.

Storm et al 2010 did 3 exper­i­ments on read­ing com­pre­hen­sion:

On a test 1 week lat­er, recall was enhanced by the expand­ing sched­ule, but only when the task between suc­ces­sive retrievals was highly inter­fer­ing with mem­ory for the pas­sage. These results sug­gest that the extent to which learn­ers ben­e­fit from expand­ing retrieval prac­tice depends on the degree to which the to-be-learned infor­ma­tion is vul­ner­a­ble to for­get­ting.

From Mnemosyne’s Prin­ci­ples page:

The Mnemosyne algo­rithm is very sim­i­lar to SM2 used in one of the early ver­sions of Super­Me­mo. There are some mod­i­fi­ca­tions that deal with early and late rep­e­ti­tions, and also to add a small, healthy dose of ran­dom­ness to the inter­vals. Super­memo now uses SM11. How­ev­er, we are a bit skep­ti­cal that the huge com­plex­ity of the newer SM algo­rithms pro­vides for a sta­tis­ti­cally rel­e­vant ben­e­fit. But, that is one of the facts we hope to find out with our data col­lec­tion. We will only make mod­i­fi­ca­tions to our algo­rithms based on com­mon sense or if the data tells us that there is a sta­tis­ti­cally rel­e­vant rea­son to do so.

Balota et al 2006:

Car­pen­ter and DeLosh (2005, Exp. 2) have recently inves­ti­gated face-­name learn­ing under massed, expanded (1-3-5), and equal inter­val (3-3-3) con­di­tions. This study also involved study and study and test pro­ce­dures dur­ing the acqui­si­tion phase. Car­pen­ter and DeLosh found a large effect of spac­ing, but no evi­dence of a ben­e­fit of expanded over equal inter­val prac­tice. In fact, Car­pen­ter and DeLosh reported a reli­able ben­e­fit of the equal inter­val con­di­tion over the expanded retrieval con­di­tion.

Balota et al 2006 again:

Rea and Modigliani (1985) tested the effec­tive­ness of expanded retrieval in a third-­grade class­room set­ting. In sep­a­rate con­di­tions, stu­dents were given new mul­ti­pli­ca­tion prob­lems or spelling words to learn. The prob­lem or word was pre­sented audio­vi­su­ally once and then tested on either a massed retrieval sched­ule of 0-0-0-0 or an expand­ing sched­ule of 0-1-2-4, in which the inter­vals involved being tested on old items or learn­ing new items. After each test trial for a given item, the item was re-p­re­sented in its entirety so stu­dents received feed­back on what they were learn­ing. Per­for­mance dur­ing the learn­ing phase was at 100% for both spelling words and mul­ti­pli­ca­tion facts. On an imme­di­ate final reten­tion test, Rea and Modigliani found a per­for­mance advan­tage for all item­s-­math and spelling- prac­ticed on an expand­ing sched­ule com­pared to the massed retrieval sched­ule. They sug­gest­ed, as have oth­ers, that spac­ing com­bined with the high suc­cess rate inher­ent in the expanded retrieval sched­ule pro­duced bet­ter reten­tion than massed retrieval prac­tice. How­ev­er, as in Spitzer’s study, Rea and Modigliani did not test an appro­pri­ate equal inter­val spac­ing con­di­tion. Hence, their find­ing that expanded retrieval is supe­rior to massed retrieval in third graders could sim­ply reflect the supe­ri­or­ity of spaced ver­sus massed rehearsal-in other words, the spac­ing effect.

Balota et al 2006.↩︎

Balota et al 2006; >1 is rare in psy­chol­o­gy, see “One Hun­dred Years of Social Psy­chol­ogy Quan­ti­ta­tively Described”, Bond et al 2003↩︎

Rohrer & Tay­lor 2006↩︎

Balota et al 2006:

…long-term reten­tion of infor­ma­tion has been demon­strated over sev­eral days in some cases (e.g., Camp et al, 1996). For exam­ple, in the lat­ter study, Camp et al employed an expand­ing retrieval strat­egy to train 23 indi­vid­u­als with mild to mod­er­ate AD to refer to a daily cal­en­dar as a cue to remem­ber to per­form var­i­ous per­sonal activ­i­ties (e.g., take med­ica­tion). Fol­low­ing a base­line phase to deter­mine whether sub­jects would spon­ta­neously use the cal­en­dar, spaced retrieval train­ing was imple­mented by repeat­edly ask­ing the sub­ject the ques­tion, ‘How are you going to remem­ber what to do each day?’ at expand­ing time inter­vals. The results indi­cated that 20/23 sub­jects did learn the strat­egy (i.e., to look at the cal­en­dar) and retained it over a 1-week peri­od.

Rohrer & Tay­lor 2006 warns us, though, about many of the other math stud­ies:

In one meta-­analy­sis by Dono­van and Rado­se­vich (1999), for instance, the size of the spac­ing effect declined sharply as con­cep­tual dif­fi­culty of the task increased from low (e.g. rotary pur­suit) to aver­age (e.g. word list recall) to high (e.g. puz­zle). By this find­ing, the ben­e­fits of spaced prac­tise may be muted for many math­e­mat­ics tasks.

What is espe­cially nice about this study was that not only did it use high­-qual­ity (in­tel­li­gent & moti­vat­ed) col­lege stu­dents (United States Air Force Acad­emy), the con­di­tions were rel­a­tively con­trolled - both groups had the same home­work (so equal test­ing effec­t), but like Rohrer & Tay­lor 2006/2007, the dis­tri­b­u­tion was what var­ied:

The course top­ics, text­book, hand­outs, read­ing assign­ments, and graded assign­ments (with the excep­tion of quiz, home­work, and par­tic­i­pa­tion points) were iden­ti­cal for the treat­ment and con­trol groups. The list­ing of home­work assign­ments in the syl­labus dif­fered between groups. The con­trol group was assigned daily home­work related to the top­ic(s) pre­sented that day in class. Peter­son (1971) calls this the ver­ti­cal model for assign­ing math­e­mat­ics home­work. The treat­ment group was assigned home­work in accor­dance with a dis­trib­uted orga­ni­za­tional pat­tern that com­bines prac­tice on cur­rent top­ics and rein­force­ment of pre­vi­ously cov­ered top­ics. Under the dis­trib­uted mod­el, approx­i­mately 40% of the prob­lems on a given topic were assigned the day the topic was first intro­duced, with an addi­tional 20% assigned on the next les­son and the remain­ing 40% of prob­lems on the topic assigned on sub­se­quent lessons (Hirsch et al, 1983). In Hirsch’s research and in this study, after the ini­tial home­work assign­ment, prob­lem(s) rep­re­sent­ing a given topic resur­faced on the 2nd, 4th, 7th, 12th, and 21st les­son. Con­se­quent­ly, treat­ment group home­work for les­son one con­sisted of only one top­ic; home­work for lessons two and three con­sisted of two top­ics; and home­work for les­son four through six con­sisted of three top­ics. This pat­tern con­tin­ued as new top­ics were added and was applied to all non-ex­am, non-lab­o­ra­tory lessons. As shown by Tables 1 and 2, the same home­work prob­lems were assigned to both groups with only the pat­tern of assign­ment dif­fer­ing. Because of the nature of the dis­trib­uted prac­tice mod­el, home­work for the treat­ment group con­tained fewer prob­lems (rel­a­tive to the con­trol group) early in the semes­ter with the num­ber of prob­lems increas­ing as the semes­ter pro­gressed. Later in the semes­ter, home­work for the treat­ment group con­tained more prob­lems (rel­a­tive to the con­trol group)….The USAFA rou­tinely col­lects study time data. After each exam, a large sam­ple of cadets (at least 60% of the course pop­u­la­tion) anony­mously reported the amount of time (in min­utes) spent study­ing for the exam. Time spent study­ing was approx­i­mately equal for both groups (see Table 5). Descrip­tive data rev­els that, for both the treat­ment and con­trol group, study time for the third exam was at least 16% greater than study time for any other exam. Study time for the final exam was at least 68% greater than study time for any of the hourly exams (see Table 5)

…The treat­ment pro­duced an effect size (f 2) of 0.013 on the first exam, 0.029 on the sec­ond exam, 0.035 on the fourth exam, and 0.040 on the final course per­cent­age grade. Although the effect sizes appear to be small, the treat­ment group outscored the con­trol group in every case. A mean dif­fer­ence of 5.13 per­cent­age points on the first, sec­ond, and fourth exam trans­lates to an advan­tage of about a third of a let­ter grade for stu­dents in the treat­ment group. In addi­tion, higher min­i­mum scores earned by the treat­ment group may indi­cate that the dis­trib­uted prac­tice treat­ment served to elim­i­nate the extremely low scores (re­fer to Table 3)….Odd­ly, the dis­trib­uted prac­tice treat­ment did not pro­duce a [sta­tis­ti­cal­ly-]sig­nif­i­cant effect on final exam scores. One pos­si­ble cause for the dis­par­ity was the USAFA pol­icy exempt­ing the top per­form­ers from the final exam. Of the 16 exempted stu­dents, 11 were from the treat­ment group with only 5 from the con­trol group.

Balch 2006 abstract:

Two intro­duc­tory psy­chol­ogy classes (N = 145) par­tic­i­pated in a coun­ter­bal­anced class­room exper­i­ment that demon­strated the spac­ing effect and, by anal­o­gy, the ben­e­fits of dis­trib­uted study. After hear­ing words pre­sented twice in either a massed or dis­trib­uted man­ner, par­tic­i­pants recalled the words and scored their recall pro­to­cols, reli­ably remem­ber­ing more dis­trib­uted than massed words. Posttest scores on a mul­ti­ple-­choice quiz cov­er­ing points illus­trated by the exper­i­ment aver­aged about twice the com­pa­ra­ble pretest scores, indi­cat­ing the effec­tive­ness of the exer­cise in con­vey­ing con­tent. Stu­dents’ sub­jec­tive rat­ings sug­gested that the exper­i­ment helped con­vince them of the ben­e­fits of dis­trib­uted study.

See Cepeda et al 2006↩︎

Com­mins, S., Cun­ning­ham, L., Har­vey, D., and Wal­sh, D. (2003). “Massed but not spaced train­ing impairs spa­tial mem­ory”. Behav­ioural Brain Research 139, 215-223↩︎

Gal­luc­cio & Rovee-­Col­lier 2006, “Nonuni­form effects of rein­state­ment within the time win­dow”. Learn­ing and Moti­va­tion, 37, 1-17.↩︎

See the pre­vi­ous sec­tions for many using chil­dren; one pre­vi­ously uncited is Top­pino 1993, “The spac­ing effect in preschool chil­dren’s free recall of pic­tures and words”; but Top­pino et al 2009 adds some inter­est­ing qual­i­fiers to spaced rep­e­ti­tion in the young:

Preschool­ers, ele­men­tary school chil­dren, and col­lege stu­dents exhib­ited a spac­ing effect in the free recall of pic­tures when learn­ing was inten­tion­al. When learn­ing was inci­den­tal and a shal­low pro­cess­ing task requir­ing lit­tle seman­tic pro­cess­ing was used dur­ing list pre­sen­ta­tion, young adults still exhib­ited a spac­ing effect, but chil­dren con­sis­tently failed to do so. Chil­dren, how­ev­er, did man­i­fest a spac­ing effect in inci­den­tal learn­ing when an elab­o­rate seman­tic pro­cess­ing task was used.

Another pre­vi­ously uncited study: Glen­berg, A. M. (1979), “Com­po­nen­t-levels the­ory of the effects of spac­ing of rep­e­ti­tions on recall and recog­ni­tion”. Mem­ory & Cog­ni­tion, 7, 95-112.↩︎

See Kor­nell et al 2010; Simone et al 2012 shows the spac­ing ben­e­fits but reduced in mag­ni­tude in its 56-74 year old sub­jects, sim­i­lar to Jack­son et al 2012 and Mad­dox 2013↩︎

Mam­marel­la, N., Rus­so, R., & Avons, S. E. (2002). "Spac­ing effects in cued-mem­ory tasks for unfa­mil­iar faces and non­words". Mem­ory & Cog­ni­tion, 30, 1238-1251↩︎

Childers, J. B., & Tomasel­lo, M. (2002). "Two-year-olds learn novel nouns, verbs, and con­ven­tional actions from massed or dis­trib­uted expo­sures". Devel­op­men­tal Psy­chol­ogy, 38, 967-978↩︎

eg. Fish­man et al 1968↩︎

The famous ‘10,000 hours of prac­tice’ fig­ure may not be as true or impor­tant as Eric­s­son and pub­li­ciz­ers like Mal­colm Glad­well imply, given the high vari­ance of exper­tise against time, and results from sports show­ing smaller time invest­ments (see also Ham­brick’s cor­pus cut­ting ‘delib­er­ate prac­tice’ down to size), and Eric­s­son absurdly deny the pow­er­ful role of genet­ics and the nec­es­sary con­di­tion of hav­ing tal­ent but the insight of ‘delib­er­ate prac­tice’ help­ing tal­ented peo­ple prob­a­bly is real. One may be able to get away with 3,000 hours rather than 10,000, but one isn’t going to do that with mind­less rep­e­ti­tion or no rep­e­ti­tions.↩︎

Gen­tner, D., Loewen­stein, J., & Thomp­son, L. (2003). “Learn­ing and trans­fer: A gen­eral role for ana­log­i­cal encod­ing”. Jour­nal of Edu­ca­tional Psy­chol­ogy, 95, 393-40↩︎

From Kor­nell et al 2010:

The ben­e­fits of spac­ing seem to dimin­ish or dis­ap­pear when to-be-learned items are not repeated exactly (Apple­ton-K­napp, Bjork, & Wick­ens, 2005)…a num­ber of stud­ies have shown that mass­ing, rather than spac­ing, pro­motes induc­tive learn­ing. These stud­ies have gen­er­ally employed rel­a­tively sim­ple per­cep­tual stim­uli that facil­i­tate exper­i­men­tal con­trol (Gag­né, 1950; Gold­stone, 1996; Kurtz & Hov­land, 1956; [Whit­man J. R., & Gar­ner, W. R. (1963). “Con­cept learn­ing as a func­tion of the form of inter­nal struc­ture”. Jour­nal of Ver­bal Learn­ing & Ver­bal Behav­ior, 2, 195-202]).

High error rates - indi­cat­ing one did­n’t actu­ally learn the card con­tents in the first place - seem to be con­nected to fail­ures of the spac­ing effect; there’s some evi­dence that peo­ple nat­u­rally choose to mass study when they don’t yet know the mate­r­i­al.↩︎

“Super­Memo as a new tool increas­ing the pro­duc­tiv­ity of a pro­gram­mer. A case study: pro­gram­ming in Object Win­dows”↩︎

The 20 years look like this (note the sci­en­tific nota­tion): [0.742675, 0.27044575182838654, 0.15275979054767388, 0.10348750000000001, 7.751290630254386e-2, 6.187922936397532e-2, 5.161829250474865e-2, 4.445884397854832e-2, 3.923055555555555e-2, 3.5275438307530015e-2, 3.219809429218694e-2, 2.9748098818459235e-2, 2.7759942051635768e-2, 2.6120309801216147e-2, 2.474928593068675e-2, 2.35890625e-2, 2.2596898475825956e-2, 2.1740583401051353e-2, 2.0995431241707652e-2, 2.0342238287817983e-2]↩︎

mod­ulo things where know­ing it is use­ful even if you don’t need it often - it can be a brick in a pyra­mid of knowl­edge; cf.page 3 of Wolf:

The prob­lem of for­get­ting might not tor­ment us so much if we could only con­vince our­selves that remem­ber­ing isn’t impor­tant. Per­haps the things we learn - words, dates, for­mu­las, his­tor­i­cal and bio­graph­i­cal details - don’t really mat­ter. Facts can be looked up. That’s what the Inter­net is for. When it comes to learn­ing, what really mat­ters is how things fit togeth­er. We mas­ter the sto­ries, the schemas, the frame­works, the par­a­digms; we rehearse the lin­go; we swim in the epis­teme.

The dis­ad­van­tage of this com­fort­ing notion is that it’s false. “The peo­ple who crit­i­cize mem­o­riza­tion - how happy would they be to spell out every let­ter of every word they read?” asks Robert Bjork, chair of UCLA’s psy­chol­ogy depart­ment and one of the most emi­nent mem­ory researchers. After all, Bjork notes, chil­dren learn to read whole words through intense prac­tice, and every time we enter a new field we become chil­dren again. “You can’t escape mem­o­riza­tion,” he says. “There is an ini­tial process of learn­ing the names of things. That’s a stage we all go through. It’s all the more impor­tant to go through it rapid­ly.” The human brain is a mar­vel of asso­cia­tive pro­cess­ing, but in order to make asso­ci­a­tions, data must be loaded into mem­o­ry.

See Stephen R. Schmidt’s web­page “The­o­ries of For­get­ting”, which cites ‘Wood­worth & Schlos­beg (1961)’ when pre­sent­ing a log graph of var­i­ous stud­ies’ for­get­ting curves.↩︎

which neatly addresses the issue of such mail­ing lists being use­less (‘who learns a word after just one expo­sure?’).↩︎

Mnemosyne in this case con­sti­tutes both a way to learn the quotes so I can use them, and a waste book; just the other day I had 3 or 4 appo­site quotes for an essay because I had entered them into Mnemosyne months or years ago.↩︎

It’s well known that any speaker of a lan­guage under­stands many more words than they will ever use or be able to explic­itly gen­er­ate, that their “read­ing vocab­u­lary” exceeds their “writ­ing vocab­u­lary”; less well-­known is that on many prob­lems, one can guess at well above ran­dom rates even while feel­ing unsure & igno­rant, neces­si­tat­ing psy­chol­o­gists to employ forced-­choice par­a­digms. Even less known is the capac­ity of recog­ni­tion mem­ory or “implicit mem­ory”; this mem­ory can apply to things like rec­og­niz­ing images or text or music, typ­ing, puz­zle solv­ing, etc. Andrew Druck­er, in “Mul­ti­ply­ing 10-digit num­bers using Flickr: The power of recog­ni­tion mem­ory”, employs visual mem­ory to cal­cu­late $9883603368\cdot 4288997768=42390752785149282624$; he cites as prece­dent Stand­ing 1973:

In one of the most wide­ly-cited stud­ies on recog­ni­tion mem­o­ry, Stand­ing showed par­tic­i­pants an epic 10,000 pho­tographs over the course of 5 days, with 5 sec­onds’ expo­sure per image. He then tested their famil­iar­i­ty, essen­tially as described above. The par­tic­i­pants showed an 83% suc­cess rate, sug­gest­ing that they had become famil­iar with about 6,600 images dur­ing their ordeal. Other vol­un­teers, trained on a smaller col­lec­tion of 1,000 images selected for vivid­ness, had a 94% suc­cess rate.

One some­times sees peo­ple argue that some­thing is inse­cure or unguess­able or free from pos­si­ble placebo effect because it involves too many objects to explic­itly mem­o­rize, but as these exam­ples make clear, recog­ni­tion mem­ory can hap­pen quickly and store sur­pris­ingly large amounts of infor­ma­tion. This could be used for authen­ti­ca­tion (see for exam­ple Boji­nov et al 2012; HN dis­cus­sion) or mes­sage since recog­ni­tion mem­ory could be exploited as a sort of secure com­mu­ni­ca­tion sys­tem. Two par­ties can share a set of 20,000 pho­tographs (10,000 pairs); to send a mes­sage, have a mes­sen­ger spend 5 days on 10,000 picked ones; and then to receive it, ask him to rec­og­nize which pho­to­graph he saw in each of the 10,000 pairs. The sub­ject not only does not know what the binary mes­sage is or what means, he can’t even pro­duce it since he can­not remem­ber the pho­tographs!

At an 80% accu­racy rate, we can even cal­cu­late how many bits of infor­ma­tion can be entrusted to the mes­sen­ger using Shan­non’s the­o­rem; a cal­cu­la­tion gives 5.8 kilo­bits as the upper lim­it: if p = 0.2 (based on the 80% suc­cess rate), then $\frac{10000}{1-(p\cdot lo{g}_{2}p+(1-p)\times (lo{g}_2(1-p)\left)\right)}=5807.44$. So we see that Frank Her­bert was right after all: the securest way to send a mes­sage is through a dis­trans mes­sen­ger. (The down­side is that the implicit recog­ni­tion mem­ory decays con­sid­er­ably; see Lan­dauer 1986 for adjusted esti­mates.)↩︎

In this vein, I am reminded of what a for­mer polypha­sic sleeper told me:

I’ve been polypha­sic for about a year. (Not any­more; kills my mem­o­ry.)…Anki reps, most­ly. I found that I could do proper review ses­sions for about 2-3 days and would hit an impen­e­tra­ble wall. I could­n’t learn a sin­gle new card and had total brain fog until I got 3 hours more sleep. That, how­ev­er, would reset my adap­ta­tion. The whole effect is a bit less pro­nounced on Every­man, but not much. It is how­ever eas­ier to add sleep when you already have a core. I did­n’t notice any other major men­tal impair­ment after the ini­tial sleep depri­va­tion.

For a more recent review, see Philips et al 2013.↩︎

Pre­sum­ably one would imme­di­ately give them all some high grade like 5 to avoid sud­denly hav­ing a daily load of 500 cards for a while.↩︎

Smaller is bet­ter.↩︎

“For Mnemosyne 2.x, Ull­rich is work­ing on an offi­cial Mnemosyne iPhone client which will have very easy sync­ing.”↩︎

Wired↩︎

See Page 4, Wolf 2008:

The spac­ing effect was one of the proud­est lab-derived dis­cov­er­ies, and it was inter­est­ing pre­cisely because it was not obvi­ous, even to pro­fes­sional teach­ers. The same year that Neisser revolt­ed, Robert Bjork, work­ing with Thomas Lan­dauer of Bell Labs, pub­lished the results of two exper­i­ments involv­ing nearly 700 under­grad­u­ate stu­dents. Lan­dauer and Bjork were look­ing for the opti­mal moment to rehearse some­thing so that it would later be remem­bered. Their results were impres­sive: The best time to study some­thing is at the moment you are about to for­get it. And yet - as Neisser might have pre­dicted - that insight was use­less in the real world.

When I first read of Super­Me­mo, I had already taken a class in cog­ni­tive psy­chol­ogy and was rea­son­ably famil­iar with Ebbing­haus’s for­get­ting curve - so my reac­tion to its method­ol­ogy was Hux­ley’s: “How extremely stu­pid not to have thought of that!”↩︎

See page 7, Wolf 2008

And yet now, as I grin broadly and wave to the gawk­ers, it occurs to me that the cold ratio­nal­ity of his approach may be only a sur­face fea­ture and that, when linked to gen­uine rewards, even the chill­i­est of sys­tems can have a cer­tain vis­ceral appeal. By pro­ject­ing the achieve­ment of extreme mem­ory back along the for­get­ting curve, by prov­ably link­ing the dis­tant future - when we will know so much - to the few min­utes we devote to study­ing today, Woz­niak has found a way to con­di­tion his tem­pera­ment along with his mem­o­ry. He is mak­ing the future notice­able. He is try­ing not just to learn many things but to warm the process of learn­ing itself with a draft of utopian ecsta­sy.