“One does not learn com­put­ing by us­ing a hand cal­cu­la­tor, but one can for­get arith­metic.” Perlis 1982↩︎

List­ing other neu­ro­pros­thet­ics is hard. It’s an in­ter­est­ing idea, but as pro­po­nents of ex­ter­nal­ism like Andy Clark have found, it’s eas­ier to feel that ex­ter­nal­ism is mean­ing­ful than to nail down a clear de­fi­n­i­tion which sep­a­rates a neu­ro­pros­thetic or part of one’s mind from a ran­dom tool you like or find use­ful. Con­sider whether a pen­cil and pa­per a neu­ro­pros­thet­ic: clearly it is not for a child learn­ing to write, who must care­fully com­pose the words in his mind and put them down one after an­oth­er, but it is not so clear for an adult who has been writ­ing all his life and can doo­dle or write down thoughts with­out think­ing about them and may even be sur­prised at what they hap­pened to write.

I like this de­fi­n­i­tion: “a neu­ro­pros­thetic is any­thing whose re­sults you use with­out fur­ther thought”. So in the clas­sic ex­am­ple, when Otto needs to go some­where, he never thinks “I am an am­ne­siac who stores lo­ca­tions in my notepad, and I must look up the lo­ca­tion” - he just looks up the lo­ca­tion. A good heuris­tic would be any­thing whose de­struc­tion leaves one feel­ing lost, slow, stu­pid, or ig­no­rant.

By this stan­dard, I can think of only a few tools I use with­out no­tice­able thought:

• key­bind­ings such as win­dow man­ager short­cuts, in par­tic­u­lar short­cuts for Google search­es; on oc­ca­sion, XMon­ad’s Prompt gets in­scrutably wedged, lock­ing it. When this hap­pens, I have to restart X be­cause I Google every­thing and the key­bind­ing is so en­grained that not us­ing it is un­bear­able. It would be like try­ing to write with your weak hand.
• Google Cal­en­dar and Pre­dic­tion­Book: it is in­cred­i­ble how many fol­lowups or re­minders or reg­u­larly hap­pen­ing tasks I can put into Google Cal­en­dar or PB. I have out­sourced many habits or thoughts to them, and I no longer think of it as any­thing spe­cial. If ei­ther were gone, I would feel fright­ened - what events were pass­ing, what be­liefs fal­si­fied, what op­por­tu­ni­ties open­ing up (or clos­ing!) that I had sud­denly be­come ig­no­rant of?
• Ever­note, for a sim­i­lar rea­son; many of my mem­o­ries have ceased to be things like “oc­to­puses see too fast to watch TV and so only HDTV or UHDTV works for them; I read this in Orion Mag­a­zine” and be­come things like “oc­to­pus TV Ever­note”, and if I want to know what it was about oc­to­puses & TV, well, I’ll have to look it up in Ever­note. Mnemosyne plays a sim­i­lar role for me, but there the mem­o­ries are much clearer on their own be­cause of the spaced rep­e­ti­tion.
• my web­site Gw­ern.net; I’ve had to say many times that I don’t know what I think about some­thing, but what­ever that is, it’s on my web­site. (A more ex­treme form of the Ever­note/M­nemosyne neu­ro­pros­thet­ic.) A com­menter once wrote that read­ing Gw­ern.net felt like he was crawl­ing around in my head. He was more right than he re­al­ized.

as quoted in “Re­trieval prac­tice and the main­te­nance of knowl­edge”, Bjork 1988↩︎

From “Close the Book. Re­call. Write It Down: That old study method still works, re­searchers say. So why don’t pro­fes­sors preach it?”; The Chron­i­cle of Higher Ed­u­ca­tion

Two psy­chol­ogy jour­nals have re­cently pub­lished pa­pers show­ing that this strat­egy works, the lat­est find­ings from a decades-old body of re­search. When stu­dents study on their own, “ac­tive re­call” - recita­tion, for in­stance, or flash­cards and other self­-quizzing - is the most effec­tive way to in­scribe some­thing in long-term mem­o­ry. Yet many col­lege in­struc­tors are only dimly fa­mil­iar with that re­search…

From “The Spac­ing Effect: A Case Study in the Fail­ure to Ap­ply the Re­sults of Psy­cho­log­i­cal Re­search” (Demp­ster 1988), whose ti­tle alone sum­ma­rizes the sit­u­a­tion (see also Kel­ley 2007, Mak­ing Minds: What’s Wrong with Ed­u­ca­tion - and What Should We Do About It?):

Sec­ond, it [the spac­ing effect] is re­mark­ably ro­bust. In many cas­es, two spaced pre­sen­ta­tions are about twice as effec­tive as two massed pre­sen­ta­tions (e.g., Hintz­man, 1974; Melton, 1970), and the differ­ence be­tween them in­creases as the fre­quency of rep­e­ti­tion in­creases (Un­der­wood, 1970)…

The spac­ing effect was known as early as 1885 when Ebbing­haus pub­lished the re­sults of his sem­i­nal work on mem­o­ry. With him­self as the sub­ject, Ebbing­haus found that for a sin­gle 12-syl­la­ble se­ries, 68 im­me­di­ately suc­ces­sive rep­e­ti­tions had the effect of mak­ing pos­si­ble an er­ror­less recital after seven ad­di­tional rep­e­ti­tions on the fol­low­ing day. How­ev­er, the same effect was achieved by only 38 dis­trib­uted rep­e­ti­tions spread over 3 days. On the ba­sis of this and other re­lated find­ings, Ebbing­haus con­cluded that ‘with any con­sid­er­able num­ber of rep­e­ti­tions a suit­able dis­tri­b­u­tion of them over a space of time is de­cid­edly more ad­van­ta­geous than the mass­ing of them at a sin­gle time’ (Eb­bing­haus, 1885/1913. p. 89)

Son & Si­mon 2012:

Fur­ther­more, even after ac­knowl­edg­ing the ben­e­fits of spac­ing, chang­ing teach­ing prac­tices proved to be enor­mously diffi­cult. De­laney et al (2010) wrote: “Anec­do­tal­ly, high school teach­ers and col­lege pro­fes­sors seem to teach in a lin­ear fash­ion with­out rep­e­ti­tion and give three or four non­cu­mu­la­tive ex­ams.” (p. 130). Fo­cus­ing on the math do­main, where one might ex­pect a very easy-to-re­view-and-to-space strat­e­gy, Rohrer (2009) points out that math­e­mat­ics text­books usu­ally present top­ics in a non-spaced, non-mixed fash­ion. Even much ear­lier, Vash (1989) had writ­ten: “Ed­u­ca­tion pol­icy set­ters know per­fectly well that [spaced prac­tice] works bet­ter [than massed prac­tice]. They don’t care. It is­n’t tidy. It does­n’t let teach­ers teach a unit and dust off their hands quickly with a nice sense of ‘Well, that’s done.’” (p. 1547).

• Rohrer, D. (2009). “The effects of spac­ing and mix­ing prac­tice prob­lems”. Jour­nal for Re­search in Math­e­mat­ics Ed­u­ca­tion, 40, 4-17
• Vash, C. L. (1989). “The spac­ing effect: A case study in the fail­ure to ap­ply the re­sults of psy­cho­log­i­cal re­search”. Amer­i­can Psy­chol­o­gist, 44, 1547 (a com­ment on Demp­ster’s ar­ti­cle?)

From Psy­chol­o­gy: An In­tro­duc­tion:

In one prac­ti­cal demon­stra­tion of the spac­ing effect, Bahrick, Bahrick, Bahrick, & Bahrick (1993) showed that re­ten­tion of for­eign lan­guage vo­cab­u­lary was greatly en­hanced if prac­tice ses­sions were spaced far apart. For ex­am­ple, “Thir­teen re­train­ing ses­sions spaced at 56 days yielded re­ten­tion com­pa­ra­ble to 26 ses­sions spaced at 14 days.” In other words, sub­jects could use half as many study ses­sions, if the study ses­sions were spread over a time pe­riod four times as long.

“Synap­tic ev­i­dence for the effi­cacy of spaced learn­ing”, Kra­mar et al 2012 (“Take your time: Neu­ro­bi­ol­ogy sheds light on the su­pe­ri­or­ity of spaced vs. massed learn­ing”):

The su­pe­ri­or­ity of spaced vs. massed train­ing is a fun­da­men­tal fea­ture of learn­ing. Here, we de­scribe unan­tic­i­pated tim­ing rules for the pro­duc­tion of long-term po­ten­ti­a­tion (LTP) in adult rat hip­pocam­pal slices that can ac­count for one tem­po­ral seg­ment of the spaced tri­als phe­nom­e­non. Suc­ces­sive bouts of nat­u­ral­is­tic theta burst stim­u­la­tion of field CA1 affer­ents markedly en­hanced pre­vi­ously sat­u­rated LTP if spaced apart by 1 h or longer, but were with­out effect when shorter in­ter­vals were used. Analy­ses of F-act­in-en­riched spines to iden­tify po­ten­ti­ated synapses in­di­cated that the added LTP ob­tained with de­layed theta trains in­volved re­cruit­ment of synapses that were “missed” by the first stim­u­la­tion bout. Sin­gle spine glu­ta­mate-uncaging ex­per­i­ments con­firmed that less than half of the spines in adult hip­pocam­pus are primed to un­dergo plas­tic­ity un­der base­line con­di­tions, sug­gest­ing that in­trin­sic vari­abil­ity among in­di­vid­ual synapses im­poses a repet­i­tive pre­sen­ta­tion re­quire­ment for max­i­miz­ing the per­cent­age of po­ten­ti­ated con­nec­tions. We pro­pose that a com­bi­na­tion of lo­cal diffu­sion from ini­tially mod­i­fied spines cou­pled with much later mem­brane in­ser­tion events dic­tate that the rep­e­ti­tions be widely spaced. Thus, the synap­tic mech­a­nisms de­scribed here pro­vide a neu­ro­bi­o­log­i­cal ex­pla­na­tion for one com­po­nent of a poorly un­der­stood, ubiq­ui­tous as­pect of learn­ing.

There are many stud­ies to the effect that ac­tive re­call is best. Here’s one re­cent study, “Re­trieval Prac­tice Pro­duces More Learn­ing than Elab­o­ra­tive Study­ing with Con­cept Map­ping”, Karpicke 2011 (cov­ered in Sci­ence Daily and the NYT):

Ed­u­ca­tors rely heav­ily on learn­ing ac­tiv­i­ties that en­cour­age elab­o­ra­tive study­ing, while ac­tiv­i­ties that re­quire stu­dents to prac­tice re­triev­ing and re­con­struct­ing knowl­edge are used less fre­quent­ly. Here, we show that prac­tic­ing re­trieval pro­duces greater gains in mean­ing­ful learn­ing than elab­o­ra­tive study­ing with con­cept map­ping. The ad­van­tage of re­trieval prac­tice gen­er­al­ized across texts iden­ti­cal to those com­monly found in sci­ence ed­u­ca­tion. The ad­van­tage of re­trieval prac­tice was ob­served with test ques­tions that as­sessed com­pre­hen­sion and re­quired stu­dents to make in­fer­ences. The ad­van­tage of re­trieval prac­tice oc­curred even when the cri­te­r­ial test in­volved cre­at­ing con­cept maps. Our find­ings sup­port the the­ory that re­trieval prac­tice en­hances learn­ing by re­trieval-spe­cific mech­a­nisms rather than by elab­o­ra­tive study process­es. Re­trieval prac­tice is an effec­tive tool to pro­mote con­cep­tual learn­ing about sci­ence.

From “For­get What You Know About Good Study Habits”. New York Times;

Cog­ni­tive sci­en­tists do not deny that hon­est-to-good­ness cram­ming can lead to a bet­ter grade on a given ex­am. But hur­riedly jam-pack­ing a brain is akin to speed-pack­ing a cheap suit­case, as most stu­dents quickly learn - it holds its new load for a while, then most every­thing falls out­….When the neural suit­case is packed care­fully and grad­u­al­ly, it holds its con­tents for far, far longer. An hour of study tonight, an hour on the week­end, an­other ses­sion a week from now: such so-called spac­ing im­proves later re­call, with­out re­quir­ing stu­dents to put in more over­all study effort or pay more at­ten­tion, dozens of stud­ies have found.

“The idea is that for­get­ting is the friend of learn­ing”, said Dr. Ko­r­nell. “When you for­get some­thing, it al­lows you to re­learn, and do so effec­tive­ly, the next time you see it.”

That’s one rea­son cog­ni­tive sci­en­tists see test­ing it­self - or prac­tice tests and quizzes - as a pow­er­ful tool of learn­ing, rather than merely as­sess­ment. The process of re­triev­ing an idea is not like pulling a book from a shelf; it seems to fun­da­men­tally al­ter the way the in­for­ma­tion is sub­se­quently stored, mak­ing it far more ac­ces­si­ble in the fu­ture.

In one of his own ex­per­i­ments, Dr. Roedi­ger and Jeffrey Karpicke, who is now at Pur­due Uni­ver­si­ty, had col­lege stu­dents study sci­ence pas­sages from a read­ing com­pre­hen­sion test, in short study pe­ri­ods. When stu­dents stud­ied the same ma­te­r­ial twice, in back­-to-back ses­sions, they did very well on a test given im­me­di­ately after­ward, then be­gan to for­get the ma­te­r­i­al. But if they stud­ied the pas­sage just once and did a prac­tice test in the sec­ond ses­sion, they did very well on one test two days lat­er, and an­other given a week lat­er.

The Math­e­mat­ics of Gam­bling, Thorp 1984, “Sec­tion Two: The Wheels”, Chap­ter 4, pg43-44:

It was the spring of 1955. I was fin­ish­ing my sec­ond year of grad­u­ate physics at U.C.L.A…I changed my field of study from physics to math­e­mat­ic­s…I at­tended classes and stud­ied from 50 to 60 hours a week, gen­er­ally in­clud­ing Sat­ur­days and Sun­days. I had read about the psy­chol­ogy of learn­ing in or­der to be able to work longer and hard­er. I found that “spaced learn­ing” worked well: study for an hour, then take a break of at least ten min­utes (show­er, meal, tea, er­rands, etc.). One Sun­day after­noon about 3 p.m., I came to the co-op din­ing room for a tea break…My head was bub­bling with physics equa­tions, and sev­eral of my good friends were sit­ting around chat­ting.

From Fi­nal Jeop­ardy: Man Vs. Ma­chine and the Quest to Know Every­thing, by Stephen Bak­er, pg 214:

The pro­gram he put to­gether tested him on cat­e­gories, gauged his strengths (sciences, NFL foot­ball) and weak­nesses (fash­ion, Broad­way shows), and then di­rected him to­ward the prepa­ra­tion most likely to pay off in his own match. To patch these holes in his knowl­edge, Craig used a free on­line tool called Anki, which pro­vides elec­tronic flash cards for hun­dreds of fields of study, from Japan­ese vo­cab­u­lary to Eu­ro­pean mon­archs. The pro­gram, in Craig’s words, is based on psy­cho­log­i­cal re­search on ‘the for­get­ting curve’. It helps peo­ple find holes in their knowl­edge and de­ter­mines how often they need those ar­eas to be re­viewed to keep them in mind. In go­ing over world cap­i­tals, for ex­am­ple, the sys­tem learns quickly that a user like Craig knows Lon­don, Paris, and Rome, so it might spend more time re­in­forc­ing the cap­i­tal of, say, Kaza­khstan. (And what would be the Kazakh cap­i­tal? ‘As­tana’, Craig said in a flash. ‘It used to be Al­maty, but they moved it.’)

“Our In­ter­view With Jeop­ardy! Cham­pion Arthur Chu”:

[Chu:] …Jeop­ardy! is aimed at the sort of av­er­age TV view­er, so they’re not go­ing to ask things that are point­lessly ob­scure…So I used a pro­gram called Anki which uses a method called “spaced rep­e­ti­tion.” It keeps track of where you’re do­ing well or poor­ly, and pushes you to study the flash­cards you don’t know as well, un­til you de­velop an even knowl­edge base about a par­tic­u­lar sub­ject, and I just made flash­cards for those spe­cific things. I mem­o­rized all the world cap­i­tals, it was­n’t that hard once I had the flash­cards and was us­ing them every day. I mem­o­rized the US State Nick­names (they’re on Wikipedi­a), mem­o­rized the ba­sic im­por­tant facts about the 44 US Pres­i­dents. I re­ally fo­cused on those. But there’s a lot more stuff to know. I went on Jeop­ardy! know­ing that there was stuff I did­n’t know. For in­stance, every­one laughs about sports - but I also knew that [s­ports clues] were the least likely to come up in Dou­ble Jeop­ardy and Fi­nal Jeop­ardy and be very im­por­tant. So I de­cided I should­n’t sweat it too much, I should just rec­og­nize that I did­n’t know them and let that go, as long as I can get the high value clues. So that was how I pre­pared.

Alan J. Perlis, “Epi­grams in Pro­gram­ming” (1982)↩︎

Web de­vel­oper Per­sol writes in Au­gust 2012:

I ac­tu­ally wrote a site that did this [spaced rep­e­ti­tion] a few months ago. I had about 4000 users who had ac­tu­ally gone through a com­plete ses­sion…As guessed, the prob­lem is that I could­n’t get peo­ple to start form­ing it as a habit. There is no im­me­di­ate pay­back. Less than 20 peo­ple out of 4000 did more than one ses­sion…Ad­di­tion­al­ly, there are at least 18 com­peti­tors. Here’s the list I made at the time. Very few seem to be suc­cess­ful. I shut the site down about a month ago. There are nu­mer­ous free com­peti­tors which don’t have any great an­noy­ances. I would­n’t sug­gest start­ing an­other of these sites un­less you fig­ured out an effec­tive way to “gam­ify” it.

…~4000 peo­ple fin­ished a ses­sion. Many more ‘tried’ than 4000…I just could­n’t de­ter­mine which users were bots that reg­is­tered ran­domly vs users that did­n’t fin­ish the first ses­sion.

• Tried: lots (but un­known)
• Fin­ished 1 ses­sion: ~4000
• Fin­ished >1 ses­sion: ~20 [0.5%]

“Play it Again: The Mas­ter Psy­chophar­ma­col­ogy Pro­gram as an Ex­am­ple of In­ter­val Learn­ing in Bite-Sized Por­tions”, Stahl et al 2010:

Since Ebbing­haus’ time, a vo­lu­mi­nous amount of re­search has con­firmed this sim­ple but im­por­tant fact: the re­ten­tion of new in­for­ma­tion de­grades rapidly un­less it is re­viewed in some man­ner. A mod­ern ex­am­ple of this loss of knowl­edge with­out rep­e­ti­tion is a study of car­diopul­monary re­sus­ci­ta­tion (CPR) skills that demon­strated rapid de­cay in the year fol­low­ing train­ing. By 3 years post-train­ing only 2.4% were able to per­form CPR suc­cess­ful­ly.⁶ An­other re­cent study of physi­cians tak­ing a tu­to­r­ial they rated as very good or ex­cel­lent showed mean knowl­edge scores in­creas­ing from 50% be­fore the tu­to­r­ial to 76% im­me­di­ately after­ward.⁷ How­ev­er, score gains were only half as great 3-8 days later and in­cred­i­bly, there was no [s­ta­tis­ti­cal­ly-]sig­nifi­cant knowl­edge re­ten­tion mea­sur­able at all at 55 days.⁷ Sim­i­lar re­sults have been re­ported by us in fol­low-up stud­ies of knowl­edge re­ten­tion from con­tin­u­ing med­ical ed­u­ca­tion pro­grams.¹ [S­tahl SM, Davis RL. Best Prac­tices for Med­ical Ed­u­ca­tors. Carls­bad, CA: NEI Press; 2009]

…This may be due to the fact that lec­tures with as­signed read­ing are the eas­i­est for teach­ers. Al­so, med­ical learn­ing is rarely mea­sured im­me­di­ately after a lec­ture or after read­ing new ma­te­r­ial for the first time and then mea­sured again a few days or weeks lat­er, so that the low re­ten­tion rates of this ap­proach may not be widely ap­pre­ci­at­ed.^1,4 No won­der for­mal med­ical ed­u­ca­tion con­fer­ences with­out en­abling or prac­tice-re­in­forc­ing strate­gies ap­pear to have rel­a­tively lit­tle im­pact on prac­tice and health­care out­comes.^8,9,10

One study look­ing at cram­ming is the 1993 “Cram­ming: A bar­rier to stu­dent suc­cess, a way to beat the sys­tem or an effec­tive learn­ing strat­e­gy?”, Vacha et al 1993, ab­stract:

Tested the hy­poth­e­sis that cram­ming is an in­effec­tive study strat­egy by ex­am­in­ing the weekly study di­aries of 166 un­der­grad­u­ates. All Ss also com­pleted an end-of-se­mes­ter ques­tion­naire mea­sur­ing study habits. Ss were clas­si­fied in the fol­low­ing study pat­terns: ide­al, con­fi­dent, zeal­ous, or cram­mer. Con­trary to the hy­poth­e­sis, re­sults sug­gest that cram­ming is an effec­tive ap­proach, most wide­spread in courses us­ing take-home es­say ex­am­i­na­tions and ma­jor re­search pa­pers. Cram­mers’ grades were as good as or bet­ter than those of Ss us­ing other strate­gies; the longer Ss were in col­lege, the more likely it was that they crammed. Cram­mers stud­ied more hours than most stu­dents and were as in­ter­ested in their courses as other stu­dents.

Note that there is no mea­sure of long-term re­ten­tion, sug­gest­ing that peo­ple who only care about grades are ra­tio­nally choos­ing to cram.↩︎

Anki has its Cram Mode and Mnemosyne 2.0 has a cram­ming plu­g­in. When a SRS does­n’t have ex­plicit sup­port, it’s al­ways pos­si­ble to ‘game’ the al­go­rithm by set­ting one’s scores ar­ti­fi­cially low, so the SR al­go­rithm thinks you are stu­pid and need to do a lot of rep­e­ti­tions.↩︎

“Ex­am­in­ing the ex­am­in­ers: Why are we so bad at as­sess­ing stu­dents?”, New­stead 2002:

Con­way, Co­hen and Stan­hope (1992) looked at long term mem­ory for the in­for­ma­tion pre­sented on a psy­chol­ogy course. They found that some types of in­for­ma­tion, es­pe­cially that re­lat­ing to re­search meth­ods, were re­mem­bered bet­ter than oth­ers. But in a fol­low up analy­sis, they found that the type of as­sess­ment used had an effect on mem­o­ry. In essence, ma­te­r­ial as­sessed by con­tin­u­ous as­sess­ment was more likely to be re­mem­bered than in­for­ma­tion as­sessed by ex­ams.

Stahl 2010:

For ex­am­ple, sim­ple restudy­ing al­lows the learner to re­ex­pe­ri­ence all of the ma­te­r­ial but ac­tu­ally pro­duces poor long-term re­ten­tion.^25,26,35 Why do stu­dents keep study­ing the orig­i­nal ma­te­ri­als? Cer­tainly if this is their only choice, then restudy­ing is a nec­es­sary tac­tic. An­other an­swer may be that re­peated study­ing falsely in­flates stu­dents’ con­fi­dence in their abil­ity to re­mem­ber in the fu­ture be­cause they sense that they un­der­stand it now, and they and their in­struc­tors may be un­aware of the many stud­ies that show poor re­ten­tion on de­layed test­ing after this form of rep­e­ti­tion.^25,26,35

From Ko­r­nell et al 2010:

Con­trary to the mass­ing-aid­s-in­duc­tion hy­poth­e­sis, fi­nal test per­for­mance was con­sis­tently and con­sid­er­ably su­pe­rior in the spaced con­di­tion. A large ma­jor­ity of par­tic­i­pants, how­ev­er, judged mass­ing to be more effec­tive than spac­ing, de­spite mak­ing the judg­ment after tak­ing the test.

…Metacog­ni­tive judg­ments-that is, judg­ments about one’s own mem­ory and cog­ni­tion-are often based on feel­ings of flu­en­cy(e.g., see Ben­jam­in, Bjork, & Schwartz, 1998; Rhodes & Castel, 2008). Be­cause mass­ing nat­u­rally leads to feel­ings of flu­ency and in­creases short­-term task per­for­mance dur­ing learn­ing, learn­ers fre­quently rate spac­ing as less effec­tive than mass­ing, even when their per­for­mance shows the op­po­site pat­tern (Bad­de­ley & Long­man 1978; Ko­r­nell & Bjork, 2008; Si­mon & Bjork, 2001; Zech­meis­ter & Shaugh­nessy, 1980). Av­er­aged across Ko­r­nell and Bjork’s (2008) ex­per­i­ments, for ex­am­ple, more than 80% of par­tic­i­pants rated mass­ing as equally or more effec­tive than spac­ing,whereas only 15% of par­tic­i­pants ac­tu­ally per­formed bet­ter in the massed con­di­tion than in the spaced con­di­tion.

…Such an il­lu­sion was ap­par­ent in the in­duc­tion con­di­tion. Con­trary to pre­vi­ous re­search, how­ev­er, par­tic­i­pants gave higher rat­ings for spac­ing than mass­ing dur­ing rep­e­ti­tion learn­ing (see, e.g., Si­mon & Bjork, 2001; Zech­meis­ter & Shaugh­nessy, 1980). This out­come may have oc­curred be­cause of a process of a ha­bit­u­a­tion: Six pre­sen­ta­tions and a to­tal of 30 s spent study­ing a sin­gle paint­ing may have come to seem in­effi­cient and point­less. Thus, there ap­pears to be a turn­ing point in metacog­ni­tive rat­ings based on flu­en­cy: As flu­ency in­creas­es, metacog­ni­tive rat­ings in­crease up to a point, but as flu­ency con­tin­ues to in­crease and en­cod­ing or re­trieval be­comes too easy, metacog­ni­tive rat­ings may be­gin to de­crease.

…In ad­vance of their re­search, Ko­r­nell and Bjork (2008) were con­vinced that such in­duc­tive learn­ing would ben­e­fit from mass­ing, yet their re­sults showed the op­po­site. Un­daunt­ed, we re­mained con­vinced that spac­ing would be more ben­e­fi­cial for rep­e­ti­tion learn­ing than for in­duc­tive learn­ing- es­pe­cially for older adults, given their over­all de­clines in episodic mem­o­ry. The cur­rent re­sults dis­con­firmed our ex­pec­ta­tions once again. If our in­tu­itions are er­ro­neous, de­spite our years spent prov­ing and prais­ing the spac­ing effec­t-in­clud­ing roughly 40 years’ worth con­tributed by Robert A. Bjork-those of the av­er­age stu­dent are surely mis­taken as well (as the in­ac­cu­racy of the par­tic­i­pants’ metacog­ni­tive rat­ings sug­gest­s). We have, per­haps, fallen vic­tim to the il­lu­sion that mak­ing learn­ing easy makes learn­ing effec­tive, rather than rec­og­niz­ing that spac­ing is a de­sir­able diffi­culty (Bjork 1994) that en­hances in­duc­tive learn­ing as well as rep­e­ti­tion learn­ing well into old age.

From Son & Si­mon 2012:

Thus, while spac­ing may boost learn­ing, it may be thought to be rel­a­tively in­effi­cient in terms of study time. As we dis­cuss lat­er, this feel­ing of in­effi­ciency may be one of the rea­sons that spac­ing is not the more pop­u­lar strat­e­gy. In­ter­est­ing­ly, in that same study (Bad­de­ley & Long­man 1978; and see also Pirolli & An­der­son 1985 and Wood­worth & Schlos­berg 1954 [Ex­per­i­men­tal Psy­chol­ogy]), there was ev­i­dence of such a thing as la­bor­ing in vain. That is, ex­ceed­ing a cer­tain num­ber of hours of prac­tice a day (more than ap­prox­i­mately 2 h) led to no in­creases in learn­ing, as might be ex­pect­ed. Re­lated to the de­fi­cien­t-pro­cess­ing the­ory men­tioned above, these re­sults are cru­cial in un­der­stand­ing in­tu­itively how the spac­ing effect works: We sim­ply get burnt out. These data are also anal­o­gous to the cog­ni­tive lit­er­a­ture on over­learn­ing, which shows that while con­tin­u­ous study over long pe­ri­ods of time might seem ben­e­fi­cial (and even feel good) in the short­-term, the ben­e­fits dis­ap­pear soon after­wards (Rohrer et al. 2005; Rohrer and Tay­lor 2006)…In the above-de­scribed Bad­de­ley and Long­man’s (1978) study, for ex­am­ple, after postal work­ers prac­ticed typ­ing in ei­ther massed or spaced study ses­sions, they had to in­di­cate how sat­is­fied they were with the train­ing. Re­sults showed that while spac­ing led to the best learn­ing, it was the least liked. Sim­i­lar­ly, Si­mon & Bjork (2001) found that peo­ple pre­ferred the mass­ing strat­egy on a mo­tor learn­ing task.

• Bad­de­ley, A. D., & Long­man, D. J. A. (1978). “The in­flu­ence of length and fre­quency of train­ing ses­sion on the rate of learn­ing to type”. Er­gonom­ics, 21, 627-635
• Pirol­li, P., & An­der­son, J. R. (1985). “The role of prac­tice in fact re­trieval”

“Study strate­gies of col­lege stu­dents: Are self­-test­ing and sched­ul­ing re­lated to achieve­ment?”, Hartwig & Dun­losky 2012:

Pre­vi­ous stud­ies, such as those by Ko­r­nell and Bjork (Psy­cho­nomic Bul­letin & Re­view, 14:219-224, 2007) and Karpicke, But­ler, and Roedi­ger (Mem­ory, 17:471-479, 2009), have sur­veyed col­lege stu­dents’ use of var­i­ous study strate­gies, in­clud­ing self­-test­ing and reread­ing. These stud­ies have doc­u­mented that some stu­dents do use self­-test­ing (but largely for mon­i­tor­ing mem­o­ry) and reread­ing, but the re­searchers did not as­sess whether in­di­vid­ual differ­ences in strat­egy use were re­lated to stu­dent achieve­ment. Thus, we sur­veyed 324 un­der­grad­u­ates about their study habits as well as their col­lege grade point av­er­age (GPA). Im­por­tant­ly, the sur­vey in­cluded ques­tions about self­-test­ing, sched­ul­ing one’s study, and a check­list of strate­gies com­monly used by stu­dents or rec­om­mended by cog­ni­tive re­search. Use of self­-test­ing and reread­ing were both pos­i­tively as­so­ci­ated with GPA. Sched­ul­ing of study time was also an im­por­tant fac­tor: Low per­form­ers were more likely to en­gage in late-night study­ing than were high per­form­ers; mass­ing (vs. spac­ing) of study was as­so­ci­ated with the use of fewer study strate­gies over­all; and all stu­dents-but es­pe­cially low per­form­er­s-were dri­ven by im­pend­ing dead­lines. Thus, self­-test­ing, reread­ing, and sched­ul­ing of study play im­por­tant roles in re­al-world stu­dent achieve­ment.

(See also Dun­losky et al 2013.) Note the self­-test­ing cor­re­la­tion ex­cludes flash­cards, a re­sult that both the au­thors and me found sur­pris­ing. The sleep con­nec­tion is in­ter­est­ing, given the hy­poth­e­sized link be­tween stronger mem­ory for­ma­tion & study­ing be­fore a good night’s sleep - you can hardly get a good night’s sleep if you are cram­ming late into the night (cor­re­lated with lower grades) but you can if you do so at a rea­son­able time in the evening (in time to get a solid night).

See also Susser & Mc­Cabe 2012:

Lab­o­ra­tory stud­ies have demon­strated the long-term mem­ory ben­e­fits of study­ing ma­te­r­ial in mul­ti­ple dis­trib­uted ses­sions as op­posed to one massed ses­sion, given an iden­ti­cal amount of over­all study time (i.e., the spac­ing effect). The cur­rent study goes be­yond the lab­o­ra­tory to in­ves­ti­gate whether un­der­grad­u­ates know about the ad­van­tage of spaced study, to what ex­tent they use it in their own study­ing, and what fac­tors might in­flu­ence its uti­liza­tion. Re­sults from a we­b-based sur­vey in­di­cated that par­tic­i­pants (n = 285) were aware of the ben­e­fits of spaced study and would use a higher level of spac­ing un­der ideal com­pared to re­al­is­tic cir­cum­stances. How­ev­er, self­-re­ported use of spac­ing was in­ter­me­di­ate, sim­i­lar to mass­ing and sev­eral other study strate­gies, and ranked well be­low com­monly used strate­gies such as reread­ing notes. Sev­eral fac­tors were en­dorsed as im­por­tant in the de­ci­sion to dis­trib­ute study time, in­clud­ing the per­ceived diffi­culty of an up­com­ing ex­am, the amount of ma­te­r­ial to learn, how heav­ily an exam is weighed in the course grade, and the value of the ma­te­r­i­al. Fur­ther, level of metacog­ni­tive self­-reg­u­la­tion and use of elab­o­ra­tion strate­gies were as­so­ci­ated with higher rates of spaced study.

An­a­lytic Cul­ture in the US In­tel­li­gence Com­mu­ni­ty: An Ethno­graphic Study, John­ston 2005, pg89:

To in­ves­ti­gate the in­ten­sity of in­struc­tional in­ter­ac­tions, Art Graesser and Na­talie Per­son 1994 com­pared ques­tion­ing and an­swer­ing in class­rooms with those in tu­to­r­ial set­tings.⁵ They found that class­room groups of stu­dents ask about three ques­tions an hour and that any sin­gle stu­dent in a class­room asks about 0.11 ques­tions per hour. In con­trast, they found that stu­dents in in­di­vid­ual tu­to­r­ial ses­sions asked 20-30 ques­tions an hour and were re­quired to an­swer 117-146 ques­tions per hour. Re­views of the in­ten­sity of in­ter­ac­tion that oc­curs in tech­nol­o­gy-based in­struc­tion have found even more ac­tive stu­dent re­sponse lev­els. [J. D. Fletcher, Tech­nol­o­gy, the Colum­bus Effect, and the Third Rev­o­lu­tion in Learn­ing.]

Al­though Graesser & Per­son 1994 also found that sheer num­ber of ques­tions was not nec­es­sar­ily im­por­tant, sug­gest­ing di­min­ish­ing mar­ginal re­turns or per­haps bad ques­tion ask­ing.↩︎

“Su­per­Memo is based on the in­sight that there is an ideal mo­ment to prac­tice what you’ve learned. Prac­tice too soon and you waste your time. Prac­tice too late and you’ve for­got­ten the ma­te­r­ial and have to re­learn it. The right time to prac­tice is just at the mo­ment you’re about to for­get. Un­for­tu­nate­ly, this mo­ment is differ­ent for every per­son and each bit of in­for­ma­tion. Imag­ine a pile of thou­sands of flash cards. Some­where in this pile are the ones you should be prac­tic­ing right now. Which are they?” Gary Wolf, “Want to Re­mem­ber Every­thing You’ll Ever Learn? Sur­ren­der to This Al­go­rithm”, Wired Mag­a­zine↩︎

“Make no mis­take about it: Com­put­ers process num­bers - not sym­bols. We mea­sure our un­der­stand­ing (and con­trol) by the ex­tent to which we can arith­me­tize an ac­tiv­i­ty.” Perlis, ibid.↩︎

this ex­po­nen­tial ex­pan­sion is how a SR pro­gram can han­dle con­tin­ual in­put of cards: if cards were sched­uled at fixed in­ter­vals, like every other day, re­view would soon be­come quite im­pos­si­ble - I have >18000 items in Mnemosyne, but I don’t have time to re­view 9000 ques­tions a day!↩︎

See the 2008 meta-analy­sis, “Learn­ing Styles: Con­cepts and Ev­i­dence” (APS press re­lease); from the ab­stract:

…in or­der to demon­strate that op­ti­mal learn­ing re­quires that stu­dents re­ceive in­struc­tion tai­lored to their pu­ta­tive learn­ing style, the ex­per­i­ment must re­veal a spe­cific type of in­ter­ac­tion be­tween learn­ing style and in­struc­tional method: Stu­dents with one learn­ing style achieve the best ed­u­ca­tional out­come when given an in­struc­tional method that differs from the in­struc­tional method pro­duc­ing the best out­come for stu­dents with a differ­ent learn­ing style. In other words, the in­struc­tional method that proves most effec­tive for stu­dents with one learn­ing style is not the most effec­tive method for stu­dents with a differ­ent learn­ing style.

Our re­view of the lit­er­a­ture dis­closed am­ple ev­i­dence that chil­dren and adults will, if asked, ex­press pref­er­ences about how they pre­fer in­for­ma­tion to be pre­sented to them. There is also plen­ti­ful ev­i­dence ar­gu­ing that peo­ple differ in the de­gree to which they have some fairly spe­cific ap­ti­tudes for differ­ent kinds of think­ing and for pro­cess­ing differ­ent types of in­for­ma­tion. How­ev­er, we found vir­tu­ally no ev­i­dence for the in­ter­ac­tion pat­tern men­tioned above, which was judged to be a pre­con­di­tion for val­i­dat­ing the ed­u­ca­tional ap­pli­ca­tions of learn­ing styles. Al­though the lit­er­a­ture on learn­ing styles is enor­mous, very few stud­ies have even used an ex­per­i­men­tal method­ol­ogy ca­pa­ble of test­ing the va­lid­ity of learn­ing styles ap­plied to ed­u­ca­tion. More­over, of those that did use an ap­pro­pri­ate method, sev­eral found re­sults that flatly con­tra­dict the pop­u­lar mesh­ing hy­poth­e­sis.

We con­clude there­fore, that at pre­sent, there is no ad­e­quate ev­i­dence base to jus­tify in­cor­po­rat­ing learn­ing-styles as­sess­ments into gen­eral ed­u­ca­tional prac­tice. Thus, lim­ited ed­u­ca­tion re­sources would bet­ter be de­voted to adopt­ing other ed­u­ca­tional prac­tices that have a strong ev­i­dence base, of which there are an in­creas­ing num­ber. How­ev­er, given the lack of method­olog­i­cally sound stud­ies of learn­ing styles, it would be an er­ror to con­clude that all pos­si­ble ver­sions of learn­ing styles have been tested and found want­i­ng; many have sim­ply not been tested at all.

Fritz, C. O., Mor­ris, P. E., Ac­ton, M., Etkind, R., & Voelkel, A. R (2007). “Com­par­ing and com­bin­ing ex­pand­ing re­trieval prac­tice and the key­word mnemonic for for­eign vo­cab­u­lary learn­ing”. Ap­plied Cog­ni­tive Psy­chol­ogy, 21, 499-526.↩︎

From Balota et al 2006, de­scrib­ing Spitzer 1939, “Stud­ies in re­ten­tion”:

Spitzer (1939) in­cor­po­rated a form of ex­panded re­trieval in a study de­signed to as­sess the abil­ity of sixth graders to learn sci­ence facts. Im­pres­sive­ly, Spitzer tested over 3600 stu­dents in Iowa-the en­tire six­th-grade pop­u­la­tion of 91 el­e­men­tary schools at the time. The stu­dents read two ar­ti­cles, one on peanuts and the other on bam­boo, and were given a 25-item mul­ti­ple choice test to as­sess their knowl­edge (such as ‘To which fam­ily of plants does bam­boo be­long?’). Spitzer tested a to­tal of nine groups, ma­nip­u­lat­ing both the tim­ing of the test (ad­min­is­tered im­me­di­ately or after var­i­ous de­lays) and the num­ber of iden­ti­cal tests stu­dents re­ceived (one to three). Spitzer did not in­cor­po­rate massed or equal in­ter­val re­trieval con­di­tions, but he had at least two groups that were tested on an ex­pand­ing sched­ule of re­trieval, in which the in­ter­vals be­tween tests were sep­a­rated by the pas­sage of time (in days) rather than by in­ter­ven­ing to-be-learned in­for­ma­tion. For ex­am­ple, in one of the groups, the first test was given im­me­di­ate­ly, the sec­ond test was given seven days after the first test, and the third test was given 63 days after the sec­ond test. Thus, in essence, this group was tested on a 0-7-63 day ex­pand­ing re­trieval sched­ule. Spitzer com­pared per­for­mance of the ex­panded re­trieval group to a group given a sin­gle test 63 days after read­ing the orig­i­nal ar­ti­cle. On the first (im­me­di­ate) test, the ex­panded re­trieval group cor­rectly an­swered 53% of the ques­tions. After 63 days and two pre­vi­ous tests, their score was still an im­pres­sive 43%. The sin­gle test group cor­rectly an­swered only 25% of the orig­i­nal items after 63 days, giv­ing the ex­panded re­trieval group an 18% re­ten­tion ad­van­tage. This is quite im­pres­sive, given that this large ben­e­fit re­mained after a 63-day re­ten­tion in­ter­val. Sim­i­lar ben­e­fi­cial effects were found in a group tested on a 0-1-21 day ex­panded re­trieval sched­ule com­pared to a group given a sin­gle test after 21 days. Of course, this study does not de­cou­ple the effects of test­ing from spac­ing or ex­pan­sion, but the re­sults do clearly in­di­cate con­sid­er­able learn­ing and re­ten­tion us­ing the ex­panded re­peated test­ing pro­ce­dure. Spitzer con­cluded that ‘…ex­am­i­na­tions are learn­ing de­vices and should not be con­sid­ered only as tools for mea­sur­ing achieve­ment of pupils’ (p. 656, ital­ics added)

“Dis­trib­ut­ing Learn­ing Over Time: The Spac­ing Effect in Chil­dren’s Ac­qui­si­tion and Gen­er­al­iza­tion of Sci­ence Con­cepts”, Vlach & Sand­hofer 2012:

The spac­ing effect de­scribes the ro­bust find­ing that long-term learn­ing is pro­moted when learn­ing events are spaced out in time, rather than pre­sented in im­me­di­ate suc­ces­sion. Stud­ies of the spac­ing effect have fo­cused on mem­ory processes rather than for other types of learn­ing, such as the ac­qui­si­tion and gen­er­al­iza­tion of new con­cepts. In this study, early el­e­men­tary school chil­dren (5-7 year-olds; N = 36) were pre­sented with sci­ence lessons on one of three sched­ules: massed, clumped, and spaced. The re­sults re­vealed that spac­ing lessons out in time re­sulted in higher gen­er­al­iza­tion per­for­mance for both sim­ple and com­plex con­cepts. Spaced learn­ing sched­ules pro­mote sev­eral types of learn­ing, strength­en­ing the im­pli­ca­tions of the spac­ing effect for ed­u­ca­tional prac­tices and cur­ricu­lum.

See also Balch 2006, who com­pared spac­ing & massed in an in­tro­duc­tory psy­chol­ogy course as well.↩︎

Roedi­ger & Karpicke 2006b again.↩︎

Balota et al 2006 re­view:

No feed­back or cor­rec­tion was given to sub­jects if they made er­rors or omit­ted an­swers. Lan­dauer & Bjork 1978 found that the ex­pand­ing-in­ter­val sched­ule pro­duced bet­ter re­call than equal-in­ter­val test­ing on a fi­nal test at the end of the ses­sion, and equal-in­ter­val test­ing, in turn, pro­duced bet­ter re­call than did ini­tial massed test­ing. Thus, de­spite the fact that massed test­ing pro­duced nearly er­ror­less per­for­mance dur­ing the ac­qui­si­tion phase, the other two sched­ules pro­duced bet­ter re­ten­tion on the fi­nal test given at the end of the ses­sion. How­ev­er, the differ­ence fa­vor­ing the ex­pand­ing re­trieval sched­ule over the equal-in­ter­val sched­ule was fairly small at around 10%. In re­search fol­low­ing up Lan­dauer and Bjork’s (1978) orig­i­nal ex­per­i­ments, prac­ti­cally all stud­ies have found that spaced sched­ules of re­trieval (whether equal-in­ter­val or ex­pand­ing sched­ules) pro­duce bet­ter re­ten­tion on a fi­nal test given later than do massed re­trieval tests given im­me­di­ately after pre­sen­ta­tion (e.g., Cull, 2000; Cull, Shaugh­nessy, & Zech­meis­ter, 1996), al­though ex­cep­tions do ex­ist. For ex­am­ple, in Ex­per­i­ments 3 and 4 of Cull et al (1996), massed test­ing pro­duced per­for­mance as good as equal-in­ter­val test­ing on a 5-5-5 sched­ule, but most other ex­per­i­ments have found that any spaced sched­ule of test­ing (ei­ther equal-in­ter­val or ex­pand­ing) is bet­ter than a massed sched­ule for per­for­mance on a de­layed test. How­ev­er, whether ex­pand­ing sched­ules are bet­ter than equal-in­ter­val sched­ules for long-term re­ten­tion-the other part of Lan­dauer and Bjork’s in­ter­est­ing find­ings-re­mains an open ques­tion. Balota, Duchek, and Lo­gan (in press) have pro­vided a thor­ough con­sid­er­a­tion of the rel­e­vant ev­i­dence and have shown that it is mixed at best, and that most re­searchers have found no differ­ence be­tween the two sched­ules of test­ing. That is, per­for­mance on a fi­nal test at the end of a ses­sion often shows no differ­ence in per­for­mance be­tween equal-in­ter­val and ex­pand­ing re­trieval sched­ules.

Cull, for those cu­ri­ous (Cull, W. L. (2000). “Un­tan­gling the ben­e­fits of mul­ti­ple study op­por­tu­ni­ties and re­peated test­ing for cued re­call”. Ap­plied Cog­ni­tive Psy­chol­ogy, 14, 215-235):

Cull (2000) com­pared ex­panded re­trieval to equal in­ter­val spaced re­trieval in a se­ries of four ex­per­i­ments de­signed to mimic typ­i­cal teach­ing or study strate­gies en­coun­tered by stu­dents. He ex­am­ined the role of test­ing ver­sus sim­ply restudy­ing the ma­te­ri­al, feed­back, and var­i­ous re­ten­tion in­ter­vals on fi­nal test per­for­mance. Paired as­so­ciates (an un­com­mon word paired with a com­mon word, such as bairn-print) were pre­sented in a man­ner sim­i­lar to the flash­card tech­niques stu­dents often use to learn vo­cab­u­lary words. The in­ter­vals be­tween re­trieval at­tempts of to-be-learned in­for­ma­tion ranged from min­utes in some ex­per­i­ments to days in oth­ers. In­ter­est­ing­ly, across four ex­per­i­ments, Cull did not find any ev­i­dence of an ad­van­tage of an ex­panded con­di­tion over a uni­form spaced con­di­tion (i.e., no [sub­stan­tial] ex­panded re­trieval effec­t), al­though both con­di­tions con­sis­tently pro­duced large ad­van­tages over massed pre­sen­ta­tions. He con­cluded that dis­trib­uted test­ing of any kind, ex­panded or equal in­ter­val, can be an effec­tive learn­ing aid for teach­ers to pro­vide for their stu­dents.

The Balota et al 2006 re­view offers a syn­the­sis of cur­rent the­o­ries on how massed and spaced differ, based on mem­ory en­cod­ing:

Ac­cord­ing to en­cod­ing vari­abil­ity the­o­ry, per­for­mance on a mem­ory test is de­pen­dent upon the over­lap be­tween the con­tex­tual in­for­ma­tion avail­able at the time of test and the con­tex­tual in­for­ma­tion avail­able dur­ing en­cod­ing. Dur­ing massed study, there is rel­a­tively lit­tle time for con­tex­tual el­e­ments to fluc­tu­ate be­tween pre­sen­ta­tions and so this con­di­tion pro­duces the high­est per­for­mance in an im­me­di­ate mem­ory test, when the test con­text strongly over­laps with the same con­tex­tual in­for­ma­tion en­coded dur­ing both of the massed pre­sen­ta­tions. In con­trast, when there is spac­ing be­tween the items, there is time for fluc­tu­a­tion to take place be­tween the pre­sen­ta­tions dur­ing study, and hence there is an in­creased like­li­hood of hav­ing mul­ti­ple unique con­texts en­cod­ed. Be­cause a de­layed test will also al­low fluc­tu­a­tion of con­text, it is bet­ter to have mul­ti­ple unique con­texts en­cod­ed, as in the spaced pre­sen­ta­tion for­mat, as op­posed to a sin­gle en­coded con­text, as in the massed pre­sen­ta­tion for­mat.

Storm et al 2010 did 3 ex­per­i­ments on read­ing com­pre­hen­sion:

On a test 1 week lat­er, re­call was en­hanced by the ex­pand­ing sched­ule, but only when the task be­tween suc­ces­sive re­trievals was highly in­ter­fer­ing with mem­ory for the pas­sage. These re­sults sug­gest that the ex­tent to which learn­ers ben­e­fit from ex­pand­ing re­trieval prac­tice de­pends on the de­gree to which the to-be-learned in­for­ma­tion is vul­ner­a­ble to for­get­ting.

From Mnemosyne’s Prin­ci­ples page:

The Mnemosyne al­go­rithm is very sim­i­lar to SM2 used in one of the early ver­sions of Su­per­Me­mo. There are some mod­i­fi­ca­tions that deal with early and late rep­e­ti­tions, and also to add a small, healthy dose of ran­dom­ness to the in­ter­vals. Su­per­memo now uses SM11. How­ev­er, we are a bit skep­ti­cal that the huge com­plex­ity of the newer SM al­go­rithms pro­vides for a sta­tis­ti­cally rel­e­vant ben­e­fit. But, that is one of the facts we hope to find out with our data col­lec­tion. We will only make mod­i­fi­ca­tions to our al­go­rithms based on com­mon sense or if the data tells us that there is a sta­tis­ti­cally rel­e­vant rea­son to do so.

Balota et al 2006:

Car­pen­ter and De­Losh (2005, Exp. 2) have re­cently in­ves­ti­gated face-name learn­ing un­der massed, ex­panded (1-3-5), and equal in­ter­val (3-3-3) con­di­tions. This study also in­volved study and study and test pro­ce­dures dur­ing the ac­qui­si­tion phase. Car­pen­ter and De­Losh found a large effect of spac­ing, but no ev­i­dence of a ben­e­fit of ex­panded over equal in­ter­val prac­tice. In fact, Car­pen­ter and De­Losh re­ported a re­li­able ben­e­fit of the equal in­ter­val con­di­tion over the ex­panded re­trieval con­di­tion.

Balota et al 2006 again:

Rea and Modigliani (1985) tested the effec­tive­ness of ex­panded re­trieval in a third-grade class­room set­ting. In sep­a­rate con­di­tions, stu­dents were given new mul­ti­pli­ca­tion prob­lems or spelling words to learn. The prob­lem or word was pre­sented au­dio­vi­su­ally once and then tested on ei­ther a massed re­trieval sched­ule of 0-0-0-0 or an ex­pand­ing sched­ule of 0-1-2-4, in which the in­ter­vals in­volved be­ing tested on old items or learn­ing new items. After each test trial for a given item, the item was re-p­re­sented in its en­tirety so stu­dents re­ceived feed­back on what they were learn­ing. Per­for­mance dur­ing the learn­ing phase was at 100% for both spelling words and mul­ti­pli­ca­tion facts. On an im­me­di­ate fi­nal re­ten­tion test, Rea and Modigliani found a per­for­mance ad­van­tage for all item­s-math and spelling- prac­ticed on an ex­pand­ing sched­ule com­pared to the massed re­trieval sched­ule. They sug­gest­ed, as have oth­ers, that spac­ing com­bined with the high suc­cess rate in­her­ent in the ex­panded re­trieval sched­ule pro­duced bet­ter re­ten­tion than massed re­trieval prac­tice. How­ev­er, as in Spitzer’s study, Rea and Modigliani did not test an ap­pro­pri­ate equal in­ter­val spac­ing con­di­tion. Hence, their find­ing that ex­panded re­trieval is su­pe­rior to massed re­trieval in third graders could sim­ply re­flect the su­pe­ri­or­ity of spaced ver­sus massed re­hearsal-in other words, the spac­ing effect.

Balota et al 2006.↩︎

Balota et al 2006; >1 is rare in psy­chol­o­gy, see “One Hun­dred Years of So­cial Psy­chol­ogy Quan­ti­ta­tively De­scribed”, Bond et al 2003↩︎

Rohrer & Tay­lor 2006↩︎

Balota et al 2006:

…long-term re­ten­tion of in­for­ma­tion has been demon­strated over sev­eral days in some cases (e.g., Camp et al, 1996). For ex­am­ple, in the lat­ter study, Camp et al em­ployed an ex­pand­ing re­trieval strat­egy to train 23 in­di­vid­u­als with mild to mod­er­ate AD to re­fer to a daily cal­en­dar as a cue to re­mem­ber to per­form var­i­ous per­sonal ac­tiv­i­ties (e.g., take med­ica­tion). Fol­low­ing a base­line phase to de­ter­mine whether sub­jects would spon­ta­neously use the cal­en­dar, spaced re­trieval train­ing was im­ple­mented by re­peat­edly ask­ing the sub­ject the ques­tion, ‘How are you go­ing to re­mem­ber what to do each day?’ at ex­pand­ing time in­ter­vals. The re­sults in­di­cated that 20/23 sub­jects did learn the strat­egy (i.e., to look at the cal­en­dar) and re­tained it over a 1-week pe­ri­od.

Rohrer & Tay­lor 2006 warns us, though, about many of the other math stud­ies:

In one meta-analy­sis by Dono­van and Ra­do­se­vich (1999), for in­stance, the size of the spac­ing effect de­clined sharply as con­cep­tual diffi­culty of the task in­creased from low (e.g. ro­tary pur­suit) to av­er­age (e.g. word list re­call) to high (e.g. puz­zle). By this find­ing, the ben­e­fits of spaced prac­tise may be muted for many math­e­mat­ics tasks.

What is es­pe­cially nice about this study was that not only did it use high­-qual­ity (in­tel­li­gent & mo­ti­vat­ed) col­lege stu­dents (United States Air Force Acad­emy), the con­di­tions were rel­a­tively con­trolled - both groups had the same home­work (so equal test­ing effec­t), but like Rohrer & Tay­lor 2006/2007, the dis­tri­b­u­tion was what var­ied:

The course top­ics, text­book, hand­outs, read­ing as­sign­ments, and graded as­sign­ments (with the ex­cep­tion of quiz, home­work, and par­tic­i­pa­tion points) were iden­ti­cal for the treat­ment and con­trol groups. The list­ing of home­work as­sign­ments in the syl­labus differed be­tween groups. The con­trol group was as­signed daily home­work re­lated to the top­ic(s) pre­sented that day in class. Pe­ter­son (1971) calls this the ver­ti­cal model for as­sign­ing math­e­mat­ics home­work. The treat­ment group was as­signed home­work in ac­cor­dance with a dis­trib­uted or­ga­ni­za­tional pat­tern that com­bines prac­tice on cur­rent top­ics and re­in­force­ment of pre­vi­ously cov­ered top­ics. Un­der the dis­trib­uted mod­el, ap­prox­i­mately 40% of the prob­lems on a given topic were as­signed the day the topic was first in­tro­duced, with an ad­di­tional 20% as­signed on the next les­son and the re­main­ing 40% of prob­lems on the topic as­signed on sub­se­quent lessons (Hirsch et al, 1983). In Hirsch’s re­search and in this study, after the ini­tial home­work as­sign­ment, prob­lem(s) rep­re­sent­ing a given topic resur­faced on the 2nd, 4th, 7th, 12th, and 21st les­son. Con­se­quent­ly, treat­ment group home­work for les­son one con­sisted of only one top­ic; home­work for lessons two and three con­sisted of two top­ics; and home­work for les­son four through six con­sisted of three top­ics. This pat­tern con­tin­ued as new top­ics were added and was ap­plied to all non-ex­am, non-lab­o­ra­tory lessons. As shown by Ta­bles 1 and 2, the same home­work prob­lems were as­signed to both groups with only the pat­tern of as­sign­ment differ­ing. Be­cause of the na­ture of the dis­trib­uted prac­tice mod­el, home­work for the treat­ment group con­tained fewer prob­lems (rel­a­tive to the con­trol group) early in the se­mes­ter with the num­ber of prob­lems in­creas­ing as the se­mes­ter pro­gressed. Later in the se­mes­ter, home­work for the treat­ment group con­tained more prob­lems (rel­a­tive to the con­trol group)….The USAFA rou­tinely col­lects study time da­ta. After each ex­am, a large sam­ple of cadets (at least 60% of the course pop­u­la­tion) anony­mously re­ported the amount of time (in min­utes) spent study­ing for the ex­am. Time spent study­ing was ap­prox­i­mately equal for both groups (see Ta­ble 5). De­scrip­tive data rev­els that, for both the treat­ment and con­trol group, study time for the third exam was at least 16% greater than study time for any other ex­am. Study time for the fi­nal exam was at least 68% greater than study time for any of the hourly ex­ams (see Ta­ble 5)

…The treat­ment pro­duced an effect size (f 2) of 0.013 on the first ex­am, 0.029 on the sec­ond ex­am, 0.035 on the fourth ex­am, and 0.040 on the fi­nal course per­cent­age grade. Al­though the effect sizes ap­pear to be small, the treat­ment group outscored the con­trol group in every case. A mean differ­ence of 5.13 per­cent­age points on the first, sec­ond, and fourth exam trans­lates to an ad­van­tage of about a third of a let­ter grade for stu­dents in the treat­ment group. In ad­di­tion, higher min­i­mum scores earned by the treat­ment group may in­di­cate that the dis­trib­uted prac­tice treat­ment served to elim­i­nate the ex­tremely low scores (re­fer to Ta­ble 3)….Odd­ly, the dis­trib­uted prac­tice treat­ment did not pro­duce a [s­ta­tis­ti­cal­ly-]sig­nifi­cant effect on fi­nal exam scores. One pos­si­ble cause for the dis­par­ity was the USAFA pol­icy ex­empt­ing the top per­form­ers from the fi­nal ex­am. Of the 16 ex­empted stu­dents, 11 were from the treat­ment group with only 5 from the con­trol group.

Balch 2006 ab­stract:

Two in­tro­duc­tory psy­chol­ogy classes (N = 145) par­tic­i­pated in a coun­ter­bal­anced class­room ex­per­i­ment that demon­strated the spac­ing effect and, by anal­o­gy, the ben­e­fits of dis­trib­uted study. After hear­ing words pre­sented twice in ei­ther a massed or dis­trib­uted man­ner, par­tic­i­pants re­called the words and scored their re­call pro­to­cols, re­li­ably re­mem­ber­ing more dis­trib­uted than massed words. Posttest scores on a mul­ti­ple-choice quiz cov­er­ing points il­lus­trated by the ex­per­i­ment av­er­aged about twice the com­pa­ra­ble pretest scores, in­di­cat­ing the effec­tive­ness of the ex­er­cise in con­vey­ing con­tent. Stu­dents’ sub­jec­tive rat­ings sug­gested that the ex­per­i­ment helped con­vince them of the ben­e­fits of dis­trib­uted study.

See Cepeda et al 2006↩︎

Com­mins, S., Cun­ning­ham, L., Har­vey, D., and Wal­sh, D. (2003). “Massed but not spaced train­ing im­pairs spa­tial mem­ory”. Be­hav­ioural Brain Re­search 139, 215-223↩︎

Gal­luc­cio & Rovee-Col­lier 2006, “Nonuni­form effects of re­in­state­ment within the time win­dow”. Learn­ing and Mo­ti­va­tion, 37, 1-17.↩︎

See the pre­vi­ous sec­tions for many us­ing chil­dren; one pre­vi­ously uncited is Top­pino 1993, “The spac­ing effect in preschool chil­dren’s free re­call of pic­tures and words”; but Top­pino et al 2009 adds some in­ter­est­ing qual­i­fiers to spaced rep­e­ti­tion in the young:

Preschool­ers, el­e­men­tary school chil­dren, and col­lege stu­dents ex­hib­ited a spac­ing effect in the free re­call of pic­tures when learn­ing was in­ten­tion­al. When learn­ing was in­ci­den­tal and a shal­low pro­cess­ing task re­quir­ing lit­tle se­man­tic pro­cess­ing was used dur­ing list pre­sen­ta­tion, young adults still ex­hib­ited a spac­ing effect, but chil­dren con­sis­tently failed to do so. Chil­dren, how­ev­er, did man­i­fest a spac­ing effect in in­ci­den­tal learn­ing when an elab­o­rate se­man­tic pro­cess­ing task was used.

An­other pre­vi­ously uncited study: Glen­berg, A. M. (1979), “Com­po­nen­t-levels the­ory of the effects of spac­ing of rep­e­ti­tions on re­call and recog­ni­tion”. Mem­ory & Cog­ni­tion, 7, 95-112.↩︎

See Ko­r­nell et al 2010; Si­mone et al 2012 shows the spac­ing ben­e­fits but re­duced in mag­ni­tude in its 56-74 year old sub­jects, sim­i­lar to Jack­son et al 2012 and Mad­dox 2013↩︎

Mam­marel­la, N., Rus­so, R., & Avons, S. E. (2002). "Spac­ing effects in cued-mem­ory tasks for un­fa­mil­iar faces and non­words". Mem­ory & Cog­ni­tion, 30, 1238-1251↩︎

Childers, J. B., & Tomasel­lo, M. (2002). "T­wo-year-olds learn novel nouns, verbs, and con­ven­tional ac­tions from massed or dis­trib­uted ex­po­sures". De­vel­op­men­tal Psy­chol­ogy, 38, 967-978↩︎

eg. Fish­man et al 1968↩︎

The fa­mous ‘10,000 hours of prac­tice’ fig­ure may not be as true or im­por­tant as Er­ic­s­son and pub­li­ciz­ers like Mal­colm Glad­well im­ply, given the high vari­ance of ex­per­tise against time, and re­sults from sports show­ing smaller time in­vest­ments (see also Ham­brick’s cor­pus cut­ting ‘de­lib­er­ate prac­tice’ down to size), and Er­ic­s­son ab­surdly deny the pow­er­ful role of ge­net­ics and the nec­es­sary con­di­tion of hav­ing tal­ent but the in­sight of ‘de­lib­er­ate prac­tice’ help­ing tal­ented peo­ple prob­a­bly is re­al. One may be able to get away with 3,000 hours rather than 10,000, but one is­n’t go­ing to do that with mind­less rep­e­ti­tion or no rep­e­ti­tions.↩︎

Gen­tner, D., Loewen­stein, J., & Thomp­son, L. (2003). “Learn­ing and trans­fer: A gen­eral role for ana­log­i­cal en­cod­ing”. Jour­nal of Ed­u­ca­tional Psy­chol­ogy, 95, 393-40↩︎

From Ko­r­nell et al 2010:

The ben­e­fits of spac­ing seem to di­min­ish or dis­ap­pear when to-be-learned items are not re­peated ex­actly (Ap­ple­ton-K­napp, Bjork, & Wick­ens, 2005)…a num­ber of stud­ies have shown that mass­ing, rather than spac­ing, pro­motes in­duc­tive learn­ing. These stud­ies have gen­er­ally em­ployed rel­a­tively sim­ple per­cep­tual stim­uli that fa­cil­i­tate ex­per­i­men­tal con­trol (Gag­né, 1950; Gold­stone, 1996; Kurtz & Hov­land, 1956; [Whit­man J. R., & Gar­ner, W. R. (1963). “Con­cept learn­ing as a func­tion of the form of in­ter­nal struc­ture”. Jour­nal of Ver­bal Learn­ing & Ver­bal Be­hav­ior, 2, 195-202]).

High er­ror rates - in­di­cat­ing one did­n’t ac­tu­ally learn the card con­tents in the first place - seem to be con­nected to fail­ures of the spac­ing effect; there’s some ev­i­dence that peo­ple nat­u­rally choose to mass study when they don’t yet know the ma­te­r­i­al.↩︎

“Su­per­Memo as a new tool in­creas­ing the pro­duc­tiv­ity of a pro­gram­mer. A case study: pro­gram­ming in Ob­ject Win­dows”↩︎

The 20 years look like this (note the sci­en­tific no­ta­tion): [0.742675, 0.27044575182838654, 0.15275979054767388, 0.10348750000000001, 7.751290630254386e-2, 6.187922936397532e-2, 5.161829250474865e-2, 4.445884397854832e-2, 3.923055555555555e-2, 3.5275438307530015e-2, 3.219809429218694e-2, 2.9748098818459235e-2, 2.7759942051635768e-2, 2.6120309801216147e-2, 2.474928593068675e-2, 2.35890625e-2, 2.2596898475825956e-2, 2.1740583401051353e-2, 2.0995431241707652e-2, 2.0342238287817983e-2]↩︎

mod­ulo things where know­ing it is use­ful even if you don’t need it often - it can be a brick in a pyra­mid of knowl­edge; cf.page 3 of Wolf:

The prob­lem of for­get­ting might not tor­ment us so much if we could only con­vince our­selves that re­mem­ber­ing is­n’t im­por­tant. Per­haps the things we learn - words, dates, for­mu­las, his­tor­i­cal and bi­o­graph­i­cal de­tails - don’t re­ally mat­ter. Facts can be looked up. That’s what the In­ter­net is for. When it comes to learn­ing, what re­ally mat­ters is how things fit to­geth­er. We mas­ter the sto­ries, the schemas, the frame­works, the par­a­digms; we re­hearse the lin­go; we swim in the epis­teme.

The dis­ad­van­tage of this com­fort­ing no­tion is that it’s false. “The peo­ple who crit­i­cize mem­o­riza­tion - how happy would they be to spell out every let­ter of every word they read?” asks Robert Bjork, chair of UCLA’s psy­chol­ogy de­part­ment and one of the most em­i­nent mem­ory re­searchers. After all, Bjork notes, chil­dren learn to read whole words through in­tense prac­tice, and every time we en­ter a new field we be­come chil­dren again. “You can’t es­cape mem­o­riza­tion,” he says. “There is an ini­tial process of learn­ing the names of things. That’s a stage we all go through. It’s all the more im­por­tant to go through it rapid­ly.” The hu­man brain is a mar­vel of as­so­cia­tive pro­cess­ing, but in or­der to make as­so­ci­a­tions, data must be loaded into mem­o­ry.

See Stephen R. Schmidt’s web­page “The­o­ries of For­get­ting”, which cites ‘Wood­worth & Schlos­beg (1961)’ when pre­sent­ing a log graph of var­i­ous stud­ies’ for­get­ting curves.↩︎

which neatly ad­dresses the is­sue of such mail­ing lists be­ing use­less (‘who learns a word after just one ex­po­sure?’).↩︎

Mnemosyne in this case con­sti­tutes both a way to learn the quotes so I can use them, and a waste book; just the other day I had 3 or 4 ap­po­site quotes for an es­say be­cause I had en­tered them into Mnemosyne months or years ago.↩︎

It’s well known that any speaker of a lan­guage un­der­stands many more words than they will ever use or be able to ex­plic­itly gen­er­ate, that their “read­ing vo­cab­u­lary” ex­ceeds their “writ­ing vo­cab­u­lary”; less well-known is that on many prob­lems, one can guess at well above ran­dom rates even while feel­ing un­sure & ig­no­rant, ne­ces­si­tat­ing psy­chol­o­gists to em­ploy forced-choice par­a­digms. Even less known is the ca­pac­ity of recog­ni­tion mem­ory or “im­plicit mem­ory”; this mem­ory can ap­ply to things like rec­og­niz­ing im­ages or text or mu­sic, typ­ing, puz­zle solv­ing, etc. An­drew Druck­er, in “Mul­ti­ply­ing 10-digit num­bers us­ing Flickr: The power of recog­ni­tion mem­ory”, em­ploys vi­sual mem­ory to cal­cu­late $9883603368\cdot 4288997768=42390752785149282624$; he cites as prece­dent Stand­ing 1973:

In one of the most wide­ly-cited stud­ies on recog­ni­tion mem­o­ry, Stand­ing showed par­tic­i­pants an epic 10,000 pho­tographs over the course of 5 days, with 5 sec­onds’ ex­po­sure per im­age. He then tested their fa­mil­iar­i­ty, es­sen­tially as de­scribed above. The par­tic­i­pants showed an 83% suc­cess rate, sug­gest­ing that they had be­come fa­mil­iar with about 6,600 im­ages dur­ing their or­deal. Other vol­un­teers, trained on a smaller col­lec­tion of 1,000 im­ages se­lected for vivid­ness, had a 94% suc­cess rate.

One some­times sees peo­ple ar­gue that some­thing is in­se­cure or unguess­able or free from pos­si­ble placebo effect be­cause it in­volves too many ob­jects to ex­plic­itly mem­o­rize, but as these ex­am­ples make clear, recog­ni­tion mem­ory can hap­pen quickly and store sur­pris­ingly large amounts of in­for­ma­tion. This could be used for au­then­ti­ca­tion (see for ex­am­ple Bo­ji­nov et al 2012; HN dis­cus­sion) or mes­sage since recog­ni­tion mem­ory could be ex­ploited as a sort of se­cure com­mu­ni­ca­tion sys­tem. Two par­ties can share a set of 20,000 pho­tographs (10,000 pairs); to send a mes­sage, have a mes­sen­ger spend 5 days on 10,000 picked ones; and then to re­ceive it, ask him to rec­og­nize which pho­to­graph he saw in each of the 10,000 pairs. The sub­ject not only does not know what the bi­nary mes­sage is or what means, he can’t even pro­duce it since he can­not re­mem­ber the pho­tographs!

At an 80% ac­cu­racy rate, we can even cal­cu­late how many bits of in­for­ma­tion can be en­trusted to the mes­sen­ger us­ing Shan­non’s the­o­rem; a cal­cu­la­tion gives 5.8 kilo­bits as the up­per lim­it: if p = 0.2 (based on the 80% suc­cess rate), then $\frac{10000}{1-(p\cdot lo{g}_{2}p+(1-p)\times (lo{g}_2(1-p)\left)\right)}=5807.44$. So we see that Frank Her­bert was right after all: the se­curest way to send a mes­sage is through a dis­trans mes­sen­ger. (The down­side is that the im­plicit recog­ni­tion mem­ory de­cays con­sid­er­ably; see Lan­dauer 1986 for ad­justed es­ti­mates.)↩︎

In this vein, I am re­minded of what a for­mer polypha­sic sleeper told me:

I’ve been polypha­sic for about a year. (Not any­more; kills my mem­o­ry.)…Anki reps, most­ly. I found that I could do proper re­view ses­sions for about 2-3 days and would hit an im­pen­e­tra­ble wall. I could­n’t learn a sin­gle new card and had to­tal brain fog un­til I got 3 hours more sleep. That, how­ev­er, would re­set my adap­ta­tion. The whole effect is a bit less pro­nounced on Every­man, but not much. It is how­ever eas­ier to add sleep when you al­ready have a core. I did­n’t no­tice any other ma­jor men­tal im­pair­ment after the ini­tial sleep de­pri­va­tion.

For a more re­cent re­view, see Philips et al 2013.↩︎

Pre­sum­ably one would im­me­di­ately give them all some high grade like 5 to avoid sud­denly hav­ing a daily load of 500 cards for a while.↩︎

Smaller is bet­ter.↩︎

“For Mnemosyne 2.x, Ull­rich is work­ing on an offi­cial Mnemosyne iPhone client which will have very easy sync­ing.”↩︎

Wired↩︎

See Page 4, Wolf 2008:

The spac­ing effect was one of the proud­est lab-derived dis­cov­er­ies, and it was in­ter­est­ing pre­cisely be­cause it was not ob­vi­ous, even to pro­fes­sional teach­ers. The same year that Neisser re­volt­ed, Robert Bjork, work­ing with Thomas Lan­dauer of Bell Labs, pub­lished the re­sults of two ex­per­i­ments in­volv­ing nearly 700 un­der­grad­u­ate stu­dents. Lan­dauer and Bjork were look­ing for the op­ti­mal mo­ment to re­hearse some­thing so that it would later be re­mem­bered. Their re­sults were im­pres­sive: The best time to study some­thing is at the mo­ment you are about to for­get it. And yet - as Neisser might have pre­dicted - that in­sight was use­less in the real world.

When I first read of Su­per­Me­mo, I had al­ready taken a class in cog­ni­tive psy­chol­ogy and was rea­son­ably fa­mil­iar with Ebbing­haus’s for­get­ting curve - so my re­ac­tion to its method­ol­ogy was Hux­ley’s: “How ex­tremely stu­pid not to have thought of that!”↩︎

See page 7, Wolf 2008

And yet now, as I grin broadly and wave to the gawk­ers, it oc­curs to me that the cold ra­tio­nal­ity of his ap­proach may be only a sur­face fea­ture and that, when linked to gen­uine re­wards, even the chill­i­est of sys­tems can have a cer­tain vis­ceral ap­peal. By pro­ject­ing the achieve­ment of ex­treme mem­ory back along the for­get­ting curve, by prov­ably link­ing the dis­tant fu­ture - when we will know so much - to the few min­utes we de­vote to study­ing to­day, Woz­niak has found a way to con­di­tion his tem­pera­ment along with his mem­o­ry. He is mak­ing the fu­ture no­tice­able. He is try­ing not just to learn many things but to warm the process of learn­ing it­self with a draft of utopian ec­sta­sy.