Efficient memorization using the spacing effect: literature review of widespread applicability, tips on use & what it’s good for.
2009-03-11–2019-05-17
finished
certainty: highly likely
importance: 9
Spaced repetition is a centuries-old psychological technique for efficient memorization & practice of skills where instead of attempting to memorize by ‘cramming’, memorization can be done far more efficiently by instead spacing out each review, with increasing durations as one learns the item, with the scheduling done by software. Because of the greater efficiency of its slow but steady approach, spaced repetition can scale to memorizing hundreds of thousands of items (while crammed items are almost immediately forgotten) and is especially useful for foreign languages & medical studies.
I review what this technique is useful for, some of the large research literature on it and the testing effect (up to ~2013, primarily), the available software tools and use patterns, and miscellaneous ideas & observations on it.
One of the most fruitful areas of computing is making up for human frailties. They do arithmetic perfectly because we can’t1. They remember terabytes because we’d forget. They make the best calendars, because they always check what there is to do today. Even if we do not remember exactly, merely remembering a reference can be just as good, like the point of reading a manual or textbook all the way through: it is not to remember everything that is in it for later but to later remember that something is in it (and skimming them, you learn the right words to search for when you actually need to know more about a particular topic).
We use any number of such neuroprosthetics2, but there are always more to be discovered. They’re worth looking for because they are so valuable: a shovel is much more effective than your hand, but a power shovel is orders of magnitude better than both - even if it requires training and expertise to use.
Spacing effect
“You can get a good deal from rehearsal,
If it just has the proper dispersal.
You would just be an ass,
To do it en masse,
Your remembering would turn out much worsal.”Ulrich Neisser3
My current favorite prosthesis is the class of software that exploits the spacing effect, a centuries-old observation in cognitive psychology, to achieve results in studying or memorization much better than conventional student techniques; it is, alas, obscure4.
The spacing effect essentially says that if you have a question (“What is the fifth letter in this random sequence you learned?”), and you can only study it, say, 5 times, then your memory of the answer (‘e’) will be strongest if you spread your 5 tries out over a long period of time - days, weeks, and months. One of the worst things you can do is blow your 5 tries within a day or two. You can think of the ‘forgetting curve’ as being like a chart of a radioactive half-life: each review bumps your memory up in strength 50% of the chart, say, but review doesn’t do much in the early days because the memory simply hasn’t decayed much! (Why does the spacing effect work, on a biological level? There are clear neurochemical differences between massed and spaced in animal models with spacing (>1 hour) enhancing long-term potentiation but not massed5, but the why and wherefore - that’s an open question; see the concept of memory traces or the sleep studies.) A graphical representation of the forgetting curve:

Even better, it’s known that active recall is a far superior method of learning than simply passively being exposed to information.6 Spacing also scales to huge quantities of information; gambler/
A graphic might help; imagine here one can afford to review a given piece of information a few times (one is a busy person). By looking at the odds we can remember the item, we can see that cramming wins in the short term, but unexercised memories decay so fast that after not too long spacing is much superior:

It’s more dramatic if we look at a video visualizing decay of a corpus of memory with random review vs most-recent review vs spaced review.
If you’re so good, why aren’t you rich
Most people find the concept of programming obvious, but the doing impossible.10
Of course, the latter strategy (cramming) is precisely what students do. They cram the night before the test, and a month later can’t remember anything. So why do people do it? (I’m not innocent myself.) Why is spaced repetition so dreadfully unpopular, even among the people who try it once?11

Because it does work. Sort of. Cramming is a trade-off: you trade a strong memory now for weak memory later. (Very weak12.) And tests are usually of all the new material, with occasional old questions, so this strategy pays off! That’s the damnable thing about it - its memory longevity & quality are, in sum, less than that of spaced repetition, but cramming delivers its goods now13. So cramming is a rational, if short-sighted, response, and even SRS software recognize its utility & support it to some degree14. (But as one might expect, if the testing is continuous and incremental, then the learning tends to also be long-lived15; I do not know if this is because that kind of testing is a disguised accidental spaced repetition system, or the students/
Across experiments, spacing was more effective than massing for 90% of the participants, yet after the first study session, 72% of the participants believed that massing had been more effective than spacing….When they do consider spacing, they often exhibit the illusion that massed study is more effective than spaced study, even when the reverse is true (Dunlosky & Nelson, 1994; Kornell & Bjork, 2008a; Simon & Bjork 2001; Zechmeister & Shaughnessy, 1980).
As one would expect if the testing and spacing effects are real things, students who naturally test themselves and study well in advance of exams tend to have higher GPAs.19 If we interpret questions as tests, we are not surprised to see that 1-on-1 tutoring works dramatically better than regular teaching and that tutored students answer orders of magnitude more questions20.
This short-term perspective is not a good thing in the long term, of course. Knowledge builds on knowledge; one is not learning independent bits of trivia. Richard Hamming recalls in “You and Your Research”:
You observe that most great scientists have tremendous drive. I worked for ten years with John Tukey at Bell Labs. He had tremendous drive. One day about three or four years after I joined, I discovered that John Tukey was slightly younger than I was. John was a genius and I clearly was not. Well I went storming into Bode’s office and said, “How can anybody my age know as much as John Tukey does?” He leaned back in his chair, put his hands behind his head, grinned slightly, and said, “You would be surprised Hamming, how much you would know if you worked as hard as he did that many years.” I simply slunk out of the office!
What Bode was saying was this: “Knowledge and productivity are like compound interest.” Given two people of approximately the same ability and one person who works 10% more than the other, the latter will more than twice outproduce the former. The more you know, the more you learn; the more you learn, the more you can do; the more you can do, the more the opportunity - it is very much like compound interest. I don’t want to give you a rate, but it is a very high rate. Given two people with exactly the same ability, the one person who manages day in and day out to get in one more hour of thinking will be tremendously more productive over a lifetime. I took Bode’s remark to heart; I spent a good deal more of my time for some years trying to work a bit harder and I found, in fact, I could get more work done.
Knowledge needs to accumulate, and flashcards with spaced repetition can aid in just that accumulation, fostering steady review even as the number of cards and intellectual prerequisites mounts into the thousands.
This long term focus may explain why explicit spaced repetition is an uncommon studying technique: the pay-off is distant & counterintuitive, the cost of self-control near & vivid. (See hyperbolic discounting.) It doesn’t help that it’s pretty difficult to figure out when one should review - the optimal point is when you’re just about to forget about it, but that’s the kicker: if you’re just about to forget about it, how are you supposed to remember to review it? You only remember to review what you remember, and what you already remember isn’t what you need to review!21
The paradox is resolved by letting a computer handle all the calculations. We can thank Hermann Ebbinghaus for investigating in such tedious detail than we can, in fact, program a computer to calculate both the forgetting curve and optimal set of reviews22. This is the insight behind spaced repetition software: ask the same question over and over, but over increasing spans of time. You start with asking it once every few days, and soon the human remembers it reasonably well. Then you expand intervals out to weeks, then months, and then years. Once the memory is formed and dispatched to long-term memory, it needs but occasional exercise to remain hale and hearty23 - I remember well the large dinosaurs made of cardboard for my 4th or 5th birthday, or the tunnel made out of boxes, even though I recollect them once or twice a year at most.
Literature review
But don’t take my word for it - Nullius in verba! We can look at the science. Of course, if you do take my word for it, you probably just want to read about how to use it and all the nifty things you can do, so I suggest you skip all the way down to that section. Everyone else, we start at the beginning:
Background: testing works!
“If you read a piece of text through twenty times, you will not learn it by heart so easily as if you read it ten times while attempting to recite from time to time and consulting the text when your memory fails.” –The New Organon, Francis Bacon
The testing effect is the established psychological observation that the mere act of testing someone’s memory will strengthen the memory (regardless of whether there is feedback). Since spaced repetition is just testing on particular days, we ought to establish that testing works better than regular review or study, and that it works outside of memorizing random dates in history. To cover a few papers:
Allen, G.A., Mahler, W.A., & Estes, W.K. (1969). “Effects of recall tests on long-term retention of paired associates”. Journal of Verbal Learning and Verbal Behavior, 8, 463-470
1 test results in memories as strong a day later as studying 5 times; intervals improve retention compared to massed presentation.
Karpicke & Roediger (2003). “The Critical Importance of Retrieval for Learning”
In learning Swahili vocabulary, students were given varying routines of testing or studying or testing and studying; this resulted in similar scores during the learning phase. Students were asked to predict what percentage they’d remember (average: 50% over all groups). One week later, the students who tested remembered ~80% of the vocabulary versus ~35% for non-testing students. Some students were tested or studied more than others; diminishing returns set in quickly once the memory had formed the first day. Students reported rarely testing themselves and not testing already learned items.
Lesson: again, testing improves memory compared to studying. Also, no student knows this.
Roediger & Karpicke (2006a). “Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention”
Students were tested (with no feedback) on reading comprehension of a passage over 5 minutes, 2 days, and 1 week. Studying beat testing over 5 minutes, but nowhere else; students believed studying superior to testing over all intervals. At 1 week, testing scores were ~60% versus ~40%.
Lesson: testing improves memory compared to studying. Everyone (teachers & students) ‘knows’ the opposite.
Karpicke & Roediger (2006a). “Expanding retrieval promotes short-term retention, but equal interval retrieval enhances long-term retention”
General scientific prose comprehension; from Roediger & Karpicke 2006b: “After 2 days, initial testing produced better retention than restudying (68% vs. 54%), and an advantage of testing over restudying was also observed after 1 week (56% vs. 42%).”
Roediger & Karpicke (2006b). “The Power of Testing Memory: Basic Research and Implications for Educational Practice”
Literature review; 7 studies before 1941 demonstrating testing improves retention, and 6 afterwards. See also the reviews “Spacing Learning Events Over Time: What the Research Says” & “Using spacing to enhance diverse forms of learning: Review of recent research and implications for instruction”, Carpenter et al 2012.
Agarwal et al 2008, “Examining the Testing Effect with Open- and Closed-Book Tests”
As with #2, the purer forms of testing (in this case, open-book versus closed-book testing) did better over the long run, and students were deluded about what worked best.
Bangert-Drowns et al 1991. “Effects of frequent classroom testing”
Meta-analysis of 35 studies (1929-1989) varying tests during school semesters. 29 found benefits; 5 found negatives; 1 null result. Meta-study found large benefits to testing even once, then diminishing returns.
Cook 2006, “Impact of self-assessment questions and learning styles in Web-based learning: a randomized, controlled, crossover trial”; final scores were higher when the doctors (residents) learned with questions.
Johnson & Kiviniemi 2009, “The Effect of Online Chapter Quizzes on Exam Performance in an Undergraduate Social Psychology Course” (“This study examined the effectiveness of compulsory, mastery-based, weekly reading quizzes as a means of improving exam and course performance. Completion of reading quizzes was related to both better exam and course performance.”); see also McDaniel et al 2012.
Metsämuuronen 2013, “Effect of Repeated Testing on the Development of Secondary Language Proficiency”
Meyer & Logan 2013, “Taking the Testing Effect Beyond the College Freshman: Benefits for Lifelong Learning”; verifies testing effect in older adults has similar effect size as younger
Larsen & Butler 2013, “Test-enhanced learning”
(One might be tempted to object that testing works for some learning styles, perhaps verbal styles. This is an unsupported assertion inasmuch as the experimental literature on learning styles is poor and the existing evidence mixed that there are such things as learning styles.24)
Subjects
The above studies often used pairs of words or words themselves. How well does the testing effect generalize?
Materials which benefited from testing:
- foreign vocabulary (eg. Karpicke & Roediger 2003, Cepeda et al 2009, Fritz et al 200725, de la Rouviere 2012)
- GRE materials (like vocab, Kornell 2009); prose passages on general scientific topics (Karpicke & Roediger, 2006a; Pashler et al, 2003)
- trivia (McDaniel & Fisher 1991)
- elementary & middle school lessons with subjects such as biographical material and science (Gates 1917; Spitzer 193926 and Vlach & Sandhofer 201227, respectively)
- Agarwal et al (2008): short-answer tests superior on textbook passages
- history textbooks; retention better with initial short-answer test rather than multiple choice (Nungester & Duchastel 1982)
- LaPorte & Voss (1975) also found better retention compared to multiple-choice or recognition problems
- Duchastel & Nungester, 1981: 6 months after testing, testing beat studying in retention of a history passage
- Duchastel (1981): free recall decisively beat short-answer & multiple choice for reading comprehension of a history passage
- Glover (1989): free recall self-test beat recognition or Cloze deletions; subject matter was the labels for parts of flowers
- Kang, McDermott, and Roediger (2007): prose passages; initial short answer testing produced superior results 3 days later on both multiple choice and short answer tests
- Leeming (2002): tests in 2 psychology courses, introductory & memory/
learning; “80% vs. 74% for the introductory psychology course and 89% vs. 80% for the learning and memory course”28
This covers a pretty broad range of what one might call ‘declarative’ knowledge. Extending testing to other fields is more difficult and may reduce to ‘write many frequent analyses, not large ones’ or ‘do lots of small exercises’, whatever those might mean in those fields:
A third issue, which relates to the second, is whether our proposal of testing is really appropriate for courses with complex subject matters, such as the philosophy of Spinoza, Shakespeare’s comedies, or creative writing. Certainly, we agree that most forms of objective testing would be difficult in these sorts of courses, but we do believe the general philosophy of testing (broadly speaking) would hold-students should be continually engaged and challenged by the subject matter, and there should not be merely a midterm and final exam (even if they are essay exams). Students in a course on Spinoza might be assigned specific readings and thought-provoking essay questions to complete every week. This would be a transfer-appropriate form of weekly ‘testing’ (albeit with take-home exams). Continuous testing requires students to continuously engage themselves in a course; they cannot coast until near a midterm exam and a final exam and begin studying only then.29
Downsides
Testing does have some known flaws:
interference in recall - ability to remember tested items drives out ability to remember similar untested items
Most/
all studies were in laboratory settings and found relatively small effects: In sum, although various types of recall interference are quite real (and quite interesting) phenomena, we do not believe that they compromise the notion of test-enhanced learning. At worst, interference of this sort might dampen positive testing effects somewhat. However, the positive effects of testing are often so large that in most circumstances they will overwhelm the relatively modest interference effects.
multiple choice tests can accidentally lead to ‘negative suggestion effects’ where having previously seen a falsehood as an item on the test makes one more likely to believe it.
This is mitigated or eliminated when there’s quick feedback about the right answer (see Butler & Roediger 2008 “Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing”). Solution: don’t use multiple choice; inferior in testing ability to free recall or short answers, anyway.
Neither problem seems major.
Distributed
A lot depends on when you do all your testing. Above we saw some benefits to testing a lot the moment you learn something, but the same number of tests could be spread out over time, to give us the spacing effect or spaced repetition. There are hundreds of studies involving the spacing effect:
- Cepeda et al 2006 is a review of 184 articles with 317 experiments; other reviews include:
- Ruch 1928, “Factors influencing the relative economy of massed and distributed practice in learning”
- Crowder 1976, Principles of learning and memory
- Dempster 1989, “Spacing effects and their implications for theory and practice”
- Delaney et al 2010, “Spacing and testing effects: A deeply critical, lengthy, and at times discursive review of the literature”
- Donovan & Radosevich 1999, “A meta-analytic review of the distribution of practice effect: Now you see it, now you don’t”
- Greene 1992, Human memory: Paradigms and paradoxes
- Janiszewski et al 2003, “A meta-analysis of the spacing effect in verbal learning: Implications for research on advertising repetition and consumer memory”
- Pavlik & Anderson 2003, “An ACT-R model of the spacing effect”
- Balota et al 2006, “Is Expanded Retrieval Practice a Superior Form of Spaced Retrieval? A Critical Review of the Extant Literature”
- Carpenter et al 2012, “Using Spacing to Enhance Diverse Forms of Learning: Review of Recent Research and Implications for Instruction”
Almost unanimously they find spacing out tests is superior to massed testing when the final test/
- Peterson, L. R., Wampler, R., Kirkpatrick, M., & Saltzman, D. (1963). “Effect of spacing presentations on retention of a paired associate over short intervals”. Journal of Experimental Psychology, 66(2), 206-209
- Glenberg, A. M. (1977). “Influences of retrieval processes on the spacing effect in free recall”. Journal of Experimental Psychology: Human Learning and Memory, 3(3), 282-294
- Balota et al 1989, “Age-related differences in the impact of spacing, lag and retention interval”. Psychology and Aging, 4, 3-9
The research literature focuses extensively on the question of what kind of spacing is best and what this implies about memory: a spacing that has static fixed intervals or a spacing which expands? This is important for understanding memory and building models of it, and would be helpful for integrating spaced repetition into classrooms (for example, Kelley & Whatson 2013’s 10 minutes studying /
For those interested, 3 of the studies that found fixed spacings better than expanding:
Carpenter, S. K., & DeLosh, E. L. (2005). “Application of the testing and spacing effects to name learning”. Applied Cognitive Psychology, 19, 619-63633
Logan, J. M. (2004). Spaced and expanded retrieval effects in younger and older adults. Unpublished doctoral dissertation, Washington University, St. Louis, MO
This thesis is interesting inasmuch as Logan found that young adults did considerably worse with an expanding spacing after a day.
Karpicke & Roediger, 2006a
The fixed vs expanding issue aside, a list of additional generic studies finding benefits to spaced vs massed:
Cepeda et al 2006 (large review used elsewhere in this page)
Karpicke & Roediger 2006a
Rohrer & Taylor 2006. “The effects of over-learning and distributed practice on the retention of mathematics knowledge”. Applied Cognitive Psychology, 20: 1209-1224 (see also Rohrer & Taylor 2007, Rohrer et al 2005)
Seabrook et al 2005. “Distributed and Massed Practice: From Laboratory to Classroom”
Keppel, Geoffrey. “A Reconsideration of the Extinction-Recovery Theory”. Journal of Verbal Learning & Verbal Behavior. 6(4) 1967, 476-486
A week later, the massed reviewers went from 5.9 correct → 2.1; the spaced reviewers went from 5.5 → 5.0. (Note the usual observation: massed was initially better, and later much worse, less than half as good.)
Bloom & Schuell 1981, “Effects of massed and distributed practice on the learning and retention of second-language vocabulary”
Four days after the 2 high school groups memorized 16 French words, the spaced group remembered 15 and the massed 11.
Rea, Cornelius P; Modigliani, Vito. “The effect of expanded versus massed practice on the retention of multiplication facts and spelling lists”. Human Learning: Journal of Practical Research & Applications. Vol 4(1) Jan-Mar 1985, 11-1834
A test immediately following the training showed superior performance for the distributed group (70% correct) compared to the massed group (53% correct). These results seem to show that the spacing effect applies to school-age children and to at least some types of materials that are typically taught in school.35
Donovan & Radosevich 1999, “A meta-analytic review of the distribution of practice effect: Now you see it, now you don’t”:
According to Donovan and Radosevich’s meta-analysis of spacing studies, the effect size for the spacing effect is d = 0.42. This means that the average person getting distributed training remembers better than about 67% of the people getting massed training. This effect size is nothing to sneeze at-in education research, effect sizes as low as d = 0.25 are considered “practically significant”, while effect sizes above d = 1 are rare.36
In one meta-analysis by Donovan and Radosevich (1999), for instance, the size of the spacing effect declined sharply as conceptual difficulty of the task increased from low (e.g. rotary pursuit) to average (e.g. word list recall) to high (e.g. puzzle). By this finding, the benefits of spaced practise may be muted for many mathematics tasks.37
The Donovan meta-analysis notes that the effect size is smaller in studies with better methodology, but still important.
Bahrick, Harry P; Phelphs, Elizabeth. “Retention of Spanish vocabulary over 8 years”. Journal of Experimental Psychology: Learning, Memory, & Cognition. Vol 13(2) April 1987, 344-349; the extremely long delay after the initial training period makes this particularly interesting:
Harry Bahrick and Elizabeth Phelps (1987) examined the retention of 50 Spanish vocabulary words after an eight-year delay. Subjects were divided into three groups. Each practiced for seven or eight sessions, separated by a few minutes, a day, or 30 days. In each session, subjects practiced until they could produce the list perfectly one time….Eight years later, people in the no-delay group could recall 6% of the words, people in the one-day delay group could remember 8%, and those in the 30-day group averaged 15%. Everyone also took a multiple choice test, and again, the spacing effect was observed. The no-delay group scored 71%, the one-day group scored 80%, and the 30-day group scored 83%.
…Bahrick and his colleagues varied both the spacing of practice and the amount of practice. Practice sessions were spaced 14, 28, or 56 days apart, and totaled 13 or 26 sessions. They tested subjects’ memory one, two, three, and five years after training. Once again, it took a bit longer to reach the criterion within each session when practice sessions were spaced farther apart, but again, this small investment paid dividends years later. It didn’t matter whether testing occurred at one, two, three, or five years after practice-the 56-day group always remembered the most, the 28-day group was next, and the 14-day group remembered the least. Further, the effect was quite large. If words were practiced every 14 days, you needed twice as much practice to reach the same level of performance as when words were practiced every 56 days!
Pashler et al, 2003; “Is Temporal Spacing of Tests Helpful Even When It Inflates Error Rates?”
Long intervals between tests necessarily means you will often err; errors were thought to intrinsically reduce learning. While the extra errors do damage accuracy in the short-run, the long intervals are powerful enough that they still win.
works in ill subpopulations:
works on short-term review conducted with Alzheimer’s patients; spacing used on the scale of seconds and minutes, with modest success in teaching object locations or daily tasks to do38:
- Camp, C. J. (1989). “Facilitation of new learning in Alzheimer’s disease”. In G. C. Gilmore, P. J. Whitehouse, & M. L. Wykle (Eds.), Memory, aging, and dementia (pp. 212-225)
- Camp, C. J., & McKitrick, L. A. (1992). “Memory interventions in Alzheimer’s-type dementia populations: Methodological and theoretical issues”. In R. L. West & J. D. Sinnott (Eds.), Everyday memory and aging: Current research and methodology (pp. 152-172) -
works with traumatic brain injury; Goverover et al 2009, “Application of the spacing effect to improve learning and memory for functional tasks in traumatic brain injury: a pilot study”
and multiple sclerosis; Goverover et al 2009, “A functional application of the spacing effect to improve learning and memory in persons with multiple sclerosis”
math39:
- multiplication (Ria & Modigliani 1985)
- permuting a sequence (Rohrer & Taylor 2006)on
- calculating the volume of polyhedrons (Rohrer & Taylor 2007)
- statistics (Smith & Rothkopf 1984)
- pre-calculus (Revak 199740 but there’s a related null ‘calculus I’ result as well) and algebra (Mayfield & Chase 2002, Patac & Patac 2013; possible null, Sutherland 2013)
medicine (Kerfoot & Brotschi 2009, Shaw et al 2012; Kerfoot 2009, a 2 year followup to Kerfoot et al 2007 and Kerfoot has a number of other relevant studies; Gyorki et al 2013) and surgery (Moulton et al 2006, “Teaching Surgical Skills: What Kind of Practice Makes Perfect? A Randomized, Controlled Trial”, distributed practice of microvascular suturing; Spruit et al 2014)
introductory psychology (Balch 2006, “Encouraging Distributed Study: A Classroom Experiment on the Spacing Effect”41. Teaching of Psychology, 33, 249-252)
8th-grade American history (Carpenter, Pashler, and Cepeda 2009)
learning to read with phonics (Seabrook et al 2005)
music (Stambaugh 2009)
biology (middle school; Kelly & Whatson 2013)
statistics (introductory; Maas et al 2015)
memorizing website passwords (Bonneau & Schechter 2014, Blocki et al 2014, Blum & Vempala 2017)
possibly not Australian constitutional law (Colbran et al 2015)
Generality of spacing effect
We have already seen that spaced repetition is effective on a variety of academic fields and mediums. Beyond that, spacing effects can be found in:
- various “domains (e.g., learning perceptual motor tasks or learning lists of words)”42 such as spatial43
- “across species (e.g., rats, pigeons, and humans [or flies or bumblebees, and sea slugs, Carew et al 1972 & Sutton et al 2002])”
- “across age groups [infancy44, childhood45, adulthood46, the elderly47] and individuals with different memory impairments”
- “and across retention intervals of seconds48 [to days49] to months” (we have already seen studies using years)
The domains are limited, however. Cepeda et al 2006:
[Moss 1995, reviewing 120 articles] concluded that longer ISIs facilitate learning of verbal information (e.g., spelling50) and motor skills (e.g., mirror tracing); in each case, over 80% of studies showed a distributed practice benefit. In contrast, only one third of intellectual skill (e.g., math computation) studies showed a benefit from distributed practice, and half showed no effect from distributed practice.
…[Donovan and Radosevich (1999)] The largest effect sizes were seen in low rigor studies with low complexity tasks (e.g., rotary pursuit, typing, and peg reversal), and retention interval failed to influence effect size. The only interaction Donovan and Radosevich examined was the interaction of ISI and task domain. It is important to note that task domain moderated the distributed practice effect; depending on task domain and lag, an increase in ISI either increased or decreased effect size. Overall, Donovan and Radosevich found that increasingly distributed practice resulted in larger effect sizes for verbal tasks like free recall, foreign language, and verbal discrimination, but these tasks also showed an inverse-U function, such that very long lags produced smaller effect sizes. In contrast, increased lags produced smaller effect sizes for skill tasks like typing, gymnastics, and music performance.
Skills like gymnastics and music performance raise an important point about the testing effect and spaced repetition: they are for the maintenance of memories or skills, they do not increase it beyond what was already learned. If one is a gifted amateur when one starts reviewing, one remains a gifted amateur. Ericsson covers what is necessary to improve and attain new expertise: deliberate practice51. From “The Role of Deliberate Practice”:
The view that merely engaging in a sufficient amount of practice—regardless of the structure of that practice—leads to maximal performance, has a long and contested history. In their classic studies of Morse Code operators, Bryan and Harter (1897, 1899) identified plateaus in skill acquisition, when for long periods subjects seemed unable to attain further improvements. However, with extended efforts, subjects could restructure their skill to overcome plateaus…Even very experienced Morse Code operators could be encouraged to dramatically increase their performance through deliberate efforts when further improvements were required…More generally, Thorndike (1921) observed that adults perform at a level far from their maximal level even for tasks they frequently carry out. For instance, adults tend to write more slowly and illegibly than they are capable of doing…The most cited condition [for optimal learning and improvement of performance] concerns the subjects’ motivation to attend to the task and exert effort to improve their performance…The subjects should receive immediate informative feedback and knowledge of results of their performance…In the absence of adequate feedback, efficient learning is impossible and improvement only minimal even for highly motivated subjects. Hence mere repetition of an activity will not automatically lead to improvement in, especially, accuracy of performance…In contrast to play, deliberate practice is a highly structured activity, the explicit goal of which is to improve performance. Specific tasks are invented to overcome weaknesses, and performance is carefully monitored to provide cues for ways to improve it further. We claim that deliberate practice requires effort and is not inherently enjoyable.
Motor skills
It should be noted that reviews conflict on how much spaced repetition applies to motor skills; Lee & Genovese 1988 find benefits, while Adams 1987 and earlier do not. The difference may be that simple motor tasks benefit from spacing as suggested by Shea & Morgan 1979 (benefits to a randomized/
The contextual interference hypothesis (Shea and Morgan 1979, Battig 1966 [“Facilitation and interference” in Acquisition of skill]) predicted the blocked condition would exhibit superior performance immediately following practice (acquisition) but the random condition would perform better at delayed retention testing. This hypothesis is generally consistent in laboratory motor learning studies (e.g.Lee & Magill 1983, Brady 2004), but less consistent in applied studies of sports skills (with a mix of positive & negative e.g.Landin & Hebert 1997, Hall et al 1994, Regal 2013) and fine-motor skills (Ollis et al 2005, Ste-Marie et al 2004).
Some of the positive spaced repetition studies (from Son & Simon 2012):
Perhaps even prior to the empirical work on cognitive learning and the spacing effect, the benefits of spaced study had been apparent in an array of motor learning tasks, including maze learning (Culler 1912), typewriting (Pyle 1915), archery (Lashley 1915), and javelin throwing (Murphy 1916; see Ruch 1928, for a larger review of the motor learning tasks which reap benefits from spacing; see also Moss 1996, for a more recent review of motor learning tasks). Thus, as in the cognitive literature, the study of practice distribution in the motor domain is long established (see reviews by Adams 1987; Schmidt and Lee 2005), and most interest has centered around the impact of varying the separation of learning trials of motor skills in learning and retention of practiced skills. Lee and Genovese (1988) conducted a review and meta-analysis of studies on distribution of practice, and they concluded that massing of practice tends to depress both immediate performance and learning, where learning is evaluated at some removed time from the practice period. Their main finding was, as in the cognitive literature, that learning was relatively stronger after spaced than after massed practice (although see Ammons 1988; Christina and Shea 1988; Newell et al. 1988 for criticisms of the review)…Probably the most widely cited example is Baddeley and Longman’s (1978) study concerning how optimally to teach postal workers to type. They had learners practice once a day or twice a day, and for session lengths of either 1 or 2 h at a time. The main findings were that learners took the fewest cumulative hours of practice to achieve a performance criterion in their typing when they were in the most distributed practice condition. This finding provides clear evidence for the benefits of spacing practice for enhancing learning. However, as has been pointed out (Newell et al. 1988; Lee and Wishart 2005), there is also trade-off to be considered in that the total elapsed time (number of days) between the beginning of practice and reaching criterion was substantially longer for the most spaced condition….The same basic results have been repeatedly demonstrated in the decades since (see reviews by Magill and Hall 1990; Lee and Simon 2004), and with a wide variety of motor tasks including different badminton serves (Goode and Magill 1986), rifle shooting (Boyce and Del Rey 1990), a pre-established skill, baseball batting (Hall et al. 1994), learning different logic gate configurations (Carlson et al. 1989; Carlson and Yaure 1990), for new users of automated teller machines (Jamieson and Rogers 2000), and for solving mathematical problems as might appear in a class homework (Rohrer and Taylor 2007; Le Blanc and Simon 2008; Taylor and Rohrer 2010).
- Culler, E. A. (1912). “The effect of distribution of practice upon learning”. Journal of Philosophical Psychology, 9, 580-583
- Pyle, W. H. (1915). “Concentrated versus distributed practice”
- Lashley 1915, “The acquisition of skill in archery”
- Murphy, H. H. (1916). “Distributions of practice periods in learning”. Journal of Educational Psychology, 7, 150-162
- Adams, J. A. (1987). “Historical review and appraisal of research on the learning, retention, and transfer of human motor skills”
- Schmidt, R. A., & Lee, T. D. (2005). Motor control and learning: A behavioral emphasis (4th ed.). Urbana-Champaign: Human Kinetics
- Lee, T. D., & Genovese, E. D. (1988). “Distribution of practice in motor skill acquisition: Learning and performance effects reconsidered”. Research Quarterly for Exercise and Sport, 59, 277-287
- Ammons, R. B. (1988). “Distribution of practice in motor skill acquisition: A few questions and comments”. Research Quarterly for Exercise and Sport, 59, 288-290
- Christina, R. W., & Shea, J. B. (1988). “The limitations of generalization based on restricted information”. Research Quarterly for Exercise and Sport, 59, 291-297
- Newell, K. M., Antoniou, A., & Carlton, L. G. (1988). “Massed and distributed practice effects: Phenomena in search of a theory?” Research Quarterly for Exercise and Sport, 59, 308-313
- Lee, T. D., & Wishart, L. R. (2005). “Motor learning conundrums (and possible solutions)”
- Lee, T. D., & Simon, D. A. (2004). “Contextual interference”
- Goode, S., & Magill, R. A. (1986). “Contextual interference effects in learning three badminton serves”. Research Quarterly for Exercise and Sport, 57, 308-314
- Boyce, B. A., & Del Rey, P. (1990). “Designing applied research in a naturalistic setting using a contextual interference paradigm”. Journal of Human Movement Studies, 18, 189-200
- Hall et al 1994, “Contextual interference effects with skilled baseball players”
- Carlson, R. A., & Yaure, R. G. (1990). “Practice schedules and the use of component skills in problem solving”
- Carlson, R. A., Sullivan, M. A., & Schneider, W. (1989). “Practice and working memory effects in building procedural skill”
- Jamieson, B. A., & Rogers, W. A. (2000). “Age-related effects of blocked and random practice schedules on learning a new technology”
- Le Blanc, K. & Simon, D. A. (2008). “Mixed practice enhances retention and JOL accuracy for mathematical skills”. Poster presented at the 2008 annual meeting of the Psychonomic Society, Chicago, IL
- Wymbs et al 2016, “Motor Skills Are Strengthened through Reconsolidation”
- Dayan & Cohen 2011, “Neuroplasticity subserving motor skill learning”
- Landin et al 1993, “The Effects of Variable Practice on the Performance of a Basketball Skill”
In this vein, it’s interesting to note that interleaving may be helpful for tasks with a mental component as well: Hatala et al 2003, Helsdingen et al 2011, and according to Huang et al 2013 the rates at which Xbox Halo: Reach video game players advance in skill matches nicely predictions from distribution: players who play 4–8 matches a week advance more in skill per match, than players who play more (distributed); but advance slower per week than players who play many more matches /
Abstraction
Another potential objection is to argue52 that spaced repetition inherently hinders any kind of abstract learning and thought because related materials are not being shown together - allowing for comparison and inference - but days or months apart. Ernst A. Rothkopf: “Spacing is the friend of recall, but the enemy of induction” (Kornell & Bjork 2008, p. 585). This is plausible based on some of the early studies53 but the 4 recent studies I know of directly examining the issue both found spaced repetition helped abstraction as well as general recall:
Kornell & Bjork 2008a, “Learning concepts and categories: Is spacing the ‘enemy of induction’?” Psychological Science, 19, 585-592
Vlach, H. A., Sandhofer, C. M., & Kornell, N. (2008). “The spacing effect in children’s memory and category induction”. Cognition, 109, 163-167
Kenney 2009. “The Spacing Effect in Inductive Learning”
Kornell, N., Castel, A. D., Eich, T. S., & Bjork, R. A. (2010). “Spacing as the friend of both memory and induction in younger and older adults”. Psychology and Aging, 25, 498-503
Vlach & Sandhofer 2012, “Distributing Learning Over Time: The Spacing Effect in Children’s Acquisition and Generalization of Science Concepts”, Child Development
Zulkiply 2012, “The spacing effect in inductive learning”; includes:
- replication of Kornell & Bjork 2008
- Zulkiply et al 2011
- Zulkiply & Burt 2012, “The exemplar interleaving effect in inductive learning: Moderation by the difficulty of category discriminations”
- unknown paper currently in peer review
McDanie et al 2013, “Effects of Spaced versus Massed Training in Function Learning”
Verkoeijen & Bouwmeester 2014, “Is spacing really the ‘friend of induction’?”
Rohrer et al 2014: 1, 2; Rorher et al 2019: “A randomized controlled trial of interleaved mathematics practice”
Vlach et al 2014, “Equal spacing and expanding schedules in children’s categorization and generalization”
Gluckman et al, “Spacing Simultaneously Promotes Multiple Forms of Learning in Children’s Science Curriculum”
Review summary
To bring it all together with the gist:
testing is effective and comes with minimal negative factors
expanding spacing is roughly as good as or better than (wide) fixed intervals, but expanding is more convenient and the default
testing (and hence spacing) is best on intellectual, highly factual, verbal domains, but may still work in many low-level domains
the research favors questions which force the user to use their memory as much as possible; in descending order of preference:
- free recall
- short answers
- multiple-choice
- Cloze deletion
- recognition
the research literature is comprehensive and most questions have been answered - somewhere.
the most common mistakes with spaced repetition are
- formulating poor questions and answers
- assuming it will help you learn, as opposed to maintain and preserve what one already learned54. (It’s hard to learn from cards, but if you have learned something, it’s much easier to then devise a set of flashcards that will test your weak points.)
Using it
One doesn’t need to use SuperMemo, of course; there are plenty of free alternatives. I like Mnemosyne (homepage) myself - Free, packaged for Ubuntu Linux, easy to use, free mobile client, long track record of development and reliability (I’ve used it since ~2008). But the SRS Anki is also popular, and has advantages in being more feature-rich and a larger & more active community (and possibly better support for East Asian language material and a better but proprietary mobile client).
OK, but what does one do with it? It’s a surprisingly difficult question, actually. It’s akin to “the tyranny of the blank page” (or blank wiki); now that I have all this power - a mechanical golem that will never forget and never let me forget whatever I chose to - what do I choose to remember?
How Much To Add
The most difficult task, beyond that of just persisting until the benefits become clear, is deciding what’s valuable enough to add in. In a 3 year period, one can expect to spend “30-40 seconds”55 on any given item. The long run theoretical predictions are a little hairier. Given a single item, the formula for daily time spent on it is . During our 20th year, we would spend , or 3.556940131083312e-4
minutes a day. This is the average daily time, so to recover the annual time spent, we simply multiply by 365. Suppose we were interested in how much time a flashcard would cost us over 20 years. The average daily time changes every year (the graph looks like an exponential decay, remember), so we have to run the formula for each year and sum them all; in Haskell:
sum $ map (\year -> ((1/500 * year**(-(1.5))) + 1/30000) * 365.25) [1..20]
# 1.8291
Which evaluates to 1.8 minutes. (This may seem too small, but one doesn’t spend much time in the first year and the time drops off quickly56.) Anki user muflax’s statistics put his per-card time at 71s, for example. But maybe Piotr Woźniak was being optimistic or we’re bad at writing flashcards, so we’ll double it to 5 minutes. That’s our key rule of thumb that lets us decide what to learn and what to forget: if, over your lifetime, you will spend more than 5 minutes looking something up or will lose more than 5 minutes as a result of not knowing something, then it’s worthwhile to memorize it with spaced repetition. 5 minutes is the line that divides trivia from useful data.57 (There might seem to be thousands of flashcards that meet the 5 minute rule. That’s fine. Spaced repetition can accommodate dozens of thousands of cards. See the next section.)
To a lesser extent, one might wonder when one is in a hurry, should one learn something with spaced repetition and with massed? How far away should the tests or deadlines be before abandoning spaced repetition? It’s hard to compare since one would need a specific regimens to compare for the crossover point, but for massed repetition, the average time after memorization at which one has a 50% chance of remembering the memorized item seems to be 3-5 days.58 Since there would be 2 or 3 repetitions in that period, presumably one would do better than 50% in recalling an item. 5 minutes and 5 days seems like a memorable enough rule of thumb: ‘don’t use spaced repetition if you need it sooner than 5 days or it’s worth less than 5 minutes’.
Overload
One common experience of new users to spaced repetition is to add too much stuff - trivialities and things they don’t really care about. But they soon learn the curse of Borges’s Funes the Memorious. If they don’t actually want to learn the material they put in, they will soon stop doing the daily reviews - which will cause reviews to pile up, which will be further discouraging, and so they stop. At least with physical fitness there isn’t a precisely dismaying number indicating how far behind you are! But if you have too little at the beginning, you’ll have few repetitions per day, and you’ll see little benefit from the technique itself - it looks like boring flash card review.
What to add
I find one of the best uses for Mnemosyne is, besides the classic use of memorizing academic material such as geography or the periodic table or foreign vocabulary or Bible/
The workload
On average, when I’m studying a new topic, I’ll add 3-20 questions a day. Combined with my particular memory, I usually review about 90 or 100 items a day (out of the total >18,300). This takes under 20 minutes, which is not too bad. (I expect the time is expanded a bit by the fact that early on, my formatting guidelines were still being developed, and I hadn’t the full panoply of categories I do now - so every so often I must stop and edit categories.)
If I haven’t been studying something recently, the exponential decaying of reviews slowly drops the daily review. For example, in March 2011, I wasn’t studying many things, so for 2011-03-24–2011-03-26, my scheduled daily reviews are 73, 83, and 74; after that, it’ll probably drop down into the 60s, and then after another week or two, into the 50s and so on until it hits the minimum plateau which will slowly shrink over years. (I haven’t gone long enough without dumping cards in to know what that might be.) By February 2012, the daily reviews are in the 40s or sometimes 50s for similar reasons, but the gradual shrinkage will continue. We can see this vividly, and we can even see a sort of analogue of the original forgetting curve, if we ask Mnemosyne 2.0 to graph the number of cards to review per day for the next year up to February 2013 (assuming no additions or missed reviews etc.):

If Mnemosyne weren’t using spaced repetition, it would be hard to keep up with 18,300+ flashcards. But because it is using spaced repetition, keeping up is easy.
Nor is 18.3k extraordinary. Many users have decks in the 6-7k range, Mnemosyne developer Peter Bienstman has >8.5k & Patrick Kenny >27k, Hugh Chen has a 73k+ deck, and in #anki, they tell me of one user who triggered bugs with his >200k deck. 200,000 may be a bit much, but for regular humans, some amount smaller seems possible - it’s interesting to compare SRS decks to the feat of memorizing Paradise Lost or to the Muslim title of ‘hafiz’, one who has memorized the ~80,000 words of the Koran, or the stricter ‘hafid’, one who had memorized the Koran and 100,000 hadiths as well. Other forms of memory are still more powerful.61 (I suspect that spaced repetition is involved in one of the few well-documented cases of “hyperthymesia”, Jill Price: reading Wired, she has ordinary fallible powers of memorization for surprise demands with no observed anatomical differences and is restricted to “her own personal history and certain categories like television and airplane crashes”; further, she is a packrat with obsessive-compulsive traits who keeps >50,000 pages of detailed diaries perhaps due to a childhood trauma & associates daily events nigh-involuntarily with past events. Marcus says the other instances of hyperthymesia resemble Price.)
When to review
When should one review? In the morning? In the evening? Any old time? The studies demonstrating the spacing effect do not control or vary the time of day, so in one sense, the answer is: it doesn’t matter - if it did matter, there would be considerable variance in how effective the effect is based on when a particular study had its subjects do their reviews.
So one reviews at whatever time is convenient. Convenience makes one more likely to stick with it, and sticking with it overpowers any temporary improvement.
If one is not satisfied with that answer, then on general considerations, one ought to review before bedtime & sleep. Memory consolidation seems to be related, and sleep is known to powerfully influence what memories enter long-term memory, strengthening memories of material learned close to bedtime and increasing creativity; interrupting sleep without affecting total sleep time or quality still damages memory formation in mice62. So reviewing before bedtime would be best. (Other mental exercises show improvement when trained before bedtime; for example, dual n-back.) One possible mechanism is that it may be that the expectancy of future reviews/
Recent advances in our understanding of the neurobiology underlying normal human memory formation have revealed that learning is not an event, but rather a process that unfolds over time.16,17,18,[Squire 2003Fundamental Neuroscience],20 Thus, it is not surprising that learning strategies that repeat materials over time enhance their retention.20,21,22,23,24,25,26
…Thousands of new cells are generated in this region every day, although many of these cells die within weeks of their creation.31 The survival of dentate gyrus neurons has been shown to be enhanced in animals when they are placed into learning situations.16-20 Animals that learn well retain more dentate gyrus neurons than do animals that do not learn well. Furthermore, 2 weeks after testing, animals trained in discrete spaced intervals over a period of time, rather than in a single presentation or a ‘massed trial’ of the same information, remember better.16-20 The precise mechanism that links neuronal survival with learning has not yet been identified. One theory is that the hippocampal neurons that preferentially survive are the ones that are somehow activated during the learning process.16-2063 The distribution of learning over a period of time may be more effective in encouraging neuronal survival by allowing more time for changes in gene expression and protein synthesis that extend the life of neurons that are engaged in the learning process.
…Transferring memory from the encoding stage, which occurs during alert wakefulness, into consolidation must thus occur at a time when interference from ongoing new memory formation is reduced.17,18 One such time for this transfer is during sleep, especially during non-rapid eye movement sleep, when the hippocampus can communicate with other brain areas without interference from new experiences.32,33,34 Maybe that is why some decisions are better made after a good night’s rest and also why pulling an all-nighter, studying with sleep deprivation, may allow you to pass an exam an hour later but not remember the material a day later.
Prospects: extended flashcards
Let’s step back for a moment. What are all our flashcards, small and large, doing for us? Why do I have a pair of flashcards for the word ‘anent’ among many others? I can just look it up.
But look ups take time compared to already knowing something. (Let’s ignore the previously discussed 5 minute rule.) If we think about this abstractly in a computer science context, we might recognize it as an old concept in algorithms & optimization discussions - the space-time tradeoff. We trade off lookup time against limited skull space.
Consider the sort of factual data already given as examples - we might one day need to know the average annual rainfall in Honolulu or Austin, but it would require too much space to memorize such data for all capitals. There are millions of English words, but in practice any more than 100,000 is excessive. More surprising is a sort of procedural knowledge. An extreme form of space-time tradeoffs in computers is when a computation is replaced by pre-calculated constants. We could take a math function and calculate its output for each possible input. Usually such a lookup table of input to output is really large. Think about how many entries would be in such a table for all possible integer multiplications between 1 and 1 billion. But sometimes the table is really small (like binary Boolean functions) or small (like trigonometric tables) or large but still useful (rainbow tables usually start in the gigabytes and easily reach terabytes).
Given an infinitely large lookup table, we could replace completely the skill of, say, addition or multiplication by the lookup table. No computation. The space-time tradeoff taken to the extreme of the space side of the continuum. (We could go the other way and define multiplication or addition as the slow computation which doesn’t know any specifics like the multiplication table - as if every time you wanted to add you had to count on 4 fingers.)
So suppose we were children who wanted to learn multiplication. SRS and Mnemosyne can’t help because multiplication is not a specific factoid? The space-time tradeoff shows us that we can de-proceduralize multiplication and turn it partly into factoids. It wouldn’t be hard for us to write a quick script or macro to generate, say, 500 random cards which ask us to multiply AB by XY, and import them to Mnemosyne.64
After all, which is your mind going to do - get good at multiplying 2 numbers (generate on-demand), or memorize 500 different multiplication problems (memoize)? From my experience with multiple subtle variants on a card, the mind gives up after just a few and falls back on a problem-solving approach - which is exactly what one wants to exercise, in this case. Congratulations; you have done the impossible.
From a software engineering point of view, we might want to modify or improve the cards, and 500 snippets of text would be a tad hard to update. So coolest would be a ‘dynamic card’. Add a markup type like <eval src="">
, and then Mnemosyne feeds the src
argument straight into the Python interpreter, which returns a tuple of the question text and the answer text. The question text is displayed to the user as usual, the user thinks, requests the answer, and grades himself. In Anki, Javascript is supported directly by the application in HTML <script>
tags (currently inline only but Anki could presumably import libraries by default), for example for kinds of syntax highlighting, so any kind of dynamic card could be written that one wants.
So for multiplication, the dynamic card would get 2 random integers, print a question like x * y = ?
and then print the result as the answer. Every so often you would get a new multiplication question, and as you get better at multiplication, you see it less often - exactly as you should. Still in a math vein, you could generate variants on formulas or programs where one version is the correct one and the others are subtly wrong; I do this by hand with my programming flashcards (especially if I make an error doing exercises, that signals a finer point to make several flashcards on), but it can be done automatically. kpreid describes one tool of his:
I have written a program (in the form of a web page) which does a specialized form of this [generating ‘damaged formulas’]. It has a set of generators of formulas and damaged formulas, and presents you with a list containing several formulas of the same type (e.g. ∫ 2x dx = x^2 + C) but with one damaged (e.g. ∫ 2x dx = 2x^2 + C).
This approach generalizes to anything you can generate random problems of or have large databases of examples of. Khan Academy apparently does something like this in associating large numbers of (algorithmicly-generated?) problems with each of its little modules and tracking retention of the skill in order to decide when to do further review of that module. For example, maybe you are studying Go and are interested in learning life-and-death positions. Those are things that can be generated by computer Go programs, or fetched from places like GoProblems.com. For even more examples, Go is rotationally invariant - the best move remains the same regardless of which way the board is oriented and since there is no canonical direction for the board (like in chess) a good player ought to be able to play the same no matter how the board looks - so each specific example can be mirrored in 3 other ways. Or one could test one’s ability to ‘read’ a board by writing a dynamic card which takes each example board/
One could learn an awful lot of things this way. Programming languages could be learned this way - someone learning Haskell could take all the functions listed in the Prelude or his Haskell textbook, and ask QuickCheck to generate random arguments for the functions and ask the GHC interpreter ghci
what the function and its arguments evaluate to. Games other than go, like chess, may work (a live example being Chess Tempo & Listudy, and see the experience of Dan Schmidt; or Super Smash Brothers). A fair bit of mathematics. If the dynamic card has Internet access, it can pull down fresh questions from an RSS feed or just a website; this functionality could be quite useful in a foreign language learning context with every day bringing a fresh sentence to translate or another exercise.
With some NLP software, one could write dynamic flashcards which test all sorts of things: if one confuses verbs, the program could take a template like “$PRONOUN $VERB $PARTICLE $OBJECT % {right: caresse, wrong: caresses}” which yields flashcards like “Je caresses le chat” or “Tu caresse le chat” and one would have to decide whether it was the correct conjugation. (The dynamicism here would help prevent memorizing specific sentences rather than the underlying conjugation.) In full generality, this would probably be difficult, but simpler approaches like templates may work well enough. Jack Kinsella:
I wish there were dynamic SRS decks for language learning (or other disciplines). Such decks would count the number of times you have reviewed an instance of an underlying grammatical rule or an instance of a particular piece of vocabulary, for example its singular/
plural/ third person conjugation/ dative form. These sophisticated decks would present users with fresh example sentences on every review, thereby preventing users from remembering specific answers and compelling them to learn the process of applying the grammatical rule afresh. Moreover, these decks would keep users entertained through novelty and would present users with tacit learning opportunities through rotating vocabulary used in non-essential parts of the example sentence. Such a system, with multiple-level review rotation, would not only prevent against overfit learning, but also increase the total amount of knowledge learned per minute, an efficiency I’d gladly invest in.
Even though these things seem like ‘skills’ and not ‘data’!
Popularity
As of 2011-05-02:
Metric | Mnemosyne | Mnemododo | Anki | iSRS | AnyMemo |
---|---|---|---|---|---|
Homepage Alexa | 383k | 27.5m | 112k | 1,766k65 | |
ML/ |
461 | 4129/ |
129 | ||
Ubuntu installs | 7k | 9k | |||
Debian installs | 164 | 364 | |||
Arch votes | 85 | 96 | |||
iPhone ratings | Unreleased66 | 193 | 69 | ||
Android ratings | 20 | 703 | 836 | ||
Android installs | 100-500 | 10-50k | 50-100k |
SuperMemo doesn’t fall under the same ratings, but it has sold in the hundreds of thousands over its 2 decades:
Biedalak is CEO of SuperMemo World, which sells and licenses Wozniak’s invention. Today, SuperMemo World employs just 25 people. The venture capital never came through, and the company never moved to California. About 50,000 copies of SuperMemo were sold in 2006, most for less than $30. Many more are thought to have been pirated.67
It seems safe to estimate the combined market-share of Anki, Mnemosyne, iSRS and other SRS apps at somewhere under 50,000 users (making due allowance for users who install multiple times, those who install and abandon it, etc.). Relatively few users seem to have migrated from SuperMemo to those newer programs, so it seems fair to simply add that 50k to the other 50k and conclude that the worldwide population is somewhere around (but probably under) 100,000.
Where was I going with this?
Nowhere, really. Mnemosyne/
See Also
In the course of using Mnemosyne, I’ve written a number of scripts to generate repetitively varying cards.
mnemo.hs
will take any newline-delimited chunk of text, like a poem, and generates every possible Cloze deletion; that is, an ABC poem will become 3 questions: _BC/ABC, A_C/ ABC, AB_/ ABC mnemo2.hs
works as above, but is more limited and is intended for long chunks of text wheremnemo.hs
would cause a combinatorial explosion of generated questions; it generates a subset: for ABCD, one gets __CD/ABCD, A__D/ ABCD, and AB__/ ABCD (it removes 2 lines, and iterates through the list). mnemo3.hs
is intended for date or name-based questions. It’ll take input like “Barack Obama is %47%.” and spit out some questions based on this: “Barack Obama is _7./47”, “Barack Obama is 4_./ 47” etc. mnemo4.hs
is intended for long lists of items. If one wants to memorize the list of US Presidents, the natural questions for flashcards goes something like “Who was the 3rd president?/Thomas Jefferson”, “Thomas Jefferson was the _rd president./ 3”, “Who was president after John Adams?/ Thomas Jefferson”, “Who was president before James Madison?/ Thomas Jefferson”. You note there’s repetition if you do this for each president - one asks the ordinal position of the item both ways (item -> position, position -> item), what precedes it, and what succeeds it.
mnemo4.hs
automates this, given a list. In order to be general, the wording is a bit odd, but it’s better than writing it all out by hand! (Example output is in the comments to the source code).
The reader might well be curious by this point what my Mnemosyne database looks like. I use Mnemosyne quite a bit, and as of 2020-02-02, I have 16,149 (active) cards in my deck. Said curious reader may find my cards & media at gwern.cards
(52M; Mnemosyne 2.x format).
The Mnemosyne project has been collecting user-submitted spaced repetition statistical data for years. The full dataset as of 2014-01-27 is available for download by anyone who wishes to analyze it.
External Links
Michael Nielsen: “Augmenting Long-term Memory”; “Using spaced repetition systems to see through a piece of mathematics”; “Quantum computing for the very curious”; “How can we develop transformative tools for thought?”
“Teaching linear algebra” (with spaced repetition), by Ben Tilly
“A Year of Spaced Repetition Software in the Classroom”; two years; cf “Easy Application of Spaced Practice in the Classroom”
AJATT table of contents -(applying SRS to learning Japanese)
SRS for programming:
- “SuperMemo as a new tool increasing the productivity of a programmer. A case study: programming in Object Windows”
- “Janki Method: Using spaced repetition systems to learn and retain technical knowledge” (Reddit discussion); SRS problems & solutions
- “Memorizing a programming language using spaced repetition software” (Derek Sivers; Hacker News)
- learning text editor shortcuts
- “Learning Go with flashcards and spaced repetition”
- “Chasing 10X: Leveraging A Poor Memory In Engineering”; “Everything I Know: Strategies, Tips, and Tricks for Anki”
- “Remembering R—Using Spaced Repetition to finally write code fluently”
- “Anki as Learning Superpower: Computer Science Edition”
“QS Primer: Spaced Repetition and Learning” -(talks on applications of spaced repetition)
Value compared to curriculums:
- Point: “Why Forgetting Can Be Good”, by Scott H. Young
- Counterpoint: “Spaced repetition in natural and artificial learning”, by Ryan Muller
My own observation is that an optimally constructed curriculum could effectively implement spaced repetition, but even if it did (most don’t), unless it is computerized it will not adapt to the user.
Bash scripts for generating vocabulary flashcards (processing multiple online dictionaries, good for having multiple examples; images; and audio)
vocabulary selection:
- “Programmed Vocabulary Learning as a Traveling Salesman Problem”
- “Teaching New Testament Greek”
- graded-reader: “A New Kind of Graded Reader” (video talk)
- Mailing list
- Programs (I took a stab at writing Haskell versions once)
“Diff revision: diff-based revision of text notes, using spaced repetition”
“A vote against spaced repetition”; “How Flashcards Fail: Confessions of a Tired Memory Guy”
“Learning Ancient Egyptian in an Hour Per Week with Beeminder”
SeRiouS: “Spaced Repetition Technology for Legal Education”, “SeRiouS: an LPTI-supported Project to Improve Students’ Learning and Bar Performance”, Gabe Teninbaum (video presentation)
“The role of digital flashcards in legal education: theory and potential”, Colbran et al 2014
“Making Summer Count: How Summer Programs Can Boost Children’s Learning”, McCombs et al 2011 (RAND MG1120)
“Factors that Influence Skill Decay And Retention: a Quantitative Review and Analysis”, Arthur et al 1998
“On The Forgetting Of College Academics: At ‘Ebbinghaus speed’?”, Subirana et al 2017
“Total recall: the people who never forget; An extremely rare condition may transform our understanding of memory” (obsessive recording & reviewing demonstrates you can recall much of your life if you live nothing worth recalling); “The Mystery of S., the Man with an Impossible Memory: The neuropsychologist Alexander Luria’s case study of Solomon Shereshevsky helped spark a myth about a man who could not forget. But the truth is more complicated”
Anki Essentials, Vermeer
“No. 126: Four Years of Spaced Repetition” (Gene Dan, actuarial studies)
“How To Remember Anything Forever-ish”: an interactive comic (Nicky Case)
“The Overfitted Brain: Dreams evolved to assist generalization”, Hoel 2020
Flashcard sources
Link Bibliography
Bibliography of page links in reading order (with annotations when available):
“Neuroprosthetics”, (2020-12-22):
Neuroprosthetics is a discipline related to neuroscience and biomedical engineering concerned with developing neural prostheses. They are sometimes contrasted with a brain–computer interface, which connects the brain to a computer rather than a device meant to replace missing biological functionality.
“Externalism”, (2020-12-22):
Externalism is a group of positions in the philosophy of mind which argues that the conscious mind is not only the result of what is going on inside the nervous system, but also what occurs or exists outside the subject. It is contrasted with internalism which holds that the mind emerges from neural activity alone. Externalism is a belief that the mind is not just the brain or functions of the brain.
“Andy Clark”, (2020-12-22):
Andrew Clark, is a British philosopher who is Professor of Cognitive Philosophy at the University of Sussex. Prior to this, he was at professor of philosophy and Chair in Logic and Metaphysics at the University of Edinburgh in Scotland, director of the Cognitive Science Program at Indiana University in Bloomington, Indiana and previously taught at Washington University in St. Louis, Missouri. Clark is one of the founding members of the CONTACT collaborative research project whose aim is to investigate the role environment plays in shaping the nature of conscious experience. Clark's papers and books deal with the philosophy of mind and he is considered a leading scientist in mind extension. He has also written extensively on connectionism, robotics and the role and nature of mental representation.
“Xmonad”, (2021-01-03):
xmonad is a dynamic window manager (tiling) for the X Window System, noted for being written in the functional programming language Haskell.
“Google Calendar”, (2021-01-02):
Google Calendar is a time-management and scheduling calendar service developed by Google. It became available in beta release April 13, 2006, and in general release in July 2009, on the web and as mobile apps for the Android and iOS platforms.
“Evernote”, (2020-12-22):
Evernote is an app designed for note taking, organizing, task management, and archiving. It is developed by the Evernote Corporation, headquartered in Redwood City, California. The app allows users to create notes, which can be text, drawings, photographs, audio, or saved web content. Notes are stored in notebooks and can be tagged, annotated, edited, searched, given attachments, and exported.
“Ultra-high-definition television”, (2021-01-03):
Ultra-high-definition television today includes 4K UHD and 8K UHD, which are two digital video formats with an aspect ratio of 16:9. These were first proposed by NHK Science & Technology Research Laboratories and later defined and approved by the International Telecommunication Union (ITU). It is a digital television (DTV) standard, and the successor to high-definition television (HDTV), which in turn was the successor to standard-definition television (SDTV).
“Power shovel”, (2021-01-03):
A power shovel is a bucket-equipped machine, usually electrically powered, used for digging and loading earth or fragmented rock and for mineral extraction. Power shovels are a type of rope/cable excavator, where the digging arm is controlled and powered by winches and steel ropes, rather than hydraulics like in the more common hydraulic excavators.Basics parts of a power shovel include the track system, cabin, cables, rack, stick, boom foot-pin, saddle block, boom, boom point sheaves and bucket.The size of bucket varies from 0.375 to 5 cubic metres.
“Spacing effect”, (2020-12-22):
The spacing effect demonstrates that learning is more effective when study sessions are spaced out. This effect shows that more information is encoded into long-term memory by spaced study sessions, also known as spaced repetition or spaced presentation, than by massed presentation (“cramming”).
https:/
/ web.archive.org/ web/ 20090430093950/ http:/ / chronicle.com/ free/ v55/ i34/ 34a00101.htm “The Chronicle of Higher Education”, (2021-01-02):
The Chronicle of Higher Education is a newspaper and website that presents news, information, and jobs for college and university faculty and student affairs professionals. A subscription is required to read some articles.
http:/
/ www.amazon.com/ Making-Minds-Whats-Education-Should/ dp/ 0415414113/ http:/
/ deepblue.lib.umich.edu/ bitstream/ 2027.42/ 32694/ 1/ 0000061.pdf http:/
/ www.psywww.com/ intropsych/ ch06_memory/ spacing_effect.html “Maintenance of Foreign Language Vocabulary and the Spacing Effect”, (1993-09-01):
In a 9-year longitudinal investigation, 4 subjects learned and relearned 300 English-foreign language word pairs. Either 13 or 26 relearning sessions were administered at intervals of 14, 28, or 56 days. Retention was tested for 1, 2, 3, or 5 years after training terminated. The longer intersession intervals slowed down acquisition slightly, but this disadvantage during training was offset by substantially higher retention. 13 retraining sessions spaced at 56 days yielded retention comparable to 26 sessions spaced at 14 days. The retention benefit due to additional sessions was independent of the benefit due to spacing, and both variables facilitated retention of words regardless of difficulty level and of the consistency of retrieval during training. The benefits of spaced retrieval practice to long-term maintenance of access to academic knowledge areas are discussed.
“Forgetting curve”, (2021-01-03):
The forgetting curve hypothesizes the decline of memory retention in time. This curve shows how information is lost over time when there is no attempt to retain it. A related concept is the strength of memory that refers to the durability that memory traces in the brain. The stronger the memory, the longer period of time that a person is able to recall it. A typical graph of the forgetting curve purports to show that humans tend to halve their memory of newly learned knowledge in a matter of days or weeks unless they consciously review the learned material.
“Half-life”, (2020-12-22):
Half-life is the time required for a quantity to reduce to half of its initial value. The term is commonly used in nuclear physics to describe how quickly unstable atoms undergo, or how long stable atoms survive, radioactive decay. The term is also used more generally to characterize any type of exponential or non-exponential decay. For example, the medical sciences refer to the biological half-life of drugs and other chemicals in the human body. The converse of half-life is doubling time.
“Long-term potentiation”, (2021-01-03):
In neuroscience, long-term potentiation (LTP) is a persistent strengthening of synapses based on recent patterns of activity. These are patterns of synaptic activity that produce a long-lasting increase in signal transmission between two neurons. The opposite of LTP is long-term depression, which produces a long-lasting decrease in synaptic strength.
http:/
/ medicalxpress.com/ news/ 2012-03-neurobiology-superiority-spaced-massed.html “Engram (neuropsychology)”, (2020-12-22):
An engram is a unit of cognitive information inside the brain, theorized to be the means by which memories are stored as biophysical or biochemical changes in the brain in response to external stimuli.
“Active recall”, (2021-01-03):
Active recall is a principle of efficient learning, which claims the need to actively stimulate memory during the learning process. It contrasts with passive review, in which the learning material is processed passively. For example, reading a text about George Washington, with no further action, is a passive review. Answering the question "Who was the first US President?", is active recall. There are several variations of active recall as a strategy and these include the 3R (read-recite-review) method by McDaniel et al. and Feynman technique.
http:/
/ www.sciencemag.org/ content/ early/ 2011/ 01/ 19/ science.1199327.abstract http:/
/ www.sciencedaily.com/ releases/ 2011/ 01/ 110121111216.htm http:/
/ memory.psych.purdue.edu/ downloads/ 2006_Roediger_Karpicke_PsychSci.pdf “Edward O. Thorp”, (2020-12-22):
Edward Oakley Thorp is an American mathematics professor, author, hedge fund manager, and blackjack researcher. He pioneered the modern applications of probability theory, including the harnessing of very small correlations for reliable financial gain.
/
docs/ statistics/ decision/ 1984-thorp-themathematicsofgambling-ch4.pdf “Roger Craig (Jeopardy! contestant)”, (2020-12-22):
Roger Alan Craig is an American game show contestant and computer scientist. He held the record for highest single-day winnings on the quiz show Jeopardy! from September 14, 2010 to April 9, 2019. In 2011, Craig returned to win the Jeopardy! Tournament of Champions. In 2014, he competed in the Battle of the Decades tournament, finishing third overall behind Brad Rutter and Ken Jennings.
“Jeopardy!”, (2021-01-03):
Jeopardy! is an American television game show created by Merv Griffin. The show features a quiz competition in which contestants are presented with general knowledge clues in the form of answers, and must phrase their responses in the form of questions. The original daytime version debuted on NBC on March 30, 1964, and aired until January 3, 1975. A weekly nighttime syndicated edition aired from September 1974 to September 1975, and a revival, The All-New Jeopardy!, ran on NBC from October 1978 to March 1979. The version airing as of 2020, a daily syndicated show produced by Sony Pictures Television, premiered on September 10, 1984.
/
docs/ spacedrepetition/ 2011-qs-rogercraigwinsjeopardy.html#comment-3004 http:/
/ www.amazon.com/ Final-Jeopardy-Machine-Quest-Everything/ dp/ 0547483163/ http:/
/ mentalfloss.com/ article/ 54853/ our-interview-jeopardy-champion-arthur-chu https:/
/ old.reddit.com/ r/ Anki/ comments/ a9s456/ what_is_the_largest_anki_deck_you_have_read_about/ http:/
/ web.archive.org/ web/ 20130128115142/ http:/ / www.anewspring.com/ storage/ memotrainerrr.png “Epigrams on Programming”, (1982-09-1):
[130 epigrams on computer science and technology, published in 1982, for ACM’s SIGPLAN journal, by noted computer scientist and programming language researcher Alan Perlis. The epigrams are a series of short, programming-language-neutral, humorous statements about computers and programming, distilling lessons he had learned over his career, which are widely quoted.]
8. A programming language is low level when its programs require attention to the irrelevant.…19. A language that doesn’t affect the way you think about programming, is not worth knowing.…54. Beware of the Turing tar-pit in which everything is possible but nothing of interest is easy.
15. Everything should be built top-down, except the first time.…30. In programming, everything we do is a special case of something more general—and often we know it too quickly.…31. Simplicity does not precede complexity, but follows it.…58. Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it.…65. Make no mistake about it: Computers process numbers—not symbols. We measure our understanding (and control) by the extent to which we can arithmetize an activity.…56. Software is under a constant tension. Being symbolic it is arbitrarily perfectible; but also it is arbitrarily changeable.
1. One man’s constant is another man’s variable. 34. The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information.
36. The use of a program to prove the 4-color theorem will not change mathematics—it merely demonstrates that the theorem, a challenge for a century, is probably not important to mathematics.
39. Re graphics: A picture is worth 10K words—but only those to describe the picture. Hardly any sets of 10K words can be adequately described with pictures.
48. The best book on programming for the layman is Alice in Wonderland; but that’s because it’s the best book on anything for the layman.
77. The cybernetic exchange between man, computer and algorithm is like a game of musical chairs: The frantic search for balance always leaves one of the 3 standing ill at ease.…79. A year spent in artificial intelligence is enough to make one believe in God.…84. Motto for a research laboratory: What we work on today, others will first think of tomorrow.
91. The computer reminds one of Lon Chaney—it is the machine of a thousand faces.
7. It is easier to write an incorrect program than understand a correct one.…93. When someone says “I want a programming language in which I need only say what I wish done,” give him a lollipop.…102. One can’t proceed from the informal to the formal by formal means.
100. We will never run out of things to program as long as there is a single program around.
108. Whenever 2 programmers meet to criticize their programs, both are silent.…112. Computer Science is embarrassed by the computer.…115. Most people find the concept of programming obvious, but the doing impossible. 116. You think you know when you can learn, are more sure when you can write, even more when you can teach, but certain when you can program. 117. It goes against the grain of modern education to teach children to program. What fun is there in making plans, acquiring discipline in organizing thoughts, devoting attention to detail and learning to be self-critical?
http:/
/ lesswrong.com/ lw/ e26/ who_wants_to_start_an_important_startup/ 7834?context=3 http:/
/ www.cnsspectrums.com/ aspx/ articledetail.aspx?articleid=2783 -
The time course of physicians' knowledge retention after learning activities has not been well characterized. Understanding the time course of retention is critical to optimizing the reinforcement of knowledge. Educational follow-up experiment with knowledge retention measured at 1 of 6 randomly assigned time intervals (0-55 days) after an online tutorial covering 2 American Diabetes Association guidelines. Internal and family medicine residents. Multiple-choice knowledge tests, subject characteristics including critical appraisal skills, and learner satisfaction. Of 197 residents invited, 91 (46%) completed the tutorial and were randomized; of these, 87 (96%) provided complete follow-up data. Ninety-two percent of the subjects rated the tutorial as "very good" or "excellent." Mean knowledge scores increased from 50% before the tutorial to 76% among those tested immediately afterward. Score gains were only half as great at 3-8 days and no significant retention was measurable at 55 days. The shape of the retention curve corresponded with a 1/4-power transformation of the delay interval. In multivariate analyses, critical appraisal skills and participant age were associated with greater initial learning, but no participant characteristic significantly modified the rate of decline in retention. Education that appears successful from immediate posttests and learner evaluations can result in knowledge that is mostly lost to recall over the ensuing days and weeks. To achieve longer-term retention, physicians should review or otherwise reinforce new learning after as little as 1 week.
http:/
/ www.amazon.com/ Practices-Medical-Educators-Stephen-Stahl/ dp/ 1422500497/ https:/
/ pdfs.semanticscholar.org/ 9731/ c11be2ef8183dd974d79ef3ad1236f33e342.pdf http:/
/ citeseerx.ist.psu.edu/ viewdoc/ download?doi=10.1.1.556.6124&rep=rep1&type=pdf http:/
/ journals.sagepub.com/ doi/ pdf/ 10.2304/ plat.2002.2.2.70 http:/
/ citeseerx.ist.psu.edu/ viewdoc/ download?doi=10.1.1.56.6901&rep=rep1&type=pdf -
Retrieving information from memory makes that information more recallable in the future than it otherwise would have been. Optimizing retrieval practice has been assumed, on the basis of evidence and arguments tracing back to Landauer and Bjork (1978), to require an expanding-interval schedule of successive retrievals, but recent findings suggest that expanding retrieval practice may be inferior to uniform-interval retrieval practice when memory is tested after a long retention interval. We report three experiments in which participants read educational passages and were then repeatedly tested, without feedback, after an expanding or uniform sequence of intervals. On a test 1 week later, recall was enhanced by the expanding schedule, but only when the task between successive retrievals was highly interfering with memory for the passage. These results suggest that the extent to which learners benefit from expanding retrieval practice depends on the degree to which the to-be-learned information is vulnerable to forgetting.
http:/
/ www.pashler.com/ Articles/ Pashler.Rohrer.Cepeda.Carpenter_2007.pdf http:/
/ citeseerx.ist.psu.edu/ viewdoc/ download?doi=10.1.1.118.1497&rep=rep1&type=pdf http:/
/ castel.bol.ucla.edu/ publications/ RhodesCastelJOLFontSize.pdf “Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom”, (2019-09-04):
Despite active learning being recognized as a superior method of instruction in the classroom, a major recent survey found that most college STEM instructors still choose traditional teaching methods. This article addresses the long-standing question of why students and faculty remain resistant to active learning. Comparing passive lectures with active learning using a randomized experimental approach and identical course materials, we find that students in the active classroom learn more, but they feel like they learn less. We show that this negative correlation is caused in part by the increased cognitive effort required during active learning. Faculty who adopt active learning are encouraged to intervene and address this misperception, and we describe a successful example of such an intervention.
We compared students’ self-reported perception of learning with their actual learning under controlled conditions in large-enrollment introductory college physics courses taught using (1) active instruction (following best practices in the discipline) and (2) passive instruction (lectures by experienced and highly rated instructors). Both groups received identical class content and handouts, students were randomly assigned, and the instructor made no effort to persuade students of the benefit of either method. Students in active classrooms learned more (as would be expected based on prior research), but their perception of learning, while positive, was lower than that of their peers in passive environments. This suggests that attempts to evaluate instruction based on students’ perceptions of learning could inadvertently promote inferior (passive) pedagogical methods. For instance, a superstar lecturer could create such a positive feeling of learning that students would choose those lectures over active learning. Most importantly, these results suggest that when students experience the increased cognitive effort associated with active learning, they initially take that effort to signify poorer learning. That disconnect may have a detrimental effect on students’ motivation, engagement, and ability to self-regulate their own learning. Although students can, on their own, discover the increased value of being actively engaged during a semester-long course, their learning may be impaired during the initial part of the course. We discuss strategies that instructors can use, early in the semester, to improve students’ response to being actively engaged in the classroom. [Keywords: scientific teaching, undergraduate education, evidence-based teaching, Constructivism]
http:/
/ sites.williams.edu/ nk2/ files/ 2011/ 08/ Kornell.2009b.pdf http:/
/ www.willatworklearning.com/ 2005/ 11/ research_review.html http:/
/ www.indiana.edu/ %7Epcl/ rgoldsto/ courses/ dunloskyimprovinglearning.pdf “Bloom's 2 sigma problem”, (2020-12-22):
Bloom's 2 sigma problem refers to an educational phenomenon observed by educational psychologist Benjamin Bloom and initially reported in 1984 in the journal Educational Researcher. Bloom found that the average student tutored one-to-one using mastery learning techniques performed two standard deviations better than students who learn via conventional instructional methods—that is, "the average tutored student was above 98% of the students in the control class". Additionally, the variation of the students' achievement changed: "about 90% of the tutored students ... attained the level of summative achievement reached by only the highest 20%" of the control class. Bloom's graduate students J. Anania and A. J. Burke conducted studies of this effect at different grade levels and in different schools, observing students with "great differences in cognitive achievement, attitudes, and academic self-concept".
“Diminishing returns”, (2021-01-03):
In economics, diminishing returns is the decrease in the marginal (incremental) output a production process as the amount of a single factor of production is incrementally increased, while the amounts of all other factors of production stay constant.
“Richard Hamming”, (2020-12-27):
Richard Wesley Hamming was an American mathematician whose work had many implications for computer engineering and telecommunications. His contributions include the Hamming code, the Hamming window, Hamming numbers, sphere-packing, and the Hamming distance.
“You and Your Research”, (1986-03-07):
[Transcript of a talk by mathematician and Bell Labs manager Richard Hamming about what he had learned about computers and how to do effective research (republished in expanded form as Art of Doing Science and Engineering: Learning to Learn; 1995 video). It is one of the most famous and most-quoted such discussions ever.]
At a seminar in the Bell Communications Research Colloquia Series, Dr. Richard W. Hamming, a Professor at the Naval Postgraduate School in Monterey, California and a retired Bell Labs scientist, gave a very interesting and stimulating talk, ‘You and Your Research’ to an overflow audience of some 200 Bellcore staff members and visitors at the Morris Research and Engineering Center on March 7, 1986. This talk centered on Hamming’s observations and research on the question “Why do so few scientists make significant contributions and so many are forgotten in the long run?” From his more than forty years of experience, thirty of which were at Bell Laboratories, he has made a number of direct observations, asked very pointed questions of scientists about what, how, and why they did things, studied the lives of great scientists and great contributions, and has done introspection and studied theories of creativity. The talk is about what he has learned in terms of the properties of the individual scientists, their abilities, traits, working habits, attitudes, and philosophy.
“John Tukey”, (2021-01-02):
John Wilder Tukey was an American mathematician best known for development of the Fast Fourier Transform (FFT) algorithm and box plot. The Tukey range test, the Tukey lambda distribution, the Tukey test of additivity, and the Teichmüller–Tukey lemma all bear his name. He is also credited with coining the term 'bit'.
“Bell Labs”, (2020-12-27):
Nokia Bell Labs is an American industrial research and scientific development company owned by Finnish company Nokia. With headquarters located in Murray Hill, New Jersey, the company operates several laboratories in the United States and around the world. Bell Labs has its origins in the complex past of the Bell System.
“Hendrik Wade Bode”, (2021-01-03):
Hendrik Wade Bode was an American engineer, researcher, inventor, author and scientist, of Dutch ancestry. As a pioneer of modern control theory and electronic telecommunications he revolutionized both the content and methodology of his chosen fields of research. His synergy with Claude Shannon, the father of information theory, laid the foundations for the technological convergence of the information age.
“Compound interest”, (2021-01-03):
Compound interest is the addition of interest to the principal sum of a loan or deposit, or in other words, interest on interest. It is the result of reinvesting interest, rather than paying it out, so that interest in the next period is then earned on the principal sum plus previously accumulated interest. Compound interest is standard in finance and economics.
“Hyperbolic discounting”, (2020-12-22):
In economics, hyperbolic discounting is a time-inconsistent model of delay discounting. It is one of the cornerstones of behavioral economics and its brain-basis is actively being studied by neuroeconomics researchers.
http:/
/ www.wired.com/ medtech/ health/ magazine/ 16-05/ ff_wozniak “Wired (magazine)”, (2021-01-03):
Wired is a monthly American magazine, published in print and online editions, that focuses on how emerging technologies affect culture, the economy, and politics. Owned by Condé Nast, it is headquartered in San Francisco, California, and has been in publication since March/April 1993. Several spin-offs have been launched, including Wired UK, Wired Italia, Wired Japan, and Wired Germany. Condé Nast's parent company Advance Publications is also the major shareholder of Reddit, an internet information conglomeration website.
https:/
/ en.wikisource.org/ wiki/ Memory:_A_Contribution_to_Experimental_Psychology “Spaced repetition”, (2021-01-03):
Spaced repetition is an evidence-based learning technique that is usually performed with flashcards. Newly introduced and more difficult flashcards are shown more frequently, while older and less difficult flashcards are shown less frequently in order to exploit the psychological spacing effect. The use of spaced repetition has been proven to increase rate of learning.
“Novum Organum”, (2021-01-03):
The Novum Organum, fully Novum Organum, sive Indicia Vera de Interpretatione Naturae or Instaurationis Magnae, Pars II, is a philosophical work by Francis Bacon, written in Latin and published in 1620. The title is a reference to Aristotle's work Organon, which was his treatise on logic and syllogism. In Novum Organum, Bacon details a new system of logic he believes to be superior to the old ways of syllogism. This is now known as the Baconian method.
“Francis Bacon”, (2021-01-02):
Francis Bacon, 1st Viscount St Alban,, also known as Lord Verulam, was an English philosopher and statesman who served as Attorney General and as Lord Chancellor of England. His works are credited with developing the scientific method and remained influential through the scientific revolution.
“Testing effect”, (2020-12-22):
The testing effect is the finding that long-term memory is often increased when some of the learning period is devoted to retrieving the to-be-remembered information. The effect is also sometimes referred to as retrieval practice, practice testing, or test-enhanced learning. Retrieval practice may be the best way to refer to the testing effect because the benefits of retrieval-related testing are not limited to tests. It can be more broad, including tools like flash cards and quizzes. The testing effect on memory should be distinguished from more general practice effects, defined in the APA Dictionary of Psychology (2007) as "any change or improvement that results from practice or repetition of task items or activities." The term testing effect is also sometimes used in a more general sense; The Oxford Dictionary of Psychology (2003) defines a testing effect as "any effect of taking tests on the respondents, a typical example being test sophistication." Whereas psychologists who develop tests for personality and intelligence want to avoid practice effects, cognitive psychologists working with educators have begun to understand how to take advantage of tests—not as an assessment tool, but as a teaching/learning tool.
http:/
/ www.wsu.edu/ ~fournier/ Teaching/ psych592/ Readings/ Karpicke_et_al_2008.pdf http:/
/ learninglab.psych.purdue.edu/ downloads/ 2007_Karpicke_Roediger_JEPLMC.pdf “The Power of Testing Memory: Basic Research and Implications for Educational Practice”, (2006-09-01):
A powerful way of improving one's memory for material is to be tested on that material. Tests enhance later retention more than additional study of the material, even when tests are given without feedback. This surprising phenomenon is called the testing effect, and although it has been studied by cognitive psychologists sporadically over the years, today there is a renewed effort to learn why testing is effective and to apply testing in educational settings. In this article, we selectively review laboratory studies that reveal the power of testing in improving retention and then turn to studies that demonstrate the basic effects in educational settings. We also consider the related concepts of dynamic testing and formative assessment as other means of using tests to improve learning. Finally, we consider some negative consequences of testing that may occur in certain circumstances, though these negative effects are often small and do not cancel out the large positive effects of testing. Frequent testing in the classroom may boost educational achievement at all levels of education.
https:/
/ pdfs.semanticscholar.org/ 7521/ c9adbe66cb2e777f37b6b00e97f5f95633c2.pdf -
Assigned textbook readings are a common requirement in undergraduate courses, but students often do not complete reading assignments or do not do so until immediately before an exam. This may have detrimental effects on learning and course performance. Regularly scheduled quizzes on reading material may increase completion of reading assignments and therefore course performance. This study examined the effectiveness of compulsory, mastery-based, weekly reading quizzes as a means of improving exam and course performance. Completion of reading quizzes was related to both better exam and course performance. The discussion includes recommendations for the use of quizzes in undergraduate courses.
http:/
/ www.ccsenet.org/ journal/ index.php/ jedp/ article/ download/ 19582/ 15080 “Learning styles”, (2020-12-22):
Learning styles refer to a range of competing and contested theories that aim to account for differences in individuals' learning. Many theories share the proposition that humans can be classified according to their 'style' of learning, but differ in how the proposed styles should be defined, categorized and assessed. A common concept is that individuals differ in how they learn.
http:/
/ www.psychologicalscience.org/ journals/ pspi/ PSPI_9_3.pdf http:/
/ www.cs.colorado.edu/ ~mozer/ Research/ Selected%20Publications/ reprints/ Cepedaetal2009.pdf http:/
/ scholar.sun.ac.za/ bitstream/ handle/ 10019.1/ 80325/ delarouviere_chinese_2012.pdf?sequence=1 “Graduate Record Examinations”, (2020-12-22):
The Graduate Record Examinations (GRE) is a standardized test that is an admissions requirement for many graduate schools in the United States and Canada. The GRE is owned and administered by Educational Testing Service (ETS). The test was established in 1936 by the Carnegie Foundation for the Advancement of Teaching.
-
The spacing effect describes the robust finding that long-term learning is promoted when learning events are spaced out in time, rather than presented in immediate succession. Studies of the spacing effect have focused on memory processes rather than for other types of learning, such as the acquisition and generalization of new concepts. In this study, early elementary school children (5–7 year-olds; n = 36) were presented with science lessons on one of three schedules: massed, clumped, and spaced. The results revealed that spacing lessons out in time resulted in higher generalization performance for both simple and complex concepts. Spaced learning schedules promote several types of learning, strengthening the implications of the spacing effect for educational practices and curriculum. [Keywords: spacing effect, distributed learning, learning and memory, generalization, cognitive development, educational curriculum and practices]
“Long-term Retention of Prose Following Testing”, (2012-01-29):
Testing enhances later recall by highschool students of a history text
“Cloze test”, (2021-01-03):
A cloze test is an exercise, test, or assessment consisting of a portion of language with certain items, words, or signs removed, where the participant is asked to replace the missing language item. Cloze tests require the ability to understand context and vocabulary in order to identify the correct language or part of speech that belongs in the deleted passages. This exercise is commonly administered for the assessment of native and second language learning and instruction.
http:/
/ www.eric.ed.gov/ ERICWebPortal/ recordDetail?accno=EJ761434 “Distributed Practice in Verbal Recall Tasks: A Review and Quantitative Synthesis”, (2006):
The authors performed a meta-analysis of the distributed practice effect to illuminate the effects of temporal variables that have been neglected in previous reviews. This review found 839 assessments of distributed practice in 317 experiments located in 184 articles. Effects of spacing (consecutive massed presentations vs. spaced learning episodes) and lag (less spaced vs. more spaced learning episodes) were examined, as were expanding inter study interval (ISI) effects. Analyses suggest that ISI and retention interval operate jointly to affect final-test retention; specifically, the ISI producing maximal retention increased as retention interval increased. Areas needing future research and theoretical implications are discussed.
http:/
/ www.amazon.com/ Principles-Learning-Memory-Experimental-Psychology/ dp/ 0898591155/ http:/
/ www.amazon.com/ Human-Memory-Paradigms-Robert-Greene/ dp/ 080580997X/ http:/
/ citeseerx.ist.psu.edu/ viewdoc/ download?doi=10.1.1.200.8846&rep=rep1&type=pdf http:/
/ act-r.psy.cmu.edu/ wordpress/ wp-content/ themes/ ACT-R/ workshops/ 2003/ proceedings/ 46.pdf -
The spacing effect is one of the most ubiquitous findings in learning and memory.Performance on a variety of tasks is better when the repetition of the to-be-learned information is distributed as opposed to massed in presentation. This observation was first formalized in Jost’s law, which states that “if two associations are of equal strength but of different age, a new repetition has a greater value for the older one” (McGeogh, 1943). Spacing effects occur across domains (e.g., learning perceptual motor tasks vs. learning lists of words), across species (e.g., rats, pigeons, and humans), across age groups and individuals with different memory impairments,and across retention intervals of seconds to months (see Cepeda, Pashler, Vul,Wixted, & Rohrer, 2006; Crowder, 1976; Dempster, 1996, for reviews). In this light, it is interesting that spacing effects have not received much attention in Cognitive Psychology textbooks. In fact, in our sampling of seven such textbooks,only one had a section dedicated to this topic, while virtually all cognitive text-books discussed mnemonic techniques such as the pegword or method of loci. Given the power and simplicity of implementing spaced practice, we clearly hope this changes in the future.
“Encoding (memory)”, (2020-12-22):
Memory has the ability to encode, store and recall information. Memories give an organism the capability to learn and adapt from previous experiences as well as build relationships. Encoding allows a perceived item of use or interest to be converted into a construct that can be stored within the brain and recalled later from long-term memory. Working memory stores information for immediate use or manipulation which is aided through hooking onto previously archived items already present in the long-term memory of an individual.
http:/
/ psychnet.wustl.edu/ coglab/ wp-content/ uploads/ 2015/ 01/ 1989-Balota.pdf http:/
/ journal.frontiersin.org/ Journal/ 10.3389/ fnhum.2013.00589/ full http:/
/ public.psych.iastate.edu/ shacarp/ Carpenter_Delosh_2005.pdf http:/
/ uweb.cas.usf.edu/ ~drohrer/ pdfs/ Rohrer&Taylor2006ACP.pdf “Effect size”, (2020-12-22):
In statistics, an effect size is a number measuring the strength of the relationship between two variables in a statistical population, or a sample-based estimate of that quantity. It can refer to the value of a statistic calculated from a sample of data, the value of a parameter of a hypothetical statistical population, or to the equation that operationalizes how statistics or parameters lead to the effect size value. Examples of effect sizes include the correlation between two variables, the regression coefficient in a regression, the mean difference, or the risk of a particular event happening. Effect sizes complement statistical hypothesis testing, and play an important role in power analyses, sample size planning, and in meta-analyses. The cluster of data-analysis methods concerning effect sizes is referred to as estimation statistics.
http:/
/ jenni.uchicago.edu/ Spencer_Conference/ Representative%20Papers/ Richard%20et%20al,%202003.pdf http:/
/ www.pashler.com/ Articles/ Pashler_Zarow_Triplett03.pdf /
docs/ spacedrepetition/ 1989-gilmore-memoryaginganddementia.pdf https:/
/ pdfs.semanticscholar.org/ 0eb4/ 8078fdc906f368d22f324e0520e4ee4f9c08.pdf “Permutation”, (2021-01-03):
In mathematics, a permutation of a set is, loosely speaking, an arrangement of its members into a sequence or linear order, or if the set is already ordered, a rearrangement of its elements. The word "permutation" also refers to the act or process of changing the linear order of an ordered set.
“Polyhedron”, (2021-01-03):
In geometry, a polyhedron is a three-dimensional shape with flat polygonal faces, straight edges and sharp corners or vertices. The word polyhedron comes from the Classical Greek πολύεδρον, as poly- + -hedron.
http:/
/ people.tamu.edu/ ~stevesmith/ SmithMemory/ SmithRothkopf1984.pdf “United States Air Force Academy”, (2021-01-02):
The United States Air Force Academy is a military academy for officer cadets of the United States Air Force and United States Space Force immediately north of Colorado Springs, Colorado. It is the youngest of the five United States service academies, having graduated its first class 62 years ago in 1959, but is the third in seniority. Graduates of the academy's four-year program receive a Bachelor of Science degree and are commissioned as second lieutenants in the U.S. Air Force or U.S. Space Force. The academy is also one of the largest tourist attractions in Colorado, attracting approximately a million visitors each year.
“The effects of cumulative practice on mathematics problem solving”, (2002):
This study compared three different methods of teaching five basic algebra rules to college students. All methods used the same procedures to teach the rules and included four 50-question review sessions interspersed among the training of the individual rules. The differences among methods involved the kinds of practice provided during the four review sessions. Participants who received cumulative practice answered 50 questions covering a mix of the rules learned prior to each review session. Participants who received a simple review answered 50 questions on one previously trained rule. Participants who received extra practice answered 50 extra questions on the rule they had just learned. Tests administered after each review included new questions for applying each rule (application items) and problems that required novel combinations of the rules (problem-solving items). On the final test, the cumulative group outscored the other groups on application and problem-solving items. In addition, the cumulative group solved the problem-solving items significantly faster than the other groups. These results suggest that cumulative practice of component skills is an effective method of training problem solving.
https:/
/ getd.libs.uga.edu/ pdfs/ sutherland_pierre_201212_ma.pdf http:/
/ qualitysafety.bmj.com/ content/ early/ 2012/ 06/ 15/ bmjqs-2011-000702.abstract https:/
/ pdfs.semanticscholar.org/ f2b5/ aed794e5d164065a184207f2663620b96ba3.pdf “Teaching surgical skills: what kind of practice makes perfect?: a randomized, controlled trial”, (2006):
Surgical skills laboratories have become an important venue for early skill acquisition. The principles that govern training in this novel educational environment remain largely unknown; the commonest method of training, especially for continuing medical education (CME), is a single multihour event. This study addresses the impact of an alternative method, where learning is distributed over a number of training sessions. The acquisition and transfer of a new skill to a life-like model is assessed. Thirty-eight junior surgical residents, randomly assigned to either massed (1 day) or distributed (weekly) practice regimens, were taught a new skill (microvascular anastomosis). Each group spent the same amount of time in practice. Performance was assessed pretraining, immediately post-training, and 1 month post-training. The ultimate test of anastomotic skill was assessed with a transfer test to a live, anesthetized rat. Previously validated computer-based and expert-based outcome measures were used. In addition, clinically relevant outcomes were assessed. Both groups showed immediate improvement in performance, but the distributed group performed significantly better on the retention test in most outcome measures (time, number of hand movements, and expert global ratings; all P values <0.05). The distributed group also outperformed the massed group on the live rat anastomosis in all expert-based measures (global ratings, checklist score, final product analysis, competency for OR; all P values <0.05). Our current model of training surgical skills using short courses (for both CME and structured residency curricula) may be suboptimal. Residents retain and transfer skills better if taught in a distributed manner. Despite the greater logistical challenge, we need to restructure training schedules to allow for distributed practice.
http:/
/ laplab.ucsd.edu/ articles/ CarpenterPashlerCepeda2009.pdf http:/
/ www.frontiersin.org/ Journal/ 10.3389/ fnhum.2013.00589/ full http:/
/ www.usenix.org/ system/ files/ conference/ usenixsecurity14/ sec14-paper-bonneau.pdf “Spaced Repetition and Mnemonics Enable Recall of Multiple Strong Passwords”, (2014-10-06):
We report on a user study that provides evidence that spaced repetition and a specific mnemonic technique enable users to successfully recall multiple strong passwords over time. Remote research participants were asked to memorize 4 Person-Action-Object (PAO) stories where they chose a famous person from a drop-down list and were given machine-generated random action-object pairs. Users were also shown a photo of a scene and asked to imagine the PAO story taking place in the scene (e.g., Bill Gates—swallowing—bike on a beach). Subsequently, they were asked to recall the action-object pairs when prompted with the associated scene-person pairs following a spaced repetition schedule over a period of 127+ days. While we evaluated several spaced repetition schedules, the best results were obtained when users initially returned after 12 hours and then in increasing intervals: 77 successfully recalled all 4 stories in 10 tests over a period of 158 days. Much of the forgetting happened in the first test period (12 hours): 89 participants who remembered their stories during the first test period successfully remembered them in every subsequent round. These findings, coupled with recent results on naturally rehearsing password schemes, suggest that 4 PAO stories could be used to create usable and strong passwords for 14 sensitive accounts following this spaced repetition schedule, possibly with a few extra upfront rehearsals. In addition, we find that there is an interference effect across multiple PAO stories: the recall rate of 100 90 stories) is significantly better than the recall rate for participants who were asked to memorize 4 PAO stories. These findings yield concrete advice for improving constructions of password management schemes and future user studies.
“The Complexity of Human Computation: A Concrete Model with an Application to Passwords”, (2017-07-05):
What can humans compute in their heads? We are thinking of a variety of Crypto Protocols, games like Sudoku, Crossword Puzzles, Speed Chess, and so on. The intent of this paper is to apply the ideas and methods of theoretical computer science to better understand what humans can compute in their heads. For example, can a person compute a function in their head so that an eavesdropper with a powerful computer—who sees the responses to random input—still cannot infer responses to new inputs? To address such questions, we propose a rigorous model of human computation and associated measures of complexity. We apply the model and measures first and foremost to the problem of (1) humanly computable password generation, and then consider related problems of (2) humanly computable "one-way functions" and (3) humanly computable "pseudorandom generators".
The theory of Human Computability developed here plays by different rules than standard computability, and this takes some getting used to. For reasons to be made clear, the polynomial versus exponential time divide of modern computability theory is irrelevant to human computation. In human computability, the step-counts for both humans and computers must be more concrete. Specifically, we restrict the adversary to at most 1024 (Avogadro number of) steps. An alternate view of this work is that it deals with the analysis of algorithms and counting steps for the case that inputs are small as opposed to the usual case of inputs large-in-the-limit.
“Deconstructing memory in Drosophila”, (2005):
Unlike most organ systems, which have evolved to maintain homeostasis, the brain has been selected to sense and adapt to environmental stimuli by constantly altering interactions in a gene network that functions within a larger neural network. This unique feature of the central nervous system provides a remarkable plasticity of behavior, but also makes experimental investigations challenging. Each experimental intervention ramifies through both gene and neural networks, resulting in unpredicted and sometimes confusing phenotypic adaptations. Experimental dissection of mechanisms underlying behavioral plasticity ultimately must accomplish an integration across many levels of biological organization, including genetic pathways acting within individual neurons, neural network interactions which feed back to gene function, and phenotypic observations at the behavioral level. This dissection will be more easily accomplished for model systems such as Drosophila, which, compared with mammals, have relatively simple and manipulable nervous systems and genomes. The evolutionary conservation of behavioral phenotype and the underlying gene function ensures that much of what we learn in such model systems will be relevant to human cognition. In this essay, we have not attempted to review the entire Drosophila memory field. Instead, we have tried to discuss particular findings that provide some level of intellectual synthesis across three levels of biological organization: behavior, neural circuitry and biochemical pathways. We have attempted to use this integrative approach to evaluate distinct mechanistic hypotheses, and to propose critical experiments that will advance this field.
-
Conditioning the proboscis extension reflex of harnessed honeybees (Apis mellifera) is used to study the effect temporal spacing between successive conditioning trials has on memory. Retention is monitored at two long-term intervals corresponding to early (1 and 2 d after conditioning) and late long-term memory (3 and 4 d). The acquisition level is varied by using different conditioned stimuli (odors, mechanical stimulation, and temperature increase at the antenna), varying strengths of the unconditioned stimulus (sucrose), and various numbers of conditioning trials. How learning trials are spaced is the dominant factor both for acquisition and retention, and although longer intertrial intervals lead to better acquisition and higher retention, the level of acquisition per se does not determine the spacing effect on retention. Rather, spaced conditioning leads to higher memory consolidation both during acquisition and later, between the early and long-term memory phases. These consolidation processes can be selectively inhibited by blocking protein synthesis during acquisition.
-
In Aplysia, three distinct phases of memory for sensitization can be dissociated based on their temporal and molecular features. A single training trial induces short-term memory (STM, lasting <30 min), whereas five trials delivered at 15-min intervals induces both intermediate-term memory (ITM, lasting >90 min) and long-term memory (LTM, lasting >24 h). Here, we explore the interaction of amount and pattern of training in establishing ITM and LTM by examining memory for sensitization after different numbers of trials (each trial = one tail shock) and different patterns of training (massed vs. spaced). Under spaced training patterns, two trials produced STM exclusively, whereas four or five trials each produced both ITM and LTM. Three spaced trials failed to induce LTM but did produce an early decaying form of ITM (E-ITM) that was significantly shorter and weaker in magnitude than the late-decaying ITM (L-ITM) observed after four to five trials. In addition, E-ITM was induced after three trials with both massed and spaced patterns of training. However, L-ITM and LTM after four to five trials require spaced training: Four or five massed trials failed to induce LTM and produced only E-ITM. Collectively, our results indicate that in addition to three identified phases of memory for sensitization—STM, ITM, and LTM—a unique temporal profile of memory, E-ITM, is revealed by varying either the amount or pattern of training.
https:/
/ link.springer.com/ content/ pdf/ 10.3758%2FBF03197590.pdf http:/
/ www.sciencedirect.com/ science/ article/ pii/ S1552526012001318 http:/
/ openscholarship.wustl.edu/ cgi/ viewcontent.cgi?article=2147&context=etd http:/
/ digitalcommons.usu.edu/ cgi/ viewcontent.cgi?article=4569&context=etd “Practice (learning method)”, (2020-12-22):
Practice is the act of rehearsing a behaviour over and over, or engaging in an activity again and again, for the purpose of improving or mastering it, as in the phrase 'practice makes perfect'. It is important to note that practise is a verb and should not be confused with the noun practice. Sports teams practise to prepare for actual games. Playing a musical instrument well takes much practice. It is a method of learning and of acquiring experience. The word derives from the Greek "πρακτική" (praktike), feminine of "πρακτικός" (praktikos), "fit for or concerned with action, practical", and that from the verb "πράσσω" (prasso), "to achieve, bring about, effect, accomplish". In English, practice is the noun and practise is the verb, but in American-English it is now common for practice to be used both as a noun and a verb.
“Variance”, (2021-01-02):
In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its mean. Informally, it measures how far a set of numbers is spread out from their average value. Variance has a central role in statistics, where some ideas that use it include descriptive statistics, statistical inference, hypothesis testing, goodness of fit, and Monte Carlo sampling. Variance is an important tool in the sciences, where statistical analysis of data is common. The variance is the square of the standard deviation, the second central moment of a distribution, and the covariance of the random variable with itself, and it is often represented by σ2, s2, or Var(X).
https:/
/ scholar.google.com/ scholar?q=author%3AHambrick%20%22deliberate%20practice%22 “The role of deliberate practice in the acquisition of expert performance”, (1993-07):
The theoretical framework presented in this article explains expert performance as the end result of individuals' prolonged efforts to improve performance while negotiating motivational and external constraints. In most domains of expertise, individuals begin in their childhood a regimen of effortful activities (deliberate practice) designed to optimize improvement. Individual differences, even among elite performers, are closely related to assessed amounts of deliberate practice. Many characteristics once believed to reflect innate talent are actually the result of intense practice extended for a minimum of 10 yrs. Analysis of expert performance provides unique evidence on the potential and limits of extreme environmental adaptation and learning.
“Studies in the physiology and psychology of the telegraphic language”, (1897):
Studied individual differences in telegraphic writing. A preliminary study was conducted, in which operators were cross-examined on aspects of psychological or physiological importance. On the basis of this, a study was undertaken on 60 Ss, who were asked to write a sentence requiring attention. There were constant differences required in the times for a given character. Further tests were made, and schools were requested to provide typical curves of improvement. Results reveal that there were distinct specialties in telegraphy. The rate of receiving varied greatly, and exceeded sending rate. Both external and subjective disturbances affected inexperienced operators. The best age to learn telegraphy was 18–30 yrs. The variations in the value of a character depended on its place in the sentence. Homotaxic variation was an inverse measure of skill, while the inflection variation increased with expertise.
“Studies on the telegraphic language: The acquisition of a hierarchy of habits”, (1899-01-01):
Investigated the different stages involved in learning telegraphy. One S was tested each week on: (1) rate of receiving letters not making words, (2) rate of receiving letters making words, but not sentences, and (3) rate of receiving letters making words and sentences. Results indicate that a hierarchy of psycho-physical habits were required to receive the telegraphic language. From an early period, letter, word and higher habits made gains together, but not equally. No plateau appeared between the learning of letters and words; the first one occurred after the learning of words. Later, there was a second ascent, representing the acquisition of higher language habits. Effective speed was largely dependent upon the mastery of these habits, which led to greater accuracy in detail. Concluded that the rate of progress, depended partly on the rate of mental and nervous processes, but far more on how much was included in each process.
/
docs/ spacedrepetition/ 1921-thorndike-educationalpsychology-v2-thepsychologyoflearning.pdf#page=188 https:/
/ pdfs.semanticscholar.org/ c747/ 336e77ce39f7f5cdbd937684a7f564e9e194.pdf http:/
/ digitalcommons.mtu.edu/ cgi/ viewcontent.cgi?article=1486&context=etds http:/
/ www.amazon.com/ Motor-Control-Learning-Behavioral-Emphasis/ dp/ 0880114843/ http:/
/ citeseerx.ist.psu.edu/ viewdoc/ download?doi=10.1.1.574.5400&rep=rep1&type=pdf http:/
/ www.cell.com/ cms/ attachment/ 2045379445/ 2056784269/ mmc2.pdf http:/
/ www.sciencedirect.com/ science/ article/ pii/ S0896627311009184 https:/
/ pdfs.semanticscholar.org/ 7e0c/ 1bb80cbc332f07bda26a75a163a9cf76d591.pdf “Halo: Reach”, (2021-01-03):
Halo: Reach is a 2010 first-person shooter developed by Bungie and published by Microsoft Game Studios, originally for the Xbox 360. The fifth installment in the Halo series and a direct prequel to Halo: Combat Evolved, Reach was released worldwide in September 2010. The game takes place in the year 2552, where humanity is locked in a war with the alien Covenant. Players control Noble Six, a member of an elite supersoldier squad, when the human world known as Reach falls under Covenant attack.
http:/
/ citeseerx.ist.psu.edu/ viewdoc/ download?doi=10.1.1.58.2647&rep=rep1&type=pdf http:/
/ cognitrn.psych.indiana.edu/ rgoldsto/ interrelated/ interrelated.html http:/
/ babytalk.psych.ucla.edu/ documents/ Vlach_Sandhofer_Kornell_2008.pdf https:/
/ pdfs.semanticscholar.org/ be1a/ 70be5a6f990cd86804ccc1be29331556ddfc.pdf http:/
/ laplab.ucsd.edu/ articles/ McDaniel.Fadler.Pashler2013.pdf “Is spacing really the "friend of induction"?”, (2014):
Inductive learning takes place when people learn a new concept or category by observing a variety of exemplars. Kornell and Bjork (2008) asked participants to learn new painting styles either by presenting different paintings of the same artist consecutively (massed presentation) or by mixing paintings of different artists (spaced presentation). In their second experiment, Kornell and Bjork (2008) showed with a final style recognition test, that spacing resulted in better inductive learning than massing. Also, by using this style recognition test, they ruled out the possibility that spacing merely resulted in a better memory for the labels of the newly learned painting styles. The findings from Kornell and Bjork's (2008) second experiment are important because they show that the benefit of spaced learning generalizes to complex learning tasks and outcomes, and that it is not confined to rote memory learning. However, the findings from Kornell and Bjork's (2008) second experiment have never been replicated. In the present study we performed an exact and high-powered replication of Kornell and Bjork's (2008) second experiment with a Web-based sample. Such a replication contributes to establish the reliability of the original finding and hence to more conclusive evidence of the spacing effect in inductive learning. The findings from the present replication attempt revealed a medium-sized advantage of spacing over massing in inductive learning, which was comparable to the original effect in the experiment by Kornell and Bjork (2008). Also, the 95% confidence intervals (CI) of the effect sizes from both experiments overlapped considerably. Hence, the findings from the present replication experiment and the original experiment clearly reinforce each other.
http:/
/ babytalk.psych.ucla.edu/ documents/ Gluckman_Vlach_Sandhofer_2014.pdf http:/
/ www.columbia.edu/ cu/ psychology/ metcalfe/ PDFs/ Son2010.pdf “Mnemosyne (software)”, (2021-01-02):
Mnemosyne is a line of spaced repetition software developed from 2003 until the present. Spaced repetition is an evidence-based learning technique that has been shown to increase the rate of memorization.
“Free software”, (2021-01-03):
Free software is computer software distributed under terms that allow users to run the software for any purpose as well as to study, change, and distribute it and any adapted versions. Free software is a matter of liberty, not price: all users are legally free to do what they want with their copies of a free software regardless of how much is paid to obtain the program. Computer programs are deemed "free" if they give end-users ultimate control over the software and, subsequently, over their devices.
“Ubuntu”, (2021-01-03):
Ubuntu is a Linux distribution based on Debian and mostly composed of free and open-source software. Ubuntu is officially released in three editions: Desktop, Server, and Core for Internet of things devices and robots. All the editions can run on the computer alone, or in a virtual machine. Ubuntu is a popular operating system for cloud computing, with support for OpenStack. Ubuntu's default desktop has been GNOME, since version 17.10.
“Anki (software)”, (2021-01-03):
Anki is a free and open-source flashcard program using spaced repetition, a technique from cognitive science for fast and long-lasting memorization. "Anki" (暗記) is the Japanese word for "memorization". The SM2 algorithm, created for SuperMemo in the late 1980s, forms the basis of the spaced repetition methods employed in the program. Anki's implementation of the algorithm has been modified to allow priorities on cards and to show flashcards in order of their urgency. The cards are presented using HTML and may include text, images, sounds, videos, and LaTeX equations. The decks of cards, along with the user's statistics, are stored in the open SQLite format.
“Scientific notation”, (2020-12-22):
Scientific notation is a way of expressing numbers that are too large or too small to be conveniently written in decimal form. It may be referred to as scientific form or standard index form, or standard form in the UK. This base ten notation is commonly used by scientists, mathematicians, and engineers, in part because it can simplify certain arithmetic operations. On scientific calculators it is usually known as "SCI" display mode.
/
docs/ spacedrepetition/ 2012-muflax-dreamingofaworldundone.html.maff “Piotr Woźniak (researcher)”, (2020-12-22):
Piotr A. Woźniak is a Polish researcher best known for his work on SuperMemo, a learning system based on spaced repetition.
http:/
/ www.wired.com/ medtech/ health/ magazine/ 16-05/ ff_wozniak?currentPage=3 http:/
/ frank.itlab.us/ forgetting/ mtsu_forgetting/ #II.%20Decay%20Theory http:/
/ frank.itlab.us/ forgetting/ mtsu_forgetting/ woodworth.jpg “Jorge Luis Borges”, (2020-12-27):
Jorge Francisco Isidoro Luis Borges Acevedo was an Argentine short-story writer, essayist, poet and translator, and a key figure in Spanish-language and universal literature. His best-known books, Ficciones (Fictions) and El Aleph, published in the 1940s, are compilations of short stories interconnected by common themes, including dreams, labyrinths, philosophers, libraries, mirrors, fictional writers, and mythology. Borges' works have contributed to philosophical literature and the fantasy genre, and have been considered by some critics to mark the beginning of the magic realist movement in 20th century Latin American literature. His late poems converse with such cultural figures as Spinoza, Camões, and Virgil.
“Funes the Memorious”, (2020-12-22):
"Funes the Memorious" is a fantasy short story by Argentine writer Jorge Luis Borges (1899–1986). First published in La Nación of June 1942, it appeared in the 1944 anthology Ficciones, part two (Artifices). The first English translation appeared in 1954 in Avon Modern Writing No. 2.
“Anu Garg”, (2021-01-03):
Anu Garg is an American author and speaker. He is also the founder of Wordsmith.org, an online community comprising word lovers from an estimated 195 countries. His books explore the joy of words. He has authored several books about language-related issues and written for magazines and newspapers. He was a columnist for MSN Encarta and Kahani magazine.
“Wiktionary”, (2021-01-03):
Wiktionary is a multilingual, web-based project to create a free content dictionary of terms in all natural languages and in a number of artificial languages. These entries may contain definitions, images for illustrations, pronunciations, etymologies, inflections, usage examples, quotations, related terms, and translations of words into other languages, among other features. It is collaboratively edited via a wiki. Its name is a portmanteau of the words wiki and dictionary. It is available in 171 languages and in Simple English. Like its sister project Wikipedia, Wiktionary is run by the Wikimedia Foundation, and is written collaboratively by volunteers, dubbed "Wiktionarians". Its wiki software, MediaWiki, allows almost anyone with access to the website to create and edit entries.
“Notebook (style)”, (2020-12-22):
Notebook is a style of writing where people jot down what they have thought or heard at the spur of moment. The contents of a notebook are unorganized, and the number of subjects covered in a notebook are unlimited: a paragraph of autobiography can be followed immediately by one on astronomy or one on history.
“Hangul”, (2021-01-02):
The Korean alphabet, known as Hangul (Hangeul) in South Korea and Chosŏn'gŭl in North Korea, is a writing system for the Korean language created by King Sejong the Great in 1443. The letters for the five basic consonants reflect the shape of the speech organs used to pronounce them, and they are systematically modified to indicate phonetic features; similarly, the vowel letters are systematically modified for related sounds, making Hangul a featural writing system.
“A. E. Housman”, (2021-01-02):
Alfred Edward Housman, usually known as A. E. Housman, was an English classical scholar and poet. His cycle of poems, A Shropshire Lad wistfully evoke the dooms and disappointments of youth in the English countryside. Their simplicity and distinctive imagery appealed strongly to Edwardian taste, and to many early 20th-century English composers both before and after the First World War. Through their song-settings, the poems became closely associated with that era, and with Shropshire itself.
http:/
/ lesswrong.com/ lw/ e26/ who_wants_to_start_an_important_startup/ 78ig http:/
/ lesswrong.com/ lw/ e26/ who_wants_to_start_an_important_startup/ 7dub?context=3 http:/
/ lesswrong.com/ lw/ e26/ who_wants_to_start_an_important_startup/ 8fgo?context=3 /
images/ spacedrepetition/ 2012-chessdata-perfectpitchspacedrepetition.webm “Memorising Milton's Paradise Lost: A study of a septuagenarian exceptional memoriser”, (2020-04-23):
At age 58, JB [John Basinger] began memorizing Milton’s epic poem Paradise Lost. 9 years and thousands of study hours later, he completed this process in 2001 and recalled from memory all 12 books of this 10,565-line poem over a 3-day period. Now 74, JB continues to recite this work. We tested his memory accuracy by cueing his recall with two lines from the beginning or middle of each book and asking JB to recall the next 10 lines. JB is an exceptional memoriser of Milton, both in our laboratory tests in which he did not know the specific tests or procedures in advance, and in our analysis of a videotaped, prepared performance. Consistent with deliberate practice theory, JB achieved this remarkable ability by deeply analysing the poem’s structure and meaning over lengthy repetitions. Our findings suggest that exceptional memorizers such as JB are made, not born, and that cognitive expertise can be demonstrated even in later adulthood. [Keywords: Exceptional memory, Prose memory, Age and memory]
“Hafiz (Quran)”, (2021-01-03):
Hafiz, literally meaning "guardian" or "memorizer", depending on the context, is a term used by Muslims for someone who has completely memorized the Quran. Hafiza is the female equivalent.
“Hadith”, (2021-01-03):
Ḥadīth or Athar in Islam refers to what Muslims believe to be a record of the words, actions, and the silent approval of the Islamic prophet Muhammad. Hadith have been called "the backbone" of Islamic civilization, and within that religion the authority of hadith as a source for religious law and moral guidance ranks second only to that of the Quran. Scriptural authority for hadith comes from the Quran which enjoins Muslims to emulate Muhammad and obey his judgments. While the number of verses pertaining to law in the Quran is relatively few, hadith give direction on everything from details of religious obligations, to the correct forms of salutations and the importance of benevolence to slaves. Thus the "great bulk" of the rules of Sharia are derived from hadith, rather than the Quran.
“Recognition memory”, (2020-12-22):
Recognition memory, a subcategory of declarative memory, is the ability to recognize previously encountered events, objects, or people. When the previously experienced event is reexperienced, this environmental content is matched to stored memory representations, eliciting matching signals. As first established by psychology experiments in the 1970s, recognition memory for pictures is quite remarkable: humans can remember thousands of images at high accuracy after seeing each only once and only for a few seconds.
“Multiplying 10-digit numbers using Flickr: The power of recognition memory”, (2011):
In this informal article, I’ll describe the “recognition method”—a simple, powerful technique for memorization and mental calculation. Compared to traditional memorization techniques, which use elaborate encoding and visualization processes [1], the recognition method is easy to learn and requires relatively little effort…The method works: using it, I was able to mentally multiply two random 10-digit numbers, by the usual grade-school algorithm, on my first attempt! I have a normal, untrained memory, and the task would have been impossible by a direct approach. (I can’t claim I was speedy: I worked slowly and carefully, using about 7 hours plus rest breaks. I practiced twice with 5-digit numbers beforehand.)
…It turns out that ordinary people are incredibly good at this task [recognizing whether a photograph has been seen before]. In one of the most widely-cited studies on recognition memory, Standing [2] showed participants an epic 10,000 photographs over the course of 5 days, with 5 seconds’ exposure per image. He then tested their familiarity, essentially as described above. The participants showed an 83% success rate, suggesting that they had become familiar with about 6,600 images during their ordeal. Other volunteers, trained on a smaller collection of 1,000 images selected for vividness, had a 94% success rate.
https:/
/ www.usenix.org/ conference/ usenixsecurity12/ technical-sessions/ presentation/ bojinov “Noisy-channel coding theorem”, (2020-12-22):
In information theory, the noisy-channel coding theorem, establishes that for any given degree of noise contamination of a communication channel, it is possible to communicate discrete data nearly error-free up to a computable maximum rate through the channel. This result was presented by Claude Shannon in 1948 and was based in part on earlier work and ideas of Harry Nyquist and Ralph Hartley.
“Frank Herbert”, (2021-01-02):
Franklin Patrick Herbert Jr. was an American science-fiction author best known for the 1965 novel Dune and its five sequels. Though he became famous for his novels, he also wrote short stories and worked as a newspaper journalist, photographer, book reviewer, ecological consultant, and lecturer.
http:/
/ www.cs.colorado.edu/ ~mozer/ Teaching/ syllabi/ 7782/ readings/ Landauer1986.pdf “Hyperthymesia”, (2020-12-22):
Hyperthymesia is a condition that leads people to be able to remember an abnormally large number of their life experiences in vivid detail.
“Jill Price”, (2020-12-22):
Jill Price is an American woman, from Southern California, who has been diagnosed with hyperthymesia. She was the first person to receive such a diagnosis, and it was her case that inspired research into hyperthymesia. She has co-authored a book on the subject.
“Memory consolidation”, (2020-12-22):
Memory consolidation is a category of processes that stabilize a memory trace after its initial acquisition. A memory trace is a change in the nervous system caused by memorizing something. Consolidation is distinguished into two specific processes. The first, synaptic consolidation, which is synonymous with late-phase long-term potentiation, occurs on a small scale in the synaptic connections and neural circuits within the first few hours after learning. The second process is systems consolidation, where hippocampus-dependent memories become independent of the hippocampus over a period of weeks to years, and it occurs on a much larger scale in the brain. Recently, a third process has become the focus of research, reconsolidation, in which previously consolidated memories can be made labile again through reactivation of the memory trace.
“Sleep and memory”, (2021-01-03):
The relationship between sleep and memory has been studied since at least the early 19th century. Memory, the cognitive process of storing and retrieving past experiences, learning and recognition, is a product of brain plasticity, the structural changes within synapses that create associations between stimuli. Stimuli are encoded within milliseconds; however, the long-term maintenance of memories can take additional minutes, days, or even years to fully consolidate and become a stable memory that is accessible. Therefore, the formation of a specific memory occurs rapidly, but the evolution of a memory is often an ongoing process.
http:/
/ www.plosone.org/ article/ info%3Adoi%2F10.1371%2Fjournal.pone.0033079 “Biphasic and polyphasic sleep”, (2020-12-22):
Biphasic sleep is the practice of sleeping during two periods over the course of 24 hours, while polyphasic sleep refers to sleeping multiple times – usually more than two. Each of these is in contrast to monophasic sleep, which is one period of sleep within 24 hours. Segmented sleep and divided sleep may refer to polyphasic or biphasic sleep, but may also refer to interrupted sleep, where the sleep has one or several shorter periods of wakefulness. A common form of biphasic or polyphasic sleep includes a nap, which is a short period of sleep, typically taken between the hours of 9 am and 9 pm as an adjunct to the usual nocturnal sleep period. Napping behavior during daytime hours is the simplest form of polyphasic sleep, especially when the nap(s) are taken on a daily basis. Therefore, biphasic sleep is considered the most quasi form of polyphasic sleep, thanks to the long sleep duration at night. Nowadays, the definition of polyphasic sleep is any sleep schedule with at least two sleeps per day, to distinguish it from monophasic sleep, which only has one sleep per day.
-
Information that is spaced over time is better remembered than the same amount of information massed together. This phenomenon, known as the spacing effect, was explored with respect to its effect on learning and neurogenesis in the adult dentate gyrus of the hippocampal formation. Because the cells are generated over time and because learning enhances their survival, we hypothesized that training with spaced trials would rescue more new neurons from death than the same number of massed trials. In the first experiment, animals trained with spaced trials in the Morris water maze outperformed animals trained with massed trials, but there was not a direct effect of trial spacing on cell survival. Rather, animals that learned well retained more cells than animals that did not learn or learned poorly. Moreover, performance during acquisition correlated with the number of cells remaining in the dentate gyrus after training. In the second experiment, the time between blocks of trials was increased. Consequently, animals trained with spaced trials performed as well as those trained with massed, but remembered the location better two weeks later. The strength of that memory correlated with the number of new cells remaining in the hippocampus. Together, these data indicate that learning, and not mere exposure to training, enhances the survival of cells that are generated 1 wk before training. They also indicate that learning over an extended period of time induces a more persistent memory, which then relates to the number of cells that reside in the hippocampus.
“Reconsolidation: maintaining memory relevance”, (2009):
The retrieval of a memory places it into a plastic state, the result of which is that the memory can be disrupted or even enhanced by experimental treatment. This phenomenon has been conceptualised within a framework of memories being reactivated and then reconsolidated in repeated rounds of cellular processing. The reconsolidation phase has been seized upon as crucial for the understanding of memory stability and, more recently, as a potential therapeutic target in the treatment of disorders such as post-traumatic stress and drug addiction. However, little is known about the reactivation process, or what might be the adaptive function of retrieval-induced plasticity. Reconsolidation has long been proposed to mediate memory updating, but only recently has this hypothesis been supported experimentally. Here, the adaptive function of memory reconsolidation is explored in more detail, with a strong emphasis on its role in updating memories to maintain their relevance.
http:/
/ www.amazon.com/ Fundamental-Neuroscience-Second-Larry-Squire/ dp/ 0126603030/ http:/
/ davidjf.free.fr/ new/ Xtech_scientific%20american_%20making%20memories%20stick.pdf https:/
/ pdfs.semanticscholar.org/ 45c2/ 7c08fbb43f8728e69a7447366d4a4f74e088.pdf https:/
/ pdfs.semanticscholar.org/ d0a7/ f06ed267f3193daab1175a65abb7a067bef4.pdf https:/
/ pdfs.semanticscholar.org/ 77fd/ 0cf03de8a6c4f56c5decc7c47ebe69cf98c1.pdf “Space–time tradeoff”, (2020-12-22):
A space–time or time–memory trade-off in computer science is a case where an algorithm or program trades increased space usage with decreased time. Here, space refers to the data storage consumed in performing a given task, and time refers to the time consumed in performing a given task.
“Function (mathematics)”, (2021-01-03):
In mathematics, a function is a binary relation between two sets that associates every element of the first set to exactly one element of the second set. Typical examples are functions from integers to integers, or from the real numbers to real numbers.
“Lookup table”, (2020-12-22):
In computer science, a lookup table is an array that replaces runtime computation with a simpler array indexing operation. The savings in processing time can be significant, because retrieving a value from memory is often faster than carrying out an "expensive" computation or input/output operation. The tables may be precalculated and stored in static program storage, calculated as part of a program's initialization phase (memoization), or even stored in hardware in application-specific platforms. Lookup tables are also used extensively to validate input values by matching against a list of valid items in an array and, in some programming languages, may include pointer functions to process the matching input. FPGAs also make extensive use of reconfigurable, hardware-implemented, lookup tables to provide programmable hardware functionality.
“Rainbow table”, (2020-12-22):
A rainbow table is a precomputed table for caching the output of cryptographic hash functions, usually for cracking password hashes. Tables are usually used in recovering a key derivation function up to a certain length consisting of a limited set of characters. It is a practical example of a space–time tradeoff, using less computer processing time and more storage than a brute-force attack which calculates a hash on every attempt, but more processing time and less storage than a simple key derivation function with one entry per hash. Use of a key derivation that employs a salt makes this attack infeasible.
“Multiplication table”, (2021-01-03):
In mathematics, a multiplication table is a mathematical table used to define a multiplication operation for an algebraic system.
“Memoization”, (2020-12-22):
In computing, memoization or memoisation is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again. Memoization has also been used in other contexts, such as in simple mutually recursive descent parsing. Although related to caching, memoization refers to a specific case of this optimization, distinguishing it from forms of caching such as buffering or page replacement. In the context of some logic programming languages, memoization is also known as tabling.
“Tuple”, (2020-12-22):
In mathematics, a tuple is a finite ordered list (sequence) of elements. An n-tuple is a sequence of n elements, where n is a non-negative integer. There is only one 0-tuple, referred to as the empty tuple. An n-tuple is defined inductively using the construction of an ordered pair.
http:/
/ ojisanseiuchi.com/ 2016/ 03/ 12/ JavaScript-in-Anki-cards/ http:/
/ lesswrong.com/ lw/ 64k/ memory_spaced_repetition_and_life/ 4e5n “Life and death”, (2021-01-03):
Life and death (死活) is a fundamental concept in the game of Go, where the status of a distinct group of stones is determined as either being "alive", where they may remain on the board indefinitely, or "dead," where the group will be lost as "captured". The basic idea can be summarized by:
- A group must have two eyes to live.
“GNU Go”, (2021-01-03):
GNU Go is a free software program by the Free Software Foundation that plays Go. Its source code is quite portable, and can be easily compiled for Linux, as well as other Unix-like systems, Microsoft Windows and macOS; ports exist for other platforms.
“Haskell (programming language)”, (2020-12-22):
Haskell is a general-purpose, statically typed, purely functional programming language with type inference and lazy evaluation. Developed to be suitable for teaching, research and industrial application, Haskell has pioneered a number of advanced programming language features such as type classes, which enable type-safe operator overloading. Haskell's main implementation is the Glasgow Haskell Compiler (GHC). It is named after logician Haskell Curry.
“QuickCheck”, (2020-12-22):
QuickCheck is a software library, specifically a combinator library, originally written in the programming language Haskell, designed to assist in software testing by generating test cases for test suites.
“Glasgow Haskell Compiler”, (2020-12-22):
The Glasgow Haskell Compiler (GHC) is an open-source native code compiler for the functional programming language Haskell. It provides a cross-platform environment for the writing and testing of Haskell code and it supports numerous extensions, libraries, and optimisations that streamline the process of generating and executing code. GHC is the most commonly used Haskell compiler. The lead developers are Simon Peyton Jones and Simon Marlow.
http:/
/ chesstempo.com/ user-guide/ en/ probSearchSpacedRepetition.html “Smash Training retrospective”, (2020-12-06):
Smash Training is a spaced-repetition training web-app I created to help my progression with Super Smash Bros. Ultimate. I released it on May 16, 2020 on Reddit to warm reception. As of December 2020, it receives 150–200 monthly users. I’d rank it as my most successful project! In this article, I discuss the choices I made for this project. The source code is available).
…I decided that I wanted to build a spaced-repetition training app, rather than reuse a general-purpose spaced-repetition flash-card system such as Anki, because the project would benefit from domain-specific knowledge. For example:
- Exercises have large numbers of variants, such as “short-hop” vs “full-hop”, or “facing left” vs “facing right”, which should be tracked separately.
- Many of the exercises have natural dependencies on others: they shouldn’t be attempted unless a certain underlying fundamental skill has been mastered.
- Exercises to train one character don’t necessarily confer the same skill for other characters. Some exercises may only be applicable to some characters.
…Stronglifts has you note down how many repetitions of the exercise you succeeded at (out of five). However, the Smash Training paradigm is different, and has you repeat the exercise for a length of time and rate your accuracy.
“RSS”, (2021-01-03):
RSS is a web feed that allows users and applications to access updates to websites in a standardized, computer-readable format. These feeds can, for example, allow a user to keep track of many different websites in a single news aggregator. The news aggregator will automatically check the RSS feed for new content, allowing the list to be automatically passed from website to website or from website to user. This passing of content is called web syndication. Websites usually use RSS feeds to publish frequently updated information, such as blog entries, news headlines, or episodes of audio and video series. RSS is also used to distribute podcasts. An RSS document includes full or summarized text, and metadata, like publishing date and author's name.
https:/
/ market.android.com/ details?id=org.liberty.android.fantastischmemo http:/
/ www.wired.com/ medtech/ health/ magazine/ 16-05/ ff_wozniak?currentPage=5 http:/
/ www.wired.com/ medtech/ health/ magazine/ 16-05/ ff_wozniak?currentPage=4 “Cognitive psychology”, (2020-12-22):
Cognitive psychology is the scientific study of mental processes such as "attention, language use, memory, perception, problem solving, creativity, and thinking".
http:/
/ www.wired.com/ medtech/ health/ magazine/ 16-05/ ff_wozniak?currentPage=7 “Cloze test”, (2021-01-03):
A cloze test is an exercise, test, or assessment consisting of a portion of language with certain items, words, or signs removed, where the participant is asked to replace the missing language item. Cloze tests require the ability to understand context and vocabulary in order to identify the correct language or part of speech that belongs in the deleted passages. This exercise is commonly administered for the assessment of native and second language learning and instruction.
“Comment (computer programming)”, (2021-01-02):
In computer programming, a comment is a programmer-readable explanation or annotation in the source code of a computer program. They are added with the purpose of making the source code easier for humans to understand, and are generally ignored by compilers and interpreters. The syntax of comments in various programming languages varies considerably.
https:/
/ www.dropbox.com/ s/ ih3aorjs9rmjvmj/ 2019-02-03-gwern-mnemosyne-export.cards?dl=0 “How Can We Develop Transformative Tools For Thought?”, (2019-10):
[Long writeup by Andy Matuschak and Michael Nielsen on experiment in integrating spaced repetition systems with a tutorial on quantum computing, Quantum Country: Quantum Computing For The Very Curious By combining explanation with spaced testing, a notoriously thorny subject may be learned more easily and then actually remembered—such a system demonstrating a possible ‘tool for thought’. Early results indicate users do indeed remember the quiz answers, and feedback has been positive.]
Part I: Memory systems
- Introducing the mnemonic medium
- The early impact of the prototype mnemonic medium
- Expanding the scope of memory systems: what types of understanding can they be used for?
- Improving the mnemonic medium: making better cards
- Two cheers for mnemonic techniques
- How important is memory, anyway?
- How to invent Hindu-Arabic numerals?
Part II: Exploring tools for thought more broadly:
Mnemonic video
Why isn’t there more work on tools for thought today?
Questioning our basic premises
- What if the best tools for thought have already been discovered?
- Isn’t this what the tech industry does? Isn’t there a lot of ongoing progress on tools for thought?
- Why not work on AGI or BCI instead?
Executable books
- Serious work and the aspiration to canonical content
- Stronger emotional connection through an inverted writing structure
Summary and Conclusion
… in Quantum Country an expert writes the cards, an expert who is skilled not only in the subject matter of the essay, but also in strategies which can be used to encode abstract, conceptual knowledge. And so Quantum Country provides a much more scalable approach to using memory systems to do abstract, conceptual learning. In some sense, Quantum Country aims to expand the range of subjects users can comprehend at all. In that, it has very different aspirations to all prior memory systems.
More generally, we believe memory systems are a far richer space than has previously been realized. Existing memory systems barely scratch the surface of what is possible. We’ve taken to thinking of Quantum Country as a memory laboratory. That is, it’s a system which can be used both to better understand how memory works, and also to develop new kinds of memory system. We’d like to answer questions such as:
- What are new ways memory systems can be applied, beyond the simple, declarative knowledge of past systems?
- How deep can the understanding developed through a memory system be? What patterns will help users deepen their understanding as much as possible?
- How far can we raise the human capacity for memory? And with how much ease? What are the benefits and drawbacks?
- Might it be that one day most human beings will have a regular memory practice, as part of their everyday lives? Can we make it so memory becomes a choice; is it possible to in some sense solve the problem of memory?
http:/
/ bentilly.blogspot.com/ 2009/ 09/ teaching-linear-algebra.html http:/
/ bentilly.blogspot.com/ 2012/ 10/ my-sons-flashcard-routine.html http:/
/ lesswrong.com/ r/ discussion/ lw/ mfm/ a_year_of_spaced_repetition_software_in_the/ http:/
/ lesswrong.com/ r/ discussion/ lw/ nkz/ a_second_year_of_spaced_repetition_software_in/ https:/
/ theeffortfuleducator.com/ 2017/ 10/ 22/ easy-application-of-spaced-practice-in-the-classroom/ https:/
/ old.reddit.com/ r/ programming/ comments/ n30hl/ janki_method_learning_programming_with_6000/ http:/
/ www.oxbridgenotes.com/ articles/ autodidactism/ note_taking “Derek Sivers”, (2021-01-02):
Derek Sivers is an American writer, musician, programmer and entrepreneur, best known for being the founder and former president of CD Baby, an online CD store for independent musicians. A professional musician since 1987, Sivers started CD Baby by accident in 1997 when he was selling his own CD on his website, and friends asked if he could sell theirs, too. CD Baby went on to become the largest seller of independent music on the web, with over $100M in sales for over 150,000 musician clients.
https:/
/ www.shortcutfoo.com/ blog/ introducing-interval-training-for-shortcuts/ https:/
/ developer.atlassian.com/ blog/ 2015/ 06/ golang-flashcards-and-spaced-repetition/ https:/
/ www.senrigan.io/ blog/ chasing-10x-leveraging-a-poor-memory-in-software-engineering/ https:/
/ senrigan.io/ blog/ everything-i-know-strategies-tips-and-tricks-for-spaced-repetition-anki/ https:/
/ empiria.io/ blog/ remembering-r-with-spaced-repetition/ https:/
/ www.gresearch.co.uk/ article/ anki-as-learning-superpower-computer-science-edition/ http:/
/ quantifiedself.com/ 2012/ 06/ spaced-repetition-and-learning/ http:/
/ www.scotthyoung.com/ blog/ 2012/ 08/ 05/ forgetting-is-good/ http:/
/ www.ummah.com/ forum/ showthread.php?381181-How-to-Memorize-the-Quran-and-Never-Forget-it http:/
/ groups.google.com/ group/ mnemosyne-proj-users/ browse_thread/ thread/ fd10b9e601fb0eb6 http:/
/ jtauber.com/ blog/ 2004/ 11/ 26/ programmed_vocabulary_learning_as_a_travelling_salesman_problem/ http:/
/ jtauber.com/ blog/ 2006/ 05/ 05/ teaching_new_testament_greek/ http:/
/ jtauber.com/ blog/ 2008/ 02/ 10/ a_new_kind_of_graded_reader/ http:/
/ lesswrong.com/ r/ discussion/ lw/ juq/ a_vote_against_spaced_repetition/ http:/
/ howtoremember.biz/ content/ how-flashcards-fail-confessions-tired-memory-guy http:/
/ archive.wired.com/ geekdad/ 2012/ 01/ everything-about-learning/ http:/
/ conference.cali.org/ 2014/ sessions/ spaced-repetition-technology-legal-education http:/
/ nautil.us/ issue/ 17/ big-bangs/ how-i-rewired-my-brain-to-become-fluent-in-math-rd http:/
/ www.newyorker.com/ books/ page-turner/ why-we-should-memorize http:/
/ www.nytimes.com/ 2014/ 11/ 23/ sunday-review/ studying-for-the-test-by-taking-it.html http:/
/ www.rand.org/ content/ dam/ rand/ pubs/ monographs/ 2011/ RAND_MG1120.pdf http:/
/ lesswrong.com/ r/ discussion/ lw/ o8e/ how_i_use_anki_to_learn_mathematics/ https:/
/ www.theguardian.com/ science/ 2017/ feb/ 08/ total-recall-the-people-who-never-forget https:/
/ www.newyorker.com/ books/ page-turner/ the-mystery-of-s-the-man-with-an-impossible-memory “The Overfitted Brain: Dreams evolved to assist generalization”, (2020-07-19):
Understanding of the evolved biological function of sleep has advanced considerably in the past decade. However, no equivalent understanding of dreams has emerged. Contemporary neuroscientific theories generally view dreams as epiphenomena, and the few proposals for their biological function are contradicted by the phenomenology of dreams themselves. Now, the recent advent of deep neural networks (DNNs) has finally provided the novel conceptual framework within which to understand the evolved function of dreams. Notably, all DNNs face the issue of overfitting as they learn, which is when performance on one data set increases but the network’s performance fails to generalize (often measured by the divergence of performance on training vs. testing data sets). This ubiquitous problem in DNNs is often solved by modelers via "noise injections" in the form of noisy or corrupted inputs. The goal of this paper is to argue that the brain faces a similar challenge of overfitting, and that nightly dreams evolved to combat the brain’s overfitting during its daily learning. That is, dreams are a biological mechanism for increasing generalizability via the creation of corrupted sensory inputs from stochastic activity across the hierarchy of neural structures. Sleep loss, specifically dream loss, leads to an overfitted brain that can still memorize and learn but fails to generalize appropriately. Herein this "overfitted brain hypothesis" is explicitly developed and then compared and contrasted with existing contemporary neuroscientific theories of dreams. Existing evidence for the hypothesis is surveyed within both neuroscience and deep learning, and a set of testable predictions are put forward that can be pursued both in vivo and in silico.
“Alan Perlis”, (2020-12-27):
Alan Jay Perlis was an American computer scientist and professor at Purdue University, Carnegie Mellon University and Yale University. He is best known for his pioneering work in programming languages and was the first recipient of the Turing Award.
“Paradise Lost”, (2020-12-27):
Paradise Lost is an epic poem in blank verse by the 17th-century English poet John Milton (1608–1674). The first version, published in 1667, consists of ten books with over ten thousand lines of verse. A second edition followed in 1674, arranged into twelve books with minor revisions throughout. It is considered to be Milton's major work, and it helped solidify his reputation as one of the greatest English poets of his time.The poem concerns the biblical story of the Fall of Man: the temptation of Adam and Eve by the fallen angel Satan and their expulsion from the Garden of Eden. Milton's purpose, as stated in Book I, is to "justify the ways of God to men."
https:/
/ www.sas.upenn.edu/ psych/ rust-lab/ publications/ standing_73.pdf “Spaced Repetition for Efficient Learning”, (2009-03-11):
Spaced repetition is a centuries-old psychological technique for efficient memorization & practice of skills where instead of attempting to memorize by ‘cramming’, memorization can be done far more efficiently by instead spacing out each review, with increasing durations as one learns the item, with the scheduling done by software. Because of the greater efficiency of its slow but steady approach, spaced repetition can scale to memorizing hundreds of thousands of items (while crammed items are almost immediately forgotten) and is especially useful for foreign languages & medical studies.
I review what this technique is useful for, some of the large research literature on it and the testing effect (up to ~2013, primarily), the available software tools and use patterns, and miscellaneous ideas & observations on it.
“One does not learn computing by using a hand calculator, but one can forget arithmetic.” Perlis 1982↩︎
Listing other neuroprosthetics is hard. It’s an interesting idea, but as proponents of externalism like Andy Clark have found, it’s easier to feel that externalism is meaningful than to nail down a clear definition which separates a neuroprosthetic or part of one’s mind from a random tool you like or find useful. Consider whether a pencil and paper a neuroprosthetic: clearly it is not for a child learning to write, who must carefully compose the words in his mind and put them down one after another, but it is not so clear for an adult who has been writing all his life and can doodle or write down thoughts without thinking about them and may even be surprised at what they happened to write.
I like this definition: “a neuroprosthetic is anything whose results you use without further thought”. So in the classic example, when Otto needs to go somewhere, he never thinks “I am an amnesiac who stores locations in my notepad, and I must look up the location” - he just looks up the location. A good heuristic would be anything whose destruction leaves one feeling lost, slow, stupid, or ignorant.
By this standard, I can think of only a few tools I use without noticeable thought:
- keybindings such as window manager shortcuts, in particular shortcuts for Google searches; on occasion, XMonad’s Prompt gets inscrutably wedged, locking it. When this happens, I have to restart X because I Google everything and the keybinding is so engrained that not using it is unbearable. It would be like trying to write with your weak hand.
- Google Calendar and PredictionBook: it is incredible how many followups or reminders or regularly happening tasks I can put into Google Calendar or PB. I have outsourced many habits or thoughts to them, and I no longer think of it as anything special. If either were gone, I would feel frightened - what events were passing, what beliefs falsified, what opportunities opening up (or closing!) that I had suddenly become ignorant of?
- Evernote, for a similar reason; many of my memories have ceased to be things like “octopuses see too fast to watch TV and so only HDTV or UHDTV works for them; I read this in Orion Magazine” and become things like “octopus TV Evernote”, and if I want to know what it was about octopuses & TV, well, I’ll have to look it up in Evernote. Mnemosyne plays a similar role for me, but there the memories are much clearer on their own because of the spaced repetition.
- my website Gwern.net; I’ve had to say many times that I don’t know what I think about something, but whatever that is, it’s on my website. (A more extreme form of the Evernote/
Mnemosyne neuroprosthetic.) A commenter once wrote that reading Gwern.net felt like he was crawling around in my head. He was more right than he realized.
as quoted in “Retrieval practice and the maintenance of knowledge”, Bjork 1988↩︎
From “Close the Book. Recall. Write It Down: That old study method still works, researchers say. So why don’t professors preach it?”; The Chronicle of Higher Education
Two psychology journals have recently published papers showing that this strategy works, the latest findings from a decades-old body of research. When students study on their own, “active recall” - recitation, for instance, or flashcards and other self-quizzing - is the most effective way to inscribe something in long-term memory. Yet many college instructors are only dimly familiar with that research…
From “The Spacing Effect: A Case Study in the Failure to Apply the Results of Psychological Research” (Dempster 1988), whose title alone summarizes the situation (see also Kelley 2007, Making Minds: What’s Wrong with Education - and What Should We Do About It?):
Second, it [the spacing effect] is remarkably robust. In many cases, two spaced presentations are about twice as effective as two massed presentations (e.g., Hintzman, 1974; Melton, 1970), and the difference between them increases as the frequency of repetition increases (Underwood, 1970)…
The spacing effect was known as early as 1885 when Ebbinghaus published the results of his seminal work on memory. With himself as the subject, Ebbinghaus found that for a single 12-syllable series, 68 immediately successive repetitions had the effect of making possible an errorless recital after seven additional repetitions on the following day. However, the same effect was achieved by only 38 distributed repetitions spread over 3 days. On the basis of this and other related findings, Ebbinghaus concluded that ‘with any considerable number of repetitions a suitable distribution of them over a space of time is decidedly more advantageous than the massing of them at a single time’ (Ebbinghaus, 1885/
1913. p. 89) Son & Simon 2012:
Furthermore, even after acknowledging the benefits of spacing, changing teaching practices proved to be enormously difficult. Delaney et al (2010) wrote: “Anecdotally, high school teachers and college professors seem to teach in a linear fashion without repetition and give three or four noncumulative exams.” (p. 130). Focusing on the math domain, where one might expect a very easy-to-review-and-to-space strategy, Rohrer (2009) points out that mathematics textbooks usually present topics in a non-spaced, non-mixed fashion. Even much earlier, Vash (1989) had written: “Education policy setters know perfectly well that [spaced practice] works better [than massed practice]. They don’t care. It isn’t tidy. It doesn’t let teachers teach a unit and dust off their hands quickly with a nice sense of ‘Well, that’s done.’” (p. 1547).
- Rohrer, D. (2009). “The effects of spacing and mixing practice problems”. Journal for Research in Mathematics Education, 40, 4-17
- Vash, C. L. (1989). “The spacing effect: A case study in the failure to apply the results of psychological research”. American Psychologist, 44, 1547 (a comment on Dempster’s article?)
From Psychology: An Introduction:
In one practical demonstration of the spacing effect, Bahrick, Bahrick, Bahrick, & Bahrick (1993) showed that retention of foreign language vocabulary was greatly enhanced if practice sessions were spaced far apart. For example, “Thirteen retraining sessions spaced at 56 days yielded retention comparable to 26 sessions spaced at 14 days.” In other words, subjects could use half as many study sessions, if the study sessions were spread over a time period four times as long.
“Synaptic evidence for the efficacy of spaced learning”, Kramar et al 2012 (“Take your time: Neurobiology sheds light on the superiority of spaced vs. massed learning”):
The superiority of spaced vs. massed training is a fundamental feature of learning. Here, we describe unanticipated timing rules for the production of long-term potentiation (LTP) in adult rat hippocampal slices that can account for one temporal segment of the spaced trials phenomenon. Successive bouts of naturalistic theta burst stimulation of field CA1 afferents markedly enhanced previously saturated LTP if spaced apart by 1 h or longer, but were without effect when shorter intervals were used. Analyses of F-actin-enriched spines to identify potentiated synapses indicated that the added LTP obtained with delayed theta trains involved recruitment of synapses that were “missed” by the first stimulation bout. Single spine glutamate-uncaging experiments confirmed that less than half of the spines in adult hippocampus are primed to undergo plasticity under baseline conditions, suggesting that intrinsic variability among individual synapses imposes a repetitive presentation requirement for maximizing the percentage of potentiated connections. We propose that a combination of local diffusion from initially modified spines coupled with much later membrane insertion events dictate that the repetitions be widely spaced. Thus, the synaptic mechanisms described here provide a neurobiological explanation for one component of a poorly understood, ubiquitous aspect of learning.
There are many studies to the effect that active recall is best. Here’s one recent study, “Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping”, Karpicke 2011 (covered in Science Daily and the NYT):
Educators rely heavily on learning activities that encourage elaborative studying, while activities that require students to practice retrieving and reconstructing knowledge are used less frequently. Here, we show that practicing retrieval produces greater gains in meaningful learning than elaborative studying with concept mapping. The advantage of retrieval practice generalized across texts identical to those commonly found in science education. The advantage of retrieval practice was observed with test questions that assessed comprehension and required students to make inferences. The advantage of retrieval practice occurred even when the criterial test involved creating concept maps. Our findings support the theory that retrieval practice enhances learning by retrieval-specific mechanisms rather than by elaborative study processes. Retrieval practice is an effective tool to promote conceptual learning about science.
From “Forget What You Know About Good Study Habits”. New York Times;
Cognitive scientists do not deny that honest-to-goodness cramming can lead to a better grade on a given exam. But hurriedly jam-packing a brain is akin to speed-packing a cheap suitcase, as most students quickly learn - it holds its new load for a while, then most everything falls out….When the neural suitcase is packed carefully and gradually, it holds its contents for far, far longer. An hour of study tonight, an hour on the weekend, another session a week from now: such so-called spacing improves later recall, without requiring students to put in more overall study effort or pay more attention, dozens of studies have found.
“The idea is that forgetting is the friend of learning”, said Dr. Kornell. “When you forget something, it allows you to relearn, and do so effectively, the next time you see it.”
That’s one reason cognitive scientists see testing itself - or practice tests and quizzes - as a powerful tool of learning, rather than merely assessment. The process of retrieving an idea is not like pulling a book from a shelf; it seems to fundamentally alter the way the information is subsequently stored, making it far more accessible in the future.
In one of his own experiments, Dr. Roediger and Jeffrey Karpicke, who is now at Purdue University, had college students study science passages from a reading comprehension test, in short study periods. When students studied the same material twice, in back-to-back sessions, they did very well on a test given immediately afterward, then began to forget the material. But if they studied the passage just once and did a practice test in the second session, they did very well on one test two days later, and another given a week later.
The Mathematics of Gambling, Thorp 1984, “Section Two: The Wheels”, Chapter 4, pg43-44:
It was the spring of 1955. I was finishing my second year of graduate physics at U.C.L.A…I changed my field of study from physics to mathematics…I attended classes and studied from 50 to 60 hours a week, generally including Saturdays and Sundays. I had read about the psychology of learning in order to be able to work longer and harder. I found that “spaced learning” worked well: study for an hour, then take a break of at least ten minutes (shower, meal, tea, errands, etc.). One Sunday afternoon about 3 p.m., I came to the co-op dining room for a tea break…My head was bubbling with physics equations, and several of my good friends were sitting around chatting.
From Final Jeopardy: Man Vs. Machine and the Quest to Know Everything, by Stephen Baker, pg 214:
The program he put together tested him on categories, gauged his strengths (sciences, NFL football) and weaknesses (fashion, Broadway shows), and then directed him toward the preparation most likely to pay off in his own match. To patch these holes in his knowledge, Craig used a free online tool called Anki, which provides electronic flash cards for hundreds of fields of study, from Japanese vocabulary to European monarchs. The program, in Craig’s words, is based on psychological research on ‘the forgetting curve’. It helps people find holes in their knowledge and determines how often they need those areas to be reviewed to keep them in mind. In going over world capitals, for example, the system learns quickly that a user like Craig knows London, Paris, and Rome, so it might spend more time reinforcing the capital of, say, Kazakhstan. (And what would be the Kazakh capital? ‘Astana’, Craig said in a flash. ‘It used to be Almaty, but they moved it.’)
“Our Interview With Jeopardy! Champion Arthur Chu”:
[Chu:] …Jeopardy! is aimed at the sort of average TV viewer, so they’re not going to ask things that are pointlessly obscure…So I used a program called Anki which uses a method called “spaced repetition.” It keeps track of where you’re doing well or poorly, and pushes you to study the flashcards you don’t know as well, until you develop an even knowledge base about a particular subject, and I just made flashcards for those specific things. I memorized all the world capitals, it wasn’t that hard once I had the flashcards and was using them every day. I memorized the US State Nicknames (they’re on Wikipedia), memorized the basic important facts about the 44 US Presidents. I really focused on those. But there’s a lot more stuff to know. I went on Jeopardy! knowing that there was stuff I didn’t know. For instance, everyone laughs about sports - but I also knew that [sports clues] were the least likely to come up in Double Jeopardy and Final Jeopardy and be very important. So I decided I shouldn’t sweat it too much, I should just recognize that I didn’t know them and let that go, as long as I can get the high value clues. So that was how I prepared.
Alan J. Perlis, “Epigrams in Programming” (1982)↩︎
Web developer Persol writes in August 2012:
I actually wrote a site that did this [spaced repetition] a few months ago. I had about 4000 users who had actually gone through a complete session…As guessed, the problem is that I couldn’t get people to start forming it as a habit. There is no immediate payback. Less than 20 people out of 4000 did more than one session…Additionally, there are at least 18 competitors. Here’s the list I made at the time. Very few seem to be successful. I shut the site down about a month ago. There are numerous free competitors which don’t have any great annoyances. I wouldn’t suggest starting another of these sites unless you figured out an effective way to “gamify” it.
…~4000 people finished a session. Many more ‘tried’ than 4000…I just couldn’t determine which users were bots that registered randomly vs users that didn’t finish the first session.
- Tried: lots (but unknown)
- Finished 1 session: ~4000
- Finished >1 session: ~20 [0.5%]
“Play it Again: The Master Psychopharmacology Program as an Example of Interval Learning in Bite-Sized Portions”, Stahl et al 2010:
Since Ebbinghaus’ time, a voluminous amount of research has confirmed this simple but important fact: the retention of new information degrades rapidly unless it is reviewed in some manner. A modern example of this loss of knowledge without repetition is a study of cardiopulmonary resuscitation (CPR) skills that demonstrated rapid decay in the year following training. By 3 years post-training only 2.4% were able to perform CPR successfully.6 Another recent study of physicians taking a tutorial they rated as very good or excellent showed mean knowledge scores increasing from 50% before the tutorial to 76% immediately afterward.7 However, score gains were only half as great 3-8 days later and incredibly, there was no [statistically-]significant knowledge retention measurable at all at 55 days.7 Similar results have been reported by us in follow-up studies of knowledge retention from continuing medical education programs.1 [Stahl SM, Davis RL. Best Practices for Medical Educators. Carlsbad, CA: NEI Press; 2009]
…This may be due to the fact that lectures with assigned reading are the easiest for teachers. Also, medical learning is rarely measured immediately after a lecture or after reading new material for the first time and then measured again a few days or weeks later, so that the low retention rates of this approach may not be widely appreciated.1,4 No wonder formal medical education conferences without enabling or practice-reinforcing strategies appear to have relatively little impact on practice and healthcare outcomes.8,9,10
One study looking at cramming is the 1993 “Cramming: A barrier to student success, a way to beat the system or an effective learning strategy?”, Vacha et al 1993, abstract:
Tested the hypothesis that cramming is an ineffective study strategy by examining the weekly study diaries of 166 undergraduates. All Ss also completed an end-of-semester questionnaire measuring study habits. Ss were classified in the following study patterns: ideal, confident, zealous, or crammer. Contrary to the hypothesis, results suggest that cramming is an effective approach, most widespread in courses using take-home essay examinations and major research papers. Crammers’ grades were as good as or better than those of Ss using other strategies; the longer Ss were in college, the more likely it was that they crammed. Crammers studied more hours than most students and were as interested in their courses as other students.
Note that there is no measure of long-term retention, suggesting that people who only care about grades are rationally choosing to cram.↩︎
Anki has its Cram Mode and Mnemosyne 2.0 has a cramming plugin. When a SRS doesn’t have explicit support, it’s always possible to ‘game’ the algorithm by setting one’s scores artificially low, so the SR algorithm thinks you are stupid and need to do a lot of repetitions.↩︎
“Examining the examiners: Why are we so bad at assessing students?”, Newstead 2002:
Conway, Cohen and Stanhope (1992) looked at long term memory for the information presented on a psychology course. They found that some types of information, especially that relating to research methods, were remembered better than others. But in a follow up analysis, they found that the type of assessment used had an effect on memory. In essence, material assessed by continuous assessment was more likely to be remembered than information assessed by exams.
Stahl 2010:
For example, simple restudying allows the learner to reexperience all of the material but actually produces poor long-term retention.25,26,35 Why do students keep studying the original materials? Certainly if this is their only choice, then restudying is a necessary tactic. Another answer may be that repeated studying falsely inflates students’ confidence in their ability to remember in the future because they sense that they understand it now, and they and their instructors may be unaware of the many studies that show poor retention on delayed testing after this form of repetition.25,26,35
From Kornell et al 2010:
Contrary to the massing-aids-induction hypothesis, final test performance was consistently and considerably superior in the spaced condition. A large majority of participants, however, judged massing to be more effective than spacing, despite making the judgment after taking the test.
…Metacognitive judgments-that is, judgments about one’s own memory and cognition-are often based on feelings of fluency(e.g., see Benjamin, Bjork, & Schwartz, 1998; Rhodes & Castel, 2008). Because massing naturally leads to feelings of fluency and increases short-term task performance during learning, learners frequently rate spacing as less effective than massing, even when their performance shows the opposite pattern (Baddeley & Longman 1978; Kornell & Bjork, 2008; Simon & Bjork, 2001; Zechmeister & Shaughnessy, 1980). Averaged across Kornell and Bjork’s (2008) experiments, for example, more than 80% of participants rated massing as equally or more effective than spacing,whereas only 15% of participants actually performed better in the massed condition than in the spaced condition.
…Such an illusion was apparent in the induction condition. Contrary to previous research, however, participants gave higher ratings for spacing than massing during repetition learning (see, e.g., Simon & Bjork, 2001; Zechmeister & Shaughnessy, 1980). This outcome may have occurred because of a process of a habituation: Six presentations and a total of 30 s spent studying a single painting may have come to seem inefficient and pointless. Thus, there appears to be a turning point in metacognitive ratings based on fluency: As fluency increases, metacognitive ratings increase up to a point, but as fluency continues to increase and encoding or retrieval becomes too easy, metacognitive ratings may begin to decrease.
…In advance of their research, Kornell and Bjork (2008) were convinced that such inductive learning would benefit from massing, yet their results showed the opposite. Undaunted, we remained convinced that spacing would be more beneficial for repetition learning than for inductive learning- especially for older adults, given their overall declines in episodic memory. The current results disconfirmed our expectations once again. If our intuitions are erroneous, despite our years spent proving and praising the spacing effect-including roughly 40 years’ worth contributed by Robert A. Bjork-those of the average student are surely mistaken as well (as the inaccuracy of the participants’ metacognitive ratings suggests). We have, perhaps, fallen victim to the illusion that making learning easy makes learning effective, rather than recognizing that spacing is a desirable difficulty (Bjork 1994) that enhances inductive learning as well as repetition learning well into old age.
From Son & Simon 2012:
Thus, while spacing may boost learning, it may be thought to be relatively inefficient in terms of study time. As we discuss later, this feeling of inefficiency may be one of the reasons that spacing is not the more popular strategy. Interestingly, in that same study (Baddeley & Longman 1978; and see also Pirolli & Anderson 1985 and Woodworth & Schlosberg 1954 [Experimental Psychology]), there was evidence of such a thing as laboring in vain. That is, exceeding a certain number of hours of practice a day (more than approximately 2 h) led to no increases in learning, as might be expected. Related to the deficient-processing theory mentioned above, these results are crucial in understanding intuitively how the spacing effect works: We simply get burnt out. These data are also analogous to the cognitive literature on overlearning, which shows that while continuous study over long periods of time might seem beneficial (and even feel good) in the short-term, the benefits disappear soon afterwards (Rohrer et al. 2005; Rohrer and Taylor 2006)…In the above-described Baddeley and Longman’s (1978) study, for example, after postal workers practiced typing in either massed or spaced study sessions, they had to indicate how satisfied they were with the training. Results showed that while spacing led to the best learning, it was the least liked. Similarly, Simon & Bjork (2001) found that people preferred the massing strategy on a motor learning task.
- Baddeley, A. D., & Longman, D. J. A. (1978). “The influence of length and frequency of training session on the rate of learning to type”. Ergonomics, 21, 627-635
- Pirolli, P., & Anderson, J. R. (1985). “The role of practice in fact retrieval”
“Study strategies of college students: Are self-testing and scheduling related to achievement?”, Hartwig & Dunlosky 2012:
Previous studies, such as those by Kornell and Bjork (Psychonomic Bulletin & Review, 14:219-224, 2007) and Karpicke, Butler, and Roediger (Memory, 17:471-479, 2009), have surveyed college students’ use of various study strategies, including self-testing and rereading. These studies have documented that some students do use self-testing (but largely for monitoring memory) and rereading, but the researchers did not assess whether individual differences in strategy use were related to student achievement. Thus, we surveyed 324 undergraduates about their study habits as well as their college grade point average (GPA). Importantly, the survey included questions about self-testing, scheduling one’s study, and a checklist of strategies commonly used by students or recommended by cognitive research. Use of self-testing and rereading were both positively associated with GPA. Scheduling of study time was also an important factor: Low performers were more likely to engage in late-night studying than were high performers; massing (vs. spacing) of study was associated with the use of fewer study strategies overall; and all students-but especially low performers-were driven by impending deadlines. Thus, self-testing, rereading, and scheduling of study play important roles in real-world student achievement.
(See also Dunlosky et al 2013.) Note the self-testing correlation excludes flashcards, a result that both the authors and me found surprising. The sleep connection is interesting, given the hypothesized link between stronger memory formation & studying before a good night’s sleep - you can hardly get a good night’s sleep if you are cramming late into the night (correlated with lower grades) but you can if you do so at a reasonable time in the evening (in time to get a solid night).
See also Susser & McCabe 2012:
Laboratory studies have demonstrated the long-term memory benefits of studying material in multiple distributed sessions as opposed to one massed session, given an identical amount of overall study time (i.e., the spacing effect). The current study goes beyond the laboratory to investigate whether undergraduates know about the advantage of spaced study, to what extent they use it in their own studying, and what factors might influence its utilization. Results from a web-based survey indicated that participants (n = 285) were aware of the benefits of spaced study and would use a higher level of spacing under ideal compared to realistic circumstances. However, self-reported use of spacing was intermediate, similar to massing and several other study strategies, and ranked well below commonly used strategies such as rereading notes. Several factors were endorsed as important in the decision to distribute study time, including the perceived difficulty of an upcoming exam, the amount of material to learn, how heavily an exam is weighed in the course grade, and the value of the material. Further, level of metacognitive self-regulation and use of elaboration strategies were associated with higher rates of spaced study.
Analytic Culture in the US Intelligence Community: An Ethnographic Study, Johnston 2005, pg89:
To investigate the intensity of instructional interactions, Art Graesser and Natalie Person 1994 compared questioning and answering in classrooms with those in tutorial settings.5 They found that classroom groups of students ask about three questions an hour and that any single student in a classroom asks about 0.11 questions per hour. In contrast, they found that students in individual tutorial sessions asked 20-30 questions an hour and were required to answer 117-146 questions per hour. Reviews of the intensity of interaction that occurs in technology-based instruction have found even more active student response levels. [J. D. Fletcher, Technology, the Columbus Effect, and the Third Revolution in Learning.]
Although Graesser & Person 1994 also found that sheer number of questions was not necessarily important, suggesting diminishing marginal returns or perhaps bad question asking.↩︎
“SuperMemo is based on the insight that there is an ideal moment to practice what you’ve learned. Practice too soon and you waste your time. Practice too late and you’ve forgotten the material and have to relearn it. The right time to practice is just at the moment you’re about to forget. Unfortunately, this moment is different for every person and each bit of information. Imagine a pile of thousands of flash cards. Somewhere in this pile are the ones you should be practicing right now. Which are they?” Gary Wolf, “Want to Remember Everything You’ll Ever Learn? Surrender to This Algorithm”, Wired Magazine↩︎
“Make no mistake about it: Computers process numbers - not symbols. We measure our understanding (and control) by the extent to which we can arithmetize an activity.” Perlis, ibid.↩︎
this exponential expansion is how a SR program can handle continual input of cards: if cards were scheduled at fixed intervals, like every other day, review would soon become quite impossible - I have >18000 items in Mnemosyne, but I don’t have time to review 9000 questions a day!↩︎
See the 2008 meta-analysis, “Learning Styles: Concepts and Evidence” (APS press release); from the abstract:
…in order to demonstrate that optimal learning requires that students receive instruction tailored to their putative learning style, the experiment must reveal a specific type of interaction between learning style and instructional method: Students with one learning style achieve the best educational outcome when given an instructional method that differs from the instructional method producing the best outcome for students with a different learning style. In other words, the instructional method that proves most effective for students with one learning style is not the most effective method for students with a different learning style.
Our review of the literature disclosed ample evidence that children and adults will, if asked, express preferences about how they prefer information to be presented to them. There is also plentiful evidence arguing that people differ in the degree to which they have some fairly specific aptitudes for different kinds of thinking and for processing different types of information. However, we found virtually no evidence for the interaction pattern mentioned above, which was judged to be a precondition for validating the educational applications of learning styles. Although the literature on learning styles is enormous, very few studies have even used an experimental methodology capable of testing the validity of learning styles applied to education. Moreover, of those that did use an appropriate method, several found results that flatly contradict the popular meshing hypothesis.
We conclude therefore, that at present, there is no adequate evidence base to justify incorporating learning-styles assessments into general educational practice. Thus, limited education resources would better be devoted to adopting other educational practices that have a strong evidence base, of which there are an increasing number. However, given the lack of methodologically sound studies of learning styles, it would be an error to conclude that all possible versions of learning styles have been tested and found wanting; many have simply not been tested at all.
Fritz, C. O., Morris, P. E., Acton, M., Etkind, R., & Voelkel, A. R (2007). “Comparing and combining expanding retrieval practice and the keyword mnemonic for foreign vocabulary learning”. Applied Cognitive Psychology, 21, 499-526.↩︎
From Balota et al 2006, describing Spitzer 1939, “Studies in retention”:
Spitzer (1939) incorporated a form of expanded retrieval in a study designed to assess the ability of sixth graders to learn science facts. Impressively, Spitzer tested over 3600 students in Iowa-the entire sixth-grade population of 91 elementary schools at the time. The students read two articles, one on peanuts and the other on bamboo, and were given a 25-item multiple choice test to assess their knowledge (such as ‘To which family of plants does bamboo belong?’). Spitzer tested a total of nine groups, manipulating both the timing of the test (administered immediately or after various delays) and the number of identical tests students received (one to three). Spitzer did not incorporate massed or equal interval retrieval conditions, but he had at least two groups that were tested on an expanding schedule of retrieval, in which the intervals between tests were separated by the passage of time (in days) rather than by intervening to-be-learned information. For example, in one of the groups, the first test was given immediately, the second test was given seven days after the first test, and the third test was given 63 days after the second test. Thus, in essence, this group was tested on a 0-7-63 day expanding retrieval schedule. Spitzer compared performance of the expanded retrieval group to a group given a single test 63 days after reading the original article. On the first (immediate) test, the expanded retrieval group correctly answered 53% of the questions. After 63 days and two previous tests, their score was still an impressive 43%. The single test group correctly answered only 25% of the original items after 63 days, giving the expanded retrieval group an 18% retention advantage. This is quite impressive, given that this large benefit remained after a 63-day retention interval. Similar beneficial effects were found in a group tested on a 0-1-21 day expanded retrieval schedule compared to a group given a single test after 21 days. Of course, this study does not decouple the effects of testing from spacing or expansion, but the results do clearly indicate considerable learning and retention using the expanded repeated testing procedure. Spitzer concluded that ‘…examinations are learning devices and should not be considered only as tools for measuring achievement of pupils’ (p. 656, italics added)
“Distributing Learning Over Time: The Spacing Effect in Children’s Acquisition and Generalization of Science Concepts”, Vlach & Sandhofer 2012:
The spacing effect describes the robust finding that long-term learning is promoted when learning events are spaced out in time, rather than presented in immediate succession. Studies of the spacing effect have focused on memory processes rather than for other types of learning, such as the acquisition and generalization of new concepts. In this study, early elementary school children (5-7 year-olds; N = 36) were presented with science lessons on one of three schedules: massed, clumped, and spaced. The results revealed that spacing lessons out in time resulted in higher generalization performance for both simple and complex concepts. Spaced learning schedules promote several types of learning, strengthening the implications of the spacing effect for educational practices and curriculum.
See also Balch 2006, who compared spacing & massed in an introductory psychology course as well.↩︎
Roediger & Karpicke 2006b again.↩︎
Balota et al 2006 review:
No feedback or correction was given to subjects if they made errors or omitted answers. Landauer & Bjork 1978 found that the expanding-interval schedule produced better recall than equal-interval testing on a final test at the end of the session, and equal-interval testing, in turn, produced better recall than did initial massed testing. Thus, despite the fact that massed testing produced nearly errorless performance during the acquisition phase, the other two schedules produced better retention on the final test given at the end of the session. However, the difference favoring the expanding retrieval schedule over the equal-interval schedule was fairly small at around 10%. In research following up Landauer and Bjork’s (1978) original experiments, practically all studies have found that spaced schedules of retrieval (whether equal-interval or expanding schedules) produce better retention on a final test given later than do massed retrieval tests given immediately after presentation (e.g., Cull, 2000; Cull, Shaughnessy, & Zechmeister, 1996), although exceptions do exist. For example, in Experiments 3 and 4 of Cull et al (1996), massed testing produced performance as good as equal-interval testing on a 5-5-5 schedule, but most other experiments have found that any spaced schedule of testing (either equal-interval or expanding) is better than a massed schedule for performance on a delayed test. However, whether expanding schedules are better than equal-interval schedules for long-term retention-the other part of Landauer and Bjork’s interesting findings-remains an open question. Balota, Duchek, and Logan (in press) have provided a thorough consideration of the relevant evidence and have shown that it is mixed at best, and that most researchers have found no difference between the two schedules of testing. That is, performance on a final test at the end of a session often shows no difference in performance between equal-interval and expanding retrieval schedules.
Cull, for those curious (Cull, W. L. (2000). “Untangling the benefits of multiple study opportunities and repeated testing for cued recall”. Applied Cognitive Psychology, 14, 215-235):
Cull (2000) compared expanded retrieval to equal interval spaced retrieval in a series of four experiments designed to mimic typical teaching or study strategies encountered by students. He examined the role of testing versus simply restudying the material, feedback, and various retention intervals on final test performance. Paired associates (an uncommon word paired with a common word, such as bairn-print) were presented in a manner similar to the flashcard techniques students often use to learn vocabulary words. The intervals between retrieval attempts of to-be-learned information ranged from minutes in some experiments to days in others. Interestingly, across four experiments, Cull did not find any evidence of an advantage of an expanded condition over a uniform spaced condition (i.e., no [substantial] expanded retrieval effect), although both conditions consistently produced large advantages over massed presentations. He concluded that distributed testing of any kind, expanded or equal interval, can be an effective learning aid for teachers to provide for their students.
The Balota et al 2006 review offers a synthesis of current theories on how massed and spaced differ, based on memory encoding:
According to encoding variability theory, performance on a memory test is dependent upon the overlap between the contextual information available at the time of test and the contextual information available during encoding. During massed study, there is relatively little time for contextual elements to fluctuate between presentations and so this condition produces the highest performance in an immediate memory test, when the test context strongly overlaps with the same contextual information encoded during both of the massed presentations. In contrast, when there is spacing between the items, there is time for fluctuation to take place between the presentations during study, and hence there is an increased likelihood of having multiple unique contexts encoded. Because a delayed test will also allow fluctuation of context, it is better to have multiple unique contexts encoded, as in the spaced presentation format, as opposed to a single encoded context, as in the massed presentation format.
Storm et al 2010 did 3 experiments on reading comprehension:
On a test 1 week later, recall was enhanced by the expanding schedule, but only when the task between successive retrievals was highly interfering with memory for the passage. These results suggest that the extent to which learners benefit from expanding retrieval practice depends on the degree to which the to-be-learned information is vulnerable to forgetting.
From Mnemosyne’s Principles page:
The Mnemosyne algorithm is very similar to SM2 used in one of the early versions of SuperMemo. There are some modifications that deal with early and late repetitions, and also to add a small, healthy dose of randomness to the intervals. Supermemo now uses SM11. However, we are a bit skeptical that the huge complexity of the newer SM algorithms provides for a statistically relevant benefit. But, that is one of the facts we hope to find out with our data collection. We will only make modifications to our algorithms based on common sense or if the data tells us that there is a statistically relevant reason to do so.
Balota et al 2006:
Carpenter and DeLosh (2005, Exp. 2) have recently investigated face-name learning under massed, expanded (1-3-5), and equal interval (3-3-3) conditions. This study also involved study and study and test procedures during the acquisition phase. Carpenter and DeLosh found a large effect of spacing, but no evidence of a benefit of expanded over equal interval practice. In fact, Carpenter and DeLosh reported a reliable benefit of the equal interval condition over the expanded retrieval condition.
Balota et al 2006 again:
Rea and Modigliani (1985) tested the effectiveness of expanded retrieval in a third-grade classroom setting. In separate conditions, students were given new multiplication problems or spelling words to learn. The problem or word was presented audiovisually once and then tested on either a massed retrieval schedule of 0-0-0-0 or an expanding schedule of 0-1-2-4, in which the intervals involved being tested on old items or learning new items. After each test trial for a given item, the item was re-presented in its entirety so students received feedback on what they were learning. Performance during the learning phase was at 100% for both spelling words and multiplication facts. On an immediate final retention test, Rea and Modigliani found a performance advantage for all items-math and spelling- practiced on an expanding schedule compared to the massed retrieval schedule. They suggested, as have others, that spacing combined with the high success rate inherent in the expanded retrieval schedule produced better retention than massed retrieval practice. However, as in Spitzer’s study, Rea and Modigliani did not test an appropriate equal interval spacing condition. Hence, their finding that expanded retrieval is superior to massed retrieval in third graders could simply reflect the superiority of spaced versus massed rehearsal-in other words, the spacing effect.
Balota et al 2006; >1 is rare in psychology, see “One Hundred Years of Social Psychology Quantitatively Described”, Bond et al 2003↩︎
Rohrer & Taylor 2006↩︎
Balota et al 2006:
…long-term retention of information has been demonstrated over several days in some cases (e.g., Camp et al, 1996). For example, in the latter study, Camp et al employed an expanding retrieval strategy to train 23 individuals with mild to moderate AD to refer to a daily calendar as a cue to remember to perform various personal activities (e.g., take medication). Following a baseline phase to determine whether subjects would spontaneously use the calendar, spaced retrieval training was implemented by repeatedly asking the subject the question, ‘How are you going to remember what to do each day?’ at expanding time intervals. The results indicated that 20/
23 subjects did learn the strategy (i.e., to look at the calendar) and retained it over a 1-week period. Rohrer & Taylor 2006 warns us, though, about many of the other math studies:
In one meta-analysis by Donovan and Radosevich (1999), for instance, the size of the spacing effect declined sharply as conceptual difficulty of the task increased from low (e.g. rotary pursuit) to average (e.g. word list recall) to high (e.g. puzzle). By this finding, the benefits of spaced practise may be muted for many mathematics tasks.
What is especially nice about this study was that not only did it use high-quality (intelligent & motivated) college students (United States Air Force Academy), the conditions were relatively controlled - both groups had the same homework (so equal testing effect), but like Rohrer & Taylor 2006/
2007, the distribution was what varied: The course topics, textbook, handouts, reading assignments, and graded assignments (with the exception of quiz, homework, and participation points) were identical for the treatment and control groups. The listing of homework assignments in the syllabus differed between groups. The control group was assigned daily homework related to the topic(s) presented that day in class. Peterson (1971) calls this the vertical model for assigning mathematics homework. The treatment group was assigned homework in accordance with a distributed organizational pattern that combines practice on current topics and reinforcement of previously covered topics. Under the distributed model, approximately 40% of the problems on a given topic were assigned the day the topic was first introduced, with an additional 20% assigned on the next lesson and the remaining 40% of problems on the topic assigned on subsequent lessons (Hirsch et al, 1983). In Hirsch’s research and in this study, after the initial homework assignment, problem(s) representing a given topic resurfaced on the 2nd, 4th, 7th, 12th, and 21st lesson. Consequently, treatment group homework for lesson one consisted of only one topic; homework for lessons two and three consisted of two topics; and homework for lesson four through six consisted of three topics. This pattern continued as new topics were added and was applied to all non-exam, non-laboratory lessons. As shown by Tables 1 and 2, the same homework problems were assigned to both groups with only the pattern of assignment differing. Because of the nature of the distributed practice model, homework for the treatment group contained fewer problems (relative to the control group) early in the semester with the number of problems increasing as the semester progressed. Later in the semester, homework for the treatment group contained more problems (relative to the control group)….The USAFA routinely collects study time data. After each exam, a large sample of cadets (at least 60% of the course population) anonymously reported the amount of time (in minutes) spent studying for the exam. Time spent studying was approximately equal for both groups (see Table 5). Descriptive data revels that, for both the treatment and control group, study time for the third exam was at least 16% greater than study time for any other exam. Study time for the final exam was at least 68% greater than study time for any of the hourly exams (see Table 5)
…The treatment produced an effect size (f 2) of 0.013 on the first exam, 0.029 on the second exam, 0.035 on the fourth exam, and 0.040 on the final course percentage grade. Although the effect sizes appear to be small, the treatment group outscored the control group in every case. A mean difference of 5.13 percentage points on the first, second, and fourth exam translates to an advantage of about a third of a letter grade for students in the treatment group. In addition, higher minimum scores earned by the treatment group may indicate that the distributed practice treatment served to eliminate the extremely low scores (refer to Table 3)….Oddly, the distributed practice treatment did not produce a [statistically-]significant effect on final exam scores. One possible cause for the disparity was the USAFA policy exempting the top performers from the final exam. Of the 16 exempted students, 11 were from the treatment group with only 5 from the control group.
Balch 2006 abstract:
Two introductory psychology classes (N = 145) participated in a counterbalanced classroom experiment that demonstrated the spacing effect and, by analogy, the benefits of distributed study. After hearing words presented twice in either a massed or distributed manner, participants recalled the words and scored their recall protocols, reliably remembering more distributed than massed words. Posttest scores on a multiple-choice quiz covering points illustrated by the experiment averaged about twice the comparable pretest scores, indicating the effectiveness of the exercise in conveying content. Students’ subjective ratings suggested that the experiment helped convince them of the benefits of distributed study.
Commins, S., Cunningham, L., Harvey, D., and Walsh, D. (2003). “Massed but not spaced training impairs spatial memory”. Behavioural Brain Research 139, 215-223↩︎
Galluccio & Rovee-Collier 2006, “Nonuniform effects of reinstatement within the time window”. Learning and Motivation, 37, 1-17.↩︎
See the previous sections for many using children; one previously uncited is Toppino 1993, “The spacing effect in preschool children’s free recall of pictures and words”; but Toppino et al 2009 adds some interesting qualifiers to spaced repetition in the young:
Preschoolers, elementary school children, and college students exhibited a spacing effect in the free recall of pictures when learning was intentional. When learning was incidental and a shallow processing task requiring little semantic processing was used during list presentation, young adults still exhibited a spacing effect, but children consistently failed to do so. Children, however, did manifest a spacing effect in incidental learning when an elaborate semantic processing task was used.
Another previously uncited study: Glenberg, A. M. (1979), “Component-levels theory of the effects of spacing of repetitions on recall and recognition”. Memory & Cognition, 7, 95-112.↩︎
See Kornell et al 2010; Simone et al 2012 shows the spacing benefits but reduced in magnitude in its 56-74 year old subjects, similar to Jackson et al 2012 and Maddox 2013↩︎
Mammarella, N., Russo, R., & Avons, S. E. (2002). "Spacing effects in cued-memory tasks for unfamiliar faces and nonwords". Memory & Cognition, 30, 1238-1251↩︎
Childers, J. B., & Tomasello, M. (2002). "Two-year-olds learn novel nouns, verbs, and conventional actions from massed or distributed exposures". Developmental Psychology, 38, 967-978↩︎
The famous ‘10,000 hours of practice’ figure may not be as true or important as Ericsson and publicizers like Malcolm Gladwell imply, given the high variance of expertise against time, and results from sports showing smaller time investments (see also Hambrick’s corpus cutting ‘deliberate practice’ down to size), and Ericsson absurdly deny the powerful role of genetics and the necessary condition of having talent but the insight of ‘deliberate practice’ helping talented people probably is real. One may be able to get away with 3,000 hours rather than 10,000, but one isn’t going to do that with mindless repetition or no repetitions.↩︎
Gentner, D., Loewenstein, J., & Thompson, L. (2003). “Learning and transfer: A general role for analogical encoding”. Journal of Educational Psychology, 95, 393-40↩︎
From Kornell et al 2010:
The benefits of spacing seem to diminish or disappear when to-be-learned items are not repeated exactly (Appleton-Knapp, Bjork, & Wickens, 2005)…a number of studies have shown that massing, rather than spacing, promotes inductive learning. These studies have generally employed relatively simple perceptual stimuli that facilitate experimental control (Gagné, 1950; Goldstone, 1996; Kurtz & Hovland, 1956; [Whitman J. R., & Garner, W. R. (1963). “Concept learning as a function of the form of internal structure”. Journal of Verbal Learning & Verbal Behavior, 2, 195-202]).
High error rates - indicating one didn’t actually learn the card contents in the first place - seem to be connected to failures of the spacing effect; there’s some evidence that people naturally choose to mass study when they don’t yet know the material.↩︎
“SuperMemo as a new tool increasing the productivity of a programmer. A case study: programming in Object Windows”↩︎
The 20 years look like this (note the scientific notation):
[0.742675, 0.27044575182838654, 0.15275979054767388, 0.10348750000000001, 7.751290630254386e-2, 6.187922936397532e-2, 5.161829250474865e-2, 4.445884397854832e-2, 3.923055555555555e-2, 3.5275438307530015e-2, 3.219809429218694e-2, 2.9748098818459235e-2, 2.7759942051635768e-2, 2.6120309801216147e-2, 2.474928593068675e-2, 2.35890625e-2, 2.2596898475825956e-2, 2.1740583401051353e-2, 2.0995431241707652e-2, 2.0342238287817983e-2]
↩︎modulo things where knowing it is useful even if you don’t need it often - it can be a brick in a pyramid of knowledge; cf.page 3 of Wolf:
The problem of forgetting might not torment us so much if we could only convince ourselves that remembering isn’t important. Perhaps the things we learn - words, dates, formulas, historical and biographical details - don’t really matter. Facts can be looked up. That’s what the Internet is for. When it comes to learning, what really matters is how things fit together. We master the stories, the schemas, the frameworks, the paradigms; we rehearse the lingo; we swim in the episteme.
The disadvantage of this comforting notion is that it’s false. “The people who criticize memorization - how happy would they be to spell out every letter of every word they read?” asks Robert Bjork, chair of UCLA’s psychology department and one of the most eminent memory researchers. After all, Bjork notes, children learn to read whole words through intense practice, and every time we enter a new field we become children again. “You can’t escape memorization,” he says. “There is an initial process of learning the names of things. That’s a stage we all go through. It’s all the more important to go through it rapidly.” The human brain is a marvel of associative processing, but in order to make associations, data must be loaded into memory.
See Stephen R. Schmidt’s webpage “Theories of Forgetting”, which cites ‘Woodworth & Schlosbeg (1961)’ when presenting a log graph of various studies’ forgetting curves.↩︎
which neatly addresses the issue of such mailing lists being useless (‘who learns a word after just one exposure?’).↩︎
Mnemosyne in this case constitutes both a way to learn the quotes so I can use them, and a waste book; just the other day I had 3 or 4 apposite quotes for an essay because I had entered them into Mnemosyne months or years ago.↩︎
It’s well known that any speaker of a language understands many more words than they will ever use or be able to explicitly generate, that their “reading vocabulary” exceeds their “writing vocabulary”; less well-known is that on many problems, one can guess at well above random rates even while feeling unsure & ignorant, necessitating psychologists to employ forced-choice paradigms. Even less known is the capacity of recognition memory or “implicit memory”; this memory can apply to things like recognizing images or text or music, typing, puzzle solving, etc. Andrew Drucker, in “Multiplying 10-digit numbers using Flickr: The power of recognition memory”, employs visual memory to calculate ; he cites as precedent Standing 1973:
In one of the most widely-cited studies on recognition memory, Standing showed participants an epic 10,000 photographs over the course of 5 days, with 5 seconds’ exposure per image. He then tested their familiarity, essentially as described above. The participants showed an 83% success rate, suggesting that they had become familiar with about 6,600 images during their ordeal. Other volunteers, trained on a smaller collection of 1,000 images selected for vividness, had a 94% success rate.
One sometimes sees people argue that something is insecure or unguessable or free from possible placebo effect because it involves too many objects to explicitly memorize, but as these examples make clear, recognition memory can happen quickly and store surprisingly large amounts of information. This could be used for authentication (see for example Bojinov et al 2012; HN discussion) or message since recognition memory could be exploited as a sort of secure communication system. Two parties can share a set of 20,000 photographs (10,000 pairs); to send a message, have a messenger spend 5 days on 10,000 picked ones; and then to receive it, ask him to recognize which photograph he saw in each of the 10,000 pairs. The subject not only does not know what the binary message is or what means, he can’t even produce it since he cannot remember the photographs!
At an 80% accuracy rate, we can even calculate how many bits of information can be entrusted to the messenger using Shannon’s theorem; a calculation gives 5.8 kilobits as the upper limit: if p = 0.2 (based on the 80% success rate), then . So we see that Frank Herbert was right after all: the securest way to send a message is through a distrans messenger. (The downside is that the implicit recognition memory decays considerably; see Landauer 1986 for adjusted estimates.)↩︎
In this vein, I am reminded of what a former polyphasic sleeper told me:
I’ve been polyphasic for about a year. (Not anymore; kills my memory.)…Anki reps, mostly. I found that I could do proper review sessions for about 2-3 days and would hit an impenetrable wall. I couldn’t learn a single new card and had total brain fog until I got 3 hours more sleep. That, however, would reset my adaptation. The whole effect is a bit less pronounced on Everyman, but not much. It is however easier to add sleep when you already have a core. I didn’t notice any other major mental impairment after the initial sleep deprivation.
For a more recent review, see Philips et al 2013.↩︎
Presumably one would immediately give them all some high grade like 5 to avoid suddenly having a daily load of 500 cards for a while.↩︎
Smaller is better.↩︎
“For Mnemosyne 2.x, Ullrich is working on an official Mnemosyne iPhone client which will have very easy syncing.”↩︎
See Page 4, Wolf 2008:
The spacing effect was one of the proudest lab-derived discoveries, and it was interesting precisely because it was not obvious, even to professional teachers. The same year that Neisser revolted, Robert Bjork, working with Thomas Landauer of Bell Labs, published the results of two experiments involving nearly 700 undergraduate students. Landauer and Bjork were looking for the optimal moment to rehearse something so that it would later be remembered. Their results were impressive: The best time to study something is at the moment you are about to forget it. And yet - as Neisser might have predicted - that insight was useless in the real world.
When I first read of SuperMemo, I had already taken a class in cognitive psychology and was reasonably familiar with Ebbinghaus’s forgetting curve - so my reaction to its methodology was Huxley’s: “How extremely stupid not to have thought of that!”↩︎
See page 7, Wolf 2008
And yet now, as I grin broadly and wave to the gawkers, it occurs to me that the cold rationality of his approach may be only a surface feature and that, when linked to genuine rewards, even the chilliest of systems can have a certain visceral appeal. By projecting the achievement of extreme memory back along the forgetting curve, by provably linking the distant future - when we will know so much - to the few minutes we devote to studying today, Wozniak has found a way to condition his temperament along with his memory. He is making the future noticeable. He is trying not just to learn many things but to warm the process of learning itself with a draft of utopian ecstasy.