- “Amateur Hour: Improving Knowledge Diversity in Psychological and Behavioral Science by Harnessing Contributions from Amateurs”, Mohlhenrich & Krpan 2021
- “Testing (Quizzing) Boosts Classroom Learning: A Systematic And Meta-Analytic Review”, Yang et al 2021
- “Spaced Mathematics Practice Improves Test Scores and Reduces Overconfidence”, Emeny et al 2021
- “Self-Directed Learning Online: An Opportunity to Binge”, LaTour & Noel 2021
- “Smash Training Retrospective”, Khan 2020
- “The Overfitted Brain: Dreams Evolved to Assist Generalization”, Hoel 2020
- “Memorising Milton's Paradise Lost: A Study of a Septuagenarian Exceptional Memoriser”, Seamon et al 2020
- “How Can We Develop Transformative Tools For Thought?”, Matuschak & Nielsen 2019
- “Measuring Actual Learning versus Feeling of Learning in Response to Being Actively Engaged in the Classroom”, Deslauriers et al 2019
- “A Randomized Controlled Trial of Interleaved Mathematics Practice”, Rohrer et al 2019
- “Inducing Self-Explanation: a Meta-Analysis”, Bisra et al 2018
- “The Complexity of Human Computation: A Concrete Model With an Application to Passwords”, Blum & Vempala 2017
- “Learning From Errors”, Metcalfe 2017
- “Relearn Faster and Retain Longer: Along With Practice, Sleep Makes Perfect”, Mazza et al 2016
- “The Impact of Student-generated Digital Flashcards on Student Learning of Constitutional Law”, Colbran et al 2015
- “Is Expanded Retrieval Practice a Superior Form of Spaced Retrieval? A Critical Review of the Extant Literature”, Balota et al 2015
- “Why Is There so Much Resistance to Direct Instruction?”, McMullen & Madelaine 2014
- “Spaced Repetition and Mnemonics Enable Recall of Multiple Strong Passwords”, Blocki et al 2014
- “Equal Spacing and Expanding Schedules in Childrenâ€™s Categorization and Generalization”, Vlach et al 2014
- “Pattern and Predictability in Memory Formation: From Molecular Mechanisms to Clinical Relevance”, Philips et al 2013
- “Chapter 38: Test-Enhanced Learning”, Larsen & Butler 2013
- “Distributing Learning Over Time: The Spacing Effect in Children’s Acquisition and Generalization of Science Concepts”, Vlach & Sandhofer 2012
- “Using Quizzes to Enhance Summative-assessment Performance in a Web-based Class: An Experimental Study”, McDaniel et al 2012
- “Spacing and Induction: Application to Exemplars Presented As Auditory and Visual Text”, Zulkiply et al 2011
- “Multiplying 10-digit Numbers Using Flickr: The Power of Recognition Memory”, Drucker 2011
- “Optimizing Retrieval As a Learning Event: When and Why Expanding Retrieval Practice Enhances Long-term Retention”, Storm et al 2010
- “A Functional Application of the Spacing Effect to Improve Learning and Memory in Persons With Multiple Sclerosis”, Goverover et al 2009
- “Dual N-Back FAQ”, Branwen 2009
- “Optimising Learning Using Flashcards: Spacing Is More Effective Than Cramming”, Kornell 2009
- “Knowledge Retention After an Online Tutorial: a Randomized Educational Experiment among Resident Physicians”, Bell et al 2008
- “A Case of Unusual Autobiographical Remembering”, Parker et al 2007
- “The Power of Testing Memory: Basic Research and Implications for Educational Practice”, III & Karpicke 2006
- “Distributed Practice in Verbal Recall Tasks: A Review and Quantitative Synthesis”, Cepeda et al 2006
- “The Role of Encoding in the Self-Explanation Effect”, Calin-Jageman & Ratner 2005
- “Application of the Testing and Spacing Effects to Name Learning”, Carpenter & DeLosh 2005
- “Chapter 2: Contextual Interference”, Lee & Simon 2004
- “Interaction between Amount and Pattern of Training in the Induction of Intermediate-Term and Long-Term Memory for Sensitization in Aplysia”, Sutton et al 2002
- “Massed and Spaced Learning in Honeybees: The Role of CS, US, the Intertrial Interval, and the Test Interval”, Menzel et al 2001
- “Previous Meta˚Analytic Review”, Hessinger 1999
- “Florida Journal of Educational Research”
- “Http://gateway.ut.ovid.com/gw2/ovidweb.cgi”, WainnerRS 1995
- “Does the Sensitivity of Judgments of Learning (JOLs) to the Effects of Various Study Activities Depend on When the JOLs Occur?”
- “Maintenance of Foreign Language Vocabulary and the Spacing Effect”, Bahrick et al 1993
- “The Role of Deliberate Practice in the Acquisition of Expert Performance”, Ericsson et al 1993
- “You and Your Research”, Hamming 1986
- “Learning 10,000 Pictures”, Standing 1973
- “Studies on the Telegraphic Language: The Acquisition of a Hierarchy of Habits”, Bryan & Harter 1899
- “Studies in the Physiology and Psychology of the Telegraphic Language”, Bryan & Harter 1897
“Amateur Hour: Improving Knowledge Diversity in Psychological and Behavioral Science by Harnessing Contributions from Amateurs”, Mohlhenrich & Krpan 2021
2021-mohlhenrich.pdf: “Amateur hour: Improving knowledge diversity in psychological and behavioral science by harnessing contributions from amateurs”, (2021-11-30; ; similar):
- Low knowledge diversity is an important issue affecting psychological science.
- We propose this issue could be resolved by harnessing contributions from amateurs.
- We outline 6 “blind spots”—neglected areas in which amateurs could contribute.
- We discuss how amateur contributions could be practically achieved.
Contemporary psychological and behavioral science suffers from a lack of diversity regarding the key intellectual activities that constitute it, including its theorizing, empirical approaches, and topics studied. We refer to this type of diversity as knowledge diversity.
To fix the knowledge diversity problem, scientists have proposed several solutions that would require transforming the field itself—an endeavor that can realistically be realized only in the long term. In this article, we propose that knowledge diversity could also be attained in the short term without transforming the field itself—by harnessing contributions from amateurs who can explore diverse aspects of psychology that are neglected in academia.
We identify 6 such “blind spot” areas within which amateurs could contribute and discuss how this could be practically achieved.
Projects (eg. theory development, research pursuit) that require dedication over a long period of time with uncertain payoffs.|
Basic observational research
Conducting observational studies that aim to identify new phenomena or characterize the generalizability of already known phenomena.
Making speculations that are not limited by current methodological or other practical considerations.|
Projects that combine diverse areas of psychology (and potentially other disciplines) and do not involve working within a specific area of expertise or topic.|
Projects that do not have pre-determined goals or planned outcomes and evolve in any direction in which pursuing psychology-related ideas takes the person.|
Uncommon research areas
Research areas that are neglected by psychological scientists.|
Table 1: Blind spots that are not incentivized in academia and could be addressed by amateur psychologists to increase knowledge diversity in psychological and behavioral science.
We hope that our article will inspire professionals and academic institutions to be more open toward amateur contributions to create a diverse body of knowledge.
[Keywords: amateurs, knowledge diversity, psychology, blind-spots, inclusivity]
“Testing (Quizzing) Boosts Classroom Learning: A Systematic And Meta-Analytic Review”, Yang et al 2021
2021-yang.pdf: “Testing (Quizzing) Boosts Classroom Learning: A Systematic And Meta-Analytic Review”, (2021-03-08; ; similar):
Testing (class quizzing) yields a variety of learning benefits, even though learners, instructors, and policymakers tend to lack full metacognitive insight into the virtues of testing. The current meta-analysis finds a reliable advantage of testing over other strategies in facilitating learning of factual knowledge, concept comprehension, and knowledge application in the classroom. Overall, testing is not only an assessment of learning but also an assessment for learning.
Over the last century hundreds of studies have demonstrated that testing is an effective intervention to enhance long-term retention of studied knowledge and facilitate mastery of new information, compared with restudying and many other learning strategies (eg. concept mapping), a phenomenon termed the testing effect. How robust is this effect in applied settings beyond the laboratory?
The current review integrated 48,478 students’ data, extracted from k = 222 independent studies, to investigate the magnitude, boundary conditions, and psychological underpinnings of test-enhanced learning in the classroom. The results show that overall testing (quizzing) raises student academic achievement to a medium extent (g = 0.499). The magnitude of the effect is modulated by a variety of factors, including learning strategy in the control condition, test format consistency, material matching, provision of corrective feedback, number of test repetitions, test administration location and timepoint, treatment duration, and experimental design.
The documented findings support 3 theories to account for the classroom testing effect: additional exposure, transfer-appropriate processing, and motivation. In addition to their implications for theory development, these results have practical importance for enhancing teaching practice and guiding education policy and highlight important directions for future research.
[Keywords: meta-analysis, motivation, academic achievement, testing effect, transfer-appropriate processing]
2021-emeny.pdf: “Spaced mathematics practice improves test scores and reduces overconfidence”, (2021-02-27; similar):
The practice assignments in a mathematics textbook or course can be arranged so that most of the problems relating to any particular concept are massed together in a single assignment, or these related problems can be distributed across many assignments—a format known as spaced practice.
Here we report the results of two classroom experiments that assessed the effects of mathematics spacing on both test scores and students’ predictions of their test scores. In each experiment, students in Year 7 (11–12 years of age) either massed their practice into a single session or divided their practice across three sessions spaced 1 week apart, followed 1 month later by a test.
In both experiments, spaced practice produced higher test scores than did massed practice, and test score predictions were relatively accurate after spaced practice yet grossly overconfident after massed practice.
2021-latour.pdf: “Self-Directed Learning Online: An Opportunity to Binge”, (2021-01-17; similar):
The online classroom is self-directed, where students decide when and how often they access their course material. Even in the traditional classroom, students have shown a propensity to shift their time allocation to the last minute, so it is not clear what happens when they have full control over their learning schedules. Our interest is whether this self-directed learning environment produces similar harmful binge behavior as observed with online television, where memory and satisfaction with the experience decrease over time. With access to clickstream data from an online e-educator, we found 62% of the sample binged their learning by concentrating their studies within the semester rather than distributing their online activity throughout. Two types of binge learning emerged as significant: Front-bingers, who accessed the majority of their education early, performed more similarly over time to those who spaced their learning activities. Back-bingers, who accessed the majority of their material late in the semester, did not perform as well. To help us better understand these findings, we used a relatively new measure of behavior called “clumpiness” to summarize their overall online activity. We discuss our findings and their implications for online education and marketing course design.
Smash Training is a spaced-repetition training web-app I created to help my progression with Super Smash Bros. Ultimate. I released it on May 16, 2020 on Reddit to warm reception. As of December 2020, it receives 150–200 monthly users. I’d rank it as my most successful project! In this article, I discuss the choices I made for this project. (The source code is available).
…I decided that I wanted to build a spaced-repetition training app, rather than reuse a general-purpose spaced-repetition flash-card system such as Anki, because the project would benefit from domain-specific knowledge. For example:
- Exercises have large numbers of variants, such as “short-hop” vs “full-hop”, or “facing left” vs “facing right”, which should be tracked separately.
- Many of the exercises have natural dependencies on others: they shouldn’t be attempted unless a certain underlying fundamental skill has been mastered.
- Exercises to train one character don’t necessarily confer the same skill for other characters. Some exercises may only be applicable to some characters.
…Stronglifts has you note down how many repetitions of the exercise you succeeded at (out of five). However, the Smash Training paradigm is different, and has you repeat the exercise for a length of time and rate your accuracy.
“The Overfitted Brain: Dreams evolved to assist generalization”, (2020-07-19; ; ; similar):
Understanding of the evolved biological function of sleep has advanced considerably in the past decade. However, no equivalent understanding of dreams has emerged. Contemporary neuroscientific theories generally view dreams as epiphenomena, and the few proposals for their biological function are contradicted by the phenomenology of dreams themselves. Now, the recent advent of deep neural networks (DNNs) has finally provided the novel conceptual framework within which to understand the evolved function of dreams. Notably, all DNNs face the issue of overfitting as they learn, which is when performance on one data set increases but the network’s performance fails to generalize (often measured by the divergence of performance on training vs. testing data sets). This ubiquitous problem in DNNs is often solved by modelers via “noise injections” in the form of noisy or corrupted inputs. The goal of this paper is to argue that the brain faces a similar challenge of overfitting, and that nightly dreams evolved to combat the brain’s overfitting during its daily learning. That is, dreams are a biological mechanism for increasing generalizability via the creation of corrupted sensory inputs from stochastic activity across the hierarchy of neural structures. Sleep loss, specifically dream loss, leads to an overfitted brain that can still memorize and learn but fails to generalize appropriately. Herein this “overfitted brain hypothesis” is explicitly developed and then compared and contrasted with existing contemporary neuroscientific theories of dreams. Existing evidence for the hypothesis is surveyed within both neuroscience and deep learning, and a set of testable predictions are put forward that can be pursued both in vivo and in silico.
“Memorising Milton's Paradise Lost: A Study of a Septuagenarian Exceptional Memoriser”, Seamon et al 2020
2010-seamon.pdf: “Memorising Milton's Paradise Lost: A study of a septuagenarian exceptional memoriser”, (2020-04-23; ; similar):
At age 58, JB [John Basinger] began memorizing Milton’s epic poem Paradise Lost. 9 years and thousands of study hours later, he completed this process in 2001 and recalled from memory all 12 books of this 10,565-line poem over a 3-day period. Now 74, JB continues to recite this work. We tested his memory accuracy by cueing his recall with two lines from the beginning or middle of each book and asking JB to recall the next 10 lines. JB is an exceptional memoriser of Milton, both in our laboratory tests in which he did not know the specific tests or procedures in advance, and in our analysis of a videotaped, prepared performance. Consistent with deliberate practice theory, JB achieved this remarkable ability by deeply analysing the poem’s structure and meaning over lengthy repetitions. Our findings suggest that exceptional memorizers such as JB are made, not born, and that cognitive expertise can be demonstrated even in later adulthood.
[Keywords: Exceptional memory, Prose memory, Age and memory]
“How Can We Develop Transformative Tools For Thought?”, (2019-10; ; ; similar):
[Long writeup by Andy Matuschak and Michael Nielsen on experiment in integrating spaced repetition systems with a tutorial on quantum computing, Quantum Country: Quantum Computing For The Very Curious By combining explanation with spaced testing, a notoriously thorny subject may be learned more easily and then actually remembered—such a system demonstrating a possible ‘tool for thought’. Early results indicate users do indeed remember the quiz answers, and feedback has been positive.]
Part I: Memory systems
- Introducing the mnemonic medium
- The early impact of the prototype mnemonic medium
- Expanding the scope of memory systems: what types of understanding can they be used for?
- Improving the mnemonic medium: making better cards
- Two cheers for mnemonic techniques
- How important is memory, anyway?
- How to invent Hindu-Arabic numerals?
Part II: Exploring tools for thought more broadly:
Why isn’t there more work on tools for thought today?
Questioning our basic premises
- What if the best tools for thought have already been discovered?
- Isn’t this what the tech industry does? Isn’t there a lot of ongoing progress on tools for thought?
- Why not work on AGI or BCI instead?
- Serious work and the aspiration to canonical content
- Stronger emotional connection through an inverted writing structure
Summary and Conclusion
… in Quantum Country an expert writes the cards, an expert who is skilled not only in the subject matter of the essay, but also in strategies which can be used to encode abstract, conceptual knowledge. And so Quantum Country provides a much more scalable approach to using memory systems to do abstract, conceptual learning. In some sense, Quantum Country aims to expand the range of subjects users can comprehend at all. In that, it has very different aspirations to all prior memory systems.
More generally, we believe memory systems are a far richer space than has previously been realized. Existing memory systems barely scratch the surface of what is possible. We’ve taken to thinking of Quantum Country as a memory laboratory. That is, it’s a system which can be used both to better understand how memory works, and also to develop new kinds of memory system. We’d like to answer questions such as:
- What are new ways memory systems can be applied, beyond the simple, declarative knowledge of past systems?
- How deep can the understanding developed through a memory system be? What patterns will help users deepen their understanding as much as possible?
- How far can we raise the human capacity for memory? And with how much ease? What are the benefits and drawbacks?
- Might it be that one day most human beings will have a regular memory practice, as part of their everyday lives? Can we make it so memory becomes a choice; is it possible to in some sense solve the problem of memory?
“Measuring Actual Learning versus Feeling of Learning in Response to Being Actively Engaged in the Classroom”, Deslauriers et al 2019
Despite active learning being recognized as a superior method of instruction in the classroom, a major recent survey found that most college STEM instructors still choose traditional teaching methods. This article addresses the long-standing question of why students and faculty remain resistant to active learning. Comparing passive lectures with active learning using a randomized experimental approach and identical course materials, we find that students in the active classroom learn more, but they feel like they learn less. We show that this negative correlation is caused in part by the increased cognitive effort required during active learning. Faculty who adopt active learning are encouraged to intervene and address this misperception, and we describe a successful example of such an intervention.
We compared students’ self-reported perception of learning with their actual learning under controlled conditions in large-enrollment introductory college physics courses taught using (1) active instruction (following best practices in the discipline) and (2) passive instruction (lectures by experienced and highly rated instructors). Both groups received identical class content and handouts, students were randomly assigned, and the instructor made no effort to persuade students of the benefit of either method. Students in active classrooms learned more (as would be expected based on prior research), but their perception of learning, while positive, was lower than that of their peers in passive environments. This suggests that attempts to evaluate instruction based on students’ perceptions of learning could inadvertently promote inferior (passive) pedagogical methods. For instance, a superstar lecturer could create such a positive feeling of learning that students would choose those lectures over active learning. Most importantly, these results suggest that when students experience the increased cognitive effort associated with active learning, they initially take that effort to signify poorer learning. That disconnect may have a detrimental effect on students’ motivation, engagement, and ability to self-regulate their own learning. Although students can, on their own, discover the increased value of being actively engaged during a semester-long course, their learning may be impaired during the initial part of the course. We discuss strategies that instructors can use, early in the semester, to improve students’ response to being actively engaged in the classroom.
[Keywords: scientific teaching, undergraduate education, evidence-based teaching, Constructivism]
2019-rohrer.pdf: “A randomized controlled trial of interleaved mathematics practice”, Doug Rohrer, Robert F. Dedrick, Marissa K. Hartwig, Chi-Ngai Cheung (2019-01-01; )
Self-explanation is a process by which learners generate inferences about causal connections or conceptual relationships.
A meta-analysis was conducted on research that investigated learning outcomes for participants who received self-explanation prompts while studying or solving problems.
Our systematic search of relevant bibliographic databases identified 69 effect sizes (from 64 research reports) which met certain inclusion criteria. The overall weighted mean effect size using a random effects model was g = 0.55.
We coded and analyzed 20 moderator variables including type of learning task (eg. solving problems, studying worked problems, and studying text), subject area, level of education, type of inducement, and treatment duration. We found that self-explanation prompts are a potentially powerful intervention across a range of instructional conditions.
Due to the limitations of relying on instructor-scripted prompts, we recommend that future research explore computer-generation of self-explanation prompts.
[Keywords: self-explanation, instructional explanation, meta-analysis, prompts]
“The Complexity of Human Computation: A Concrete Model With an Application to Passwords”, Blum & Vempala 2017
What can humans compute in their heads? We are thinking of a variety of Crypto Protocols, games like Sudoku, Crossword Puzzles, Speed Chess, and so on. The intent of this paper is to apply the ideas and methods of theoretical computer science to better understand what humans can compute in their heads. For example, can a person compute a function in their head so that an eavesdropper with a powerful computer—who sees the responses to random input—still cannot infer responses to new inputs? To address such questions, we propose a rigorous model of human computation and associated measures of complexity. We apply the model and measures first and foremost to the problem of (1) humanly computable password generation, and then consider related problems of (2) humanly computable “one-way functions” and (3) humanly computable “pseudorandom generators”.
The theory of Human Computability developed here plays by different rules than standard computability, and this takes some getting used to. For reasons to be made clear, the polynomial versus exponential time divide of modern computability theory is irrelevant to human computation. In human computability, the step-counts for both humans and computers must be more concrete. Specifically, we restrict the adversary to at most 1024 (Avogadro number of) steps. An alternate view of this work is that it deals with the analysis of algorithms and counting steps for the case that inputs are small as opposed to the usual case of inputs large-in-the-limit.
Although error avoidance during learning appears to be the rule in American classrooms, laboratory studies suggest that it may be a counterproductive strategy, at least for neurologically typical students.
Experimental investigations indicate that errorful learning followed by corrective feedback is beneficial to learning. Interestingly, the beneficial effects are particularly salient when individuals strongly believe that their error is correct: Errors committed with high confidence are corrected more readily than low-confidence errors. Corrective feedback, including analysis of the reasoning leading up to the mistake, is crucial.
Aside from the direct benefit to learners, teachers gain valuable information from errors, and error tolerance encourages students’ active, exploratory, generative engagement. If the goal is optimal performance in high-stakes situations, it may be worthwhile to allow and even encourage students to commit and correct errors while they are in low-stakes learning situations rather than to assiduously avoid errors at all costs.
[Keywords: errorless learning, generation effect, hypercorrection effect, feedback, after-action review (AAR), error management training (EMT), formative assessment, reconsolidation, prediction error]
- Encouraging Versus Discouraging Errors In The Classroom
- Error Generation And Memory For Correct Responses In The Lab
- Confidence In Errors
- Implications Of The Hypercorrection Effect
- Theories Of Why Errors Enhance Learning
- Secondary Benefits Of Encouraging Errors
- Origin Of The Idea That Errorless Learning Is A Good Thing
- Emotional Consequences Of Errors
- Using Errors To Improve Learning
2016-mazza.pdf: “Relearn Faster and Retain Longer: Along With Practice, Sleep Makes Perfect”, (2016-08-16; ; similar):
Both repeated practice and sleep improve long-term retention of information. The assumed common mechanism underlying these effects is memory reactivation, either on-line and effortful or off-line and effortless.
In the study reported here, we investigated whether sleep-dependent memory consolidation could help to save practice time during relearning. During two sessions occurring 12 hr apart, 40 participants practiced foreign vocabulary until they reached a perfect level of performance. Half of them learned in the morning and relearned in the evening of a single day. The other half learned in the evening of one day, slept, and then relearned in the morning of the next day. Their retention was assessed 1 week later and 6 months later. We found that interleaving sleep between learning sessions not only reduced the amount of practice needed by half but also ensured much better long-term retention.
Sleeping after learning is definitely a good strategy, but sleeping between two learning sessions is a better strategy.
[Keywords: learning, sleep-wake cycle, relearning, sleep-dependent memory consolidation, repeated practice]
“The Impact of Student-generated Digital Flashcards on Student Learning of Constitutional Law”, Colbran et al 2015
2015-colbran.pdf: “The impact of student-generated digital flashcards on student learning of constitutional law”, (2015-10-01; ; similar):
This article describes, evaluates and reflects upon student creation of cloud-based digital flashcards as an authentic formative and summative assessment task designed for the deep learning of constitutional law.
The usefulness of digital flashcards in online legal education is explored. The undergraduate law student participants in the study responded differently to the assessment task depending upon the constitutional law topic they were assigned, the perceived relevance of constructing digital flashcards to professional practice and how they reacted to this creative task.
Building digital flashcards provides a potentially powerful authentic assessment task for the study of constitutional law provided it is designed to support semester long creation, validation and sharing of digital flashcards that students perceive as professionally relevant and educationally useful.
Student recommendations for designing an assessment task involving the creation of digital flashcards are evaluated.
[Keywords: Flashcards, student directed learning, authentic learning, law assignments, online student flashcards, online legal education, constitutional law]
“Is Expanded Retrieval Practice a Superior Form of Spaced Retrieval? A Critical Review of the Extant Literature”, Balota et al 2015
The spacing effect is one of the most ubiquitous findings in learning and memory. Performance on a variety of tasks is better when the repetition of the to-be-learned information is distributed as opposed to massed in presentation. This observation was first formalized in Jost’s law, which states that “if two associations are of equal strength but of different age, a new repetition has a greater value for the older one” (McGeogh, 1943). Spacing effects occur across domains (eg. learning perceptual motor tasks vs. learning lists of words), across species (eg. rats, pigeons, and humans), across age groups and individuals with different memory impairments, and across retention intervals of seconds to months (see Cepeda et al 2006; Crowder 1976; Dempster 1996, for reviews).
In this light, it is interesting that spacing effects have not received much attention in Cognitive Psychology textbooks. In fact, in our sampling of 7 such textbooks, only one had a section dedicated to this topic, while virtually all cognitive text-books discussed mnemonic techniques such as the pegword or method of loci. Given the power and simplicity of implementing spaced practice, we clearly hope this changes in the future.
2014-mcmullen.pdf: “Why is there so much resistance to Direct Instruction?”, (2014-12-04; similar):
Direct Instruction (DI) has been the subject of empirical research since its inception in the 1960s and has garnered a strong research base to support it. Despite its proven efficacy, Direct Instruction is not widely implemented and draws much criticism from some educators. This literature review details the components of Direct Instruction, research to support it and reported attitudes towards it. The aspects of Direct Instruction that attract the most criticism are broken down to determine just what it is that educators do not like about it. In addition, this review attempts to outline possible ways to improve the landscape for Direct Instruction by reviewing research on how best to achieve a shift in beliefs when adopting change in schools. This includes pre-service teacher education and professional development and support for practising teachers as a means of improving rates of implementation of Direct Instruction.
We report on an user study that provides evidence that spaced repetition and a specific mnemonic technique enable users to successfully recall multiple strong passwords over time. Remote research participants were asked to memorize 4 Person-Action-Object (PAO) stories where they chose a famous person from a drop-down list and were given machine-generated random action-object pairs. Users were also shown a photo of a scene and asked to imagine the PAO story taking place in the scene (eg. Bill Gates—swallowing—bike on a beach). Subsequently, they were asked to recall the action-object pairs when prompted with the associated scene-person pairs following a spaced repetition schedule over a period of 127+ days. While we evaluated several spaced repetition schedules, the best results were obtained when users initially returned after 12 hours and then in 1.5× increasing intervals: 77% of the participants successfully recalled all 4 stories in 10 tests over a period of 158 days. Much of the forgetting happened in the first test period (12 hours): 89% of participants who remembered their stories during the first test period successfully remembered them in every subsequent round. These findings, coupled with recent results on naturally rehearsing password schemes, suggest that 4 PAO stories could be used to create usable and strong passwords for 14 sensitive accounts following this spaced repetition schedule, possibly with a few extra upfront rehearsals. In addition, we find that there is an interference effect across multiple PAO stories: the recall rate of 100% (resp. 90%) for participants who were asked to memorize 1 PAO story (resp. 2 PAO stories) is statistically-significantly better than the recall rate for participants who were asked to memorize 4 PAO stories. These findings yield concrete advice for improving constructions of password management schemes and future user studies.
“Equal Spacing and Expanding Schedules in Childrenâ€™s Categorization and Generalization”, Vlach et al 2014
2014-vlach.pdf: “Equal spacing and expanding schedules in childrenâ€™s categorization and generalization”, Haley A. Vlach, Catherine M. Sandhofer, Robert A. Bjork (2014-01-01; )
“Pattern and Predictability in Memory Formation: From Molecular Mechanisms to Clinical Relevance”, Philips et al 2013
2013-philips.pdf: “Pattern and predictability in memory formation: From molecular mechanisms to clinical relevance”, Gary T. Philips, Ashley M. Kopec, Thomas J. Carew (2013-01-01; )
2013-larsen.pdf: “Chapter 38: Test-Enhanced Learning”, Douglas P. Larsen, Andrew C. Butler (2013-01-01; )
“Distributing Learning Over Time: The Spacing Effect in Children’s Acquisition and Generalization of Science Concepts”, Vlach & Sandhofer 2012
The spacing effect describes the robust finding that long-term learning is promoted when learning events are spaced out in time, rather than presented in immediate succession. Studies of the spacing effect have focused on memory processes rather than for other types of learning, such as the acquisition and generalization of new concepts. In this study, early elementary school children (5–7 year-olds; n = 36) were presented with science lessons on one of three schedules: massed, clumped, and spaced. The results revealed that spacing lessons out in time resulted in higher generalization performance for both simple and complex concepts. Spaced learning schedules promote several types of learning, strengthening the implications of the spacing effect for educational practices and curriculum.
[Keywords: spacing effect, distributed learning, learning and memory, generalization, cognitive development, educational curriculum and practices]
“Using Quizzes to Enhance Summative-assessment Performance in a Web-based Class: An Experimental Study”, McDaniel et al 2012
2012-mcdaniel.pdf: “Using quizzes to enhance summative-assessment performance in a web-based class: An experimental study”, Mark A. McDaniel, Kathleen M. Wildman, Janis L. Anderson (2012-01-01; )
“Spacing and Induction: Application to Exemplars Presented As Auditory and Visual Text”, Zulkiply et al 2011
2011-zulkiply.pdf: “Spacing and induction: Application to exemplars presented as auditory and visual text”, Norehan Zulkiply, John McLean, Jennifer S. Burt, Debra Bath (2011-01-01; )
In this informal article, I’ll describe the “recognition method”—a simple, powerful technique for memorization and mental calculation. Compared to traditional memorization techniques, which use elaborate encoding and visualization processes 1, the recognition method is easy to learn and requires relatively little effort…The method works: using it, I was able to mentally multiply two random 10-digit numbers, by the usual grade-school algorithm, on my first attempt! I have a normal, untrained memory, and the task would have been impossible by a direct approach. (I can’t claim I was speedy: I worked slowly and carefully, using about 7 hours plus rest breaks. I practiced twice with 5-digit numbers beforehand.)
…It turns out that ordinary people are incredibly good at this task [recognizing whether a photograph has been seen before]. In one of the most widely-cited studies on recognition memory, Standing 1973 showed participants an epic 10,000 photographs over the course of 5 days, with 5 seconds’ exposure per image. He then tested their familiarity, essentially as described above. The participants showed an 83% success rate, suggesting that they had become familiar with about 6,600 images during their ordeal. Other volunteers, trained on a smaller collection of 1,000 images selected for vividness, had a 94% success rate.
“Optimizing Retrieval As a Learning Event: When and Why Expanding Retrieval Practice Enhances Long-term Retention”, Storm et al 2010
Retrieving information from memory makes that information more recallable in the future than it otherwise would have been. Optimizing retrieval practice has been assumed, on the basis of evidence and arguments tracing back to Landauer and Bjork (1978), to require an expanding-interval schedule of successive retrievals, but recent findings suggest that expanding retrieval practice may be inferior to uniform-interval retrieval practice when memory is tested after a long retention interval.
We report three experiments in which participants read educational passages and were then repeatedly tested, without feedback, after an expanding or uniform sequence of intervals. On a test 1 week later, recall was enhanced by the expanding schedule, but only when the task between successive retrievals was highly interfering with memory for the passage. These results suggest that the extent to which learners benefit from expanding retrieval practice depends on the degree to which the to-be-learned information is vulnerable to forgetting.
“A Functional Application of the Spacing Effect to Improve Learning and Memory in Persons With Multiple Sclerosis”, Goverover et al 2009
2009-goverover.pdf: “A functional application of the spacing effect to improve learning and memory in persons with multiple sclerosis”, (2009-05-20; ; similar):
The present study examined the utility of using spaced learning trials (when trials are distributed over time) versus massed learning trials (consecutive learning trials) in the acquisition of everyday functional tasks.
In a within-subjects design, 20 participants with multiple sclerosis (MS) and 18 healthy controls (HC) completed 2 route learning tasks and 2 paragraph reading tasks. One task in each area was presented in the “spaced” condition, in which the task was presented to the participants 3 times with 5-minutes break between each trial, and the second task in each area was presented in the “massed” condition, in which the task was presented 3 consecutive times to the participants. The dependent variables consisted of recall and recognition of the paragraphs and routes both immediately and 30 minutes following initial learning.
Results showed that for paragraph learning, the spaced condition statistically-significantly enhanced memory performance for this task relative to the massed condition. However, this effect was not demonstrated in the route learning task. Thus, the spacing effect can be beneficial to enhance recall and performance of activities of daily living for individuals with MS; however, this effect was statistically-significant for verbal tasks stimuli, but not for visual tasks stimuli.
It will be important during future investigations to better characterize the factors that maximize the spacing effect.
[Keywords: memory, activities of daily living, cognitive rehabilitation, multiple sclerosis, spacing effect]
A compendium of DNB, WM, IQ information up to 2015.
Between 2008 and 2011, I collected a number of anecdotal reports about the effects of n-backing; there are many other anecdotes out there, but the following are a good representation—for what they’re worth.
- The Argument
- Notes from the author
- N-back training
- What’s some relevant research?
- Moody 2009 (re: Jaeggi 2008)
- Seidler 2010
- Jonasson 2011
- Chooi 2011
- Preece 2011 / Palmer 2011
- Kundu et al 2012
- Salminen 2012
- Redick et al 2012
- Rudebeck 2012
- Heinzel et al 2013
- Thompson et al 2013
- Smith et al 2013
- Nussbaumer et al 2013
- Oelhafen et al 2013
- Sprenger et al 2013
- Colom et al 2013
- Burki et al 2014
- Pugin et al 2014
- Heffernan 2014
- Hancock 2013
- Waris et al 2015
- Baniqued et al 2015
- Kuper & Karbach 2015
- Lindeløv et al 2016
- Schwarb et al 2015
- Lawlor-Savage & Goghari 2016
- Studer-Luethi et al 2015
- Minear et al 2016
- Studer-Luethi et al 2016
- Does it really work?
- Non-IQ or non-DNB gains
- Lucid dreaming
- What else can I do?
- See Also
“Optimising Learning Using Flashcards: Spacing Is More Effective Than Cramming”, (2009-01-19; ; ; similar):
The spacing effect—that is, the benefit of spacing learning events apart rather than massing them together—has been demonstrated in hundreds of experiments, but is not well known to educators or learners.
I investigated the spacing effect in the realistic context of flashcard use. Learners often divide flashcards into relatively small stacks, but compared to a large stack, small stacks decrease the spacing between study trials. In three experiments, participants used a web-based study programme to learn GRE-type word pairs.
Studying one large stack of flashcards (ie. spacing) was more effective than studying four smaller stacks of flashcards separately (ie. massing). Spacing was also more effective than cramming—that is, massing study on the last day before the test. Across experiments, spacing was more effective than massing for 90% of the participants, yet after the first study session, 72% of the participants believed that massing had been more effective than spacing.
“Knowledge Retention After an Online Tutorial: a Randomized Educational Experiment among Resident Physicians”, Bell et al 2008
Background: The time course of physicians’ knowledge retention after learning activities has not been well characterized. Understanding the time course of retention is critical to optimizing the reinforcement of knowledge.
Design: Educational follow-up experiment with knowledge retention measured at 1 of 6 randomly assigned time intervals (0–55 days) after an online tutorial covering 2 American Diabetes Association guidelines.
Participants: Internal and family medicine residents.
Measurements: Multiple-choice knowledge tests, subject characteristics including critical appraisal skills, and learner satisfaction.
Results: Of 197 residents invited, 91 (46%) completed the tutorial and were randomized; of these, 87 (96%) provided complete follow-up data. Ninety-two percent of the subjects rated the tutorial as “very good” or “excellent.” Mean knowledge scores increased from 50% before the tutorial to 76% among those tested immediately afterward. Score gains were only half as great at 3–8 days and no significant retention was measurable at 55 days. The shape of the retention curve corresponded with a 1/4-power transformation of the delay interval. In multivariate analyses, critical appraisal skills and participant age were associated with greater initial learning, but no participant characteristic significantly modified the rate of decline in retention.
Conclusions: Education that appears successful from immediate post-tests and learner evaluations can result in knowledge that is mostly lost to recall over the ensuing days and weeks. To achieve longer-term retention, physicians should review or otherwise reinforce new learning after as little as 1 week.
This report describes AJ, a woman whose remembering dominates her life. Her memory is “nonstop, uncontrollable, and automatic.” AJ spends an excessive amount of time recalling her personal past with considerable accuracy and reliability. If given a date, she can tell you what she was doing and what day of the week it fell on. She differs from other cases of superior memory who use practiced mnemonics to remember vast amounts of personally irrelevant information.
We propose the name hyperthymestic syndrome, from the Greek word thymesis meaning remembering, and that AJ is the first reported case. [Since renamed Highly Superior Autobiographical Memory (HSAM).]
“The Power of Testing Memory: Basic Research and Implications for Educational Practice”, III & Karpicke 2006
A powerful way of improving one’s memory for material is to be tested on that material. Tests enhance later retention more than additional study of the material, even when tests are given without feedback. This surprising phenomenon is called the testing effect, and although it has been studied by cognitive psychologists sporadically over the years, today there is a renewed effort to learn why testing is effective and to apply testing in educational settings. In this article, we selectively review laboratory studies that reveal the power of testing in improving retention and then turn to studies that demonstrate the basic effects in educational settings. We also consider the related concepts of dynamic testing and formative assessment as other means of using tests to improve learning. Finally, we consider some negative consequences of testing that may occur in certain circumstances, though these negative effects are often small and do not cancel out the large positive effects of testing. Frequent testing in the classroom may boost educational achievement at all levels of education.
“Distributed Practice in Verbal Recall Tasks: A Review and Quantitative Synthesis”, Cepeda et al 2006
The authors performed a meta-analysis of the distributed practice effect to illuminate the effects of temporal variables that have been neglected in previous reviews. This review found 839 assessments of distributed practice in 317 experiments located in 184 articles. Effects of spacing (consecutive massed presentations vs. spaced learning episodes) and lag (less spaced vs. more spaced learning episodes) were examined, as were expanding inter study interval (ISI) effects. Analyses suggest that ISI and retention interval operate jointly to affect final-test retention; specifically, the ISI producing maximal retention increased as retention interval increased. Areas needing future research and theoretical implications are discussed.
2005-calinjageman.pdf: “The Role of Encoding in the Self-Explanation Effect”, (2005-04; ; similar):
We examined the relation between self-explaining and encoding among kindergartners.
For 5 days, children (n = 27) took turns solving addition problems with an adult expert who always used an advanced addition strategy. During the game, children explained the expert’s answers (Explain-Expert), explained their own answers (Explain-Novice), or did not generate explanations (Control). Encoding of the expert’s strategy was measured each day by asking children to describe how the expert had solved the last problem.
Explain-Expert children encoded more and learned more than children in the Control group; Explain-Novice children showed neither advantage. The Explain-Expert group also acquired the expert’s strategy more rapidly and used it more frequently than the other groups.
These results suggest that explanations enhance learning in part by facilitating encoding.
2005-carpenter.pdf: “Application of the testing and spacing effects to name learning”, (2005-03-14; ; similar):
4 experiments investigated the effects of testing and spacing on the learning of face-name stimulus-response pairs:
- Experiments 1a and 1b compared the recall of names following intervening tests versus additional study opportunities and found that testing produced better retention of names.
- Experiments 2 and 3 explored the effects of repeated tests versus study for massed, uniform, or expanded spacing intervals.
Tested names were better retained than studied names, spaced names were better retained than massed names, and memory was best for items tested at spaced intervals. Contrary to past findings, expanded schedules did not yield better memory than uniform schedules in either experiment.
Theoretical implications for the testing and spacing effects are discussed, along with effective name-learning techniques based on these principles.
2004-lee.pdf: “Chapter 2: Contextual interference”, Timothy D. Lee, Dominic A. Simon (2004-01-01; )
“Interaction between Amount and Pattern of Training in the Induction of Intermediate-Term and Long-Term Memory for Sensitization in Aplysia”, Sutton et al 2002
In Aplysia, three distinct phases of memory for sensitization can be dissociated based on their temporal and molecular features. A single training trial induces short-term memory (STM, lasting <30 min), whereas five trials delivered at 15-min intervals induces both intermediate-term memory (ITM, lasting >90 min) and long-term memory (LTM, lasting >24 h). Here, we explore the interaction of amount and pattern of training in establishing ITM and LTM by examining memory for sensitization after different numbers of trials (each trial = one tail shock) and different patterns of training (massed vs. spaced). Under spaced training patterns, two trials produced STM exclusively, whereas four or five trials each produced both ITM and LTM. Three spaced trials failed to induce LTM but did produce an early decaying form of ITM (E-ITM) that was statistically-significantly shorter and weaker in magnitude than the late-decaying ITM (L-ITM) observed after four to five trials. In addition, E-ITM was induced after three trials with both massed and spaced patterns of training. However, L-ITM and LTM after four to five trials require spaced training: Four or five massed trials failed to induce LTM and produced only E-ITM. Collectively, our results indicate that in addition to three identified phases of memory for sensitization—STM, ITM, and LTM—a unique temporal profile of memory, E-ITM, is revealed by varying either the amount or pattern of training.
“Massed and Spaced Learning in Honeybees: The Role of CS, US, the Intertrial Interval, and the Test Interval”, Menzel et al 2001
Conditioning the proboscis extension reflex of harnessed honeybees (Apis mellifera) is used to study the effect temporal spacing between successive conditioning trials has on memory. Retention is monitored at two long-term intervals corresponding to early (1 and 2 d after conditioning) and late long-term memory (3 and 4 d). The acquisition level is varied by using different conditioned stimuli (odors, mechanical stimulation, and temperature increase at the antenna), varying strengths of the unconditioned stimulus (sucrose), and various numbers of conditioning trials.
How learning trials are spaced is the dominant factor both for acquisition and retention, and although longer intertrial intervals lead to better acquisition and higher retention, the level of acquisition per se does not determine the spacing effect on retention. Rather, spaced conditioning leads to higher memory consolidation both during acquisition and later, between the early and long-term memory phases. These consolidation processes can be selectively inhibited by blocking protein synthesis during acquisition.
2000-jamieson.pdf: “99202” (2000-01-01; )
1999-donovan.pdf: “Previous Meta˚Analytic Review”, Roy Hessinger (1999-01-01; )
1997-revak.pdf: “Florida Journal of Educational Research” (1997-01-01; )
1995-davis.pdf: “http: / / gateway.ut.ovid.com / gw2 / ovidweb.cgi”, WainnerRS (1995-01-01; )
“Does the Sensitivity of Judgments of Learning (JOLs) to the Effects of Various Study Activities Depend on When the JOLs Occur?”
1994-dunlosky.pdf: “Does the Sensitivity of Judgments of Learning (JOLs) to the Effects of Various Study Activities Depend on When the JOLs Occur?” (1994-01-01; ; )
1993-bahrick.pdf: “Maintenance of Foreign Language Vocabulary and the Spacing Effect”, (1993-09-01; ; similar):
In a 9-year longitudinal investigation, 4 subjects learned and relearned 300 English-foreign language word pairs. Either 13 or 26 relearning sessions were administered at intervals of 14, 28, or 56 days. Retention was tested for 1, 2, 3, or 5 years after training terminated. The longer intersession intervals slowed down acquisition slightly, but this disadvantage during training was offset by substantially higher retention. 13 retraining sessions spaced at 56 days yielded retention comparable to 26 sessions spaced at 14 days. The retention benefit due to additional sessions was independent of the benefit due to spacing, and both variables facilitated retention of words regardless of difficulty level and of the consistency of retrieval during training. The benefits of spaced retrieval practice to long-term maintenance of access to academic knowledge areas are discussed.
1993-ericsson.pdf: “The role of deliberate practice in the acquisition of expert performance”, (1993-07; ; ; similar):
The theoretical framework presented in this article explains expert performance as the end result of individuals’ prolonged efforts to improve performance while negotiating motivational and external constraints. In most domains of expertise, individuals begin in their childhood a regimen of effortful activities (deliberate practice) designed to optimize improvement. Individual differences, even among elite performers, are closely related to assessed amounts of deliberate practice. Many characteristics once believed to reflect innate talent are actually the result of intense practice extended for a minimum of 10 years. Analysis of expert performance provides unique evidence on the potential and limits of extreme environmental adaptation and learning.
[Transcript of a talk by mathematician and Bell Labs manager Richard Hamming about what he had learned about computers and how to do effective research (republished in expanded form as Art of Doing Science and Engineering: Learning to Learn; 1995 video). It is one of the most famous and most-quoted such discussions ever.]
At a seminar in the Bell Communications Research Colloquia Series, Dr. Richard W. Hamming, a Professor at the Naval Postgraduate School in Monterey, California and a retired Bell Labs scientist, gave a very interesting and stimulating talk, ‘You and Your Research’ to an overflow audience of some 200 Bellcore staff members and visitors at the Morris Research and Engineering Center on March 7, 1986. This talk centered on Hamming’s observations and research on the question “Why do so few scientists make substantial contributions and so many are forgotten in the long run?” From his more than 40 years of experience, 30 of which were at Bell Laboratories, he has made a number of direct observations, asked very pointed questions of scientists about what, how, and why they did things, studied the lives of great scientists and great contributions, and has done introspection and studied theories of creativity. The talk is about what he has learned in terms of the properties of the individual scientists, their abilities, traits, working habits, attitudes, and philosophy.
Four experiments are reported which examined memory capacity and retrieval speed for pictures and for words. Single-trial learning tasks were employed throughout, with memory performance assessed by forced-choice recognition, recall measures or choice reaction-time tasks. The main experimental findings were: (1) memory capacity, as a function of the amount of material presented, follows a general power law with a characteristic exponent for each task; (2) pictorial material obeys this power law and shows an overall superiority to verbal material. The capacity of recognition memory for pictures is almost limitless, when measured under appropriate conditions; (3) when the recognition task is made harder by using more alternatives, memory capacity stays constant and the superiority of pictures is maintained; (4) picture memory also exceeds verbal memory in terms of verbal recall; comparable recognition/recall ratios are obtained for pictures, words and nonsense syllables; (5) verbal memory shows a higher retrieval speed than picture memory, as inferred from reaction-time measures. Both types of material obey a power law, when reaction-time is measured for various sizes of learning set, and both show very rapid rates of memory search.
From a consideration of the experimental results and other data it is concluded that the superiority of the pictorial mode in recognition and free recall learning tasks is well established and cannot be attributed to methodological artifact.
“Studies on the Telegraphic Language: The Acquisition of a Hierarchy of Habits”, Bryan & Harter 1899
1899-william.pdf: “Studies on the telegraphic language: The acquisition of a hierarchy of habits”, (1899; ; ; similar):
Investigated the different stages involved in learning telegraphy. One S was tested each week on: (1) rate of receiving letters not making words, (2) rate of receiving letters making words, but not sentences, and (3) rate of receiving letters making words and sentences. Results indicate that a hierarchy of psycho-physical habits were required to receive the telegraphic language. From an early period, letter, word and higher habits made gains together, but not equally. No plateau appeared between the learning of letters and words; the first one occurred after the learning of words. Later, there was a second ascent, representing the acquisition of higher language habits. Effective speed was largely dependent upon the mastery of these habits, which led to greater accuracy in detail. Concluded that the rate of progress, depended partly on the rate of mental and nervous processes, but far more on how much was included in each process.
1897-bryan.pdf: “Studies in the physiology and psychology of the telegraphic language”, (1897; ; ; similar):
Studied individual differences in telegraphic writing. A preliminary study was conducted, in which operators were cross-examined on aspects of psychological or physiological importance. On the basis of this, a study was undertaken on 60 Ss, who were asked to write a sentence requiring attention. There were constant differences required in the times for a given character. Further tests were made, and schools were requested to provide typical curves of improvement. Results reveal that there were distinct specialties in telegraphy. The rate of receiving varied greatly, and exceeded sending rate. Both external and subjective disturbances affected inexperienced operators. The best age to learn telegraphy was 18–30 yrs. The variations in the value of a character depended on its place in the sentence. Homotaxic variation was an inverse measure of skill, while the inflection variation increased with expertise.