2021-silver.pdf: “Wise teamwork: Collective confidence calibration predicts the effectiveness of group discussion”, (2021-09-01):
‘Crowd wisdom’ refers to the surprising accuracy that can be attained by averaging judgments from independent individuals. However, independence is unusual; people often discuss and collaborate in groups. When does group interaction improve vs. degrade judgment accuracy relative to averaging the group’s initial, independent answers?
Two large laboratory studies explored the effects of 969 face-to-face discussions on the judgment accuracy of 211 teams facing a range of numeric estimation problems from geographic distances to historical dates to stock prices. Although participants nearly always expected discussions to make their answers more accurate, the actual effects of group interaction on judgment accuracy were decidedly mixed. Importantly, a novel, group-level measure of collective confidence calibration robustly predicted when discussion helped or hurt accuracy relative to the group’s initial independent estimates. When groups were collectively calibrated prior to discussion, with more accurate members being more confident in their own judgment and less accurate members less confident, subsequent group interactions were likelier to yield increased accuracy.
We argue that collective calibration predicts improvement because groups typically listen to their most confident members. When confidence and knowledge are positively associated across group members, the group’s most knowledgeable members are more likely to influence the group’s answers.
[Keywords: crowd wisdom, group judgment, calibration, teamwork, confidence, advice-taking, estimation]
2018-morgan.pdf: “The Wisdom of Crowds Approach to Influenza-Rate Forecasting”, (2018-11-09):
Influenza is an important public health concern. Influenza leads to the death or hospitalization of thousands of people around the globe every year. However, the flu-season varies every year viz. when it starts, when it peaks, and the severity of the outbreak. Knowing the trajectory of the epidemic outbreak is important for taking appropriate mitigation strategies. Starting with the 2013–2014 flu season, the Influenza Division of the Centers for Disease Control and Prevention (CDC) has held a “Predict the Influenza Season Challenge” to encourage the scientific community to make advances in the field of influenza forecasting. A key observation from these challenges is that a simple average of the submitted forecasts outperformed nearly all of the individual models. Further, ongoing efforts seek ways to assign weights to individual models to create high-performing ensemble models. Given the sheer number of models, as well as variation in methodology followed among teams contributing influenza-risk forecasts, multiple forecasting models can be combined, by capturing human judgment, to outperform a simple average of these same models. This project exploits such a “wisdom of crowds” approach, using public votes acquired with the help of an R/
Shiny based web-application platform in order to assign weights to individual forecasting models on a week-over-week basis, in an effort to improve overall ILI risk prediction accuracy. We describe a strategy for improving the accuracy of influenza risk forecast modeling based on a crowd-sourced set of team-specific forecast votes and the results of the 2017–2018 season. Our approach to assigning weights based on crowd-sourced votes on individual models outperformed an average forecasts of the individual models. The crowd was statistically-significantly more accurate than the average model and all but one of the individual models.
[Keywords: Diseases, Modeling, Risk, Teams, Trajectories (Physics)]
2010-anderson.pdf: “Applying the Fermi Estimation Technique to Business Problems”, (2010-03-01):
It is often necessary to make quick estimates when neither time nor resources are available for making traditional assessments. This is particularly true at the idea stage of product development when even a gross estimate could be useful for heading off ill-advised expenditures.
The Fermi question, with which the scientific and engineering community has long been comfortable, is a helpful starting point for gaining insight into order of magnitude estimation. Although numerous worked-out solutions to Fermi questions are available, a systematic approach to solving them is not.
This led the authors to develop a methodology which could be easily implemented in a traditional business course. A more pragmatic reason for introducing Fermi questions to business students is that corporations now employ these questions in the job interview process, as a means of gauging applicants’ analytical skills.
On this basis alone, business students should be taught the methodology, as it may immediately have relevance to furthering their careers.
2010-denrell.pdf: “Predicting the Next Big Thing: Success as a Signal of Poor Judgment”, Jerker Denrell, Christina Fang
2001-armstrong-principlesforecasting.pdf: “Principles of Forecasting: A Handbook for Researchers and Practitioners”, (2001; ):
Forecasting is important in many aspects of our lives. As individuals, we try to predict success in our marriages, occupations, and investments. Organizations invest enormous amounts based on forecasts for new products, factories, retail outlets, and contracts with executives. Government agencies need forecasts of the economy, environmental impacts, new sports stadiums, and effects of proposed social programs.
The purpose of this book is to summarize knowledge of forecasting as a set of principles. These “principles” represent advice, guidelines, prescriptions, condition-action statements, and rules. We expect principles to be supported by empirical evidence. For this book, however, I asked authors to be ambitious in identifying principles for forecasting by including those based on expert judgment and even those that might be speculative. The authors describe the evidence so that you can judge how much confidence can be placed in the principles.
To summarize the findings, I invited 39 leading researchers to describe principles in their areas of expertise…Most of the book is devoted to descriptions of forecasting methods, discussions of the conditions under which they are most useful, and summaries of the evidence.