Technically, rewriting the site to use Hakyll-4 as a static site generator and converting the repository from Darcs to Git were major pieces of technical debt I needed to pay off. Painful, but it had to be done. I was also pleased to finish my annotated ebook of Radiance: I think it’s an unjustly obscure piece of literary scientific fiction which shines even more when the historical context is laid out.
My McCaleb email interview on MtGox helped correct Wikipedia about the history of MtGox. And not a moment too soon, given how toxic the topic is now for anyone previously associated with it.
Putting them together was a lot of work, but thus far it seems to be worth it: I am up to ~450 subscribers, most of whom click on at least one or two links, and I suspect most would not follow the RSS feed or my Google+ profile.
2014 was an eventful year. Bitcoin continued evolving towards respectability despite the MtGox disaster (which I think surprised even the more cynical observers), and the technical possibilities continue to be explored (I particularly like sidechains and Truthcoin). The darknet market scene saw dizzying turnover as markets continued to rise and fall in the post-SR1 vacuum (although in many cases, the fall was no less than merited) with 42 new markets and more closures, not to mention various prosecutions. Monitoring them all, much less scraping them, has proven to be quite a challenge, but I now have fairly complete archives which I have distributed to a number of academics; with luck, I can do a public release in 2015. This all drew some media attention as well; I did interviews in person or over email with Mike Power, the NHK, and Erica Fink.
Programming folklore notes that one way to get better lossless compression efficiency is to rearrange files inside the archive to group ‘similar’ files together and expose redundancy to the compressor, in accordance with information-theoretical principles. A particularly easy and broadly-applicable way of doing this, which does not require using any unusual formats or tools and is fully compatible with the default archive methods, is to sort the files by filename and especially file extension. I show how to do this with the standard Unix command-line sort tool, using the so-called “sort --key trick”, and give examples of the large space-savings possible from my archiving work for personal website mirrors and for making darknet market mirror datasets where the redundancy at the file level is particularly extreme and the sort --key trick shines compared to the naive approach.
It is widely understood that statistical correlation between two variables ≠ causation. Despite this admonition, people are overconfident in claiming correlations to support favored causal interpretations and are surprised by the results of randomized experiments, suggesting that they are biased & systematically underestimate the prevalence of confounds / common-causation. I speculate that in realistic causal networks or DAGs, the number of possible correlations grows faster than the number of possible causal relationships. So confounds really are that common, and since people do not think in realistic DAGs but toy models, the imbalance also explains overconfidence.
I ran a randomized experiment with a free program (Redshift) which reddens screens at night to avoid tampering with melatonin secretion & the sleep from 2012–2013, measuring sleep changes with my Zeo. With 533 days of data, the main result is that Redshift causes me to go to sleep half an hour earlier but otherwise does not improve sleep quality.
This is a compilation of my book reviews. Book reviews are sorted by star, and sorted by length of review within each star level, under the assumption that longer reviews are of more interest to readers.
Catch-22 is a satirical war novel by American author Joseph Heller. He began writing it in 1953; the novel was first published in 1961. Often cited as one of the most significant novels of the twentieth century, it uses a distinctive non-chronological third-person omniscient narration, describing events from the points of view of different characters. The separate storylines are out of sequence so the timeline develops along with the plot.
A Clockwork Orange is a dystopian satirical black comedy novel by English writer Anthony Burgess, published in 1962. It is set in a near-future society that has a youth subculture of extreme violence. The teenage protagonist, Alex, narrates his violent exploits and his experiences with state authorities intent on reforming him. The book is partially written in a Russian-influenced argot called "Nadsat", which takes its name from the Russian suffix that is equivalent to '-teen' in English. According to Burgess, it was a jeu d'esprit written in just three weeks.
Past, Present, and Future of Statistical Science was commissioned in 2013 by the Committee of Presidents of Statistical Societies (COPSS) to celebrate its 50th anniversary and the International Year of Statistics. COPSS consists of five charter member statistical societies in North America and is best known for sponsoring prestigious awards in statistics, such as the COPSS Presidents’ award. Through the contributions of a distinguished group of 50 statisticians who are past winners of at least one of the five awards sponsored by COPSS, this volume showcases the breadth and vibrancy of statistics, describes current challenges and new opportunities, highlights the exciting future of statistical science, and provides guidance to future generations of statisticians. The book is not only about statistics and science but also about people and their passion for discovery. Distinguished authors present expository articles on a broad spectrum of topics in statistical education, research, and applications. Topics covered include reminiscences and personal reflections on statistical careers, perspectives on the field and profession, thoughts on the discipline and the future of statistical science, and advice for young statisticians. Many of the articles are accessible not only to professional statisticians and graduate students but also to undergraduate students interested in pursuing statistics as a career and to all those who use statistics in solving real-world problems. A consistent theme of all the articles is the passion for statistics enthusiastically shared by the authors. Their success stories inspire, give a sense of statistics as a discipline, and provide a taste of the exhilaration of discovery, success, and professional accomplishment.
“This collection of reminiscences, musings on the state of the art, and advice for young statisticians makes for compelling reading. There are 52 contributions from eminent statisticians who have won a Committee of Presidents of Statistical Societies award. Each is a short, focused chapter and so one could even say this is ideal bedtime (or coffee break) reading. Anyone interested in the history of statistics will know that much has been written about the early days but little about the field since the Second World War. This book goes some way to redress this and is all the more valuable for coming from the horse’s mouth…the closing chapter, the shortest of all, from Brad Efron: a list of”thirteen rules for giving a really bad talk“. This made me laugh out loud and should be posted on the walls of all conferences. I shall leave the final word to Peter Bickel:”We should glory in this time when statistical thinking pervades almost every field of endeavor. It is really a lot of fun."
―Robert Grant, in Significance, April 2017
The History of COPSS: “A brief history of the Committee of Presidents of Statistical Societies (COPSS)”, Ingram Olkin
Reminiscences and Personal Reflections on Career Paths
“Reminiscences of the Columbia University Department of Mathematical Statistics in the late 1940s”, Ingram Olkin·“A career in statistics”, Herman Chernoff·“. . . how wonderful the field of statistics is . . .”, David R. Brillinger·“An unorthodox journey to statistics: Equity issues, remarks on multiplicity”, Juliet Popper Shaffer·“Statistics before and after my COPSS Prize”, Peter J. Bickel·“The accidental biostatistics professor”, Donna Brogan·“Developing a passion for statistics”, Bruce G. Lindsay·“Reflections on a statistical career and their implications”, R. Dennis Cook·“Science mixes it up with statistics”, Kathryn Roeder·“Lessons from a twisted career path”, Jeffrey S. Rosenthal·“Promoting equity”, Mary Gray
Perspectives on the Field and Profession
“Statistics in service to the nation”, Stephen E. Fienberg·“Where are the majors?”, Iain M. Johnstone·“We live in exciting times”, Peter Hall·“The bright future of applied statistics”, Rafael A. Irizarry·“The road travelled: From a statistician to a statistical scientist”, Nilanjan Chatterjee·“Reflections on a journey into statistical genetics and genomics”, Xihong Lin·“Reflections on women in statistics in Canada”, Mary E. Thompson·“The whole women thing”, Nancy Reid·“Reflections on diversity”, Louise Ryan
Reflections on the Discipline
“Why does statistics have two theories?”, Donald A.S. Fraser·“Conditioning is the issue”, James O. Berger·“Statistical inference from a Dempster-Shafer perspective”, Arthur P. Dempster·“Nonparametric Bayes”, David B. Dunson·“How do we choose our default methods?”, Andrew Gelman·“Serial correlation and Durbin-Watson bounds”, T.W. Anderson·“A non-asymptotic walk in probability and statistics”, Pascal Massart·“The past’s future is now: What will the present’s future bring?”, Lynne Billard·“Lessons in biostatistics”, Norman E. Breslow·“A vignette of discovery”, Nancy Flournoy·“Statistics and public health research”, Ross L. Prentice·“Statistics in a new era for finance and health care”, Tze Leung Lai·“Meta-analyses: Heterogeneity can be a good thing”, Nan M. Laird·“Good health: Statistical challenges in personalizing disease prevention”, Alice S. Whittemore·“Buried treasures”, Michael A. Newton·“Survey sampling: Past controversies, current orthodoxy, future paradigms”, Roderick J.A. Little·“Environmental informatics: Uncertainty quantification in the environmental sciences”, Noel A. Cressie·“A journey with statistical genetics”, Elizabeth Thompson·“Targeted learning: From MLE to TMLE”, Mark van der Laan·“Statistical model building, machine learning, and the ah-ha moment”, Grace Wahba·“In praise of sparsity and convexity”, Robert J. Tibshirani·“Features of Big Data and sparsest solution in high confidence set”, Jianqing Fan·“Rise of the machines”, Larry A. Wasserman·“A trio of inference problems that could win you a Nobel Prize in statistics (if you help fund it)”, Xiao-Li Meng
Advice for the Next Generation
“Inspiration, aspiration, ambition”, C.F. Jeff Wu·“Personal reflections on the COPSS Presidents’ Award”, Raymond J. Carroll·“Publishing without perishing and other career advice”, Marie Davidian·“Converting rejections into positive stimuli”, Donald B. Rubin·“The importance of mentors”, Donald B. Rubin·“Never ask for or give advice, make mistakes, accept mediocrity, enthuse”, Terry Speed·“Thirteen rules”, Bradley Efron
With due allowance for style and age, Hadamard ably describes and defends the basic model of ‘work, incubation, illumination, verification’, with reference to his own discoveries, his many famous acquaintances, Poincaré’s lecture, and a very interesting survey of mathematicians. In fact, it’s a little depressing that we don’t seem to have gone much beyond that in the half-century since this was published back in 1945 or so. While at least we no longer need his defense of the unconscious as a meaningful part of cognition, much of the rest is depressingly familiar—for example, his acute observations on mental imagery & people who solely think in words, and mention of Francis Galton’s survey (little-known outside of psychology), could be usefully read by many who commit the typical mind fallacy.
If Hadamard comes to no hard and fast conclusions, but merely raises many interesting points and criticizes a number of theories, we can hardly hold that against him, as we can do little better and so it becomes our failing to followup, not his.]
Aoi Honō is a Japanese coming-of-age manga series written and illustrated by Kazuhiko Shimamoto. It is a fictionalized account of his time as a student at the Osaka University of Arts, which he attended alongside Hideaki Anno, Hiroyuki Yamaga, and Takami Akai. It was adapted into a Japanese television drama that aired in July 2014 and ended in October 2014. The live action drama is legally streaming on Viki with English subtitles.
Kill la Kill is a 2013 Japanese anime television series produced by Trigger. The series follows vagrant schoolgirl Ryuko Matoi on her search for her father's killer, which brings her into violent conflict with Satsuki Kiryuin, the iron-willed student council president of Honnouji Academy, and her mother's fashion empire. In the show, characters obtain superpowers from specialized clothing due to the presence of a material known as Life Fibers.
Silver Spoon is a Japanese coming-of-age manga series written and illustrated by Hiromu Arakawa, serialized in Shogakukan's Weekly Shōnen Sunday from April 2011 to November 2019. The story is set in the fictional Ooezo Agricultural High School in Hokkaido, and depicts the daily life of Yuugo Hachiken, a high school student from Sapporo who enrolled at Ezo fleeing from the demands of his strict father. However, he soon learns that life on an agricultural school is not as easy as he initially believed. Unlike his new classmates, he has no intention of following an agricultural career after graduating, although he envies them for already having set goals for their lives and the pursuit of their dreams.
The Eccentric Family is a Japanese urban fantasy comedy-drama novel written by Tomihiko Morimi, originally published by Gentosha in 2007, with a sequel published in 2015. An anime television series adaptation based on the first book aired from July 7 to September 29, 2013 and was simulcast by Crunchyroll. A second season based on the second book aired from April 9 to June 25, 2017.
Kyousougiga is an original net animation created by Izumi Todo and produced by Toei Animation in collaboration with Banpresto. The animation was released on Nico Nico Douga on December 6, 2011, followed by a release on YouTube on December 10, 2011. Five additional episodes were streamed between August 31, 2012 and December 22, 2012. An anime television series aired between October and December 2013. The series is named after the Chōjū-giga scrolls.
Fusé: Teppō Musume no Torimonochō is a 2012 Japanese animated film directed by Masayuki Miyaji based on Kazuki Sakuraba's book Fusé Gansaku: Satomi Hakkenden. Both novel and film are an adaptation of Kyokutei Bakin's Nansō Satomi Hakkenden, focusing on a female hunter named Hamaji.
God Is an Astronaut are an Irish post-rock band from County Wicklow, formed in 2002 by Niels and Torsten Kinsella. Their style employs elements of electronic music, krautrock, and space rock, reminiscent of Tangerine Dream. The band currently consists of Niels and Torsten Kinsella and Lloyd Hanney. They have released 9 studio albums to date.
Dark Net Markets (DNM) are online markets typically hosted as Tor hidden services providing escrow services between buyers & sellers transacting in Bitcoin or other cryptocoins, usually for drugs or other illegal/regulated goods; the most famous DNM was Silk Road 1, which pioneered the business model in 2011.
From 2013–2015, I scraped/mirrored on a weekly or daily basis all existing English-language DNMs as part of my research into their usage, lifetimes/characteristics, & legal riskiness; these scrapes covered vendor pages, feedback, images, etc. In addition, I made or obtained copies of as many other datasets & documents related to the DNMs as I could.
This uniquely comprehensive collection is now publicly released as a 50GB (~1.6TB uncompressed) collection covering 89 DNMs & 37+ related forums, representing <4,438 mirrors, and is available for any research.
This page documents the download, contents, interpretation, and technical methods behind the scrapes.
Incubation is one of the four proposed stages of creativity, which are preparation, incubation, illumination, and verification. Incubation is defined as, when attending to a different task, humans forget about the previous unsuccessful attempts and can engage with the task anew, often leading to finding the solution. Incubation is related to intuition and insight in that it is the unconscious part of a process whereby an intuition may become validated as an insight. Incubation substantially increases the odds of solving a problem, and benefits from long incubation periods with low cognitive workloads.
[Alexander defines the “typical mind fallacy”: everyone reasons about their mental experiences as if they are universal. People with vivid visual imagery assume everyone can see things in “the mind’s eye” while ‘aphantasics’ assume that this is simply a poetic metaphor; people with color-blindness wonder why other people get so worked up about various shades of gray, and people with anosmia are puzzled by the focus on flowers etc. Further examples include maladaptive daydreaming, pain insensitivity, the prevalence of visual & auditory hallucinations in mentally-healthy individuals like ‘scintillating scotoma’, misophonia, hearing voices, inner monologues, facial self-awareness, trypophobia, Severely Deficient Autobiographical Memory, hypermnesia, ASMR, face blindness/prosospagnosia, musical anhedonia, ‘the call of the void’/intrusive thoughts, hypnagogia, the nasal dilation cycle…
This phenomenon for visual imagery was discovered only recently by Francis Galton, who asked if the interminable debate between philosophers/psychologists like Berkeley or Behaviorists like Skinner, where neither could accept that there was (or was not) visual imagery, was because both were right—some people have extremely vivid mental imagery, while others have none at all. He simply circulated a survey and asked. Turned out, most people do but some don’t.
The typical mind fallacy may explain many interpersonal conflicts and differences in advice: we underappreciate the sheer cognitive diversity of mankind, because we only have access to our limited personal anecdote, and people typically do not discuss all their differences because they don’t realize they exist nor have a vocabulary/name.]
The cypherpunk movement laid the ideological roots of Bitcoin and the online drug market Silk Road; balancing previous emphasis on cryptography, I emphasize the non-cryptographic market aspects of Silk Road which is rooted in cypherpunk economic reasoning, and give a fully detailed account of how a buyer might use market information to rationally buy, and finish by discussing strengths and weaknesses of Silk Road, and what future developments are predicted by cypherpunk ideas.
I compile a dataset of 87 public English-language darknet markets (DNMs) 2011–2016 in the vein of the famous Silk Road 1, recording their openings/closing and relevant characteristics. A survival analysis indicates the markets follow a Type TODO lifespan, with a median life of TODO months. Risk factors include TODO. With the best model, I generate estimates for the currently-operating markets.
I compile a table and discussion of all known arrests and prosecutions related to English-language Tor-Bitcoin darknet markets (DNMs) such as Silk Road 1, primarily 2011–2015, along with discussion of how they came to be arrested.
Newsletter tag: archive of all issues back to 2013 for the gwern.net newsletter (monthly updates, which will include summaries of projects I’ve worked on that month (the same as the changelog), collations of links or discussions from my subreddit, and book/movie reviews.)
Aphantasia is a condition characterized by an inability to voluntarily visualize mental imagery. Many people with aphantasia also report an inability to recall sounds, smells, or sensations of touch. Some also report prosopagnosia, the inability to recognize faces.
Anosmia, also known as smell blindness, is the loss of the ability to detect one or more smells. Anosmia may be temporary or permanent. It differs from hyposmia, which is a decreased sensitivity to some or all smells.
Sir Francis Galton, FRS, was an English Victorian era polymath: a statistician, sociologist, psychologist, anthropologist, eugenicist, tropical explorer, geographer, inventor, meteorologist, proto-geneticist, and psychometrician. He was knighted in 1909.
This page is a changelog for Gwern.net: a monthly reverse chronological list of recent major writings/changes/additions.
Following my writing can be a little difficult because it is often so incremental. So every month, in addition to my regular /r/Gwern subreddit submissions, I write up reasonably-interesting changes and send it out to the mailing list in addition to a compilation of links & reviews (archives).
A subreddit for posting links of interest and also for announcing updates to gwern.net (which can be used as a RSS feed). Submissions are categorized similar to the monthly newsletter and typically will be collated there.
Subscription page for the monthly gwern.net newsletter. There are monthly updates, which will include summaries of projects I’ve worked on that month (the same as the changelog), collations of links or discussions from my subreddit, and book/movie reviews. You can also browse the archives since December 2013.