Link Bibliography

1. “February 2019 News”, Gw­ern Bran­wen (2019-01-20):

   February 2019 gwern.net newsletter with 2 essays/projects, site improvements; links on genetics, AI, propaganda, and typography; and 1 opera review.

2. https://gwern.substack.com

3. “January 2019 News”, Gw­ern Bran­wen (2018-12-18):

   January 2019 gwern.net newsletter with Danbooru2018 release announcement, links on genetic engineering, reinforcement learning, compression, and 3 book/movie reviews

4. “Gwern.net newsletter archives”, Gw­ern Bran­wen (2013-12-01):

   Newsletter tag: archive of all issues back to 2013 for the gwern.net newsletter (monthly updates, which will include summaries of projects I’ve worked on that month (the same as the changelog), collations of links or discussions from my subreddit, and book/movie reviews.)

5. “Changelog”, Gw­ern Bran­wen (2013-09-15):

   This page is a changelog for Gw­ern.net: a monthly re­verse chrono­log­i­cal list of re­cent ma­jor writings/changes/additions.

   Fol­low­ing my writ­ing can be a lit­tle diffi­cult be­cause it is often so in­cre­men­tal. So every mon­th, in ad­di­tion to my reg­u­lar /r/Gwern sub­red­dit sub­mis­sions, I write up rea­son­ably-in­ter­est­ing changes and send it out to the mail­ing list in ad­di­tion to a com­pi­la­tion of links & re­views (archives).

6. “/r/gwern subreddit”, Gw­ern Bran­wen (2018-10-01):

   A subreddit for posting links of interest and also for announcing updates to gwern.net (which can be used as a RSS feed). Submissions are categorized similar to the monthly newsletter and typically will be collated there.

7. https://www.patreon.com/gwern

8. “ThisWaifuDoesNotExist.net”, Gw­ern Bran­wen (2019-02-19):

   ThisWaifuDoesNotExist.net (TWDNE) is a static website which uses JS to display random anime faces generated by StyleGAN neural networks, along with GPT-3-generated anime plot summaries.

   

9. “This Waifu Does Not Exist”, Gw­ern Bran­wen (2019-02-19):

   

   Gen­er­at­ing high­-qual­ity anime faces has long been a task neural net­works strug­gled with. The in­ven­tion of StyleGAN in 2018 has effec­tively solved this task and I have trained a StyleGAN model which can gen­er­ate high­-qual­ity anime faces at 512px res­o­lu­tion. To show off the re­cent pro­gress, I made a web­site, “This Waifu Does Not Ex­ist” for dis­play­ing ran­dom StyleGAN 2 faces. TWDNE dis­plays a differ­ent neu­ral-net-gen­er­ated face & plot sum­mary every 15s. The site was pop­u­lar and went vi­ral on­line, es­pe­cially in Chi­na. The model can also be used in­ter­ac­tively for ex­plo­ration & edit­ing in the Art­breeder on­line ser­vice.

   TWDNE faces have been used as screen­savers, user avatars, char­ac­ter art for game packs or on­line games, up­loaded to Pix­iv, given away in streams, and used in a re­search pa­per (Noguchi & Harada 2019). TWDNE re­sults also helped in­spired Sizigi Stu­dio’s on­line in­ter­ac­tive waifu GAN, Waifu Labs, which gen­er­ates even bet­ter anime faces than my StyleGAN re­sults.

10. “Making Anime Faces With StyleGAN”, Gw­ern Bran­wen (2019-02-04):

    

    Gen­er­a­tive neural net­works, such as GANs, have strug­gled for years to gen­er­ate de­cen­t-qual­ity anime faces, de­spite their great suc­cess with pho­to­graphic im­agery such as real hu­man faces. The task has now been effec­tively solved, for anime faces as well as many other do­mains, by the de­vel­op­ment of a new gen­er­a­tive ad­ver­sar­ial net­work, StyleGAN, whose source code was re­leased in Feb­ru­ary 2019.

    I show off my StyleGAN 1/2 CC-0-li­censed anime faces & videos, pro­vide down­loads for the fi­nal mod­els & anime por­trait face dataset, pro­vide the ‘miss­ing man­ual’ & ex­plain how I trained them based on Danbooru2017/2018 with source code for the data pre­pro­cess­ing, doc­u­ment in­stal­la­tion & con­fig­u­ra­tion & train­ing tricks.

    For ap­pli­ca­tion, I doc­u­ment var­i­ous scripts for gen­er­at­ing im­ages & videos, briefly de­scribe the web­site “This Waifu Does Not Ex­ist” I set up as a pub­lic demo (see also Art­breeder), dis­cuss how the trained mod­els can be used for trans­fer learn­ing such as gen­er­at­ing high­-qual­ity faces of anime char­ac­ters with small datasets (eg Holo or Asuka Souryuu Lan­g­ley), and touch on more ad­vanced StyleGAN ap­pli­ca­tions like en­coders & con­trol­lable gen­er­a­tion.

    The ap­pen­dix gives sam­ples of my fail­ures with ear­lier GANs for anime face gen­er­a­tion, and I pro­vide sam­ples & model from a rel­a­tively large-s­cale BigGAN train­ing run sug­gest­ing that BigGAN may be the next step for­ward to gen­er­at­ing ful­l-s­cale anime im­ages.

    A minute of read­ing could save an hour of de­bug­ging!

11. /Littlewood#origin-of-littlewoods-law-of-miracles

12. https://edwardtufte.github.io/tufte-css/#epigraphs

13. “Source Sans Pro”, Eng­lish Wikipedia (2020-12-28):

    

    Source Sans Pro is a sans serif typeface created by Paul D. Hunt for Adobe. It is the first open-source font family from Adobe, distributed under the SIL Open Font License.

14. “Text figures”, Eng­lish Wikipedia (2020-12-28):

    

    Text figures are numerals designed with varying heights in a fashion that resembles a typical line of running text, hence the name. They are contrasted with lining figures, which are the same height as upper-case letters. Georgia is an example of a popular typeface that employs text figures by default.

15. “Typographic alignment”, Eng­lish Wikipedia (2020-12-22):

    In typesetting and page layout, alignment or range is the setting of text flow or image placement relative to a page, column (measure), table cell, or tab. The type alignment setting is sometimes referred to as text alignment, text justification, or type justification. The edge of a page or column is known as a margin, and a gap between columns is known as a gutter.

16. http://share.obormot.net/misc/gwern/image-focus.js

17. http://share.obormot.net/misc/gwern/image-focus.css

18. https://journals.sagepub.com/doi/pdf/10.1177/0956797617744542

19. “Genome-wide association study of antisocial personality disorder”, M-R. Rauti­ainen, T. Pau­nio, E. Re­po-Ti­iho­nen, M. Virkkunen, H. M. Ollila, S. Sulka­va, O. Jolanki, A. Palotie, J. Ti­iho­nen (2016-09-06):

    The pathophysiology of antisocial personality disorder (ASPD) remains unclear. Although the most consistent biological finding is reduced grey matter volume in the frontal cortex, about 50% of the total liability to developing ASPD has been attributed to genetic factors. The contributing genes remain largely unknown. Therefore, we sought to study the genetic background of ASPD. We conducted a genome-wide association study (GWAS) and a replication analysis of Finnish criminal offenders fulfilling DSM-IV criteria for ASPD (n = 370, n = 5850 for controls, GWAS; n = 173, n = 3766 for controls and replication sample). The GWAS resulted in suggestive associations of two clusters of single-nucleotide polymorphisms at 6p21.2 and at 6p21.32 at the human leukocyte antigen (HLA) region. Imputation of HLA alleles revealed an independent association with DRB1*01:01 (odds ratio (OR)=2.19 (1.53–3.14), p = 1.9 × 10^-5). Two polymorphisms at 6p21.2 LINC00951–LRFN2 gene region were replicated in a separate data set, and rs4714329 reached genome-wide significance (OR = 1.59 (1.37–1.85), p = 1.6 × 10⁻⁹) in the meta-analysis. The risk allele also associated with antisocial features in the general population conditioned for severe problems in childhood family (β = 0.68, p = 0.012). Functional analysis in brain tissue in open access GTEx and Braineac databases revealed eQTL associations of rs4714329 with LINC00951 and LRFN2 in cerebellum. In humans, LINC00951 and LRFN2 are both expressed in the brain, especially in the frontal cortex, which is intriguing considering the role of the frontal cortex in behavior and the neuroanatomical findings of reduced gray matter volume in ASPD. To our knowledge, this is the first study showing genome-wide significant and replicable findings on genetic variants associated with any personality disorder.

20. https://pdfs.semanticscholar.org/f116/27888a7e0b1cb7077dff91c9d97df1cad062.pdf

21. “Exploring the genetic correlations of antisocial behavior and life history traits”, Jorim J. Tiel­beek, J.C. Bar­nes, Arne Pop­ma, Tinca J.C. Pol­d­er­man, James J. Lee, John R.B. Per­ry, Danielle Posthu­ma, Brian B. Boutwell (2018-08-23):

    Prior evolutionary theory provided reason to suspect that measures of development and reproduction would be correlated with antisocial behaviors in human and non-human species. Behavioral genetics has revealed that most quantitative traits are heritable, suggesting that these phenotypic correlations may share genetic etiologies. We use GWAS data to estimate the genetic correlations between various measures of reproductive development (N = 52,776 – 318,863) and antisocial behavior (N = 31,968). Our genetic correlation analyses demonstrate that alleles associated with higher reproductive output (number of children ever born, r_g=0.50, p = .0065) were positively correlated with alleles associated with antisocial behavior, whereas alleles associated with more delayed reproductive onset (age of first birth, r_g=-.64, p = .0008) were negatively associated with alleles linked to antisocial behavior. Ultimately, these findings coalesce with evolutionary theories suggesting that increased antisocial behaviors may partly represent a faster life history approach, which may be significantly calibrated by genes.

22. /docs/genetics/heritable/2019-odintsova.pdf

23. /docs/genetics/heritable/2005-baud.pdf

24. /docs/genetics/heritable/2005-moffitt.pdf

25. /docs/genetics/heritable/2012-craig.pdf

26. “Human aggression across the lifespan: genetic propensities and environmental moderators.”, Tu­vblad, Cather­ine Bak­er, Laura A (2011):

    This chapter reviews the recent evidence of genetic and environmental influences on human aggression. Findings from a large selection of the twin and adoption studies that have investigated the genetic and environmental architecture of aggressive behavior are summarized. These studies together show that about half (50%) of the variance in aggressive behavior is explained by genetic influences in both males and females, with the remaining 50% of the variance being explained by environmental factors not shared by family members. Form of aggression (reactive, proactive, direct/physical, indirect/relational), method of assessment (laboratory observation, self-report, ratings by parents and teachers), and age of the subjects-all seem to be significant moderators of the magnitude of genetic and environmental influences on aggressive behavior. Neither study design (twin vs. sibling adoption design) nor sex (male vs. female) seems to impact the magnitude of the genetic and environmental influences on aggression. There is also some evidence of gene-environment interaction (G × E) from both twin/adoption studies and molecular genetic studies. Various measures of family adversity and social disadvantage have been found to moderate genetic influences on aggressive behavior. Findings from these G × E studies suggest that not all individuals will be affected to the same degree by experiences and exposures, and that genetic predispositions may have different effects depending on the environment.

27. /docs/genetics/heritable/2012-anholt.pdf

28. https://www.nature.com/articles/mp201331

29. https://jeb.biologists.org/content/218/1/123

30. “Unraveling the Genetic Etiology of Adult Antisocial Behavior: A Genome-Wide Association Study”, Jorim J. Tiel­beek, Sarah E. Med­land, Beben Benyam­in, Enda M. Byrne, An­drew C. Heath, Pamela A. F. Mad­den, Nicholas G. Mar­t­in, Naomi R. Wray, Karin J. H. Ver­weij (2012-08-17):

    Crime poses a major burden for society. The heterogeneous nature of criminal behavior makes it difficult to unravel its causes. Relatively little research has been conducted on the genetic influences of criminal behavior. The few twin and adoption studies that have been undertaken suggest that about half of the variance in antisocial behavior can be explained by genetic factors. In order to identify the specific common genetic variants underlying this behavior, we conduct the first genome-wide association study (GWAS) on adult antisocial behavior. Our sample comprised a community sample of 4816 individuals who had completed a self-report questionnaire. No genetic polymorphisms reached genome-wide significance for association with adult antisocial behavior. In addition, none of the traditional candidate genes can be confirmed in our study. While not genome-wide significant, the gene with the strongest association (p = 8.7×10⁻⁵) was DYRK1A, a gene previously related to abnormal brain development and mental retardation. Future studies should use larger, more homogeneous samples to disentangle the etiology of antisocial behavior. Biosocial criminological research allows a more empirically grounded understanding of criminal behavior, which could ultimately inform and improve current treatment strategies.

31. “A genome-wide association study of behavioral disinhibition.”, McGue, Matt Zhang, Yi­wei Miller, Michael B. Ba­su, Saonli Vrieze, Scott Hicks, Brian Mal­one, Steve Oet­ting, William S. Ia­cono, William G (2013):

    We report results from a genome wide association study (GWAS) of five quantitative indicators of behavioral disinhibition: nicotine, alcohol consumption, alcohol dependence, illicit drugs, and non-substance related behavioral disinhibition. The sample, consisting of 7,188 Caucasian individuals clustered in 2,300 nuclear families, was genotyped on over 520,000 SNP markers from Illumina's Human 660W-Quad Array. Analysis of individual SNP associations revealed only one marker-component phenotype association, between rs1868152 and illicit drugs, with a p value below the standard genome-wide threshold of 5 × 10(-8). Because we had analyzed five separate phenotypes, we do not consider this single association to be significant. However, we report 13 SNPs that were associated at p < 10(-5) for one phenotype and p < 10(-3) for at least two other phenotypes, which are potential candidates for future investigations of variants associated with general behavioral disinhibition. Biometric analysis of the twin and family data yielded estimates of additive heritability for the component phenotypes ranging from 49 to 70%, GCTA estimates of heritability for the same phenotypes ranged from 8 to 37%. Consequently, even though the common variants genotyped on the GWAS array appear in aggregate to account for a sizable proportion of heritable effects in multiple indicators of behavioral disinhibition, our data suggest that most of the additive heritability remains "missing".

32. “Genetic background of extreme violent behavior.”, Ti­iho­nen, J. Rauti­ainen, M-R Ollila, H. M Re­po-Ti­iho­nen, E. Virkkunen, M. Palotie, A. Pietiläi­nen, O. Kris­tiansson, K. Jouka­maa, M. Lauer­ma, H. Saare­la, J. Tyni, S. Var­ti­ainen, H. Paana­nen, J. Gold­man, D. Pau­nio, T (2015):

    In developed countries, the majority of all violent crime is committed by a small group of antisocial recidivistic offenders, but no genes have been shown to contribute to recidivistic violent offending or severe violent behavior, such as homicide. Our results, from two independent cohorts of Finnish prisoners, revealed that a monoamine oxidase A (MAOA) low-activity genotype (contributing to low dopamine turnover rate) as well as the CDH13 gene (coding for neuronal membrane adhesion protein) are associated with extremely violent behavior (at least 10 committed homicides, attempted homicides or batteries). No substantial signal was observed for either MAOA or CDH13 among non-violent offenders, indicating that findings were specific for violent offending, and not largely attributable to substance abuse or antisocial personality disorder. These results indicate both low monoamine metabolism and neuronal membrane dysfunction as plausible factors in the etiology of extreme criminal violent behavior, and imply that at least about 5-10% of all severe violent crime in Finland is attributable to the aforementioned MAOA and CDH13 genotypes.

33. “Genome-Wide Association Study of Proneness to Anger”, Eric Mick, James Mc­Gough, Cur­tis K. Deutsch, Jean A. Fra­zier, David Kennedy, Robert J. Gold­berg (2013-12-27):

    Background: Community samples suggest that approximately 1 in 20 children and adults exhibit clinically significant anger, hostility, and aggression. Individuals with dysregulated emotional control have a greater lifetime burden of psychiatric morbidity, severe impairment in role functioning, and premature mortality due to cardiovascular disease.

    Methods: With publicly available data secured from dbGaP, we conducted a genome-wide association study of proneness to anger using the Spielberger State-Trait Anger Scale in the Atherosclerosis Risk in Communities (ARIC) study (n = 8,747).

    Results: Subjects were, on average, 54 (range 45–64) years old at baseline enrollment, 47% (n = 4,117) were male, and all were of European descent by self-report. The mean Angry Temperament and Angry Reaction scores were 5.8±1.8 and 7.6±2.2. We observed a nominally significant finding (p = 2.9E-08, λ = 1.027—corrected p_gc = 2.2E-07, λ = 1.0015) on chromosome 6q21 in the gene coding for the non-receptor protein-tyrosine kinase, Fyn.

    Conclusions: Fyn interacts with NDMA receptors and inositol-1,4,5-trisphosphate (IP3)-gated channels to regulate calcium influx and intracellular release in the post-synaptic density. These results suggest that signaling pathways regulating intracellular calcium homeostasis, which are relevant to memory, learning, and neuronal survival, may in part underlie the expression of Angry Temperament.

34. https://www.nature.com/articles/tp201536

35. https://onlinelibrary.wiley.com/doi/pdf/10.1002/ajmg.b.32363

36. https://onlinelibrary.wiley.com/doi/pdf/10.1002/ajmg.b.32419

37. http://biology-web.nmsu.edu/~houde/genetics%20of%20aggression.pdf

38. https://onlinelibrary.wiley.com/doi/pdf/10.1002/ajmg.b.32388

39. /docs/genetics/heritable/2017-manchia.pdf

40. “On the Genetic and Genomic Basis of Aggression, Violence, and Antisocial Behavior”, Kevin M. Beaver, Eric J. Con­nol­ly, Joseph L. Ned­elec, Joseph A. Schwartz (2018-04-01):

    There is a great deal of interest in examining the genetic and environmental architecture to aggression, violence, and antisocial behaviors. This interest has resulted in hundreds of studies being published that estimate genetic and environmental effects on antisocial phenotypes. The results generated from these studies have been remarkably consistent and have contributed greatly to the knowledge base on the etiology of antisocial behavior. This chapter reviews the research on the genetic basis to antisocial phenotypes by presenting the results related to the heritability of antisocial phenotypes. It also discusses some of the molecular genetic association studies as well as genome-wide association studies that focus on the development of antisocial behaviors. In doing so, it also reviews findings related to gene-environment interactions. The chapter concludes by discussing some of the ways in which these findings could be used for intervention and prevention programs.

41. /docs/genetics/heritable/2018-tremblay.pdf

42. https://pdfs.semanticscholar.org/b15f/026aadf93744168fe9992cb86002a85647d2.pdf

43. /docs/genetics/heritable/2019-gard.pdf

44. https://pdfs.semanticscholar.org/2cb9/e5fe8009b934faa127778bf6aae028e30b62.pdf

45. https://duepublico.uni-duisburg-essen.de/servlets/DerivateServlet/Derivate-28060/Oades2008Does.pdf

46. https://duepublico.uni-duisburg-essen.de/servlets/DerivateServlet/Derivate-28837/Anney_ea_2008.pdf

47. /docs/genetics/heritable/2010-viding.pdf

48. “Genome-wide association study of conduct disorder symptomatology”, Dick, D. M Aliev, F. Krueger, R. F Ed­wards, A. Agrawal, A. Lynskey, M. Lin, P. Schuck­it, M. Hes­sel­brock, V. Nurn­berg­er, J. Al­masy, L. Por­jesz, B. Eden­berg, H. J Bu­cholz, K. Kramer, J. Ku­per­man, S. Bierut, L (2011):

    Conduct disorder (CD) is one of the most prevalent childhood psychiatric conditions, and is associated with a number of serious concomitant and future problems. CD symptomatology is known to have a considerable genetic component, with heritability estimates in the range of 50%. Despite this, there is a relative paucity of studies aimed at identifying genes involved in the susceptibility to CD. In this study, we report results from a genome-wide association study of CD symptoms. CD symptoms were retrospectively reported by a psychiatric interview among a sample of cases and controls, in which cases met the criteria for alcohol dependence. Our primary phenotype was the natural log transformation of the number of CD symptoms that were endorsed, with data available for 3963 individuals who were genotyped on the Illumina Human 1M beadchip array. Secondary analyses are presented for case versus control status, in which caseness was established as endorsing three or more CD symptoms (n = 872 with CD and N = 3091 without CD). We find four markers that meet the criteria for genome-wide significance (p < 5 × 10⁻⁸) with the CD symptom count, two of which are located in the gene C1QTNF7 (C1q and tumor necrosis factor-related protein 7). There were six additional SNPs in the gene that yielded converging evidence of association. These data provide the first evidence of a specific gene that is associated with CD symptomatology. None of the top signals resided in traditional candidate genes, underscoring the importance of a genome-wide approach for identifying novel variants involved in this serious childhood disorder.

49. https://www.nature.com/articles/tp201113

50. “Genome-wide association study of the child behavior checklist dysregulation profile”, Mick, Eric Mc­Gough, James Loo, San­dra Doyle, Alysa E. Woz­ni­ak, Janet Wilens, Tim­o­thy E. Smal­l­ey, Su­san Mc­Crack­en, James Bie­der­man, Joseph Faraone, Stephen V (2011):

    A potentially useful tool for understanding the distribution and determinants of emotional dysregulation in children is a Child Behavior Checklist profile, comprising the Attention Problems, Anxious/Depressed, and Aggressive Behavior clinical subscales (CBCL-DP). The CBCL-DP indexes a heritable trait that increases susceptibility for later psychopathology, including severe mood problems and aggressive behavior. We have conducted a genome-wide association study of the CBCL-DP in children with attention-deficit/hyperactivity disorder (ADHD). Families were ascertained at Massachusetts General Hospital and University of California, Los Angeles. Genotyping was conducted with the Illumina Human1M or Human1M-Duo BeadChip platforms. Genome-wide association analyses were conducted with the MQFAM multivariate extension of PLINK. CBCL data were available for 341 ADHD offspring from 339 ADHD affected trio families from the UCLA (n = 128) and the MGH (n = 213) sites. We found no genome-wide statistically significant associations but identified several plausible candidate genes among findings at p < 5E-05: TMEM132D, LRRC7, SEMA3A, ALK, and STIP1. We found suggestive evidence for developmentally expressed genes operant in hippocampal dependent memory and learning with the CBCL-DP.

51. https://onlinelibrary.wiley.com/doi/pdf/10.1002/ajmg.b.32333

52. https://moffittcaspi.trinity.duke.edu/sites/moffittcaspi.trinity.duke.edu/files/Agression_ViolentBehavior.pdf

53. “Predicting educational achievement from genomic measures and socioeconomic status”, So­phie von Stumm, Emily Smith-Wool­ley, Zi­ada Ay­orech, An­drew McMil­lan, Kaili Rim­feld, Philip S. Dale, Robert Plomin (2019-02-04):

    The two best predictors of children’s educational achievement available from birth are parents’ socioeconomic status (SES) and, recently, children’s inherited DNA differences that can be aggregated in genome-wide polygenic scores (GPS). Here we chart for the first time the developmental interplay between these two predictors of educational achievement at ages 7, 11, 14 and 16 in a sample of almost 5,000 UK school children. We show that the prediction of educational achievement from both GPS and SES increases steadily throughout the school years. Using latent growth curve models, we find that GPS and SES not only predict educational achievement in the first grade but they also account for systematic changes in achievement across the school years. At the end of compulsory education at age 16, GPS and SES respectively predict 14% and 23% of the variance of educational achievement; controlling for genetic influence on SES reduces its predictive power to 16%. Analyses of the extremes of GPS and SES highlight their influence and interplay: In children who have high GPS and come from high SES families, 77% go to university, whereas 21% of children with low GPS and from low SES backgrounds attend university. We find that the effects of GPS and SES are primarily additive, suggesting that their joint impact is particularly dramatic for children at the extreme ends of the distribution.

54. https://www.nature.com/articles/d41586-019-00395-4

55. https://quillette.com/2019/01/30/the-death-of-a-dreamer/

56. https://www.technologyreview.com/s/612838/the-transhumanist-diy-designer-baby-funded-with-bitcoin/

57. “Better Language Models and Their Implications: We've trained a large-scale unsupervised language model [GPT-2] which generates coherent paragraphs of text, achieves state-of-the-art performance on many language modeling benchmarks, and performs rudimentary reading comprehension, machine translation, question answering, and summarization—all without task-specific training”, Alec Rad­ford, Jeffrey Wu, Dario Amod­ei, Daniela Amod­ei, Jack Clark, Miles Brundage, Ilya Sutskever (Ope­nAI) (2019-02-14):

    Our model, called GPT-2 (a successor to GPT), was trained simply to predict the next word in 40GB of Internet text. Due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for researchers to experiment with, as well as a technical paper.

    GPT-2 is a large Transformer-based language model with 1.5 billion parameters, trained on a dataset of 8 million web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some text. The diversity of the dataset causes this simple goal to contain naturally occurring demonstrations of many tasks across diverse domains. GPT-2 is a direct scale-up of GPT, with more than 10× the parameters and trained on more than 10× the amount of data.

    GPT-2 displays a broad set of capabilities, including the ability to generate conditional synthetic text samples of unprecedented quality, where we prime the model with an input and have it generate a lengthy continuation. In addition, GPT-2 outperforms other language models trained on specific domains (like Wikipedia, news, or books) without needing to use these domain-specific training datasets. On language tasks like question answering, reading comprehension, summarization, and translation, GPT-2 begins to learn these tasks from the raw text, using no task-specific training data. While scores on these downstream tasks are far from state-of-the-art, they suggest that the tasks can benefit from unsupervised techniques, given sufficient (unlabeled) data and compute.

58. “GPT-1: Improving Language Understanding with Unsupervised Learning”, Ope­nAI (June 11, 2018):

    We’ve obtained state-of-the-art results on a suite of diverse language tasks with a scalable, task-agnostic system, which we’re also releasing. Our approach is a combination of two existing ideas: transformers and unsupervised pre-training. These results provide a convincing example that pairing supervised learning methods with unsupervised pre-training works very well; this is an idea that many have explored in the past, and we hope our result motivates further research into applying this idea on larger and more diverse datasets.

59. “Language Models are Unsupervised Multitask Learners”, Alec Rad­ford, Jeffrey Wu, Re­won Child, David Lu­an, Dario Amod­ei, Ilya Sutskever (2019-02-14):

    Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on task-specific datasets.

    We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText. When conditioned on a document plus questions, the answers generated by the language model reach 55 F1 on the CoQA dataset—matching or exceeding the performance of 3 out of 4 baseline systems without using the 127,000+ training examples.

    The capacity of the language model is essential to the success of zero-shot task transfer and increasing it improves performance in a log-linear fashion across tasks. Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting but still underfits WebText. Samples from the model reflect these improvements and contain coherent paragraphs of text.

    These findings suggest a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.

60. “Real-time Continuous Transcription with Live Transcribe”, Sagar Savla (Google) (2019-02-04):

    The World Health Organization (WHO) estimates that there are 466 million people globally that are deaf and hard of hearing. A crucial technology in empowering communication and inclusive access to the world’s information to this population is automatic speech recognition (ASR), which enables computers to detect audible languages and transcribe them into text for reading. Google’s ASR is behind automated captions in Youtube, presentations in Slides and also phone calls…Today, we’re announcing Live Transcribe, a free Android service that makes real-world conversations more accessible by bringing the power of automatic captioning into everyday, conversational use. Powered by Google Cloud, Live Transcribe captions conversations in real-time, supporting over 70 languages and more than 80% of the world’s population. You can launch it with a single tap from within any app, directly from the accessibility icon on the system tray.

    …Relying on cloud ASR provides us greater accuracy, but we wanted to reduce the network data consumption that Live Transcribe requires. To do this, we implemented an on-device neural network-based speech detector, built on our previous work with AudioSet. This network is an image-like model, similar to our published VGGish model, which detects speech and automatically manages network connections to the cloud ASR engine, minimizing data usage over long periods of use.

    …Known as the cocktail party problem, understanding a speaker in a noisy room is a major challenge for computers. To address this, we built an indicator that visualizes the volume of user speech relative to background noise. This also gives users instant feedback on how well the microphone is receiving the incoming speech from the speaker, allowing them to adjust the placement of the phone…Potential future improvements in mobile-based automatic speech transcription include on-device recognition, speaker-separation, and speech enhancement. Relying solely on transcription can have pitfalls that can lead to miscommunication. Our research with Gallaudet University shows that combining it with other auditory signals like speech detection and a loudness indicator, makes a tangibly meaningful change in communication options for our users.

61. https://www.blog.google/outreach-initiatives/accessibility/making-audio-more-accessible-two-new-apps/

62. https://github.com/NVlabs/stylegan

63. https://www.punchlinedesign.net/pun_generator

64. https://nips2018creativity.github.io/doc/entendrepreneur.pdf

65. https://slatestarcodex.com/2019/02/24/ot122-openne-thread/#comment-724333

66. https://news.ycombinator.com/item?id=19241236

67. https://old.reddit.com/r/gwern/comments/aulo8q/entendrepreneur_a_portmanteau_rhyme_generator/

68. https://llamasandmystegosaurus.blogspot.com/2017/05/alpha.html

69. /Faces#progan

70. “Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?”, Hugues Lor­tie-Forgues, Matthew In­glis (2019-03-11):

    There are a growing number of large-scale educational randomized controlled trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the UK-based Education Endowment Foundation and the U.S.-based National Center for Educational Evaluation and Regional Assistance, which evaluated interventions aimed at improving academic achievement in K–12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations. These sat within relatively large confidence intervals (mean width = 0.30 SDs), which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects. [Keywords: educational policy, evaluation, meta-analysis, program evaluation.]

71. “The Iron Law Of Evaluation And Other Metallic Rules”, Pe­ter H. Rossi (2012-09-18):

    Problems with social experiments and evaluating them, loopholes, causes, and suggestions; non-experimental methods systematically deliver false results, as most interventions fail or have small effects.

72. /docs/nootropics/2018-hanley.pdf

73. “The Real War 1939-1945”, Paul Fussell (1989-08 (The At­lantic)):

    [Hard-hitting longform piece on WWII about demystifying the ‘good war’ and bringing home the chaos, folly, incompetence, suffering, death, destruction visited on soldiers, and propaganda or silence which covered it all up.]

    On its fiftieth anniversary, how should we think of the Second World War? What is its contemporary meaning? One possible meaning, reflected in every line of what follows, is obscured by that oddly minimizing term “conventional war.” With our fears focused on nuclear destruction, we tend to be less mindful of just what conventional war between modern industrial powers is like. This article describes such war, in a stark, unromantic manner

74. “Avoid News: Towards a Healthy News Diet”, Rolf Do­belli (2010):

    This article is the antidote to news. It is long, and you probably won’t be able to skim it. Thanks to heavy news consumption, many people have lost the reading habit and struggle to absorb more than four pages straight. This article will show you how to get out of this trap—if you are not already too deeply in it.

75. /docs/sociology/2019-poulos.pdf

76. “Notes on Nggwal”, William Buck­ner (Tra­di­tions of Con­flict) (2019-02-23):

    During an undetermined time period preceding European contact, a gargantuan, humanoid spirit-God conquered parts of the Sepik region of Papua New Guinea. With a voracious appetite for pork and yams—and occasional demands of ritual murder—Nggwal was the tutelary spirit for a number of Sepik horticulturalist societies…what specific demands does Nggwal make? The first is for food. Nggwal must be fed, and while it is the men who are his most devoted servants and the keepers of his great secrets, it is often the responsibility of the women to provide for his subsistence, “Women are well aware of Nggwal’s hunger, for to them falls much of the gardening, hauling and cooking needed to feed him,” Tuzin writes. But how does Nggwal consume the food offered to him? “Needless to say, it is not the Tambaran [Nggwal himself] which eats the pork but the men themselves, in secret conclaves,” and Tuzin continues describing the "feasts among Tambaran Cult members in secret seclusion, during which non-members are under the impression that the food is being given directly to the spirits.

    …Despite the playful, Halloween-like aspects of this practice, the hangahiwa wandafunei [violent spirits] were a much more serious matter. 10% of the male masks portrayed hangahiwa wandafunei, and they were associated with the commission of ritually sanctioned murder. These murders committed by the violent spirits were always attributed to Nggwal.

    …Traditionally, hangahiwa wandafunei sought out victims who were alone in their garden or on the forest paths at dusk. Pigs, dogs and chickens were also fair game. After spearing the victim, the offending hangamu’w would escape back to its spirit house. The wearer would replace it with the other costumes and emerge without fear of detection—in time to join the general alarm aroused by the discovery of the body.

    Sometimes the wearer would not put the mask away, however, and instead he would take it to a nearby enemy village, where a relative or other acquaintance of his would take the mask and keep it in their own community’s spirit house, until it was time to be used and transferred once more. Through these ritual killings and the passage of costumes between communities, Nggwal impels cooperation between men of even hostile villages, and unites them in cult secrecy.

    Nggwal, who travels in structures of fiber and bone atop rivers of blood.

77. “Suppression of the Society of Jesus”, Eng­lish Wikipedia (2020-12-28):

    

    The suppression of the Jesuits was a politically instigated removal of all members of the Society of Jesus from most of the countries of Western Europe and their colonies, beginning in 1759, and ultimately approved by The Holy See in 1773. In 1814, Pope Pius VII restored the Society to its previous provinces and Jesuits began resuming their work in those countries.

78. “Taxes, Lawyers, and the Decline of Witch Trials in France”, Noel D. John­son, Mark Koyama (2014-02-01):

    How is rule of law established? We address this question by exploring the effect of increases in fiscal capacity on the establishment of well-enforced, formal, legal standards in a preindustrial economy. Between 1550 and 1700, there were over 2,000 witch trials in France. Prosecuting a witch required local judges to significantly deviate from formal rules of evidence. Hence, we exploit the significant variation across time and space in witch trials and fiscal capacity across French regions between 1550 and 1700 to show that increases in fiscal capacity were associated with increased adherence to the formal rule of law. As fiscal capacity increased, local judges increasingly upheld de jure rules, and the frequency of witch trials declined.

79. https://old.reddit.com/r/gwern/comments/c4uznu/abandoned_footnotes_the_good_parts/

80. “Aztec Political Thought ”, Xavier Mar­quez (2013-11-21):

    A footnote on Inga Clendinnen’s extraordinary Aztecs: An Interpretation. If there’s a better book on the Aztecs than this, I want to read it…Consider this passage Clendinnen quotes from the Florentine Codex (one of the main sources for pre-conquest Mexica thought and culture), coming after the speech with which the Mexica greeted a new tlatoani (ruler; literally, the “Great Speaker”) and exhorted him to good behaviour:

    Those early and anxious exhortations to benevolent behaviour were necessary, ‘for it was said when we replaced one, when we selected someone…he was already our lord, our executioner and our enemy.’ (p. 80; the quote is from Book 6, chapter 10, in Dibble and Anderson’s translation from the Nahuatl).

    It’s an arresting thought: “he was already our lord, our executioner, and our enemy.” (Clendinnen comments on the “desolate cadence” of these words). The ruler is not understood by the Mexica as normally benevolent though potentially dangerous; he is the enemy, and yet as the enemy he is indispensable. There is something profoundly alien in this thought, with its unsettling understanding of “legitimacy,” something I do not find anywhere in the classical Western tradition of political thought…But Aztec cosmology, it turns out, goes much further than this. The ruler embodies or channels Tezcatlipoca, who is often vaguely characterized as a god of “fate and war” (and normally downplayed in favor of Huizilopochtli, e.g., in the current Te Papa exhibit on the Aztecs here in Wellington, who is more understandable as a straightforward god of war, and is viewed as the “patron” of the Tenochtitlan Mexica). But Tezcatlipoca is the more important deity: he is described at the beginning of Book 6 of the Florentine Codex as “the principal god” of the Mexica. And he is not a merciful or benevolent god; on the contrary, he represents a kind of arbitrary malice that is visited on all alike, and is variously addressed as the Enemy on Both Sides, the Mocker, He Whose Slaves We Are, and the Lord of the Smoking Mirror (for the smoky reflections in dark obsidian mirrors used by the shamans, “obscure intimations of what was to come endlessly dissolving back into obscurity,” as Clendinnen puts it [p. 148])…Clendinnen notes many other examples of the “shared and steady vision common to the different social groupings in Tenochtitlan” concerning “the casual, inventive, tireless malice of the only sacred force concerned with the fates of men,” p. 148

    …When reading these passages, I cannot help but think: how could the Mexica be reconciled to their social and natural worlds with such an arbitrary, even malignant conception of divine and political authority? How is a ruler or a deity who is simultaneously seen as an enemy inspire support and commitment? As Clendinnen puts it, the puzzle is that “submission to a power which is caprice embodied is a taxing enterprise, yet it is that which the most devoted Mexica appear to have striven to achieve” (p. 76). Yet she hits on the right answer, I think, when she interprets these statements in the context of the rituals of Mexica society. In particular, she shows the Aztec state as an extraordinary example of what Clifford Geertz, referring to pre-colonial Bali, once called the “theatre state.”

    I mentioned earlier that human sacrifice was one of the central practices of Mexica society. But this does not quite capture what was going on. Human sacrifice was the most intense part of the pervasive ritual practices that structured Mexica society, but it was never merely sacrifice. Sacrifice was the culminating act of a set of amazing spectacles, enormously powerful intensifiers of emotion that made use of the entire register of Aztec symbols and pharmacopeia, and drew on the full resources of the empire.

81. https://abandonedfootnotes.blogspot.com/2011/04/qaddafis-chickens.html

82. https://abandonedfootnotes.blogspot.com/2011/12/flattery-inflation.html

83. https://abandonedfootnotes.blogspot.com/2012/10/ten-thousand-melodies-cannot-express.html

84. https://abandonedfootnotes.blogspot.com/2014/02/francisco-franco-robust-action-and.html

85. http://www.stats.ox.ac.uk/~snijders/PadgettAnsell1993.pdf

86. https://abandonedfootnotes.blogspot.com/2014/03/the-varieties-of-electoral-experience.html

87. https://abandonedfootnotes.blogspot.com/2015/02/electoral-parodies.html

88. https://abandonedfootnotes.blogspot.com/2014/07/the-good-tsar-bias.html

89. https://abandonedfootnotes.blogspot.com/2015/01/the-saudi-monarchy-as-family-firm.html

90. https://abandonedfootnotes.blogspot.com/2015/07/propaganda-as-signaling_76.html

91. /docs/history/1988-veyne.pdf

92. https://abandonedfootnotes.blogspot.com/2010/06/footnotes-on-things-ive-been-reading.html

93. https://abandonedfootnotes.blogspot.com/2010/08/most-surprising-sentence-i-read-today.html

94. https://abandonedfootnotes.blogspot.com/2011/01/weather-under-ceausescu.html

95. https://abandonedfootnotes.blogspot.com/2015/04/principal-agent-problems-in-soviet.html

96. https://abandonedfootnotes.blogspot.com/2012/09/the-great-norm-shift-and-triumph-of.html

97. https://abandonedfootnotes.blogspot.com/2013/04/the-sun-was-once-new.html

98. “Deep Intellect”, Sy Mont­gomery (O­rion Mag­a­zine) (2011-10-25):

    [Discussion of the remarkable abilities & intelligence of octopuses, despite being small, fragile, asocial beings. With hundreds of millions of neurons (most in its arms, which appear to be able to think and act independently, coordinating with the other arms/mouth, with their immensely-strong suckers), octopus are able to recognize individuals and bear grudges (squirting water at the foe), somehow imitate color despite being color-blind, use tools, solve puzzles, and manipulate rocks to create shelters, they are noted escape artists: one octopus was found breaking out of its aquarium at night to feast in other tanks, sneaking back before humans returned.]

99. http://learning.mpi-sws.org/memorize/

100. https://www.newyorker.com/magazine/2019/02/11/a-suspense-novelists-trail-of-deceptions

101. “Congregations of Tear Drinking Bees at Human Eyes: Foraging Strategies for an Invaluable Resource by Lisotrigona in Thailand (Apidae, Meliponini)”, Hans Bänziger (2018-05-04):

     Wild Lisotrigona cacciae (Nurse) and L. furva Engel were studied in their natural forest habitat at three sites in northern Thailand, May 2013–November 2014. The author, both experimenter and tear source, marked the minute bees while they drank from his eyes viewed in a mirror. All marked workers, 34 L. cacciae and 23 L. furva, came repeatedly to engorge, 34 and 27 times on average, respectively. The maximum number of times the same L. cacciae and L. furva came was 78 and 144 visits in one day, respectively; the maximum over two days was 145 visits by one L. cacciae; the maximum number of visiting days by the same bee was four over seven days by one L. furva which made 65 visits totally. The same forager may collect tears for more than 10h in a day, on average for 3h15min and 2h14min for L. cacciae and L. furva, respectively. Engorging from the inner eye corner averaged 3.1 and 2.2 min, respectively, but only 1.3 and 0.9 min when settled on the lower eye lid/ciliae. The interval between consecutive visits averaged 3.3 min and 3.8 min, respectively. Lachryphagy occurred during all months of the year, with 91–320 foragers a day during the hot season and 6–280 foragers during the rainy season; tear collecting resumed after a downpour. During the cold season eye visitation was reduced to 3–64 foragers, but none left her nest when the temperature was below 22°C. Flying ranges were greater than in comparable non-lachryphagous meliponines. It is proposed that Lisotrigona colonies have workers that are, besides nectar and pollen foragers, specialized tear collectors. Tears are 200 times richer in proteins than sweat, a secretion well-known to be imbibed by many meliponines. Digestion of proteins dissolved in tears is not hampered by an exine wall as in pollen, and they have bactericidal properties. These data corroborate the inference that Lisotrigona, which also visit other mammals, birds and reptiles, harvest lachrymation mainly for its content of proteins rather than only for salt and water.

102. “What Color are your bits?”, Matthew Skala (2004-06-10):

     [Philosophy piece attempting to explain, via an amusing analogy to classic RPG game Paranoia, to programmers how the rest of the world sees information: as tainted, in a dualist immaterial sense, by their history. Two bits are not identical even if they are identical, because they may have different histories; these are recorded and enforced by consensual society-wide hallucinations, such as intellectual property law. This may be insane, like in Paranoia, but that is how the human world works, and why many clever copyright hacks will fail.]

103. https://www.lowtechmagazine.com/2015/12/fruit-walls-urban-farming.html

104. http://code4k.blogspot.com/2010/12/crinkler-secrets-4k-intro-executable.html

105. “Alien”, Dave Addey (Type­set In The Fu­ture) (2014-12-01):

     [Discussion with screenshots of the classic Ridley Scott SF movie Alien, which makes extensive use of Helvetica, Futura, Eurostile Bold Extended, and other "modern" fonts to give a futuristic industrial feel to all of the (multilingual) spaceship/computer displays, controls, and credits; Alien also makes intriguing use of many logos, icons, and symbols for quick communication, tracing back to the Semiotic Standard.]

106. “Blade Runner”, Dave Addey (Type­set In The Fu­ture) (2016-06-19):

     [Discussion with screenshots of the classic Ridley Scott SF movie Blade Runner, which employs typography to disconcert the viewer, with unexpected choices, random capitalization and small caps, corporate branding/advertising, and the mashed-up creole multilingual landscape of noir cyberpunk LA (plus discussion of the buildings and sets, and details such as call costs being correctly inflation-adjusted).]

107. http://blog.archive.org/2011/08/17/scanning-a-braille-playboy/

108. /docs/economics/1957-nutter.pdf

109. http://neojaponisme.com/2011/11/28/the-great-shift-in-japanese-pop-culture-part-one/

110. http://neojaponisme.com/2011/11/29/the-great-shift-in-japanese-pop-culture-part-two/

111. http://neojaponisme.com/2011/11/30/the-great-shift-in-japanese-pop-culture-part-three/

112. http://neojaponisme.com/2011/12/01/the-great-shift-in-japanese-pop-culture-part-four/

113. http://neojaponisme.com/2011/12/02/the-great-shift-in-japanese-pop-culture-part-five/

114. https://www.vanityfair.com/style/2012/01/prisoners-of-style-201201

115. https://steve-yegge.blogspot.com/2012/03/borderlands-gun-collectors-club.html

116. https://www.theatlantic.com/magazine/archive/1973/11/the-force-that-drives-the-flower/308963/

117. http://www.infinityplus.co.uk/stories/colderwar.htm

118. https://slatestarcodex.com/2019/02/13/the-proverbial-murder-mystery/

119. https://slatestarcodex.com/2014/04/03/the-study-of-anglophysics/

120. http://www.slate.com/articles/news_and_politics/low_concept/2003/04/the_poetry_of_dh_rumsfeld.single.html

121. “Olbers' paradox”, Eng­lish Wikipedia (2020-12-28):

     

     In astrophysics and physical cosmology, Olbers' paradox, named after the German astronomer Heinrich Wilhelm Olbers (1758–1840), also known as the "dark night sky paradox", is the argument that the darkness of the night sky conflicts with the assumption of an infinite and eternal static universe. In the hypothetical case that the universe is static, homogeneous at a large scale, and populated by an infinite number of stars, any line of sight from Earth must end at the surface of a star and hence the night sky should be completely illuminated and very bright. This contradicts the observed darkness and non-uniformity of the night.

122. https://fivebooks.com/best-books/favourite-art-books-andrew-graham-dixon/

123. “Benjamin Haydon”, Eng­lish Wikipedia (2020-12-28):

     

     Benjamin Robert Haydon was a British painter who specialised in grand historical pictures, although he also painted a few contemporary subjects and portraits. His commercial success was damaged by his often tactless dealings with patrons, and by the enormous scale on which he preferred to work. He was troubled by financial problems throughout his life, which led to several periods of imprisonment for debt. He died by suicide in 1846.

124. “Gravity Research Foundation”, Eng­lish Wikipedia (2020-12-28):

     

     The Gravity Research Foundation is an organization established in 1948 by businessman Roger Babson to find ways to implement gravitational shielding. Over time, the foundation turned away from trying to block gravity and began trying to understand it. It holds an annual contest rewarding essays by scientific researchers on gravity-related topics. The contest, which awards prizes of up to $4,000, has been won by at least six people who later won the Nobel Prize in physics.

125. https://practicaltypography.com/

126. “Matthew Butterick”, Eng­lish Wikipedia (2020-12-28):

     

     Matthew Coffin Butterick is an American typographer, lawyer, writer, and computer programmer. He received the 2012 Golden Pen Award from the Legal Writing Institute for his book Typography for Lawyers, which started as a website in 2008 based on his experience as a practicing attorney. He has worked for The Font Bureau and founded his own website design company, Atomic Vision. Expanding Typography for Lawyers, Butterick published Practical Typography as a "web-based book" in July 2013.

127. “Carmen”, Eng­lish Wikipedia (2020-12-28):

     

     Carmen is an opera in four acts by French composer Georges Bizet. The libretto was written by Henri Meilhac and Ludovic Halévy, based on a novella of the same title by Prosper Mérimée. The opera was first performed by the Opéra-Comique in Paris on 3 March 1875, where its breaking of conventions shocked and scandalized its first audiences.

128. “Review of Carmen (opera)”, Gw­ern Bran­wen (2019-03-02):

     First opera review. How does a Met HD live opera broadcast work? Extremely well! They are lavishly produced, with multiple cameras and live directing, subtitling, and bonus features like seeing backstage. The opera itself is wonderful: it makes a great impression to sing and play orchestral music the entire time through, and the character motivations are painfully realistic and believable. I understand now the description of opera as a totalizing Gesamtkunstwerk: it combines all the art forms in an extremely technically-demanding to overpower the viewer and their emotions, overriding the lack of realism to produce an exaggerated effect, rendering it the most prestigious art of its era. The opera was interestingly "red pill"-like in depicting Carmen as demanding ever more sacrifices and ultimately abandoning her lover, leading to the final murder-suicide. Watching Carmen made me a believer in the Met HD broadcasts, and I resolved to watch more.

129. “Gwern.net newsletter (Substack subscription page)”, Gw­ern Bran­wen (2013-12-01):

     Subscription page for the monthly gwern.net newsletter. There are monthly updates, which will include summaries of projects I’ve worked on that month (the same as the changelog), collations of links or discussions from my subreddit, and book/movie reviews. You can also browse the archives since December 2013.

130. https://old.reddit.com/r/gwern/.rss

131. “GPT-3 Creative Fiction”, Gw­ern Bran­wen (2020-06-19):

     

     I con­tinue my AI po­etry gen­er­a­tion ex­per­i­ments with Ope­nAI’s 2020 GPT-3, which is 116× larg­er, and much more pow­er­ful, than the 2019 GPT-2. GPT-3, how­ev­er, is not merely a quan­ti­ta­tive tweak yield­ing “GPT-2 but bet­ter”—it is qual­i­ta­tively differ­ent, ex­hibit­ing eerie run­time learn­ing ca­pa­bil­i­ties al­low­ing even the raw mod­el, with zero fine­tun­ing, to “meta-learn” many tex­tual tasks purely by ex­am­ple or in­struc­tion. One does not train or pro­gram GPT-3 in a nor­mal way, but one en­gages in di­a­logue and writes prompts to teach GPT-3 what one wants.

     Ex­per­i­ment­ing through the Ope­nAI Beta API in June 2020, I find that GPT-3 does not just match my fine­tuned GPT-2-1.5b-poetry for po­em-writ­ing qual­i­ty, but ex­ceeds it, while be­ing ver­sa­tile in han­dling po­etry, Tom Swifty puns, sci­ence fic­tion, di­a­logue like Tur­ing’s Tur­ing-test di­a­logue, lit­er­ary style par­o­dies… As the pièce de ré­sis­tance, I recre­ate Stanis­law Lem’s Cy­be­riad’s “Trurl’s Elec­tronic Bard” po­etry us­ing GPT-3. (A­long the way, I doc­u­ment in­stances of how the BPE text en­cod­ing un­nec­es­sar­ily dam­ages GPT-3’s per­for­mance on a va­ri­ety of tasks, how to best elicit the high­est-qual­ity re­spons­es, com­mon er­rors peo­ple make in us­ing GPT-3, and test out GPT-3’s im­prove­ments in NN weak points like logic or com­mon­sense knowl­edge.)

     GPT-3’s sam­ples are not just close to hu­man lev­el: they are cre­ative, wit­ty, deep, meta, and often beau­ti­ful. They demon­strate an abil­ity to han­dle ab­strac­tions, like style par­o­dies, I have not seen in GPT-2 at all. Chat­ting with GPT-3 feels un­can­nily like chat­ting with a hu­man. I was im­pressed by the re­sults re­ported in the GPT-3 pa­per, and after spend­ing a week try­ing it out, I re­main im­pressed.

     This page records GPT-3 sam­ples I gen­er­ated in my ex­plo­rations, and thoughts on how to use GPT-3 and its re­main­ing weak­nesses. I hope you en­joy them even a tenth as much as I en­joyed test­ing GPT-3 and watch­ing the com­ple­tions scroll across my screen.

132. “A Style-Based Generator Architecture for Generative Adversarial Networks”, Tero Kar­ras, Samuli Laine, Timo Aila (2018-12-12):

     We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.

133. “Language Models are Few-Shot Learners”, Tom B. Brown, Ben­jamin Mann, Nick Ry­der, Melanie Sub­bi­ah, Jared Ka­plan, Pra­fulla Dhari­wal, Arvind Nee­lakan­tan, Pranav Shyam, Girish Sas­try, Amanda Askell, Sand­hini Agar­wal, Ariel Her­bert-Voss, Gretchen Krueger, Tom Henighan, Re­won Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Win­ter, Christo­pher Hes­se, Mark Chen, Eric Sigler, Ma­teusz Litwin, Scott Gray, Ben­jamin Chess, Jack Clark, Christo­pher Bern­er, Sam Mc­Can­dlish, Alec Rad­ford, Ilya Sutskev­er, Dario Amodei (2020-05-28):

     Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions—something which current NLP systems still largely struggle to do.

     Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10× more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora.

     Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

134. “Artbreeder”, Joel Si­mon (2019-09-09):

     [Artbreeder is an interactive GAN generator website. Originally named “Ganbreeder” and providing only the 256px BigGAN generator, it now provides a variety of BigGAN & StyleGAN models, including the anime portrait StyleGAN model. (It is more general than the similar Waifu Labs, but my anime model is not as good.) Users can generate random samples and explore slight variants of them to gradually explore the “latent space” and find interesting images, but they can also edit images more directly, upload existing images to find the most similar image produced by the model, etc. A popular website, it has generated >56m images from September 2019 to January 2020.]

135. https://klimaleksus.github.io/FindTwin/

136. https://github.com/darabos/high-five-trading

137. https://www.youtube.com/watch?v=FmTrF1dT12I&t=429

138. “Image Generation From Small Datasets via Batch Statistics Adaptation”, At­suhiro Noguchi, Tat­suya Harada (2019-04-03):

     Thanks to the recent development of deep generative models, it is becoming possible to generate high-quality images with both fidelity and diversity. However, the training of such generative models requires a large dataset. To reduce the amount of data required, we propose a new method for transferring prior knowledge of the pre-trained generator, which is trained with a large dataset, to a small dataset in a different domain. Using such prior knowledge, the model can generate images leveraging some common sense that cannot be acquired from a small dataset. In this work, we propose a novel method focusing on the parameters for batch statistics, scale and shift, of the hidden layers in the generator. By training only these parameters in a supervised manner, we achieved stable training of the generator, and our method can generate higher quality images compared to previous methods without collapsing, even when the dataset is small ( 100). Our results show that the diversity of the filters acquired in the pre-trained generator is important for the performance on the target domain. Our method makes it possible to add a new class or domain to a pre-trained generator without disturbing the performance on the original domain.

139. “Waifu Labs”, Sizigi Stu­dios (2019-07-23):

     [Waifu Labs is an interactive website for generating (1024px?) anime faces using a customized StyleGAN trained on Danbooru2018. Similar to Artbreeder, it supports face exploration and face editing, and at the end, a user can purchase prints of a particular face.]

     We taught a world-class artificial intelligence how to draw anime. All the drawings you see were made by a non-human artist! Wild, right? It turns out machines love waifus almost as much as humans do. We proudly present the next chapter of human history: lit waifu commissions from the world's smartest AI artist. In less than 5 minutes, the artist learns your preferences to make the perfect waifu just for you.

140. https://www.gwern.net/Faces#why-dont-gans-work

141. “A Style-Based Generator Architecture for Generative Adversarial Networks”, Tero Kar­ras, Samuli Laine, Timo Aila (2018-12-12):

     We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.

142. https://www.gwern.net/Faces#examples

143. “Danbooru2019 Portraits”, Gw­ern Bran­wen (2019):

     Danbooru2019 Portraits is a dataset of n = 302,652 (16GB) 512px anime faces cropped from ‘solo’ SFW Danbooru2019 images in a relatively broad ‘portrait’ style encompassing necklines/ears/hats/etc rather than tightly focused on the face, upscaled to 512px as necessary, and low-quality images deleted by manual review using Discriminator ranking. This dataset has been used for creating TWDNE.

144. “Danbooru2019: A Large-Scale Crowdsourced and Tagged Anime Illustration Dataset”, Gw­ern Bran­wen (2015-12-15):

     

     Deep learn­ing for com­puter re­vi­sion re­lies on large an­no­tated datasets. Classification/categorization has ben­e­fited from the cre­ation of Im­a­geNet, which clas­si­fies 1m pho­tos into 1000 cat­e­gories. But classification/categorization is a coarse de­scrip­tion of an im­age which lim­its ap­pli­ca­tion of clas­si­fiers, and there is no com­pa­ra­bly large dataset of im­ages with many tags or la­bels which would al­low learn­ing and de­tect­ing much richer in­for­ma­tion about im­ages. Such a dataset would ide­ally be >1m im­ages with at least 10 de­scrip­tive tags each which can be pub­licly dis­trib­uted to all in­ter­ested re­searchers, hob­by­ists, and or­ga­ni­za­tions. There are cur­rently no such pub­lic datasets, as Im­a­geNet, Birds, Flow­ers, and MS COCO fall short ei­ther on im­age or tag count or re­stricted dis­tri­b­u­tion. I sug­gest that the “im­age -boorus” be used. The im­age boorus are long­stand­ing web data­bases which host large num­bers of im­ages which can be ‘tagged’ or la­beled with an ar­bi­trary num­ber of tex­tual de­scrip­tions; they were de­vel­oped for and are most pop­u­lar among fans of ani­me, who pro­vide de­tailed an­no­ta­tions.

     The best known booru, with a fo­cus on qual­i­ty, is Dan­booru. We pro­vide a torrent/rsync mir­ror which con­tains ~3tb of 3.69m im­ages with 108m tag in­stances (of 392k de­fined tags, ~29/image) cov­er­ing Dan­booru from 2005-05-24–2019-12-31 (fi­nal ID: #3,734,659), pro­vid­ing the im­age files & a JSON ex­port of the meta­da­ta. We also pro­vide a smaller tor­rent of SFW im­ages down­scaled to 512×512px JPGs (295GB; 2,828,400 im­ages) for con­ve­nience.

     Our hope is that a Dan­booru2019 dataset can be used for rich large-s­cale classification/tagging & learned em­bed­dings, test out the trans­fer­abil­ity of ex­ist­ing com­puter vi­sion tech­niques (pri­mar­ily de­vel­oped us­ing pho­tographs) to illustration/anime-style im­ages, pro­vide an archival backup for the Dan­booru com­mu­ni­ty, feed back meta­data im­prove­ments & cor­rec­tions, and serve as a test­bed for ad­vanced tech­niques such as con­di­tional im­age gen­er­a­tion or style trans­fer.

145. https://www.gwern.net/Faces#data-preparation

146. https://www.gwern.net/Faces#installation

147. https://www.gwern.net/Faces#configuration

148. https://www.gwern.net/Faces#running

149. https://www.gwern.net/Faces#sampling

150. https://www.gwern.net/Faces#twdne

151. https://artbreeder.com

152. https://www.gwern.net/Faces#transfer-learning

153. https://www.gwern.net/Faces#holo

154. https://www.gwern.net/Faces#asuka

155. “Making Anime Faces With StyleGAN: Reversing StyleGAN To Control & Modify Images”, Gw­ern Bran­wen (2019-03-24):

     Discussion of how to modify existing images with GANs. There are several possibilities: train another NN to turn an image back into the original encoding; run blackbox search on encodings, repeatedly tweaking it to approximate a target face; or the whitebox approach, directly backpropagating through the model from the image to the encoding while holding the model fixed. All of these have been implemented for StyleGAN, and a combination works best. There are even GUIs for editing StyleGAN anime faces!

156. https://www.gwern.net/Faces#appendix

157. “Generating Anime Faces with BigGAN”, Gw­ern Bran­wen (2019-06-04):

     I explore BigGAN, another recent GAN with SOTA results on the most complex image domain tackled by GANs so far, ImageNet. BigGAN’s capabilities come at a steep compute cost, however. I experiment with 128px ImageNet transfer learning (successful) with ~6 GPU-days, and from-scratch 256px anime portraits of 1000 characters on a 8×2080ti machine for a month (mixed results). My BigGAN results are good but compromised by practical problems with the released BigGAN code base. While BigGAN is not yet superior to StyleGAN for many purposes, BigGAN-like approaches may turn out to be necessary to scale to whole anime images.

158. https://github.com/openai/gpt-2

159. “Attention Is All You Need”, Ashish Vaswani, Noam Shazeer, Niki Par­mar, Jakob Uszko­r­e­it, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Il­lia Polo­sukhin (2017-06-12):

     The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

160. “Semi-supervised Sequence Learning”, An­drew M. Dai, Quoc V. Le (2015-11-04):

     We present two approaches that use unlabeled data to improve sequence learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a conventional language model in natural language processing. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a "pretraining" step for a later supervised sequence learning algorithm. In other words, the parameters obtained from the unsupervised step can be used as a starting point for other supervised training models. In our experiments, we find that long short term memory recurrent networks after being pretrained with the two approaches are more stable and generalize better. With pretraining, we are able to train long short term memory recurrent networks up to a few hundred timesteps, thereby achieving strong performance in many text classification tasks, such as IMDB, DBpedia and 20 Newsgroups.

161. /GPT-2#gpt-2-1.5b

162. /GPT-3#poetry

163. /GPT-3#tom-swifties

164. /GPT-3#turing-dialogue

165. /GPT-3#literary-parodies

166. /GPT-3#stanislaw-lems-cyberiad

167. “GPT-3 Weaknesses: Byte-Pair Encodings (BPEs)”, Gw­ern Bran­wen (2020-06-23):

     Compared to GPT-2, GPT-3 improves performance on character-level tasks like rhyming, alliteration, punning, anagrams or permutations, acrostic poems, and arithmetic less than expected, despite being very good at many other closely-related kinds of writings like satire.

     Why? A plausible explanation is an obscure technical detail: as a performance optimization, GPT does not see characters but sub-word-chunks called “byte-pair encodings” (BPEs). Because GPTs never see characters but opaque partial-words, which vary chaotically based on the specific word and even the surrounding context, they are unable to easily learn about character-level aspects of language, like similar spellings or sounds, and are forced to learn relationships much more indirectly, like by brute-force memorizing of pairs of words.

     Some experiments with reformatting GPT-3’s poorest-performing tasks to avoid inconsistent BPE encodings of strings shows small to large performance gains, consistent with this theory.

168. “GPT-3: Prompts As Programming”, Gw­ern Bran­wen (2020-06-23):

     The GPT-3 neural network is so large a model in terms of power and dataset that it exhibits qualitatively different behavior: you do not apply it to a fixed set of tasks which were in the training dataset, requiring retraining on additional data if one wants to handle a new task (as one would have to retrain GPT-2); instead, you interact with it, expressing any task in terms of natural language descriptions, requests, and examples, tweaking the prompt until it “understands” & it meta-learns the new task based on the high-level abstractions it learned from the pretraining.

     This is a rather different way of using a DL model, and it’s better to think of it as a new kind of programming, where the prompt is now a “program” which programs GPT-3 to do new things.

169. /GPT-3#weaknesses

170. “Large Scale GAN Training for High Fidelity Natural Image Synthesis”, An­drew Brock, Jeff Don­ahue, Karen Si­monyan (2018-09-28):

     Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick," allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator’s input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128×128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.5 and Frechet Inception Distance (FID) of 7.4, improving over the previous best IS of 52.52 and FID of 18.6.

171. “Generating Anime Faces with StyleGAN: Using a trained Discriminator to Rank and Clean Data”, Gw­ern Bran­wen (2019-04-22):

     The Discriminator of a GAN is trained to detect outliers or bad datapoints. So it can be used for cleaning the original dataset of aberrant samples. This works reasonably well and I obtained BigGAN/StyleGAN quality improvements by manually deleting the worst samples (typically badly-cropped or low-quality faces), but has peculiar behavior which indicates that the Discriminator is not learning anything equivalent to a “quality” score but may be doing some form of memorization of specific real datapoints. What does this mean for how GANs work?

172. https://danbooru.donmai.us/

173. “GPT-2 Neural Network Poetry”, Gw­ern Bran­wen (2019-03-03):

     In Feb­ru­ary 2019, fol­low­ing up on my 2015–2016 tex­t-gen­er­a­tion ex­per­i­ments with char-RNNs, I ex­per­i­ment with the cut­ting-edge Trans­former NN ar­chi­tec­ture for lan­guage mod­el­ing & text gen­er­a­tion. Us­ing Ope­nAI’s GPT-2-117M (117M) model pre-trained on a large In­ter­net cor­pus and nshep­perd’s fine­tun­ing code, I re­train GPT-2-117M on a large (117MB) Project Guten­berg po­etry cor­pus. I demon­strate how to train 2 vari­ants: “GPT-2-poetry”, trained on the po­ems as a con­tin­u­ous stream of text, and “GPT-2-poetry-prefix”, with each line pre­fixed with the meta­data of the PG book it came from. In May 2019, I trained the nex­t-largest GPT-2, GPT-2-345M, sim­i­lar­ly, for a fur­ther qual­ity boost in gen­er­ated po­ems. In Oc­to­ber 2019, I re­trained GPT-2-117M on a Project Guten­berg cor­pus with im­proved for­mat­ting, and com­bined it with a con­tem­po­rary poem dataset based on Po­etry Foun­da­tion’s web­site. .> With just a few GPU-days on 1080ti GPUs, GPT-2-117M fine­tun­ing can pro­duce high­-qual­ity po­etry which is more the­mat­i­cally con­sis­tent than my char-RNN po­ems, ca­pa­ble of mod­el­ing sub­tle fea­tures like rhyming, and some­times even a plea­sure to read. I list the many pos­si­ble ways to im­prove poem gen­er­a­tion and fur­ther ap­proach hu­man-level po­ems. For the high­est-qual­ity AI po­etry to date, see my fol­lowup page, “GPT-3 Cre­ative Writ­ing”.

     For anime plot sum­maries, see TWDNE; for gen­er­at­ing ABC-formatted folk mu­sic, see “GPT-2 Folk Mu­sic” & “GPT-2 Pref­er­ence Learn­ing for Mu­sic and Po­etry Gen­er­a­tion”; for play­ing chess, see “A Very Un­likely Chess Game”; for the Red­dit com­ment gen­er­a­tor, see SubSimulatorGPT-2; for fan­fic­tion, the Ao3; and for video games, the walk­through model. For Ope­nAI’s GPT-3 fol­lowup, see “GPT-3: Lan­guage Mod­els are Few-Shot Learn­ers”.

174. “RNN metadata for mimicking individual author style”, Gw­ern Bran­wen (2015-09-12):

     Char-RNNs are un­su­per­vised gen­er­a­tive mod­els which learn to mimic text se­quences. I sug­gest ex­tend­ing char-RNNs with in­line meta­data such as genre or au­thor pre­fixed to each line of in­put, al­low­ing for bet­ter & more effi­cient meta­data, and more con­trol­lable sam­pling of gen­er­ated out­put by feed­ing in de­sired meta­da­ta. A 2015 ex­per­i­ment us­ing torch-rnn on a set of ~30 Project Guten­berg e-books (1 per au­thor) to train a large char-RNN shows that a char-RNN can learn to re­mem­ber meta­data such as au­thors, learn as­so­ci­ated prose styles, and often gen­er­ate text vis­i­bly sim­i­lar to that of a spec­i­fied au­thor.

     I fur­ther try & fail to train a char-RNN on Geoc­i­ties HTML for un­clear rea­sons.

     More suc­cess­ful­ly, I ex­per­i­ment in 2019 with a re­cent­ly-de­vel­oped al­ter­na­tive to char-RNNs, the Trans­former NN ar­chi­tec­ture, by fine­tun­ing train­ing Ope­nAI’s GPT-2-117M Trans­former model on a much larger (117MB) Project Guten­berg po­etry cor­pus us­ing both un­la­beled lines & lines with in­line meta­data (the source book). The gen­er­ated po­etry is much bet­ter. And GPT-3 is bet­ter still.

175. “Poetry Foundation”, Eng­lish Wikipedia (2020-12-27):

     

     The Poetry Foundation is a Chicago-based American foundation created to promote poetry in the wider culture. It was formed from Poetry magazine, which it continues to publish, with a 2003 gift of $200 million from philanthropist Ruth Lilly.

176. https://www.poetryfoundation.org/poems

177. /TWDE#text

178. “GPT-2 Folk Music”, Gw­ern Bran­wen (2019-11-01):

     

     In No­vem­ber 2019, I ex­per­i­mented with train­ing a GPT-2 neural net model to gen­er­ate folk mu­sic in the high­-level ABC mu­sic text for­mat, fol­low­ing pre­vi­ous work in 2016 which used a char-RNN trained on a ‘The Ses­sion’ dataset. A GPT-2 hy­po­thet­i­cally can im­prove on an RNN by bet­ter global co­her­ence & copy­ing of pat­terns, with­out prob­lems with the hid­den-s­tate bot­tle­neck.

     I en­coun­tered prob­lems with the stan­dard GPT-2 mod­el’s en­cod­ing of text which dam­aged re­sults, but after fix­ing that, I suc­cess­fully trained it on n = 205,304 ABC mu­sic pieces taken from The Ses­sion & ABCnotation.com. The re­sult­ing mu­sic sam­ples are in my opin­ion quite pleas­ant. (A sim­i­lar model was later re­trained by Geer­lings & Meroño-Peñuela 2020.)

     The ABC folk model & dataset are avail­able for down­load, and I pro­vide for lis­ten­ing se­lected mu­sic sam­ples as well as med­leys of ran­dom sam­ples from through­out train­ing.

     We fol­lowed the ABC folk model with an ABC-MIDI model: a dataset of 453k ABC pieces de­com­piled from MIDI pieces, which fit into GPT-2-117M with an ex­panded con­text win­dow when trained on TPUs. The MIDI pieces are far more di­verse and chal­leng­ing, and GPT-2 un­der­fits and strug­gles to pro­duce valid sam­ples but when sam­pling suc­ceeds, it can gen­er­ate even bet­ter mu­si­cal sam­ples.

179. “GPT-2 Preference Learning for Music Generation”, Gw­ern Bran­wen (2019-12-16):

     

     Stan­dard lan­guage gen­er­a­tion neural net­work mod­els, like GPT-2, are trained via like­li­hood train­ing to im­i­tate hu­man text cor­pus­es. Gen­er­ated text suffers from per­sis­tent flaws like rep­e­ti­tion, due to my­opic gen­er­a­tion word-by-word, and can­not im­prove on the train­ing data be­cause they are trained to pre­dict ‘re­al­is­tic’ com­ple­tions of the train­ing da­ta.

     A pro­posed al­ter­na­tive is to use re­in­force­ment learn­ing to train the NNs, to en­cour­age global prop­er­ties like co­her­ence & lack of rep­e­ti­tion, and po­ten­tially im­prove over the orig­i­nal cor­pus’s av­er­age qual­i­ty. Pref­er­ence learn­ing trains a re­ward func­tion on hu­man rat­ings, and uses that as the ‘en­vi­ron­ment’ for a black­box DRL al­go­rithm like PPO.

     Ope­nAI re­leased a code­base im­ple­ment­ing this du­al-model pref­er­ence learn­ing ap­proach for tex­tual gen­er­a­tion, based on GPT-2. Hav­ing pre­vi­ously used GPT-2 for po­etry & mu­sic gen­er­a­tion, I ex­per­i­mented with GPT-2 pref­er­ence learn­ing for un­con­di­tional mu­sic and po­etry gen­er­a­tion.

     I found that pref­er­ence learn­ing seemed to work bet­ter for mu­sic than po­et­ry, and seemed to re­duce the pres­ence of rep­e­ti­tion ar­ti­facts, but the re­sults, at n≅7,400 rat­ings com­piled over 23 it­er­a­tions of train­ing+sam­pling No­vem­ber 2019–­Jan­u­ary 2020, are not dra­mat­i­cally bet­ter than al­ter­na­tive im­prove­ments like scal­ing up mod­els or more thor­ough data-clean­ing or more strin­gent sam­ple cu­ra­tion. My blind rat­ings us­ing n≅200 com­par­isons showed no large ad­van­tage for the RL-tuned sam­ples (win­ning only 93 of 210 com­par­isons, or 46%).

     This may be due to in­suffi­cient rat­ings, bad hy­per­pa­ra­me­ters, or not us­ing sam­ples gen­er­ated with com­mon pre­fix­es, but I sus­pect it’s the for­mer, as some NLP tasks in Ziegler et al 2019 re­quired up to 60k rat­ings for good per­for­mance, and the re­ward model ap­peared to achieve poor per­for­mance & suc­cumb to ad­ver­sar­ial ex­am­ples eas­i­ly.

     Work­ing with it, I sus­pect that pref­er­ence learn­ing is un­nec­es­sar­ily sam­ple-in­effi­cient & data-in­effi­cient, and that the black­box re­in­force­ment learn­ing ap­proach is in­fe­rior to di­rectly us­ing the re­ward model to op­ti­mize text sam­ples, and pro­pose two ma­jor ar­chi­tec­tural over­hauls: have the re­ward model di­rectly model the im­plied rank­ing of every dat­a­point, and drop the agent model en­tirely in fa­vor of back­prop-pow­ered gra­di­ent as­cent which op­ti­mizes se­quences to max­i­mize the re­ward mod­el’s out­put.

180. “A Very Unlikely Chess Game”, Scott Alexan­der (2020-01-06):

     …Black is GPT-2. Its excuse [for this chess blunder] is that it’s a text prediction program with no concept of chess. As far as it knows, it’s trying to predict short alphanumeric strings like “e2e4” or “Nb7”. Nobody told it this represents a board game. It doesn’t even have a concept of 2D space that it could use to understand such a claim. But it still captured my rook! Embarrassing!…Last month, I asked him if he thought GPT-2 could play chess. I wondered if he could train it on a corpus of chess games written in standard notation (where, for example, e2e4 means “move the pawn at square e2 to square e4”). There are literally millions of games written up like this. GPT-2 would learn to predict the next string of text, which would correspond to the next move in the chess game. Then you would prompt it with a chessboard up to a certain point, and it would predict how the chess masters who had produced its training data would continue the game – ie make its next move using the same heuristics they would. Gwern handed the idea to his collaborator Shawn Presser, who had a working GPT-2 chess engine running within a week:…You can play against GPT-2 yourself by following the directions in the last tweet, though it won’t be much of a challenge for anyone better than I am.

     …What does this imply? I’m not sure (and maybe it will imply more if someone manages to make it actually good). It was already weird to see something with no auditory qualia learn passable poetic meter. It’s even weirder to see something with no concept of space learn to play chess. Is any of this meaningful? How impressed should we be that the same AI can write poems, compose music, and play chess, without having been designed for any of those tasks? I still don’t know.

     [See also the much later Noever et al 2020a/Noever et al 2020b who do the exact same thing in applying GPT-2 to Go SGF/chess PGN games.]

181. /docs/www/old.reddit.com/7eaaa81a26404ef60df4279ee1f1b0c829d73be5.html

182. /GPT-2#archive-of-our-own-ao3-gpt-2-1.5b

183. /GPT-2#video-game-walkthrough-gpt-2-1.5b

184. /RNN-metadata#geocities-char-rnn

185. “GPT-2 Folk Music: Training a Spaceless Model”, Gw­ern Bran­wen (2019-12-12):

     While training a GPT-2-117M on a folk music corpus written in ABC format, persistent syntax errors kept being generated by an otherwise-high-quality model: random spaces would be generated, rendering a music piece either erroneous or lower-quality. Why? It seems to be some issue with the GPT BPE encoder handling of spaces which makes it difficult to emit the right space-separated characters. We found that ABC does not actually require spaces, and we simply removed all spaces from the corpus—noticeably improving quality of generated pieces.

186. “Interacting with GPT–2 to Generate Controlled and Believable Musical Sequences in ABC Notation”, Car­iña Geer­lings, Al­bert Meroño-Peñuela (2020-10-16):

     Generating symbolic music with language models is a promising research area, with potential applications in automated music composition. Recent work shows that Transformer architectures can learn to generate compelling four-instrument scores from large MIDI datasets. In this paper, we re-train the small (117M) GPT-2 model with a large dataset in ABC notation, and generate samples of single-instrument folk music. Our BLEU and ROUGE based quantitative, and survey based qualitative, evaluations suggest that ABC notation is learned with syntactical and semantic correctness, and that samples contain robust and believable n-grams.

187. /GPT-2-music#combined-model-the-session-abcnotation.com

188. /GPT-2-music#samples

189. “Generating MIDI Music With GPT-2: Generating MIDI by converting to ABC and expanding the GPT-2 context window—works, if only just”, Gw­ern Bran­wen (2020-04-25):

     To expand the ABC GPT-2 model to cover a wider variety of musical genres, I turn to the next-most compact widespread music encoding format: MIDI. There are hundreds of thousands of MIDIs which can be decompiled to ABC format, averaging ~10k BPEs—within GPT-2-117M’s feasible context window when trained on TPUs (which permit training of context windows up to 30k wide).

     We compile the ABC from before and 2 large MIDI datasets, and convert to ABC, yielding ~453k usable ABC-MIDI musical files (~5.1GB of text). We trained January–April 2020 on our TPU swarm (with many interruptions), achieving a final loss of ~0.2 (underfit).

     Sampling from the final model is hit-or-miss as it is prone to the likelihood repetition trap and it generates instruments one-by-one so it is common for instruments to be cut off or otherwise broken during sampling (indicating that sampling is increasingly a bigger problem than training for long-range sequence modeling). However, successful pieces are possible, and are musically far more diverse than the folk ABC corpus, with many pleasingly complex samples.

190. /GPT-2-music#midi-dataset

191. /GPT-2-music#midi-samples

192. /GPT-2-preference-learning#bradley-terry-preference-learning

193. /GPT-2-preference-learning#optimization-by-backprop-not-blackbox

194. https://slatestarcodex.com/2019/02/28/meaningful/

195. “The Go Transformer: Natural Language Modeling for Game Play”, Matthew Cioli­no, David No­ev­er, Josh Kalin (2020-07-07):

     This work applies natural language modeling to generate plausible strategic moves in the ancient game of Go. We train the Generative Pretrained Transformer (GPT-2) to mimic the style of Go champions as archived in Smart Game Format (SGF), which offers a text description of move sequences. The trained model further generates valid but previously unseen strategies for Go. Because GPT-2 preserves punctuation and spacing, the raw output of the text generator provides inputs to game visualization and creative patterns, such as the Sabaki project’s game engine using auto-replays. Results demonstrate that language modeling can capture both the sequencing format of championship Go games and their strategic formations. Compared to random game boards, the GPT-2 fine-tuning shows efficient opening move sequences favoring corner play over less advantageous center and side play. Game generation as a language modeling task offers novel approaches to more than 40 other board games where historical text annotation provides training data (e.g., Amazons & Connect 4/6).

196. “The Chess Transformer: Mastering Play using Generative Language Models”, David No­ev­er, Matt Cioli­no, Josh Kalin (2020-08-02):

     This work demonstrates that natural language transformers can support more generic strategic modeling, particularly for text-archived games. In addition to learning natural language skills, the abstract transformer architecture can generate meaningful moves on a chessboard. With further fine-tuning, the transformer learns complex gameplay by training on 2.8 million chess games in Portable Game Notation. After 30,000 training steps, OpenAI’s Generative Pre-trained Transformer (GPT-2) optimizes weights for 774 million parameters. This fine-tuned Chess Transformer generates plausible strategies and displays game formations identifiable as classic openings, such as English or the Slav Exchange. Finally, in live play, the novel model demonstrates a human-to-transformer interface that correctly filters illegal moves and provides a novel method to challenge the transformer’s chess strategies. We anticipate future work will build on this transformer’s promise, particularly in other strategy games where features can capture the underlying complex rule syntax from simple but expressive player annotations.

197. “Decompiler”, Eng­lish Wikipedia (2020-12-22):

     A decompiler is a computer program that takes an executable file as input, and attempts to create a high level source file which can be recompiled successfully. It is therefore the opposite of a compiler, which takes a source file and makes an executable. Decompilers are usually unable to perfectly reconstruct the original source code, and as such, will frequently produce obfuscated code. Nonetheless, decompilers remain an important tool in the reverse engineering of computer software.