Link Bibliography

1. https://gwern.substack.com

2. “April 2015 News”, Gw­ern Bran­wen (2015-03-31):

   N/A

3. “Changelog”, Gw­ern Bran­wen (2013-09-15):

   This page is a changelog for Gw­ern.net: a monthly re­verse chrono­log­i­cal list of re­cent ma­jor writings/changes/additions.

   Fol­low­ing my writ­ing can be a lit­tle diffi­cult be­cause it is often so in­cre­men­tal. So every mon­th, in ad­di­tion to my reg­u­lar /r/Gwern sub­red­dit sub­mis­sions, I write up rea­son­ably-in­ter­est­ing changes and send it out to the mail­ing list in ad­di­tion to a com­pi­la­tion of links & re­views (archives).

4. “/r/gwern subreddit”, Gw­ern Bran­wen (2018-10-01):

   A subreddit for posting links of interest and also for announcing updates to gwern.net (which can be used as a RSS feed). Submissions are categorized similar to the monthly newsletter and typically will be collated there.

5. “Tor DNM-related arrests, 2011-2015”, Gw­ern Bran­wen (2012-07-14):

   I com­pile a ta­ble and dis­cus­sion of all known ar­rests and pros­e­cu­tions re­lated to Eng­lish-lan­guage Tor-Bit­coin dark­net mar­kets (DNMs) such as Silk Road 1, pri­mar­ily 2011–2015, along with dis­cus­sion of how they came to be ar­rest­ed.

6. “LWer Effective Altruism donations, 2013-2014”, Gw­ern Bran­wen (2015-05-12):

   A LW critic noted that the an­nual LW sur­vey re­ported a me­dian do­na­tion for “effec­tive al­tru­ists” of $0, though the EA move­ment en­cour­ages strongly do­na­tions. I look closer at the 2013-2014 LW sur­veys and find in mul­ti­ple re­gres­sion that iden­ti­fy­ing as an EA does pre­dict more do­na­tions after con­trol­ling for age and in­come, sug­gest­ing that the low EA me­dian do­na­tion may be due to EAers hav­ing low in­come and youth (eg be­ing a stu­dent) rather than be­ing un­usu­ally or even av­er­agely selfish.

7. “Meta-analysis of the heritability of human traits based on fifty years of twin studies”, Tinca J. C. Pol­d­er­man, Beben Benyam­in, Chris­ti­aan A. de Leeuw, Patrick F. Sul­li­van, Ar­jen van Bo­choven, Pe­ter M. Viss­cher, Danielle Posthuma (2015-05-18):

   Despite a century of research on complex traits in humans, the relative importance and specific nature of the influences of genes and environment on human traits remain controversial. We report a meta-analysis of twin correlations and reported variance components for 17,804 traits from 2,748 publications including 14,558,903 partly dependent twin pairs, virtually all published twin studies of complex traits. Estimates of heritability cluster strongly within functional domains, and across all traits the reported heritability is 49%. For a majority (69%) of traits, the observed twin correlations are consistent with a simple and parsimonious model where twin resemblance is solely due to additive genetic variation. The data are inconsistent with substantial influences from shared environment or non-additive genetic variation. This study provides the most comprehensive analysis of the causes of individual differences in human traits thus far and will guide future gene-mapping efforts. All the results can be visualized using the MaTCH webtool.

8. http://www.unz.com/gnxp/through-the-wormhole-are-we-here-for-a-reason-premier-may-13th/

9. /docs/economics/2018-gimpelson.pdf

10. http://econlog.econlib.org/archives/2015/05/systematically_3.html

11. http://digitalcommons.law.yale.edu/cgi/viewcontent.cgi?article=1532&context=fss_papers

12. http://www.theatlantic.com/magazine/archive/1996/05/the-nitrous-oxide-philosopher/376581/

13. http://www.newyorker.com/magazine/2013/01/14/the-science-of-sex-abuse?currentPage=all

14. https://blogs.discovermagazine.com/gnxp/2013/01/why-the-future-wont-be-genetically-homogeneous/

15. /docs/longevity/1992-nelson.pdf

16. “End-to-End Training of Deep Visuomotor Policies”, Sergey Levine, Chelsea Finn, Trevor Dar­rell, Pieter Abbeel (2015-04-02):

    Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot’s motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a partially observed guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods.

17. https://www.youtube.com/watch?v=JeVppkoloXs

18. https://www.youtube.com/watch?v=EtMyH_--vnU

19. “The Unreasonable Effectiveness of Recurrent Neural Networks”, An­drej Karpa­thy (2015-05-21):

    [Exploration of char-RNN neural nets for generating text. Karpathy codes a simple recurrent NN which generates character-by-character, and discovers that it is able to generate remarkably plausible text (at the syntactic level) for Paul Graham, Shakespeare, Wikipedia, LaTeX, Linux C code, and baby names—all using the same generic architecture. Visualizing the internal activity of the char-RNNs, they seem to be genuinely understanding some of the recursive syntactic structure of the text in a way that other text-generation methods like n-grams cannot. Inspired by this post, I began tinkering with char-RNNs for poetry myself; as of 2019, char-RNNs have been largely obsoleted by the new Transformer architecture, but recurrency will make a comeback and Karpathy’s post is still a valuable and fun read.]

    There’s something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience I’ve in fact reached the opposite conclusion). Fast forward about a year: I’m training RNNs all the time and I’ve witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me. This post is about sharing some of that magic with you.We’ll train RNNs to generate text character by character and ponder the question “how is that even possible?”

20. “Using N-of-1 trials to improve patient management and save costs.”, Scuffham, Paul A. Nikles, Jane Mitchell, Ge­offrey K. Yel­land, Michael J. Vine, Norma Poulos, Christo­pher J. Pil­lans, Pe­ter I. Bash­ford, Guy del Mar, Chris Schluter, Philip J. Glasziou, Paul (2010):

    N-of-1 trials test treatment effectiveness within an individual patient. To assess (1) the impact of three different N-of-1 trials on both clinical and economic outcomes over 12 months and (2) whether the use of N-of-1 trials to target patients' access to high-cost drugs might be cost-effective in Australia. Descriptive study of management change, persistence, and costs summarizing three N-of-1 trials. Volunteer patients with osteoarthritis, chronic neuropathic pain or ADHD whose optimal choice of treatment was uncertain. Double-blind cyclical alternative medications for the three conditions. Detailed resource use, treatment and health outcomes (response) data collected by postal and telephone surveys immediately before and after the trial and at 3, 6 and 12 months. Estimated costs to the Australian healthcare system for the pre-trial vs. 12 months post-trial. Participants persisting with the joint patient-doctor decision 12 months after trial completion were 32% for osteoarthritis, 45% for chronic neuropathic pain and 70% for the ADHD trials. Cost-offsets were obtained from reduced usage of non-optimal drugs, and reduced medical consultations. Drug costs increased for the chronic neuropathic pain and ADHD trials due to many patients being on either low-cost or no pharmaceuticals before the trial. N-of-1 trials are an effective method to identify optimal treatment in patients in whom disease management is uncertain. Using this evidence-based approach, patients and doctors tend to persist with optimal treatment resulting in cost-savings. N-of-1 trials are clinically acceptable and may be an effective way of rationally prescribing some expensive long-term medicines.

21. http://lesswrong.com/lw/gbi/assessing_kurzweil_the_results/

22. http://projecteuclid.org/DPubS/Repository/1.0/Disseminate?view=body&id=pdfview_1&handle=euclid.ssu/1356628931

23. http://pps.sagepub.com/content/7/6/643.full

24. http://home.gwu.edu/~forrest/fmdeathincongressps.pdf

25. https://izbicki.me/blog/nuclear-weapon-statistics-using-monoids-groups-and-modules-in-haskell

26. https://arcsecond.wordpress.com/2010/09/08/flat-priors-and-other-improbable-tales/

27. http://online.liebertpub.com/doi/pdf/10.1089/rej.2014.1636

28. https://www.biomedcentral.com/content/pdf/1475-2891-9-4.pdf

29. /docs/spacedrepetition/1987-adams.pdf

30. http://blog.ninlabs.com/2013/01/programmer-interrupted/

31. http://www.thefreelibrary.com/E+unibus+pluram%3A+television+and+U.S.+fiction.-a013952319

32. “2,4-Dinitrophenol”, Eng­lish Wikipedia (2020-12-28):

    

    2,4-Dinitrophenol (2,4-DNP or simply DNP) is an organic compound with the formula HOC₆H₃(NO₂)₂. It is a yellow, crystalline solid that has a sweet, musty odor. It sublimes, is volatile with steam, and is soluble in most organic solvents as well as aqueous alkaline solutions. When in a dry form, it is a high explosive and has an instantaneous explosion hazard. It is a precursor to other chemicals and is biochemically active, inhibiting adenosine triphosphate (ATP) production in cells with mitochondria. Its use as a dieting aid has been identified with severe side-effects, including a number of deaths.

33. “What’s in a Color? The Unique Human Health Effects of Blue Light”, David C. Holz­man (2010-01):

    In 1958, J. Woodland Hastings and Beatrice M. Sweeney tested the ability of different wavelengths of light—corresponding to different colors—to shift the circadian rhythm in the photosynthetic marine dinoflagellate Gonyaulax polyedra. The greatest power to reset the organism’s daily meter lay in the blues, with a precipitous decline into the greens and a modest boost in the reds.

    Hastings and Sweeney’s paper, published in the December 1958 Biological Bulletin, gathered dust for decades. No one thought these findings might hold any relevance for humans, whose circadian rhythms were then widely believed to be relatively insensitive to light.

    But scientific discoveries in the past two decades have changed all that. Not only does light reset the human circadian rhythm, but the same blue light that has the strongest impact on dinoflagellates has equal power to reset our own clocks—although most visible wavelengths can reset the clock, the blues do the job with the greatest efficiency.

    Now researchers are finding increasingly that an out-of-phase circadian rhythm is a health hazard. “Maintaining synchronized circadian rhythms is important to health and well-being,” says Dieter Kunz, director of the Sleep Research and Clinical Chronobiology Research Group at Charité–Universitätsmedizin Berlin. “A growing body of evidence suggests that a desynchronization of circadian rhythms may play a role in various tumoral diseases, diabetes, obesity, and depression.”

    Shift workers, whom Kunz calls “a model for internal desynchronization,” are known to experience increased morbidity and mortality for a number of diseases, including cardiovascular disorders and cancer. In fact, in 2007, the World Health Organization decreed that shift work is a risk factor for breast cancer, and on that basis, in 2009, the Danish government began compensating some female shift workers with breast cancer.

    At the same time, researchers have repeatedly shown that bright white light has the power to mitigate depression and other maladies of mood. An emergent recent literature suggests that blue light may be particularly potent for such applications.

34. /docs/technology/2013-02-07-nyrb-rich-divingdeepintodanger.html

35. https://arcsecond.wordpress.com/2011/02/08/late-night-thoughts-on-reading-scientology/

36. “Epigrams on Programming”, Alan J. Perlis (1982-09-1):

    [130 epigrams on computer science and technology, published in 1982, for ACM’s SIGPLAN journal, by noted computer scientist and programming language researcher Alan Perlis. The epigrams are a series of short, programming-language-neutral, humorous statements about computers and programming, distilling lessons he had learned over his career, which are widely quoted.]

    8. A programming language is low level when its programs require attention to the irrelevant.…19. A language that doesn’t affect the way you think about programming, is not worth knowing.…54. Beware of the Turing tar-pit in which everything is possible but nothing of interest is easy.

    15. Everything should be built top-down, except the first time.…30. In programming, everything we do is a special case of something more general—and often we know it too quickly.…31. Simplicity does not precede complexity, but follows it.…58. Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it.…65. Make no mistake about it: Computers process numbers—not symbols. We measure our understanding (and control) by the extent to which we can arithmetize an activity.…56. Software is under a constant tension. Being symbolic it is arbitrarily perfectible; but also it is arbitrarily changeable.

    1. One man’s constant is another man’s variable. 34. The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information.

    36. The use of a program to prove the 4-color theorem will not change mathematics—it merely demonstrates that the theorem, a challenge for a century, is probably not important to mathematics.

    39. Re graphics: A picture is worth 10K words—but only those to describe the picture. Hardly any sets of 10K words can be adequately described with pictures.

    48. The best book on programming for the layman is Alice in Wonderland; but that’s because it’s the best book on anything for the layman.

    77. The cybernetic exchange between man, computer and algorithm is like a game of musical chairs: The frantic search for balance always leaves one of the 3 standing ill at ease.…79. A year spent in artificial intelligence is enough to make one believe in God.…84. Motto for a research laboratory: What we work on today, others will first think of tomorrow.

    91. The computer reminds one of Lon Chaney—it is the machine of a thousand faces.

    7. It is easier to write an incorrect program than understand a correct one.…93. When someone says “I want a programming language in which I need only say what I wish done,” give him a lollipop.…102. One can’t proceed from the informal to the formal by formal means.

    100. We will never run out of things to program as long as there is a single program around.

    108. Whenever 2 programmers meet to criticize their programs, both are silent.…112. Computer Science is embarrassed by the computer.…115. Most people find the concept of programming obvious, but the doing impossible. 116. You think you know when you can learn, are more sure when you can write, even more when you can teach, but certain when you can program. 117. It goes against the grain of modern education to teach children to program. What fun is there in making plans, acquiring discipline in organizing thoughts, devoting attention to detail and learning to be self-critical?

37. https://pdfs.semanticscholar.org/2066/e63f9f668d9e9789949af2f2d8a55bf72c49.pdf

38. “How Much of the Web Is Archived?”, Scott G. Ainsworth, Ahmed Al­Sum, Hany Sala­hEldeen, Michele C. Wei­gle, Michael L. Nel­son (2012-12-26):

    Although the Internet Archive’s Wayback Machine is the largest and most well-known web archive, there have been a number of public web archives that have emerged in the last several years. With varying resources, audiences and collection development policies, these archives have varying levels of overlap with each other. While individual archives can be measured in terms of number of URIs, number of copies per URI, and intersection with other archives, to date there has been no answer to the question "How much of the Web is archived?" We study the question by approximating the Web using sample URIs from DMOZ, Delicious, Bitly, and search engine indexes; and, counting the number of copies of the sample URIs exist in various public web archives. Each sample set provides its own bias. The results from our sample sets indicate that range from 35 between 2-5 copies, 1 in public web archives. The number of URI copies varies as a function of time, but no more than 31.3

39. http://actualfacebookgraphsearches.tumblr.com/

40. http://www.newyorker.com/magazine/2015/05/11/overkill-atul-gawande

41. http://www.theatlantic.com/magazine/archive/2012/12/the-insourcing-boom/309166/?single_page=true

42. http://www.bloomberg.com/bw/articles/2012-11-29/the-march-of-robots-into-chinese-factories#r=else-fs

43. http://www.econtalk.org/archives/2013/01/jerven_on_measu.html

44. https://web.archive.org/web/20130114231801/http://33bits.org/2013/01/08/online-price-discrimination-conspicuous-by-its-absence/

45. https://web.archive.org/web/20130222033025/http://33bits.org:80/2013/01/22/price-discrimination-and-the-illusion-of-fairness/

46. https://www.theverge.com/2015/5/13/8592817/rat-patrol-new-york-alberta-canada-south-georgia-eradication

47. /docs/economics/2013-sachs.pdf

48. http://marginalrevolution.com/marginalrevolution/2013/01/the-ricardo-effect-in-europe-germany-fact-of-the-day.html

49. https://www.jacobinmag.com/2011/12/four-futures/

50. http://www.economist.com/news/briefing/21569381-idea-innovation-and-new-technology-have-stopped-driving-growth-getting-increasing

51. http://kk.org/thetechnium/2013/01/the-post-produc

52. http://lesswrong.com/lw/th/harder_choices_matter_less/

53. http://rsta.royalsocietypublishing.org/content/369/1956/4913.full.pdf

54. http://blog.regehr.org/archives/881

55. http://famouspoetsandpoems.com/poets/william_butler_yeats/poems/10349

56. http://thoughtcrime.crummy.com/2009/Error.html

57. /docs/culture/2003-ramsay.pdf

58. http://fernwanderweg.blogspot.ch/2013/01/a-rockslide-in-action-arch-falls-into.html

59. “Ready Player One”, Eng­lish Wikipedia (2020-12-28):

    

    Ready Player One is a 2011 science fiction novel, and the debut novel of American author Ernest Cline. The story, set in a dystopia in 2045, follows protagonist Wade Watts on his search for an Easter egg in a worldwide virtual reality game, the discovery of which would lead him to inherit the game creator's fortune. Cline sold the rights to publish the novel in June 2010, in a bidding war to the Crown Publishing Group. The book was published on August 16, 2011. An audiobook was released the same day; it was narrated by Wil Wheaton, who was mentioned briefly in one of the chapters. In 2012, the book received an Alex Award from the Young Adult Library Services Association division of the American Library Association and won the 2011 Prometheus Award.A film adaptation, screenwritten by Cline and Zak Penn and directed by Steven Spielberg, was released on March 29, 2018. A sequel, Ready Player Two, was released on November 24, 2020.

60. /Book-reviews#ready-player-one-cline-2011

61. “A Shropshire Lad”, Eng­lish Wikipedia (2020-12-22):

    A Shropshire Lad is a collection of sixty-three poems by the English poet Alfred Edward Housman, published in 1896. Selling slowly at first, it then rapidly grew in popularity, particularly among young readers. Composers began setting the poems to music less than ten years after their first appearance, and many parodists have satirised Housman's themes and poetic style.

62. /Book-reviews#a-shropshire-lad-housman-1990

63. https://archiveofourown.org/works/2372021/chapters/5238359

64. https://slatestarcodex.com/2015/05/08/links-515-tall-and-linky/

65. http://www.amazon.com/Drift-into-Failure-Sidney-Dekker/dp/1409422216

66. /Book-reviews#drift-into-failure-dekker-2011

67. “Pumping Iron”, Eng­lish Wikipedia (2020-12-28):

    

    Pumping Iron is a 1977 American docudrama about the world of professional bodybuilding, with a focus on the 1975 IFBB Mr. Universe and 1975 Mr. Olympia competitions. Directed by George Butler and Robert Fiore and edited by Geof Bartz and Larry Silk, it is inspired by a book of the same name by Butler and Charles Gaines, and nominally centers on the competition between Arnold Schwarzenegger and one of his primary competitors for the title of Mr. Olympia, Lou Ferrigno. The film also features segments on bodybuilders Franco Columbu and Mike Katz, in addition to appearances by Ken Waller, Ed Corney, Serge Nubret, and other famous bodybuilders of the era.

68. /Movie-reviews#pumping-iron

69. “Monthly Girls' Nozaki-kun”, Eng­lish Wikipedia (2020-12-28):

    

    Monthly Girls' Nozaki-kun is a Japanese four-panel romantic comedy manga written and illustrated by Izumi Tsubaki. The chapters are serialized online in Gangan Online, and have been published in both physical and digital releases of Shoujo Romance Girly and tankōbon volumes by Square Enix. An anime adaptation by Doga Kobo aired in July 2014.

70. /Anime-reviews#gekkan-shoujo-nozaki-kun

71. https://www.youtube.com/watch?v=WxFQWI7Thsw

72. http://otokei-douj.in/tlmc//%5B%E3%81%97%E3%81%97%E3%81%BE%E3%81%84%E3%83%96%E3%83%A9%E3%82%B6%E3%83%BC%E3%82%BA%5D%20Sisimai-brothers/2009.12.30%20%5BSSM-009%5D%20%E5%A4%A7%E6%A5%BD%E7%AC%A6%20%5BC77%5D/01%20%5B%E3%81%97%E3%81%97%E3%81%BE%E3%81%84%E4%B8%89%E5%8F%B7%5D%20%E6%81%8B%E3%81%A8%E6%B0%B4%E3%81%A8%E7%9F%B3%E7%82%AD%E3%81%A8.mp3

73. http://otokei-douj.in/tlmc//%5B%E3%81%97%E3%81%97%E3%81%BE%E3%81%84%E3%83%96%E3%83%A9%E3%82%B6%E3%83%BC%E3%82%BA%5D%20Sisimai-brothers/2009.12.30%20%5BSSM-009%5D%20%E5%A4%A7%E6%A5%BD%E7%AC%A6%20%5BC77%5D/03%20%5B%E3%81%97%E3%81%97%E3%81%BE%E3%81%84%E4%B8%89%E5%8F%B7%5D%20%E6%B0%B4%E5%BD%A9%E4%BA%BA%E5%BD%A2.mp3

74. http://otokei-douj.in/tlmc//%5B%E3%81%97%E3%81%97%E3%81%BE%E3%81%84%E3%83%96%E3%83%A9%E3%82%B6%E3%83%BC%E3%82%BA%5D%20Sisimai-brothers/2009.12.30%20%5BSSM-009%5D%20%E5%A4%A7%E6%A5%BD%E7%AC%A6%20%5BC77%5D/09%20%5B%E3%81%97%E3%81%97%E3%81%BE%E3%81%84%E4%B8%89%E5%8F%B7%5D%20%E7%A9%BA%E3%81%AE%E9%96%93%E3%81%AB%E9%96%93%E3%81%AB.mp3

75. https://www.youtube.com/watch?v=a_nQMmi_cH8

76. https://www.dropbox.com/s/31lgdtw0hjk18mu/tam%C3%97keigokanzaki-%E6%9D%B1%E6%96%B9%E3%83%90%E3%82%A4%E3%82%AA%E3%83%AA%E3%83%B3%E3%83%AD%E3%83%83%E3%82%AF%E6%B0%B8nagarae-maidenscapriccio.ogg

77. https://www.dropbox.com/s/86m42ktwm90ofhq/tam%C3%97keigokanzaki-%E6%9D%B1%E6%96%B9%E3%83%90%E3%82%A4%E3%82%AA%E3%83%AA%E3%83%B3%E3%83%AD%E3%83%83%E3%82%AF%E6%B0%B8nagarae-extendash.ogg

78. https://www.dropbox.com/s/hj9k4x5wcq9bgri/tam%C3%97keigokanzaki-%E6%9D%B1%E6%96%B9%E3%83%90%E3%82%A4%E3%82%AA%E3%83%AA%E3%83%B3%E3%83%AD%E3%83%83%E3%82%AF%E6%B0%B8nagarae-eveningprimrose.ogg

79. https://www.dropbox.com/s/znixxhsporh9lqq/%E4%BC%8A%E8%97%A4%E7%BF%BC-kantaiphilharmonicorchestra-%E9%8E%AE%E5%AE%88%E5%BA%9C%E3%81%AE%E5%BA%8F%E6%9B%B2.ogg

80. “Gwern.net newsletter (Substack subscription page)”, Gw­ern Bran­wen (2013-12-01):

    Subscription page for the monthly gwern.net newsletter. There are monthly updates, which will include summaries of projects I’ve worked on that month (the same as the changelog), collations of links or discussions from my subreddit, and book/movie reviews. You can also browse the archives since December 2013.

81. “Gwern.net newsletter archives”, Gw­ern Bran­wen (2013-12-01):

    Newsletter tag: archive of all issues back to 2013 for the gwern.net newsletter (monthly updates, which will include summaries of projects I’ve worked on that month (the same as the changelog), collations of links or discussions from my subreddit, and book/movie reviews.)

82. https://old.reddit.com/r/gwern/.rss

83. “RNN metadata for mimicking individual author style”, Gw­ern Bran­wen (2015-09-12):

    Char-RNNs are un­su­per­vised gen­er­a­tive mod­els which learn to mimic text se­quences. I sug­gest ex­tend­ing char-RNNs with in­line meta­data such as genre or au­thor pre­fixed to each line of in­put, al­low­ing for bet­ter & more effi­cient meta­data, and more con­trol­lable sam­pling of gen­er­ated out­put by feed­ing in de­sired meta­da­ta. A 2015 ex­per­i­ment us­ing torch-rnn on a set of ~30 Project Guten­berg e-books (1 per au­thor) to train a large char-RNN shows that a char-RNN can learn to re­mem­ber meta­data such as au­thors, learn as­so­ci­ated prose styles, and often gen­er­ate text vis­i­bly sim­i­lar to that of a spec­i­fied au­thor.

    I fur­ther try & fail to train a char-RNN on Geoc­i­ties HTML for un­clear rea­sons.

    More suc­cess­ful­ly, I ex­per­i­ment in 2019 with a re­cent­ly-de­vel­oped al­ter­na­tive to char-RNNs, the Trans­former NN ar­chi­tec­ture, by fine­tun­ing train­ing Ope­nAI’s GPT-2-117M Trans­former model on a much larger (117MB) Project Guten­berg po­etry cor­pus us­ing both un­la­beled lines & lines with in­line meta­data (the source book). The gen­er­ated po­etry is much bet­ter. And GPT-3 is bet­ter still.

84. “GPT-2 Neural Network Poetry”, Gw­ern Bran­wen (2019-03-03):

    In Feb­ru­ary 2019, fol­low­ing up on my 2015–2016 tex­t-gen­er­a­tion ex­per­i­ments with char-RNNs, I ex­per­i­ment with the cut­ting-edge Trans­former NN ar­chi­tec­ture for lan­guage mod­el­ing & text gen­er­a­tion. Us­ing Ope­nAI’s GPT-2-117M (117M) model pre-trained on a large In­ter­net cor­pus and nshep­perd’s fine­tun­ing code, I re­train GPT-2-117M on a large (117MB) Project Guten­berg po­etry cor­pus. I demon­strate how to train 2 vari­ants: “GPT-2-poetry”, trained on the po­ems as a con­tin­u­ous stream of text, and “GPT-2-poetry-prefix”, with each line pre­fixed with the meta­data of the PG book it came from. In May 2019, I trained the nex­t-largest GPT-2, GPT-2-345M, sim­i­lar­ly, for a fur­ther qual­ity boost in gen­er­ated po­ems. In Oc­to­ber 2019, I re­trained GPT-2-117M on a Project Guten­berg cor­pus with im­proved for­mat­ting, and com­bined it with a con­tem­po­rary poem dataset based on Po­etry Foun­da­tion’s web­site. .> With just a few GPU-days on 1080ti GPUs, GPT-2-117M fine­tun­ing can pro­duce high­-qual­ity po­etry which is more the­mat­i­cally con­sis­tent than my char-RNN po­ems, ca­pa­ble of mod­el­ing sub­tle fea­tures like rhyming, and some­times even a plea­sure to read. I list the many pos­si­ble ways to im­prove poem gen­er­a­tion and fur­ther ap­proach hu­man-level po­ems. For the high­est-qual­ity AI po­etry to date, see my fol­lowup page, “GPT-3 Cre­ative Writ­ing”.

    For anime plot sum­maries, see TWDNE; for gen­er­at­ing ABC-formatted folk mu­sic, see “GPT-2 Folk Mu­sic” & “GPT-2 Pref­er­ence Learn­ing for Mu­sic and Po­etry Gen­er­a­tion”; for play­ing chess, see “A Very Un­likely Chess Game”; for the Red­dit com­ment gen­er­a­tor, see SubSimulatorGPT-2; for fan­fic­tion, the Ao3; and for video games, the walk­through model. For Ope­nAI’s GPT-3 fol­lowup, see “GPT-3: Lan­guage Mod­els are Few-Shot Learn­ers”.

85. “Alan Perlis”, Eng­lish Wikipedia (2020-12-27):

    

    Alan Jay Perlis was an American computer scientist and professor at Purdue University, Carnegie Mellon University and Yale University. He is best known for his pioneering work in programming languages and was the first recipient of the Turing Award.

86. /RNN-metadata#geocities-char-rnn

87. “GPT-3 Creative Fiction”, Gw­ern Bran­wen (2020-06-19):

    

    I con­tinue my AI po­etry gen­er­a­tion ex­per­i­ments with Ope­nAI’s 2020 GPT-3, which is 116× larg­er, and much more pow­er­ful, than the 2019 GPT-2. GPT-3, how­ev­er, is not merely a quan­ti­ta­tive tweak yield­ing “GPT-2 but bet­ter”—it is qual­i­ta­tively differ­ent, ex­hibit­ing eerie run­time learn­ing ca­pa­bil­i­ties al­low­ing even the raw mod­el, with zero fine­tun­ing, to “meta-learn” many tex­tual tasks purely by ex­am­ple or in­struc­tion. One does not train or pro­gram GPT-3 in a nor­mal way, but one en­gages in di­a­logue and writes prompts to teach GPT-3 what one wants.

    Ex­per­i­ment­ing through the Ope­nAI Beta API in June 2020, I find that GPT-3 does not just match my fine­tuned GPT-2-1.5b-poetry for po­em-writ­ing qual­i­ty, but ex­ceeds it, while be­ing ver­sa­tile in han­dling po­etry, Tom Swifty puns, sci­ence fic­tion, di­a­logue like Tur­ing’s Tur­ing-test di­a­logue, lit­er­ary style par­o­dies… As the pièce de ré­sis­tance, I recre­ate Stanis­law Lem’s Cy­be­riad’s “Trurl’s Elec­tronic Bard” po­etry us­ing GPT-3. (A­long the way, I doc­u­ment in­stances of how the BPE text en­cod­ing un­nec­es­sar­ily dam­ages GPT-3’s per­for­mance on a va­ri­ety of tasks, how to best elicit the high­est-qual­ity re­spons­es, com­mon er­rors peo­ple make in us­ing GPT-3, and test out GPT-3’s im­prove­ments in NN weak points like logic or com­mon­sense knowl­edge.)

    GPT-3’s sam­ples are not just close to hu­man lev­el: they are cre­ative, wit­ty, deep, meta, and often beau­ti­ful. They demon­strate an abil­ity to han­dle ab­strac­tions, like style par­o­dies, I have not seen in GPT-2 at all. Chat­ting with GPT-3 feels un­can­nily like chat­ting with a hu­man. I was im­pressed by the re­sults re­ported in the GPT-3 pa­per, and after spend­ing a week try­ing it out, I re­main im­pressed.

    This page records GPT-3 sam­ples I gen­er­ated in my ex­plo­rations, and thoughts on how to use GPT-3 and its re­main­ing weak­nesses. I hope you en­joy them even a tenth as much as I en­joyed test­ing GPT-3 and watch­ing the com­ple­tions scroll across my screen.

88. “Poetry Foundation”, Eng­lish Wikipedia (2020-12-27):

    

    The Poetry Foundation is a Chicago-based American foundation created to promote poetry in the wider culture. It was formed from Poetry magazine, which it continues to publish, with a 2003 gift of $200 million from philanthropist Ruth Lilly.

89. https://www.poetryfoundation.org/poems

90. /TWDE#text

91. “GPT-2 Folk Music”, Gw­ern Bran­wen (2019-11-01):

    

    In No­vem­ber 2019, I ex­per­i­mented with train­ing a GPT-2 neural net model to gen­er­ate folk mu­sic in the high­-level ABC mu­sic text for­mat, fol­low­ing pre­vi­ous work in 2016 which used a char-RNN trained on a ‘The Ses­sion’ dataset. A GPT-2 hy­po­thet­i­cally can im­prove on an RNN by bet­ter global co­her­ence & copy­ing of pat­terns, with­out prob­lems with the hid­den-s­tate bot­tle­neck.

    I en­coun­tered prob­lems with the stan­dard GPT-2 mod­el’s en­cod­ing of text which dam­aged re­sults, but after fix­ing that, I suc­cess­fully trained it on n = 205,304 ABC mu­sic pieces taken from The Ses­sion & ABCnotation.com. The re­sult­ing mu­sic sam­ples are in my opin­ion quite pleas­ant. (A sim­i­lar model was later re­trained by Geer­lings & Meroño-Peñuela 2020.)

    The ABC folk model & dataset are avail­able for down­load, and I pro­vide for lis­ten­ing se­lected mu­sic sam­ples as well as med­leys of ran­dom sam­ples from through­out train­ing.

    We fol­lowed the ABC folk model with an ABC-MIDI model: a dataset of 453k ABC pieces de­com­piled from MIDI pieces, which fit into GPT-2-117M with an ex­panded con­text win­dow when trained on TPUs. The MIDI pieces are far more di­verse and chal­leng­ing, and GPT-2 un­der­fits and strug­gles to pro­duce valid sam­ples but when sam­pling suc­ceeds, it can gen­er­ate even bet­ter mu­si­cal sam­ples.

92. “GPT-2 Preference Learning for Music Generation”, Gw­ern Bran­wen (2019-12-16):

    

    Stan­dard lan­guage gen­er­a­tion neural net­work mod­els, like GPT-2, are trained via like­li­hood train­ing to im­i­tate hu­man text cor­pus­es. Gen­er­ated text suffers from per­sis­tent flaws like rep­e­ti­tion, due to my­opic gen­er­a­tion word-by-word, and can­not im­prove on the train­ing data be­cause they are trained to pre­dict ‘re­al­is­tic’ com­ple­tions of the train­ing da­ta.

    A pro­posed al­ter­na­tive is to use re­in­force­ment learn­ing to train the NNs, to en­cour­age global prop­er­ties like co­her­ence & lack of rep­e­ti­tion, and po­ten­tially im­prove over the orig­i­nal cor­pus’s av­er­age qual­i­ty. Pref­er­ence learn­ing trains a re­ward func­tion on hu­man rat­ings, and uses that as the ‘en­vi­ron­ment’ for a black­box DRL al­go­rithm like PPO.

    Ope­nAI re­leased a code­base im­ple­ment­ing this du­al-model pref­er­ence learn­ing ap­proach for tex­tual gen­er­a­tion, based on GPT-2. Hav­ing pre­vi­ously used GPT-2 for po­etry & mu­sic gen­er­a­tion, I ex­per­i­mented with GPT-2 pref­er­ence learn­ing for un­con­di­tional mu­sic and po­etry gen­er­a­tion.

    I found that pref­er­ence learn­ing seemed to work bet­ter for mu­sic than po­et­ry, and seemed to re­duce the pres­ence of rep­e­ti­tion ar­ti­facts, but the re­sults, at n≅7,400 rat­ings com­piled over 23 it­er­a­tions of train­ing+sam­pling No­vem­ber 2019–­Jan­u­ary 2020, are not dra­mat­i­cally bet­ter than al­ter­na­tive im­prove­ments like scal­ing up mod­els or more thor­ough data-clean­ing or more strin­gent sam­ple cu­ra­tion. My blind rat­ings us­ing n≅200 com­par­isons showed no large ad­van­tage for the RL-tuned sam­ples (win­ning only 93 of 210 com­par­isons, or 46%).

    This may be due to in­suffi­cient rat­ings, bad hy­per­pa­ra­me­ters, or not us­ing sam­ples gen­er­ated with com­mon pre­fix­es, but I sus­pect it’s the for­mer, as some NLP tasks in Ziegler et al 2019 re­quired up to 60k rat­ings for good per­for­mance, and the re­ward model ap­peared to achieve poor per­for­mance & suc­cumb to ad­ver­sar­ial ex­am­ples eas­i­ly.

    Work­ing with it, I sus­pect that pref­er­ence learn­ing is un­nec­es­sar­ily sam­ple-in­effi­cient & data-in­effi­cient, and that the black­box re­in­force­ment learn­ing ap­proach is in­fe­rior to di­rectly us­ing the re­ward model to op­ti­mize text sam­ples, and pro­pose two ma­jor ar­chi­tec­tural over­hauls: have the re­ward model di­rectly model the im­plied rank­ing of every dat­a­point, and drop the agent model en­tirely in fa­vor of back­prop-pow­ered gra­di­ent as­cent which op­ti­mizes se­quences to max­i­mize the re­ward mod­el’s out­put.

93. “A Very Unlikely Chess Game”, Scott Alexan­der (2020-01-06):

    …Black is GPT-2. Its excuse [for this chess blunder] is that it’s a text prediction program with no concept of chess. As far as it knows, it’s trying to predict short alphanumeric strings like “e2e4” or “Nb7”. Nobody told it this represents a board game. It doesn’t even have a concept of 2D space that it could use to understand such a claim. But it still captured my rook! Embarrassing!…Last month, I asked him if he thought GPT-2 could play chess. I wondered if he could train it on a corpus of chess games written in standard notation (where, for example, e2e4 means “move the pawn at square e2 to square e4”). There are literally millions of games written up like this. GPT-2 would learn to predict the next string of text, which would correspond to the next move in the chess game. Then you would prompt it with a chessboard up to a certain point, and it would predict how the chess masters who had produced its training data would continue the game – ie make its next move using the same heuristics they would. Gwern handed the idea to his collaborator Shawn Presser, who had a working GPT-2 chess engine running within a week:…You can play against GPT-2 yourself by following the directions in the last tweet, though it won’t be much of a challenge for anyone better than I am.

    …What does this imply? I’m not sure (and maybe it will imply more if someone manages to make it actually good). It was already weird to see something with no auditory qualia learn passable poetic meter. It’s even weirder to see something with no concept of space learn to play chess. Is any of this meaningful? How impressed should we be that the same AI can write poems, compose music, and play chess, without having been designed for any of those tasks? I still don’t know.

    [See also the much later Noever et al 2020a/Noever et al 2020b who do the exact same thing in applying GPT-2 to Go SGF/chess PGN games.]

94. /docs/www/old.reddit.com/7eaaa81a26404ef60df4279ee1f1b0c829d73be5.html

95. /GPT-2#archive-of-our-own-ao3-gpt-2-1.5b

96. /GPT-2#video-game-walkthrough-gpt-2-1.5b

97. “Language Models are Few-Shot Learners”, Tom B. Brown, Ben­jamin Mann, Nick Ry­der, Melanie Sub­bi­ah, Jared Ka­plan, Pra­fulla Dhari­wal, Arvind Nee­lakan­tan, Pranav Shyam, Girish Sas­try, Amanda Askell, Sand­hini Agar­wal, Ariel Her­bert-Voss, Gretchen Krueger, Tom Henighan, Re­won Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Win­ter, Christo­pher Hes­se, Mark Chen, Eric Sigler, Ma­teusz Litwin, Scott Gray, Ben­jamin Chess, Jack Clark, Christo­pher Bern­er, Sam Mc­Can­dlish, Alec Rad­ford, Ilya Sutskev­er, Dario Amodei (2020-05-28):

    Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions—something which current NLP systems still largely struggle to do.

    Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10× more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora.

    Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

98. /GPT-2#gpt-2-1.5b

99. /GPT-3#poetry

100. /GPT-3#tom-swifties

101. /GPT-3#turing-dialogue

102. /GPT-3#literary-parodies

103. /GPT-3#stanislaw-lems-cyberiad

104. “GPT-3 Weaknesses: Byte-Pair Encodings (BPEs)”, Gw­ern Bran­wen (2020-06-23):

     Compared to GPT-2, GPT-3 improves performance on character-level tasks like rhyming, alliteration, punning, anagrams or permutations, acrostic poems, and arithmetic less than expected, despite being very good at many other closely-related kinds of writings like satire.

     Why? A plausible explanation is an obscure technical detail: as a performance optimization, GPT does not see characters but sub-word-chunks called “byte-pair encodings” (BPEs). Because GPTs never see characters but opaque partial-words, which vary chaotically based on the specific word and even the surrounding context, they are unable to easily learn about character-level aspects of language, like similar spellings or sounds, and are forced to learn relationships much more indirectly, like by brute-force memorizing of pairs of words.

     Some experiments with reformatting GPT-3’s poorest-performing tasks to avoid inconsistent BPE encodings of strings shows small to large performance gains, consistent with this theory.

105. “GPT-3: Prompts As Programming”, Gw­ern Bran­wen (2020-06-23):

     The GPT-3 neural network is so large a model in terms of power and dataset that it exhibits qualitatively different behavior: you do not apply it to a fixed set of tasks which were in the training dataset, requiring retraining on additional data if one wants to handle a new task (as one would have to retrain GPT-2); instead, you interact with it, expressing any task in terms of natural language descriptions, requests, and examples, tweaking the prompt until it “understands” & it meta-learns the new task based on the high-level abstractions it learned from the pretraining.

     This is a rather different way of using a DL model, and it’s better to think of it as a new kind of programming, where the prompt is now a “program” which programs GPT-3 to do new things.

106. /GPT-3#weaknesses

107. “GPT-2 Folk Music: Training a Spaceless Model”, Gw­ern Bran­wen (2019-12-12):

     While training a GPT-2-117M on a folk music corpus written in ABC format, persistent syntax errors kept being generated by an otherwise-high-quality model: random spaces would be generated, rendering a music piece either erroneous or lower-quality. Why? It seems to be some issue with the GPT BPE encoder handling of spaces which makes it difficult to emit the right space-separated characters. We found that ABC does not actually require spaces, and we simply removed all spaces from the corpus—noticeably improving quality of generated pieces.

108. “Interacting with GPT–2 to Generate Controlled and Believable Musical Sequences in ABC Notation”, Car­iña Geer­lings, Al­bert Meroño-Peñuela (2020-10-16):

     Generating symbolic music with language models is a promising research area, with potential applications in automated music composition. Recent work shows that Transformer architectures can learn to generate compelling four-instrument scores from large MIDI datasets. In this paper, we re-train the small (117M) GPT-2 model with a large dataset in ABC notation, and generate samples of single-instrument folk music. Our BLEU and ROUGE based quantitative, and survey based qualitative, evaluations suggest that ABC notation is learned with syntactical and semantic correctness, and that samples contain robust and believable n-grams.

109. /GPT-2-music#combined-model-the-session-abcnotation.com

110. /GPT-2-music#samples

111. “Generating MIDI Music With GPT-2: Generating MIDI by converting to ABC and expanding the GPT-2 context window—works, if only just”, Gw­ern Bran­wen (2020-04-25):

     To expand the ABC GPT-2 model to cover a wider variety of musical genres, I turn to the next-most compact widespread music encoding format: MIDI. There are hundreds of thousands of MIDIs which can be decompiled to ABC format, averaging ~10k BPEs—within GPT-2-117M’s feasible context window when trained on TPUs (which permit training of context windows up to 30k wide).

     We compile the ABC from before and 2 large MIDI datasets, and convert to ABC, yielding ~453k usable ABC-MIDI musical files (~5.1GB of text). We trained January–April 2020 on our TPU swarm (with many interruptions), achieving a final loss of ~0.2 (underfit).

     Sampling from the final model is hit-or-miss as it is prone to the likelihood repetition trap and it generates instruments one-by-one so it is common for instruments to be cut off or otherwise broken during sampling (indicating that sampling is increasingly a bigger problem than training for long-range sequence modeling). However, successful pieces are possible, and are musically far more diverse than the folk ABC corpus, with many pleasingly complex samples.

112. /GPT-2-music#midi-dataset

113. /GPT-2-music#midi-samples

114. /GPT-2-preference-learning#bradley-terry-preference-learning

115. /GPT-2-preference-learning#optimization-by-backprop-not-blackbox

116. “Language Models are Unsupervised Multitask Learners”, Alec Rad­ford, Jeffrey Wu, Re­won Child, David Lu­an, Dario Amod­ei, Ilya Sutskever (2019-02-14):

     Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on task-specific datasets.

     We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText. When conditioned on a document plus questions, the answers generated by the language model reach 55 F1 on the CoQA dataset—matching or exceeding the performance of 3 out of 4 baseline systems without using the 127,000+ training examples.

     The capacity of the language model is essential to the success of zero-shot task transfer and increasing it improves performance in a log-linear fashion across tasks. Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting but still underfits WebText. Samples from the model reflect these improvements and contain coherent paragraphs of text.

     These findings suggest a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.

117. https://slatestarcodex.com/2019/02/28/meaningful/

118. “The Go Transformer: Natural Language Modeling for Game Play”, Matthew Cioli­no, David No­ev­er, Josh Kalin (2020-07-07):

     This work applies natural language modeling to generate plausible strategic moves in the ancient game of Go. We train the Generative Pretrained Transformer (GPT-2) to mimic the style of Go champions as archived in Smart Game Format (SGF), which offers a text description of move sequences. The trained model further generates valid but previously unseen strategies for Go. Because GPT-2 preserves punctuation and spacing, the raw output of the text generator provides inputs to game visualization and creative patterns, such as the Sabaki project’s game engine using auto-replays. Results demonstrate that language modeling can capture both the sequencing format of championship Go games and their strategic formations. Compared to random game boards, the GPT-2 fine-tuning shows efficient opening move sequences favoring corner play over less advantageous center and side play. Game generation as a language modeling task offers novel approaches to more than 40 other board games where historical text annotation provides training data (e.g., Amazons & Connect 4/6).

119. “The Chess Transformer: Mastering Play using Generative Language Models”, David No­ev­er, Matt Cioli­no, Josh Kalin (2020-08-02):

     This work demonstrates that natural language transformers can support more generic strategic modeling, particularly for text-archived games. In addition to learning natural language skills, the abstract transformer architecture can generate meaningful moves on a chessboard. With further fine-tuning, the transformer learns complex gameplay by training on 2.8 million chess games in Portable Game Notation. After 30,000 training steps, OpenAI’s Generative Pre-trained Transformer (GPT-2) optimizes weights for 774 million parameters. This fine-tuned Chess Transformer generates plausible strategies and displays game formations identifiable as classic openings, such as English or the Slav Exchange. Finally, in live play, the novel model demonstrates a human-to-transformer interface that correctly filters illegal moves and provides a novel method to challenge the transformer’s chess strategies. We anticipate future work will build on this transformer’s promise, particularly in other strategy games where features can capture the underlying complex rule syntax from simple but expressive player annotations.

120. “Decompiler”, Eng­lish Wikipedia (2020-12-22):

     A decompiler is a computer program that takes an executable file as input, and attempts to create a high level source file which can be recompiled successfully. It is therefore the opposite of a compiler, which takes a source file and makes an executable. Decompilers are usually unable to perfectly reconstruct the original source code, and as such, will frequently produce obfuscated code. Nonetheless, decompilers remain an important tool in the reverse engineering of computer software.