About This Website

Meta page describing gwern.net site ideals of stable long-term essays which improve over time; technical decisions using Markdown and static hosting; idea sources and writing methodology; metadata definitions; semi-annual web traffic statistics; copyright license
topics: personal, psychology, archiving, statistics, predictions, meta, shell, R, Bayes, Google
created: 01 Oct 2010; modified: 4 Jan 2019; status: finished; confidence: highly likely; importance: 3

This page is about gwern.net; for information about me, see Links.

The Content

“Ah! let not Censure term our fate our choice, / The stage but echoes back the public’s voice; / The drama’s laws the drama’s patrons give, / For we that live to please must please to live.”

Samuel Johnson (“Prologue at the Opening of Drury Lane Theatre”)

The content here varies from statistics to psychology to self-experiments/Quantified Self to philosophy to poetry to programming to anime to investigations of online drug markets or leaked movie scripts (or two topics at once: anime & statistics or anime & criticism or heck anime & statistics & criticism!). It is everything I felt worth writing for the past few years that didn’t fit somewhere like Wikipedia or was already written—“…I realised that I wanted to read about them what I myself knew. More than this—what only I knew. Deprived of this possibility, I decided to write about them. Hence this book.”1 I never expected to write so much, but I discovered that once I had a hammer, nails were everywhere, and that supply creates its own demand2. I believe that someone who has been well-educated will think of something worth writing at least once a week; to a surprising extent, this has been true. (I added ~130 documents to this repository over the first 3 years.) There are many benefits to keeping notes as they allow one to accumulate confirming and especially contradictory evidence3, and even drafts can be useful so you Don’t Repeat Yourself or simply decently respect the opinions of mankind:

“Special knowledge can be a terrible disadvantage if it leads you too far along a path you cannot explain anymore.”

Brian Herbert (Dune: House Harkonnen)

One of my personal interests is applying the idea of the Long Now. What and how do you write a personal site with the long-term in mind? We live most of our lives in the future, and the actuarial tables give me until the 2070–2080s, excluding any benefits from caloric restriction/intermittent fasting or projects like SENS. It is a common-place in science fiction4 that longevity would cause widespread risk aversion. But on the other hand, it could do the opposite: the longer you live, the more long-shots you can afford to invest in. Someone with a timespan of 70 years has reason to protect against black swans—but also time to look for them.5 It’s worth noting that old people make many short-term choices, as reflected in increased suicide rates and reduced investment in education or new hobbies, and this is not due solely to the ravages of age but the proximity of death—the HIV-infected (but otherwise in perfect health) act similarly short-term.6

What sort of writing could you create if you worked on it (be it ever so rarely) for the next 60 years? What could you do if you started now?7

"Of all the books I have delivered to the presses, none, I think, is as personal as the straggling collection mustered for this hodgepodge, precisely because it abounds in reflections and interpolations. Few things have happened to me, and I have read a great many. Or rather, few things have happened to me more worth remembering than Schopenhauer’s thought or the music of England’s words.

A man sets himself the task of portraying the world. Through the years he peoples a space with images of provinces, kingdoms, mountains, bays, ships, islands, fishes, rooms, instruments, stars, horses, and people. Shortly before his death, he discovers that that patient labyrinth of lines traces the image of his face."8

Long Site

“The Internet is self destructing paper. A place where anything written is soon destroyed by rapacious competition and the only preservation is to forever copy writing from sheet to sheet faster than they can burn. If it’s worth writing, it’s worth keeping. If it can be kept, it might be worth writing…If you store your writing on a third party site like Blogger, Livejournal or even on your own site, but in the complex format used by blog/wiki software du jour you will lose it forever as soon as hypersonic wings of Internet labor flows direct people’s energies elsewhere. For most information published on the Internet, perhaps that is not a moment too soon, but how can the muse of originality soar when immolating transience brushes every feather?”

Julian Assange (“Self destructing paper”, 5 December 2006)

Keeping the site running that long is a challenge, and leads to the recommendations for Resilient Haskell Software: 100% FLOSS software9, open standards for data, textual human-readability, avoiding external dependencies1011, and staticness12.

Preserving the content is another challenge. Keeping the content in a DVCS like git protects against file corruption and makes it easier to mirror the content; regular backups13 help. I have taken additional measures: WebCitation has archived most pages and almost all external links; the Internet Archive is also archiving pages & external links14. (For details, read Archiving URLs.)

One could continue in this vein, devising ever more powerful & robust storage methods (perhaps combine the DVCS with forward error correction through PAR2, a la bup), but what is one to fill the storage with?

Long Content

“What has been done, thought, written, or spoken is not culture; culture is only that fraction which is remembered.”

Gary Taylor (The Clock of the Long Now; emphasis added)15

‘Blog posts’ might be the answer. But I have read blogs for many years and most blog posts are the triumph of the hare over the tortoise. They are meant to be read by a few people on a weekday in 2004 and never again, and are quickly abandoned—and perhaps as Assange says, not a moment too soon. (But isn’t that sad? Isn’t it a terrible ROI for one’s time?) On the other hand, the best blogs always seem to be building something: they are rough drafts—works in progress16. So I did not wish to write a blog. Then what? More than just “evergreen content”, what would constitute Long Content as opposed to the existing culture of Short Content? How does one live in a Long Now sort of way?17

It’s shocking to find how many people do not believe they can learn, and how many more believe learning to be difficult. Muad’Dib knew that every experience carries its lesson.18

My answer is that one uses such a framework to work on projects that are too big to work on normally or too tedious. (Conscientiousness is often lacking online or in volunteer communities19 and many useful things go undone.) Knowing your site will survive for decades to come gives you the mental wherewithal to tackle long-term tasks like gathering information for years, and such persistence can be useful20—if one holds onto every glimmer of genius for years, then even the dullest person may look a bit like a genius himself21. (Even experienced professionals can only write at their peak for a few hours a day—usually first thing in the morning, it seems.) Half the challenge of fighting procrastination is the pain of starting—I find when I actually get into the swing of working on even dull tasks, it’s not so bad. So this suggests a solution: never start. Merely have perpetual drafts, which one tweaks from time to time. And the rest takes care of itself. I have a few examples of this:

  1. DNB FAQ:

    When I read in Wired in 2008 that the obscure working memory exercise called dual n-back (DNB) had been found to increase IQ substantially, I was shocked. IQ is one of the most stubborn properties of one’s mind, one of the most fragile22, the hardest to affect positively23, but also one of the most valuable traits one could have24; if the technique panned out, it would be huge. Unfortunately, DNB requires a major time investment (as in, half an hour daily); which would be a bargain—if it delivers. So, to do DNB or not?

    Questions of great import like this are worth studying carefully. The wheels of academia grind exceeding slow, and only a fool expects unanimous answers from fields like psychology. Any attempt to answer the question ‘is DNB worthwhile?’ will require years and cover a breadth of material. This FAQ on DNB is my attempt to cover that breadth over those years.

  2. Neon Genesis Evangelion notes and essay draft:

    I have been discussing NGE since 2004. The task of interpreting Eva is very difficult; the source works themselves are a major time-sink25, and there are thousands of primary, secondary, and tertiary works to consider—personal essays, interviews, reviews, etc. The net effect is that many Eva fans ‘know’ certain things about Eva, such as End of Evangelion not being a grand ‘screw you’ statement by Hideaki Anno or that the TV series was censored, but they no longer have proof. Because each fan remembers a different subset, they have irreconcilable interpretations. (Half the value of the page for me is having a place to store things I’ve said in countless fora which I can eventually turn into something more systematic.)

    To compile claims from all those works, to dig up forgotten references, to scroll through microfilms, buy issues of defunct magazines—all this is enough work to shatter the heart of the stoutest salaryman. Which is why I began years ago and expect not to finish for years to come. (Finishing by 2020 seems like a good prediction.)

  3. Cloud Nine: Years ago I was reading the papers of the economist Robin Hanson. I recommend his work highly; even if they are wrong, they are imaginative and some of the finest speculative fiction I have read. (Except they were non-fiction.) One night I had a dream in which I saw in a flash a medieval city run in part on Hansonian grounds; a steampunk version of his futarchy. A city must have another city as a rival, and soon I had remembered the strange ’90s idea of assassination markets, which was easily tweaked to work in a medieval setting. Finally, between them, was one of my favorite proposals, Buckminster Fuller’s cloud nine megastructure.

    I wrote several drafts but always lost them. Sad26 and discouraged, I abandoned it for years. This fear leads straight into the next example.

  4. A Book reading list:

    Once, I didn’t have to keep reading lists. I simply went to the school library shelf where I left off and grabbed the next book. But then I began reading harder books, and they would cite other books, and sometimes would even have horrifying lists of hundreds of other books I ought to read (‘bibliographies’). I tried remembering the most important ones but quickly forgot. So I began keeping a book list on paper. I thought I would throw it away in a few months when I read them all, but somehow it kept growing and growing. I didn’t trust computers to store it before27, but now I do, and it lives on in digital form (currently on Goodreads—because they have export functionality). With it, I can track how my interests evolved over time28, and what I was reading at the time. I sometimes wonder if I will read them all even by 2070.

What is next? So far the pages will persist through time, and they will gradually improve over time. But a truly Long Now approach would be to make them be improved by time—make them more valuable the more time passes. (Stewart Brand remarks in The Clock of the Long Now that a group of monks carved thousands of scriptures into stone, hoping to preserve them for posterity—but posterity would value far more a carefully preserved collection of monk feces, which would tell us countless valuable things about important phenomenon like global warming.)

One idea I am exploring is adding long-term predictions like the ones I make on PredictionBook.com. Many29 pages explicitly or implicitly make predictions about the future. As time passes, predictions would be validated or falsified, providing feedback on the ideas.30

For example, the Evangelion essay’s paradigm implies many things about the future movies in Rebuild of Evangelion31; The Melancholy of Kyon is an extended prediction32 of future plot developments in The Melancholy of Haruhi Suzumiya series; Haskell Summer of Code has suggestions about what makes good projects, which could be turned into predictions by applying them to predict success or failure when the next Summer of Code choices are announced. And so on.

I don’t think “Long Content” is simply for working on things which are equivalent to a “monograph” (a work which attempts to be an exhaustive exposition of all that is known—and what has been recently discovered—on a single topic), although monographs clearly would benefit from such an approach. If I write a short essay cynically remarking on, say, Al Gore and predicting he’d sell out and registered some predictions and came back 20 years later to see how it worked out, I would consider this “Long Content” (it gets more interesting with time, as the predictions reach maturation); but one couldn’t consider this a “monograph” in any ordinary sense of the word.

One of the ironies of this approach is that as a transhumanist, I assign non-trivial probability to the world undergoing massive change during the 21st century due to any of a number of technologies such as artificial intelligence (such as mind uploading33) or nanotechnology; yet here I am, planning as if I and the world were immortal.

I personally believe that one should “think Less Wrong and act Long Now”, if you follow me. I diligently do my daily spaced-repetition review and n-backing; I carefully design my website and writings to last decades, actively think about how to write material that improves with time, and work on writings that will not be finished for years (if ever). It’s a bit schizophrenic since both are totalized worldviews with drastically conflicting recommendations about where to invest my time. It’s a case of high discount rates versus low discount rates; and one could fairly accuse me of committing the sunk cost fallacy, but then, I’m not sure that sunk cost fallacy is a fallacy (certainly, I have more to show for my wasted time than most people).

The Long Now views its proposals like the Clock and the Long Library and seedbanks as insurance—in case the future turns out to be surprisingly unsurprising. I view these writings similarly. If Ray Kurzweil’s most ambitious predictions turn out right and the Singularity happens by 2050 or so, then much of my writings will be moot, but I will have all the benefits of said Singularity; if the Singularity never happens or ultimately pays off in a very disappointing way, then my writings will be valuable to me. By working on them, I hedge my bets.

Finding my ideas

To the extent I personally have any method for ‘getting started’ on writing something, it’s to pay attention to anytime you find yourself thinking, “how irritating that there’s no good webpage/Wikipedia article on X” or “I wonder if Y” or “has anyone done Z” or “huh, I just realized that A!” or “this is the third time I’ve had to explain this, jeez.”

The DNB FAQ started because I was irritated people were repeating themselves on the dual n-back mailing list; the modafinil article started because it was a pain to figure out where one could order modafinil; the trio of Death Note articles (Anonymity, Ending, Script) all started because I had an amusing thought about information theory; the Silk Road 1 page was commissioned after I growsed about how deeply sensationalist & shallow & ill-informed all the mainstream media articles on the Silk Road drug marketplace were (similarly for Bitcoin is Worse is Better); my Google survival analysis was based on thinking it was a pity that Arthur’s Guardian analysis was trivially & fatally flawed; and so on and so forth.

None of these seems special to me. Anyone could’ve compiled the DNB FAQ; anyone could’ve kept a list of online pharmacies where one could buy modafinil; someone tried something similar to my Google shutdown analysis before me (and the fancier statistics were all standard tools). If I have done anything meritorious with them, it was perhaps simply putting more work into them than someone else would have; to quote Teller:

“I think you’ll see what I mean if I teach you a few principles magicians employ when they want to alter your perceptions…Make the secret a lot more trouble than the trick seems worth. You will be fooled by a trick if it involves more time, money and practice than you (or any other sane onlooker) would be willing to invest.”

“My partner, Penn, and I once produced 500 live cockroaches from a top hat on the desk of talk-show host David Letterman. To prepare this took weeks. We hired an entomologist who provided slow-moving, camera-friendly cockroaches (the kind from under your stove don’t hang around for close-ups) and taught us to pick the bugs up without screaming like preadolescent girls. Then we built a secret compartment out of foam-core (one of the few materials cockroaches can’t cling to) and worked out a devious routine for sneaking the compartment into the hat. More trouble than the trick was worth? To you, probably. But not to magicians.”

Besides that, I think after a while writing/research can be a virtuous circle or autocatalytic. If one were to look at my repo statistics, you see that I haven’t always been writing as much. What seems to happen is that as I write more:

  • I learn more tools

    eg. I learned basic meta-analysis in R to answer what all the positive & negative n-back studies summed to, but then I was able to use it for iodine; I learned linear models for analyzing MoR reviews but now I can use them anywhere I want to, like in my Touhou draft material.

    The “Feynman method” has been facetiously described as “find a problem; think very hard; write down the answer”, but Gian-Carlo Rota gives the real one:

    Richard Feynman was fond of giving the following advice on how to be a genius. You have to keep a dozen of your favorite problems constantly present in your mind, although by and large they will lay in a dormant state. Every time you hear or read a new trick or a new result, test it against each of your twelve problems to see whether it helps. Every once in a while there will be a hit, and people will say: “How did he do it? He must be a genius!”

  • I internalize a habit of noticing interesting questions that flit across my brain

    eg. in March 2013 while meditating: “I wonder if more doujin music gets released when unemployment goes up and people may have more spare time or fail to find jobs? Hey! That giant Touhou music torrent I downloaded, with its 45000 songs all tagged with release year, could probably answer that!” (One could argue that these questions probably should be ignored and not investigated in depth—Teller again—nevertheless, this is how things work for me.)

  • if you aren’t writing, you’ll ignore useful links or quotes; but if you stick them in small asides or footnotes as you notice them, eventually you’ll have something bigger.

    I grab things I see on Google Alerts & Scholar, Pubmed, Reddit, Hacker News, my RSS feeds, books I read, and note them somewhere until they amount to something. (An example would be my slowly accreting citations on IQ and economics.)

  • people leave comments, ping me on IRC, send me emails, or leave anonymous messages, all of which help

    Some examples of this come from my most popular page, on Silk Road 1:

    1. an anonymous message led me to investigate a vendor in depth and ponder the accusation leveled against them; I wrote it up and gave my opinions and thus I got another short essay to add to my SR page which I would not have had otherwise (and I think there’s a <20% chance that in a few years this will pay off and become a very interesting essay).
    2. CMU’s Nicholas Christin, who wrote a paper by scraping SR for many months and giving all sorts of overall statistics, emailed me to point out I was citing inaccurate figures from the first version of his paper. I thanked him for the correction and while I was replying, mentioned I had a hard time believing his paper’s claims about the extreme rarity of scams on SR as estimated through buyer feedback. After some back and forth and suggesting specific mechanisms how the estimates could be positively biased, he was able to check his database and confirmed that there was at least one very large omission of scams in the scraped data and there was probably a general undersampling; so now I have a more accurate feedback estimate for my SR page (important for estimating risk of ordering) and he said he’ll acknowledge me in the/a paper, which is nice.

Information organizing

Occasionally people ask how I manage information and read things.

  1. For quotes or facts which are very important, I employ spaced repetition by adding them to my Mnemosyne

  2. I keep web clippings in Evernotes; I also excerpt from research papers & books, and miscellaneous sources. This is useful for targeted searches when I remember a fact but not where I learned it, and for storing things which I don’t want to memorize but which have no logical home in my website or LW or elsewhere. It is also helpful for writing my book reviews and the monthly newsletter, as I can read through my book excerpts to remind myself of the highlights and at the end of the month review clippings from papers/webpages to find good things to reshare which I was too busy at the time to do so or was unsure of its importance. I don’t make any use of more complex Evernote features.

    I periodically back up my Evernote using the Linux client Nixnote’s export feature. (I made sure there was a working export method before I began using Evernote, and use it only as long as Nixnote continues to work.)

    My workflow for dealing with PDFs, as of late 2014, is:

    1. if necessary, jailbreak the paper using Libgen or a university proxy, then upload a copy to Dropbox, named year-author.pdf
    2. read the paper, making excerpts as I go
    3. store the metadata & excerpts in Evernote, possibly copying over to my Google+ account if I think it’s good or otherwise interesting
    4. if useful, integrate into gwern.net with its title/year/author metadata, adding a local fulltext copy if the paper had to be jailbroken, otherwise rely on my custom archiving setup to preserve the remote URL
    5. hence, any future searches for the filename / title / key contents should result in hits either in my Evernote or gwern.net
  3. Web pages are archived & backed up by my custom archiving setup. This is intended mostly for fixing dead links (eg to recover the fulltext of the original URL of an Evernote clipping).

  4. I don’t have any special book reading techniques. For really good books I excerpt from each chapter and stick the quotes into Evernote.

  5. I store insights and thoughts in various pages as parenthetical comments, footnotes, and appendices. If they don’t fit anywhere, I dump them in Notes.

  6. Larger masses of citations and quotes typically get turned into pages.

  7. I make heavy use of RSS subscriptions for news. For that, I am currently using Liferea. (Not that I’m hugely thrilled about it. Google Reader was much better.)

  8. For projects and followups, I use reminders in Google Calendar.

  9. For recording personal data, I automate as much as possible (eg Zeo and arbtt) and I make a habit of the rest—getting up in the morning is a great time to build a habit of recording data because it’s a time of habits like eating breakfast and getting dressed.

Hence, to refind information, I use a combination of Google, Evernote, grep (on the gwern.net files), occasionally Mnemosyne, and a good visual memory.

As far as writing goes, I do not use note-taking software or things like FreeMind or org-mode—not that I think they are useless but I am worried about whether they would ever repay the large upfront investments of learning/tweaking or interfere with other things. Instead, I occasionally compile outlines of articles from comments on LW/Reddit/IRC, keep editing them with stuff as I remember them, search for relevant parts, allow little thoughts to bubble up while meditating, and pay attention to when I am irritated at people being wrong or annoyed that a particular topic hasn’t been written down yet.

Confidence tags

Most of the metadata in each page is self-explanatory: the date is the last time the page was meaningfully modified34, the tags are categorization, etc. The “status” tag describes the state of completion: whether it’s a pile of links & snippets & “notes”, or whether it is a “draft” which at least has some structure and conveys a coherent thesis, or it’s a well-developed draft which could be described as “in progress”, and finally when a page is done—in lieu of additional material turning up—it is simply “finished”.

The “confidence” tag is a little more unusual. I stole the idea from Muflax’s “epistemic state” tags; I use the same meaning for “log” for collections of data or links (“log entries that simply describe what happened without any judgment or reflection”) personal or reflective writing can be tagged “emotional” (“some cluster of ideas that got itself entangled with a complex emotional state, and I needed to externalize it to even look at it; in no way endorsed, but occasionally necessary (similar to fiction)”), and “fiction” needs no explanation (every author has some reason for writing the story or poem they do, but not even they always know whether it is an expression of their deepest fears, desires, history, or simply random thoughts). I drop his other tags in favor of giving my subjective probability using the “Kesselman List of Estimative Words”:

  1. “certain”
  2. “highly likely”
  3. “likely”
  4. “possible” (my preference over Kesselman’s “Chances a Little Better [or Less]”)
  5. “unlikely”
  6. “highly unlikely”
  7. “remote”
  8. “impossible”

These are used to express my feeling about how well-supported the essay is, or how likely it is the overall ideas are right. (Of course, an interesting idea may be worth writing about even if very wrong, and even a long shot may be profitable to examine if the potential payoff is large enough.)

Importance tags

An additional useful bit of metadata would be distinction between things which are trivial and those which are about more important topics which might change your life. Using my interactive sorting tool Resorter, I’ve ranked pages in deciles from 0–10 on how important the topic is to myself, the intended reader, or the world. For example, topics like embryo selection for traits such as intelligence or evolutionary pressures towards autonomous AI are vastly more important, and be ranked 10, than some poems or a dream or someone’s small nootropics self-experiment, which would be ranked 0–1.

Writing checklist

It turns out that writing essays (technical or philosophical) is a lot like writing code—there are so many ways to err that you need a process with as much automation as possible. My current checklist for finishing an essay:

Markdown checker

I’ve found that many errors in my writing can be caught by some simple scripts, which I’ve compiled into a shell script, markdown-lint.sh.

My linter does:

  1. checks for corrupted non-text binary files

  2. checks a blacklist of domains which are either dead (eg Google+) or have a history of being unreliable (eg ResearchGate, NBER, PNAS); such links need35 to either be fixed, pre-emptively mirrored, or removed entirely.

    • a special case is PDFs hosted on IA; the IA is reliable, but I try to rehost such PDFs so they’ll show up in Google/Google Scholar for everyone else.
  3. Broken syntax: I’ve noticed that when I make Markdown syntax errors, they tend to be predictable and show up either in the original Markdown source, or in the rendered HTML. Two common source errors:


    And the following should rarely show up in the final rendered HTML:

     " _ "
     "<-- "

    Similarly, I sometimes slip up in writing image/document links so any link starting https://www.gwern.net or ~/wiki/ or /home/gwern/ is probably wrong. There are a few Pandoc-specific issues that should be checked for too, like duplicate footnote names and images without separating newlines or unescaped dollar signs (which can accidentally lead to sentences being rendered as TeX).

    A final pass with htmltidy finds many errors which slip through, like incorrectly-escaped URLs.

  4. Flag dangerous language: Imperial units are deprecated, but so too is the misleading language of NHST statistics (if one must talk of “significance” I try to flag it as “statistically-significant” to warn the reader). I also avoid some other dangerous words like “obvious” (if it is really is, why do I need to say it?).

  5. Bad habits:

    • proselint (with some checks disabled because they play badly with Markdown documents)
    • Another static warning is checking for too-long lines (most common in code blocks, although sometimes broken indentation will cause this) which will cause browsers to use scrollbars, for which I’ve written a Pandoc script,
    • one for a bad habit of mine—too-long footnotes
  6. duplicate and hidden-PDF URLs: a URL being linked multiple times is sometimes an error (too much copy-paste or insufficiently edited sections); PDF URLs should receive a visual annotation warning the reader it’s a PDF, but the CSS rules, which catch cases like .pdf$, don’t cover cases where the host quietly serves a PDF anyway, so all URLs are checked. (A URL which is a PDF can be made to trigger the PDF rule by appending #pdf.)

  7. broken links are detected with linkchecker. The best time to fix broken links is when you’re already editing a page.

While this throws many false positives, those are easy to ignore, and the script fights bad habits of mine while giving me much greater confidence that a page doesn’t have any merely technical issues that screw it up (without requiring me to constantly reread pages every time I modify them, lest an accidental typo while making an edit breaks everything).

Anonymous feedback

Back in November 2011, lukeprog posted “Tell me what you think of me” where he described his use of a Google Docs form for anonymous receipt of textual feedback or comments. Typically, most forms of communication are non-anonymous, or if they are anonymous, they’re public. One can set up pseudonyms and use those for private contact, but it’s not always that easy, and is definitely a series of trivial inconveniences (if anonymous feedback is not solicited, one has to feel it’s important enough to do and violate implicit norms against anonymous messages; one has to set up an identity; one has to compose and send off the message, etc).

I thought it was a good idea to try out, and on 8 November 2011, I set up my own anonymous feedback form and stuck it in the footer of all pages on gwern.net where it remains to this day. I did wonder if anyone would use the form, especially since I am easy to contact via email, use multiple sites like Reddit or Lesswrong, and even my Disqus comments allow anonymous comments—so who, if anyone, would be using this form? I scheduled a followup in 2 years on 30 November 2013 to review how the form fared.

754 days, 2.884m page views, and 1.350m unique visitors later, I have received 116 pieces of feedback (mean of 24.8k visits per feedback). I categorize them as follows in descending order of frequency:

  • Corrections, problems (technical or otherwise), suggested edits: 34
  • Praise: 31
  • Question/request (personal, tech support, etc): 22
  • Misc (eg gibberish, socializing, Japanese): 13
  • Criticism: 9
  • News/suggestions: 5
  • Feature request: 4
  • Request for cybering: 1
  • Extortion: 1 (see my blackmail page dealing with the September 2013 incident)

Some submissions cover multiple angles (they can be quite long), sometimes people double-submitted or left it blank, etc, so the numbers won’t sum to 116.

In general, a lot of the corrections were usable and fixed issues of varying importance, from typos to the entire site’s CSS being broken due to being uploaded with the wrong MIME type. One of the news/suggestion feedbacks was very valuable, as it lead to writing the Silk Road mini-essay “A Mole?” A lot of the questions were a waste of my time; I’d say half related to Tor/Bitcoin/Silk-Road. (I also got an irritating number of emails from people asking me to, say, buy LSD or heroin off SR for them.) The feature requests were usually for a better RSS feed, which I tried to oblige by starting the Changelog page. The cybering and extortion were amusing, if nothing else. The praise was good for me mentally, as I don’t interact much with people.

I consider the anonymous feedback form to have been a success, I’m glad lukeprog brought it up on LW, and I plan to keep the feedback form indefinitely.

Feedback causes

One thing I wondered is whether feedback was purely a function of traffic (the more visits, the more people who could see the link in the footer and decide to leave a comment), or more related to time (perhaps people returning regularly and eventually being emboldened or noticing something to comment on). So I compiled daily hits, combined with the feedback dates, and looked at a graph of hits:

Hits over time for gwern.net
Hits over time for gwern.net

The hits are heavily skewed by Hacker News & Reddit traffic spikes, and probably should be log transformed. Then I did a logistic regression on hits, log hits, and a simple time index:

The logged hits works out better than regular hits, and survives to the simplified model. And the traffic influence seems much larger than the time variable (which is, curiously, negative).

Technical aspects


October 2010–February 2011

My editing activity, as generated by darcs-graph:

Plot of patch creations (y-axis) versus date (x-axis): October to February
Plot of patch creations (y-axis) versus date (x-axis): October to February


“An audience, even an audience of one, is always to be treasured and respected.”

Adalric Brandl (“Uhl Eharl Khoehng”, Patricia A. Jackson)

Popularity-wise, Google Analytics reports that over the 150 days between 1 October 2010 and 28 February 2011, there were 4,346 page-views (average 30/day):

Plot of page-hits (y-axis) versus date (x-axis)
Plot of page-hits (y-axis) versus date (x-axis)

The most popular pages were36:

  1. DNB FAQ: 1,180
  2. Modafinil: 644
  3. Haskell Summer of Code: 241
  4. Nootropics experiments log: 108
  5. The Melancholy of Kyon: 104
  6. Spaced repetition: 101
  7. Links: 96

The rankings are not as I would prefer (I imagine Internet archivist Jason Scott Sadofsky feels much the same way about Sockington), but it’s pretty clear that people enjoy my more practical articles the most.

February 2011–July 2011

darcs-graph for this period:

Plot of patch creations (y-axis) versus date (x-axis): Repository creation to July
Plot of patch creations (y-axis) versus date (x-axis): Repository creation to July


“Streaming in the wind / the smoke from Fuji / vanishes in the sky; / I know not where / these thoughts of mine go, either.”

the monk Saigyô (Shin Kokin Wakashu XVII: #1615)

Google Analytics reports that over the 124 days between 28 February 2011 and 2 July 2011, there were 42,410 page-views (average 342/day):

Plot of page-hits (y-axis) versus date (x-axis)
Plot of page-hits (y-axis) versus date (x-axis)

The most popular pages ranking changed considerably; while the DNB FAQ maintained its pre-eminent popularity, 3 new pages bumped out ‘Links’, ‘Spaced repetition’, ‘The Melancholy of Kyon’, and ‘Haskell Summer of Code’. I am a little surprised that my 2 Death Note essays seemed to’ve struck a chord, and even more surprised that my sloppy & random & un-rigorous notes about nootropics would be consistently popular:

  1. DNB FAQ: 10,406
  2. home/main page: 4,189
  3. Modafinil: 3,231
  4. Girl Scouts and good governance: 2,633
  5. Death Note’s Ending: 1,779
  6. Death Note Anonymity: 1,366
  7. Nootropics: 2,056
  8. Archiving GitHub: 706


“They accumulate / but there are none to buy them— / these leaves of words / piling up like wares for sale / beneath the Sumiyoshi Pine.”

Shōtetsu; (#180, “Famous Market Town”, Unforgotten Dreams; trans. Steven D. Carter)37

As a writer, I desire feedback. I also want to feel that my work has been of use to people. So while it would be nice if the world beat a path to my website, I recognize that I have to put some effort into marketing my work. I’ve tried a number of methods.

  1. Witcoin: I submitted any number of fairly popular articles but my total Witcoin traffic over this period was 132 visits—a traffic total I could have gotten with one slightly popular link on Reddit or a few links in comments. While I didn’t lose any Bitcoins (because my registration was funded by Kiba’s donation of ฿1) and actually profited ฿2.77, I have spent at least 6 hours figuring out how to use Witcoin, submitting articles, responding to comments, and voting. Not the best use of time.
  2. Google AdWords: initially disappointing, with after 3010 impression, there were still no clicks! It was funded by the $100 coupon for signing up for Google’s Webmaster Tools. Interface is decent given complexity of task, but deeply frustrating to have to wait many weeks for the DNB FAQ and Modafinil ads to be approved or rejected. Finally, almost in June, the DNB FAQ ads were approved and the modafinil ads rejected. From 23 March to 2 July 2011, I paid $21.07 for 98,900 impressions yielding 63 clicks through. (Those visitors only spent an average of <1.5 minutes on the site, too.) Again, not a great investment of time.
  3. StumbleUpon: with just 3 articles ‘stumbled’ (included in the database), specifically DNB FAQ, In Defense Of Inclusionism & Nootropics, StumbleUpon was responsible for 161 visits or 2.77% of all traffic in the period I looked at. How much traffic could I expect with 30 or 40 articles stumbled? Quite a bit. SU has no ‘front page’ like other social news aggregators so traffic is more of a trickle than flood; nevertheless, Death Note Ending” clicked with SU readers and I got >500 readers out of it in a day or two. In total over this time, SU drove 2,257 visits. SU tended to give a pretty steady 30–50 visits a day with rare spikes when an article clicked. The downside is that after looking at SU comments and at how much time they spend on pages38, I have to agree with Arvind Narayanan’s “StumbleUpon Considered Harmful”—SUers do not want quality content but quick content, for the dopamine boost.
  4. Hacker News: “Girl Scouts and good governance” made it to the front page, resulting in 1,727 visits & setting gwern.net traffic records (it is that giant spike in the traffic graph), but apparently minimal viewing of other pages. Further, while I seem to get a modest amount of Reddit traffic from even unsuccessful submissions, HN submissions will sink without a trace. Kiba calls Hacker News a ‘lottery’, but it seems to be one worth playing.
  5. LessWrong is a natural place to post many of my writings. And perhaps unsurprisingly, LW is my second-largest source of traffic, coming in after SU with 1,857 visits. While few of my submissions get upvoted all that highly, most of them drove a fair amount of traffic even in the Discussion ghetto. (Linking in comments also drives a surprising amount of traffic over long periods to my practical articles like on n-back or melatonin.) At some point I hope to have a good Article and see how much of a disparity there is.

July 2011–December 2011

“Res audita perit, litera scripta manet.”


darcs-graph for this period (including 1 January 2012):

Plot of patch creations (y-axis) versus date (x-axis): July 2011 to 2 January 2012
Plot of patch creations (y-axis) versus date (x-axis): July 2011 to 2 January 2012

I ran into a cool post by Christopher Done on a tool that does detailed analysis of patch patterns on a Git repository, GitStats, and this spurred me to create a Git mirror of gwern.net using darcs-to-git. GitStats produces a whole bundle of graphs and figures, some of which I found surprising. (I did not expect to see a large spike on Wednesday and relatively few patches on Saturday, or a spike around 5 PM, as opposed to the early morning.) I think I will update the GitStats output with each output, as a (large) adjunct to the darcs-graph plots.


“…prompt no more the follies you decry, / As tyrants doom their tools of guilt to die; / ’Tis yours this night to bid the reign commence / Of rescu’d Nature, and reviving Sense; / …Bid scenic Virtue form the rising age, / And Truth diffuse her radiance from the stage.”

Samuel Johnson (“Prologue”)

Google Analytics reports, that over the 185 days between 2 July 2011 and 2 January 2012, there were 191,015 page-views (average 1,032/day) by 79,346 visitors for a total of 115,585 visits (average 624/day). This is better than I expected and makes me wonder about my prediction for <2000 average daily visits by 2013 (but it still seems unlikely traffic will triple over the next year).

Plot of page-hits (y-axis) versus date (x-axis)
Plot of page-hits (y-axis) versus date (x-axis)

The main change in page popularity did not surprise me; when I was writing “Silk Road 1”, I knew it would almost certainly be popular given how very popular the Gawker article was but also how lacking in practical details it was, and I also suspected that “Bitcoin is Worse is Better” would be fairly popular as it argued an interesting and controversial thesis (the original and promoted version is hosted on BitcoinWeekly.com, so its hit-count ought to be low). I’m surprised at how much the page gwern.net still gets; a surprising number of people must either visit the main page after reading another article or click on my various blog comments.

  1. Silk Road: 62,167
  2. DNB FAQ: 25,541
  3. home/main page: 16,967
  4. Modafinil: 10,437
  5. Nootropics: 10,219
  6. Spaced repetition: 6,570
  7. Bitcoin is Worse is Better: 4,297
  8. Links: 3,490

More interesting is the other signals of popularity: Zeo Inc. gave me a free set of headbands (worth ~$50) because they liked my Zeo self-experiments, a software engineer/manager contacted me to see about recruiting me, ThinkGum offered me some of their eponymous product for my Nootropics page, and my request for Bitcoin donations has paid off a little with a few donations ฿0.1–1 and one generous donation of ฿20 (worth a bit upwards of $100 at the time; I spent it on modafinil). (This is all intrinsically helpful but I value it mostly because money speaks louder than words.)


I’ve done relatively little in this period compared with the previous period:

  1. I abandoned Witcoin not long after my experiment with it; and now Witcoin is dead, pending a possible open-sourcing of the codebase.
  2. My AdWords credit is mostly expired. For some reason, my click-through rates kept dropping.
  3. StumbleUpon remains a good traffic source (1,430 visits). I continue to ‘stumble’ my new articles when I remember to do so.
  4. Hacker News was responsible for a great deal of my traffic in this period (4,175 visits). Most of it was not my doing, however—whenever I submit links, they do poorly.
  5. LessWrong remains a major traffic driver (6,961 visits); I continue to see a lot of referrals from old posts and comments. Nor do all the links seem to be perceived negatively or as self-promotion by the LW community: I posted an article describing site updates and the article was received well to my surprise, eliciting very favorable reviews of my writings in general. That was nice.
  6. For this period, I did spend a little more effort submitting stuff to Reddit; and I was handsomely rewarded with the Silk Road submission skyrocketing and become one of the all-time most popular articles in the Bitcoin subreddit. Between that and my nootropics articles, Reddit sent me 20,842 visits.

The largest traffic sources are Google at 36,625 visits and direct/no-referrals at 24,118 visits. As I have no idea how to improve these two figures, I ignore it. I write good content, submit it places, supply metadata, and abide by my hackerly principles; I hope that that is all the SEO I need.

January 2012–July 2012

“Uproot your questions from their ground and the dangling roots will be seen. More questions!”

Frank Herbert (“Mentat Zensufi admonition”, Chapterhouse Dune)

darcs-graph for this period (3 January 2012–2 July 2012):

Plot of patch creations (y-axis) versus date (x-axis): January 2012 to 2 July 2012
Plot of patch creations (y-axis) versus date (x-axis): January 2012 to 2 July 2012


“If it were not for the intellectual snobs who pay—in solid cash—the tribute which philistinism owes to culture, the arts would perish with their starving practitioners. Let us thank heaven for hypocrisy.”

Aldous Huxley (“Indian & Burma”, Jesting Pilate, 1926)

The Analytics report records traffic over those 182 days as being substantially increased: to 570,997 page-views (average 3,137/day) - almost 5 times the previous six month period—by 268,031 unique visitors in 366,301 visits (average 2012/day). If this rate continues, I will likely lose my traffic prediction (which doesn’t bother me very much). In particular, the lifetime total page-views is now at 809,000, which would seem to imply that I will break the million page-view mark by the next update! I would be very pleased by that milestone.

Plot of page-hits (y-axis) versus date (x-axis), early 2012
Plot of page-hits (y-axis) versus date (x-axis), early 2012

The popularity ranking remains mainly the same. 2 differences from the past stand out: the sudden popularity of the Zeo and “Death Note Anonymity” articles. Both owe their inclusion to 2 successful front-page appearances on Hacker News. DNA was submitted by someone I don’t know, but I submitted the Zeo page at the conclusion of my first Vitamin D experiment where I concluded that that Vitamin D consumed in the evening did indeed damage my sleep (I then followed up with a second experiment which found that Vitamin D consumed in the morning did not damage my sleep.) I am proud of these two experiments and so I was gratified that Hacker News found them worthwhile too.

  1. Silk Road: 308,895
  2. DNB FAQ: 36,488
  3. home/main page: 32,259
  4. Modafinil: 23,801
  5. Nootropics: 21,970
  6. Death Note Anonymity: 15,860
  7. Zeo sleep self-experiments: 15,841
  8. Spaced repetition: 8,628
  9. Links: 7,431

Donation-wise, I received ~฿2, and a number of Paypal donations: $5, $4, $5, and $100 from a particularly generous LessWronger who wished me to backup my files more securely & remotely. Alexandra Carmichael was impressed by my Zeo experiments and asked me to write an ebook on sleep for an upcoming Quantified Self series of O’Reilly ebooks; we will see how that goes.


Hacker News, Reddit, and LessWrong remain major referral drivers. For recent new pages, I’ve been trying a checklist which includes submission to SU, Google+, Hacker News, Reddit, and LessWrong as appropriate; I am not sure how well it is working since popularity seems very random.

July 2012–January 2013

“All I say is by way of discourse, and nothing by way of advice. I should not speak so boldly if it were my due to be believed.”

Michel de Montaigne (“Of Cripples”, Essays)

darcs-graph for this period (3 July 2012–2 January 2013):

Plot of patch creations (y-axis) versus date (x-axis): July 2012 to 2 January 2013
Plot of patch creations (y-axis) versus date (x-axis): July 2012 to 2 January 2013


Analytics traffic records 758,843 page views by 366,028 unique visitors over the 184 days for a daily average of 4,124.1 page-views, which is double the previous half-year average of 2,012 daily page-views; traffic growth is clearly slowing, though, since the previous half-year had quadruple the traffic compared to its predecessor. My prediction of breaking the million page-view mark came true, by a very large margin: the lifetime total page-views is now 1,568,957 page-views.

Plot of page-hits (y-axis) versus date (x-axis), late 2012
Plot of page-hits (y-axis) versus date (x-axis), late 2012

Popularity rankings have changed a bit: the Death Note essay and my sleep experiments have fallen out of the top 10 (the former because not many people are still linking it, and the latter probably because my latest experiments were relatively boring), replaced by a sidebar link and one of my terrorism-related essays:

  1. Silk Road: 491,934
  2. home/main page: 37,853
  3. Modafinil: 31,047
  4. DNB FAQ: 27,433
  5. Nootropics: 22,015
  6. Drug heuristics/Algernon’s Law: 18,991
  7. Spaced repetition: 11,693
  8. Links: 8,123
  9. About: 8,075
  10. “Terrorism is not about Terror”: 7,369

I am quite surprised that my Slowing Moore’s Law essay does not even make the top 50 pages, given that it deals with an novel thesis on which there’s many interesting things to think and which is easily misunderstood.

Donations: the ebook fell through when O’Reilly decided to cancel the entire series, which was a disappointment; Carmichael had finished her book on mood and is self-publishing, but I haven’t seen tremendous interest in sleep and will probably just roll my draft material into the existing Zeo page. Paypal donations performed outstandingly: I received $10, $25, $10, $15, $200, & $10 ($270). Bitcoiners were not so generous: ฿0.25, ฿0.32, & ฿1 (฿1.57, or $21 at the 2 January 2013 Mt.Gox exchange rate).


To Hacker News, Reddit, and LessWrong, I can add as a major referrer Wikipedia—primarily to the Silk Road article, but also to a few Evangelion-related pages. StumbleUpon has declined to the 10th largest referrer:

  1. HN: 16,886
  2. Reddit: 15,219
  3. Wikipedia: 7,531
  4. LessWrong: 6,332
  5. Facebook: 3,952
  6. brainworkshop.sourceforge.net: 3,733
  7. google.com: 2,649
  8. youtube.com: 1,798
  9. mainstreamlos.tumblr.com: 1,694
  10. stumbleupon.com: 1,239

I haven’t spent much time promoting my content, but improving the site with metadata & writing new content: for example, I tripled the size of my anime-manga hafu census database.

January 2013–July 2013

“I was decimated. To program any more would be pointless. My programs would never live as long as The Trial. A computer will never live as long as The Trial. …What if Amerika was only written for 32-bit PowerPC?”

_why the lucky stiff (CLOSURE)

darcs-graph for this period (3 January 2013–2 July 2013):

Plot of patch creations (y-axis) versus date (x-axis): January 2013 to 2 July 2013
Plot of patch creations (y-axis) versus date (x-axis): January 2013 to 2 July 2013


Analytics traffic records 832,415 page-views by 422,285 unique visitors over 181 days, or 4598 page-views per day. (The lifetime total page-views has thus reached 2,403,807.) This represents ~11% growth in traffic compared to the previous 6-month period of 4,124 dailies, continuing the slowdown trend—possibly the next half-year won’t see any growth, or a decline.

Plot of page-hits (y-axis) versus date (x-axis), early-mid 2013
Plot of page-hits (y-axis) versus date (x-axis), early-mid 2013

Popularity rankings have changed: Silk Road lost a lot of Wikipedia traffic when, during an editing dispute, an administrator deleted it from their article on Silk Road, and it was only restored on 2 July. 2 new statistical essays (Death Note script & Google survival analysis) enter the top 10, but neither were able to dethrone Silk Road as that darknet market continued to thrive & receive media coverage:

  1. Silk Road: 396,993
  2. Modafinil: 58,856
  3. Google shutdowns: 49,907
  4. DNB FAQ: 25,979
  5. Death Note script: 25,792
  6. Nootropics: 22,702
  7. home/main page: 20,051
  8. Links: 16,789
  9. About: 14,746
  10. Spaced repetition: 11,711

Financial: this period was a remarkable period. My contracting work dried up (largely my own fault) and pressed for money, I began exploring alternatives:

  1. the easiest strategy was to turn all my Amazon links for books & nootropics & miscellaneous into affiliate links; they are already there, do not affect readers, and coding it up was as simple as appending a string to links in hakyll.hs by a simple modification of my existing code for easily linking to Wikipedia. This worked as far as it went, which was not very far ($47 in Q1 2013 and $130 Q2 2013).
  2. more painfully, I decided to try Google’s AdSense. While AdWords hadn’t been the most pleasant experience in the world, it was fairly decent, and Google’s ads always seemed reasonable to me in the search engine (before I learned of AdBlock). AdSense has scary language in its terms of service forbidding detailed discussion of revenue, but I should be safe when I say that the mean CTR was 0.17% & the mean CPC was $0.57, ~40% of visitors did not have ad filtering enabled, and so over the 80 days AdSense was enabled, I earned ~$260. Helpful, but not rent-paying.
  3. donations made the difference. After “Google shutdowns” hit the front page of Hacker News, an acquaintance mentioned I could use some money. I received $408 in 14 Paypal donations; through Bitcoin, 1.62btc (then $189) in 4 donations (followup). I am grateful to all the HNers who donated.
  4. Related to that, I was contacted in May by Nava Whiteford with an offer: he had heard my financial target of $300/month and offered that (in bitcoins) in exchange for a sponsorship/banner linking to his graphical terminal HTerm—a combination purchase/donation. This was both larger & more stable than AdSense, so I accepted, and 3 months have thus far passed.


Wikipedia declines in this period due to the aforementioned removal, and more obscure sites become chief referrers:

  1. news.ycombinator.com: 45,186
  2. reddit.com: 34,225
  3. en.wikipedia.org: 16,527
  4. lesswrong.com: 11,177
  5. Twitter: 6,717
  6. facebook.com: 5,675
  7. mainstreamlos.tumblr.com: 5,576
  8. boingboing.net: 4,982
  9. motherboard.vice.com: 4,496
  10. bulletproofexec.com: 4,336

July 2013–January 2014

“The great globe reels in the solar fire, / Spinning the trivial and unique away. / (How all things flash! How all things flare!) / …Time is the school in which we learn / Time is the fire in which we burn…”

Delmore Schwartz (“Calmly We Walk Through This April’s Day”)

darcs-graph for this period (3 July 2013–2 January 2014):

Plot of patch creations (y-axis) versus date (x-axis): July 2013 to 2 January 2014
Plot of patch creations (y-axis) versus date (x-axis): July 2013 to 2 January 2014


Analytics traffic records 770,264 page-views by 340,104 unique visitors over 184 days, or 4186 page-views per day. (Lifetime total: 3,173,172 page-views by 1,472,606 unique visitors.) Since the previous period was 4598/day, that means traffic fell by ~9%. (Traffic would have fallen even more if I hadn’t happened to be engaged in a successful campaign of Hacker News submissions at the time.) My guess is that this is largely attributable to the fall of Silk Road in early October rendering my page of much less interest, and that was one of my most popular pages. It was inevitable, really, but I had hoped SR would last more than <3 years. Traffic may fall or increase during the next period.

Plot of page-hits (y-axis) versus date (x-axis), mid-late 2013
Plot of page-hits (y-axis) versus date (x-axis), mid-late 2013

This rebalancing is evident in the traffic rankings, where SR is still #1 and bigger than #2 (Modafinil), but by a much smaller factor this time (1.26x rather than 6.66x):

  1. Silk Road: 80,233
  2. Modafinil: 63,966
  3. home/main page: 65,203
  4. Melatonin: 50,952
  5. Blackmail attempts: 36,573
  6. Spaced repetition: 35768
  7. LSD microdosing self-experiment: 26,825
  8. Nicotine: 25,370
  9. DNB FAQ: 23,169
  10. Nootropics: 20,883

My finances continued to improve in part due to Whiteford, additional donations (I explored Flattr & Gratipay; the latter seems to be working out better), another more-targeted banner on the modafinil page, and a major appreciation in Bitcoin. (Takes a real load off me.) I thank all my donators. I think the results show in my Changelog.


  1. reddit.com: 148,940
  2. news.ycombinator.com: 84,163
  3. lesswrong.com: 41,511
  4. en.wikipedia.org: 31,967
  5. facebook.com: 20,394
  6. t.co [Twitter]: 19,628
  7. brainworkshop.sourceforge.net: 16,808
  8. bulletproofexec.com: 11,135
  9. google.com [Google+?]: 10,154
  10. mainstreamlos.tumblr.com: 10,128

Interestingly, despite my Hacker News experiment (which resulted in dozens of my pages reaching its main page), news.ycombinator.com is still beaten out traffic-wise by Reddit, by a huge factor (almost double).

January–July 2014

“A courteous handsome young man / well-versed in the Classics and Histories / people address him sir / everyone calls him a scholar / but he hasn’t found a position yet / and he doesn’t know how to farm / in winter he wears a tattered robe / this is how books fool us”

Han-Shan (#28, The Collected Songs of Cold Mountain)

Activity graph39 for this period (3 January 2014–2 July 2014):

Plot of patch creations (y-axis) versus date (x-axis): January to July 2014
Plot of patch creations (y-axis) versus date (x-axis): January to July 2014


Analytics traffic records 370,592 page-views by 162,032 unique visitors over 181 days, or 2047 page-views per day. (Lifetime total: 3,544,489 page-views by 1,628,173 unique visitors.) Since the previous period was 4186/day, that means traffic fell by a disastrous ~50%, defeating my hopes that traffic might recover post-Silk-Road & Hacker News experiment

Plot of page-views (y-axis) versus date (x-axis), early-mid 2014
Plot of page-views (y-axis) versus date (x-axis), early-mid 2014

The traffic rankings show a continue in the plummet of SR, and its dethroning by Modafinil and even the index page, while hosting some early Satoshi Nakamoto drew considerable interest & my “Intellectual Mistakes I Have Made” page was surprisingly linked on Tyler Cowen’s Marginal Revolution blog:

  1. Modafinil: 54,992
  2. home/main page: 43,984
  3. Silk Road: 27,998
  4. Spaced repetition: 27,680
  5. DNB FAQ: 17,833
  6. Nootropics: 15,935
  7. Mistakes: 15,267
  8. Nicotine: 11,936
  9. Wei Dai/Satoshi Nakamoto emails: 9,827
  10. Links: 9,391

On the plus side, it’s hard to see how traffic could fall much more.

Finance-wise, Gratipay has worked out much better than I expected. I thank all my donators.


Hacker News turns out to no longer show up in the referrals analytics due to stripping referrals (apparently related to HTTPS use?), which is unfortunate; I suspect other sites are also “going dark”, given that the numbers are more than halving:

  1. reddit.com: 24,176
  2. marginalrevolution.com: 5,695
  3. bulletproofexec.com: 5,504
  4. disqus.com: 4,051
  5. lesswrong.com: 3,954
  6. longecity.org: 3,051
  7. Twitter: 2,771
  8. boldanddetermined.com: 2,54
  9. vice.com: 1,812
  10. brainworkshop.sourceforge.net: 1,625

I also began a monthly mailing list; with 300 people on it, it doesn’t seem to make a big difference to traffic but it’s a good way to curate my links and non-gwern.net writings, and may be quite useful.

July 2014–January 2015

“Writing is in itself a joy, / Yet saints and sages have long since held it in awe.”

“For it is being, created from a void; / It is sound rung out of profound silence. / In a sheet of paper is contained the infinite, / And, evolved from an inch-sized heart, an endless panorama.”

Lu Chi’s Wen fu (section 4, ‘The Joy of Writing’, of “The Art of Writing”).

Activity graph40 for this period (3 July 2014–2 January 2015):

Plot of patch creations (y-axis) versus date (x-axis): July 2014 to January 2015
Plot of patch creations (y-axis) versus date (x-axis): July 2014 to January 2015


Analytics traffic records 347,482 page-views by 154,599 unique visitors over the 184 days, or 1888 page-views per day for a daily drop of 159 compared to the previous six months. (Lifetime total: 3,891,971 page-views by 1,778,197 unique visitors.) This drop is not nearly as bad before, and may represent a bottom.

Plot of page-views (y-axis) versus date (x-axis), late 2014
Plot of page-views (y-axis) versus date (x-axis), late 2014

Page rankings:

  1. Modafinil: 44,420
  2. home/main page: 41,499
  3. Nootropics: 41,499
  4. Silk Road: 21,863
  5. Spaced repetition: 17,620
  6. Causality: 15,396
  7. DNB FAQ: 14,463
  8. Time-lock crypto: 11,313
  9. Nicotine: 9,804
  10. Links: 8,823

Silk Road declines a little more; the Wei Dai/Satoshi Nakamoto emails & Mistakes pages drop out after their moments in the limelight; and we see 2 new entries: I wrote a statistical essay on quantifying how often correlation≠causality (a topic of burning importance in almost any health-related research), and an older cryptography essay became abruptly more relevant when one of my proposals was implemented using Bitcoin.

Gratipay continues to work well, and I am grateful for all my donators.


I did minimal active promotion of my content, letting my AdWords credit be used up and not spending much effort on Reddit or Hacker News,

Confusingly, HN is back in my referrals:

  1. reddit.com: 27,182
  2. news.ycombinator.com: 18,420
  3. lesswrong.com: 4,583
  4. Twitter: 3,559
  5. slatestarcodex.com: 2,327
  6. Longecity: 2,143
  7. bulletproofexec.com: 2,123
  8. boldanddetermined.com: 1,488
  9. brainworkshop.sourceforge.net: 1,346
  10. disqus.com: 1,276

My monthly mailing list continued, having reached 456 subscribers.

January 2015–July 2015

“The horror of that moment,” the King went on, “I shall never never forget!” “You will, though”, the Queen said, “if you don’t make a memorandum of it.”

Lewis Carroll (Through the Looking-Glass, and What Alice Found There)

Activity graph for this period (3 January 2015–2 July 2015):

Plot of patch creations (y-axis) versus date (x-axis): January 2015 to July 2015
Plot of patch creations (y-axis) versus date (x-axis): January 2015 to July 2015


Analytics traffic records 378,580 page-views by 175,303 unique visitors over the 181 days, or 2091 page-views per day for a daily increase of +203 compared to the previous six months. (Lifetime total: 4,270,551 page-views by 1,949,700 unique visitors.) This is better than the previous drop but may be due mostly to some additional spikes from HN.

Plot of page-views (y-axis) versus date (x-axis), early 2015
Plot of page-views (y-axis) versus date (x-axis), early 2015

Page rankings:

  1. Modafinil: 48,259
  2. home/main page: 48,614
  3. Darknet Market (DNM) arrests: 20,312
  4. Spaced repetition: 17,522
  5. DNB FAQ: 16,076
  6. Silk Road: 15,116
  7. LSD microdosing: 13,673
  8. Nicotine: 9,804
  9. Links: 9,234
  10. Death Note Anonymity: 9,234

The new entry here, “DNM arrests”, is a long-term project in which I compile all the several hundred known and inferable arrests/prosecutions related to Silk Road 1 and its successor DNMs; it can be very difficult to find details for individual cases (often requiring hours reading court documents for a key detail) or keep track of them, and only recently was I able to clear out my backlog and present it to the community. But it’s useful in giving people an authoritative idea of the risks and what people are arrested for, and I’m pleased to see that it’s being read & consulted. “LSD microdosing” seems to be getting more traffic due to some recent media coverage of the idea and people linking to my writeup as (still) the only blinded self-experiment on the topic.

Gratipay, unfortunately, has been forced to change due to financial regulations and is no longer useful for donations. It had a good run but I will probably be switching to Patreon.


As before, I did minimal active promotion of my content.

  1. reddit.com: 27,783
  2. news.ycombinator.com: 11,538
  3. lesswrong.com: 3,264
  4. Twitter (t.co): 2,700
  5. Slate Star Codex: 2,357
  6. Longecity: 1,871
  7. brainworkshop.sourceforge.net: 1,403
  8. Deep Dot Web: 1,226
  9. rs.io: 1,197
  10. Facebook: 1,148

My monthly mailing list has reached 606 subscribers.

July 2015–January 2016

“We will never run out of things to program as long as there is a single program around.”

Alan Perlis (“Epigrams in Programming” 1982)

Activity graph for this period (3 July 2015–3 January 2016):

Plot of patch creations (y-axis) versus date (x-axis): July 2015 to January 2016
Plot of patch creations (y-axis) versus date (x-axis): July 2015 to January 2016


Analytics traffic records 472,244 page-views by 400,305 unique visitors over the 185 days, or 2552 page-views per day for a daily increase of +461 compared to the previous six months. (Lifetime total: 4,742,795 page-views by 2,191,138 unique visitors.)

Plot of page-views (y-axis) versus date (x-axis), late 2015
Plot of page-views (y-axis) versus date (x-axis), late 2015

Page rankings:

  1. Modafinil: 67,152
  2. home/main page: 50708
  3. DNM archives 2011–2015: 44,164
  4. Surprisingly Turing-complete systems: 23,879
  5. LSD microdosing: 22,078
  6. char-rnn for mimicking CSS: 20,581
  7. DNM arrests: 18,524
  8. Nootropics: 15,709
  9. Terrorism is not about Terror: 15,223
  10. Spaced repetition: 14,903
  11. DNB FAQ: 13,954
  12. Silk Road: 11,606

The new entry here, “DNM archives”, is the culmination of my DNM work: mirrors/scrapes of dozens of markets, along with all sorts of archives and datasets related to the DNMs, covering years of market activity (1.2tb in total, to give an idea of scope). The other 2 surprises were a tutorial on how to set up char-rnn on an Amazon EC2 instance and train it to generate remarkably valid CSS (suggesting that it may be possible to train a reinforcement-learning deep neural network to design the visual appearance of websites and optimize the appearance based on visitor experiments), and an old little compilation I made of things exhibiting Turing-completeness which I found bizarre or surprising or disturbing (which naturally led to a lot of nitpicking and debate over definitions).

My Patreon profile, set up after I was forced out of Gratipay, has been a smashing success, with generally ~$300/month in donations.


As before, I did minimal active promotion of my content.

  1. news.ycombinator.com: 51,256
  2. reddit.com: 25,479
  3. Twitter (t.co): 7,748
  4. Slate Star Codex: 3,166
  5. Facebook: 3,044
  6. lesswrong.com: 3,035
  7. Disqus: 2,042
  8. Bullet Proof Exec: 1,931
  9. Longecity: 1,887
  10. Wired: 1,854

(I have no idea why Facebook traffic increased so much, since the referrers are useless.)

My monthly mailing list has reached 857 subscribers.

January 2016–July 2016

“It is easy enough to say that man is immortal simply because he will endure: that when the last ding-dong of doom has clanged and faded from the last worthless rock hanging tideless in the last red and dying evening, that even then there will still be one more sound: that of his puny inexhaustible voice, still talking. I refuse to accept this. I believe that man will not merely endure: he will prevail.”

William Faulkner (1950 Nobel Prize speech)

Activity graph for this period (4 January 2016–3 July 2016):

Plot of patch creations (y-axis) versus date (x-axis): January 2016 to July 2016
Plot of patch creations (y-axis) versus date (x-axis): January 2016 to July 2016


Analytics traffic records 294,009 page-views by 141,795 unique visitors over the 182 days, or 1615 page-views per day for a daily decrease of 937 compared to the previous six months. (Lifetime total: 5,036,804 page-views by 2,329,713 unique visitors.)

Plot of page-views (y-axis) versus date (x-axis), early 2016
Plot of page-views (y-axis) versus date (x-axis), early 2016

Page rankings:

  1. homepage/index: 35,673
  2. Modafinil: 33,928
  3. LSD microdosing: 21,712
  4. DNM arrests: 14,322
  5. Spaced repetition: 14,044
  6. Nootropics: 12,117
  7. DNB FAQ: 10,983
  8. Silk Road: 7,654
  9. About: 6,642
  10. Links: 6,410

My Patreon profile continues to be a success.


As before, I did minimal active promotion of my content.

  1. Reddit: 17,792
  2. Facebook: 4119
  3. Twitter: 2,572
  4. HN: 2,518
  5. Slate Star Codex: 2,467
  6. LessWrong: 2,135
  7. Longecity: 1,555
  8. rs.io: 1,440
  9. brainworkshop.sourceforge.net: 1,035
  10. MetaFilter: 1,018

My monthly mailing list has reached 1053 subscribers.

July 2016–January 2017

“With the people of the past / How I wish I could share the beauty / Of these cherry blossoms. // Have each of us across the years / Left behind this very thought?”

Fujiwara no Teika (“Shogaku Hyakushu”, 1181 AD, trans. Bundy 1990)

Activity graph for this period (4 July 2016–3 January 2017):

Plot of patch creations (y-axis) versus date (x-axis): July 2016–January 2017
Plot of patch creations (y-axis) versus date (x-axis): July 2016–January 2017


Analytics traffic records 339,529 page-views by 172,203 unique visitors over the 184 days, or 1845 page-views per day for a daily increase of 230 compared to the previous six months. (Lifetime total: 5,376,333 page-views by 2,499,949 unique visitors.)

Plot of page-views (y-axis) versus date (x-axis), late 2016
Plot of page-views (y-axis) versus date (x-axis), late 2016

Page rankings:

  1. Nootropics: 43,266
  2. index/main page: 39,161
  3. Spaced repetition: 26,965
  4. Modafinil: 26,849
  5. DNM arrests: 16,279
  6. LSD microdosing: 12,943
  7. DNB FAQ: 10,501
  8. Why Tool AIs Want to Be Agent AIs: 8,026
  9. In Defense of Inclusionism: 7,037
  10. Links: 6,677

My Patreon profile has largely leveled off at a constant $310.


As before, I did minimal active promotion of my content.

  1. HN: 42,341

    Traffic was driven primarily by a few big HN submissions by other people.

  2. Reddit: 16,151

  3. Twitter: 4,915

  4. Facebook: 4,702

  5. rs.io: 1,585

  6. Longecity: 1,175

  7. thethirdwave.co: 1,171

  8. DuckDuckGo search: 990

  9. brainworkshop.sourceforge.net: 912

  10. Wikipedia: 794

My monthly mailing list has reached 1212 subscribers, up ~200.

January 2017–July 2017

“Authors write things down so as to have to think of them less.”

Alain de Botton

Activity graph for this period (4 January 2017–3 July 2017):

Plot of patch creations (y-axis) versus date (x-axis): January 2017–July 2017
Plot of patch creations (y-axis) versus date (x-axis): January 2017–July 2017


Analytics traffic records 228,652 page-views by 112,814 unique visitors over the 181 days, or 1263 page-views per day for a daily decrease of 582 visits compared to the previous six months. (Lifetime total: 5,604,985 page-views by 2,609,657 unique visitors.)

Plot of page-views (y-axis) versus date (x-axis), early 2017
Plot of page-views (y-axis) versus date (x-axis), early 2017

Page rankings:

  1. Modafinil: 16,980
  2. Spaced repetition: 14,484
  3. LSD microdosing: 13,394
  4. DNB FAQ: 10,145
  5. “Ted Chiang’s ‘Story Of Your Life’ Is Not A Time-Travel Story”: 9,760
  6. Nootropics: 9,309
  7. DNM arrests: 8,148
  8. Silk Road: 6,456
  9. Links: 5,668
  10. About: 5,176

Aside from a critical exegesis of “Story Of Your Life” (prompted by its successful movie adaptation), I spent much of early 2017 tweaking gwern.net and working on unpopular statistics projects (particularly “The Kelly Coin-Flipping Game: Exact Solutions via Decision Trees”, which though thorough and highly educational for myself, like “When Should I Check The Mail?”, will never be popular reading). In late 2017, I hope to finish up many draft or in-progress works, which may help maintain traffic.

After rewriting & including social proof & tweeting, my Patreon profile has increased to $600.


As before, I did minimal active promotion of my content. Referrals:

  1. HN: 10,881 (“Story Of Your Life” was an unexpected hit)
  2. Reddit: 9,538
  3. Twitter: 3,702
  4. Facebook: 2,274
  5. LessWrong: 1,568
  6. Slate Star Codex: 1,462
  7. rs.io: 1,315
  8. Longecity: 1,071
  9. Brain Workshop: 861
  10. DuckDuckGo: 820

My monthly mailing list has reached 1419 subscribers, up ~207.

July 2017–January 2018

“It is only the attempt to write down your ideas that enables them to develop.”

Wittgenstein to Drury (Recollections of Wittgenstein, Rhees 1984)

Activity graph for this period (4 July 2017–3 January 2018):

Plot of patch creations (y-axis) versus date (x-axis): July 2017–January 2018
Plot of patch creations (y-axis) versus date (x-axis): July 2017–January 2018


Analytics traffic records 326,852 page-views by 155,532 unique visitors over the 184 days, or 1776 page-views per day for a daily increase of 513 visits compared to the previous six months. (Lifetime total: 5,931,837 page-views by 2,763,402 unique visitors.)

Plot of page-views (y-axis) versus date (x-axis), late 2017
Plot of page-views (y-axis) versus date (x-axis), late 2017

Page rankings (somewhat scrambled due to the big renaming and GA not merging analytics across URLs):

  1. index: 43,863
  2. Banner Ads Considered Harmful: 20,013
  3. Modafinil: 14,340
  4. DNM arrests: 14,115
  5. The Neural Net Tank Urban Legend: 13,808
  6. DNB FAQ: 13,650
  7. Bitcoin Is Worse Is Better: 11,401
  8. Spaced repetition: 12,428
  9. Reasons of State: Why Didn’t Denmark Sell Greenland?: 11,150
  10. Nootropics: 7,615

My A/B test of banner ads turned in an unexpected result of substantial harm, which was of considerable general interest (especially to anyone who runs an ad-supported website), so no surprise that had a lot of traffic. I wrote off the obscure neural net/tank story years ago after some cursory investigation showed it had all the classic traits of an urban legend; but it has kept coming up, and at Anders Sandberg’s request, I wrote it up in considerably more detail, and was surprised that people took such an interest in it and even defended it as probably genuine, in the absence of all evidence—I guess people like debunkings? The final oddity was the resurgence of my old essay on an extremely obscure bit of US historical trivia, that it had tried to acquire Greenland in the early Cold War, where I noted that Denmark’s refusal was a costly one; I don’t know why anyone was reading it, but I was annoyed at the idiotic objections people made, typically either pure fantasy, already refuted like the many suggestions that Greenland oil would yet pay off, or illogical virtue signaling—ironically demonstrating my point. (Considerable effort invested in defining hundreds of redirects for 404 errors didn’t pay off as people kept discovering ever new ways to misspell URLs, but it did allow me to rename pages without destroying traffic by breaking most incoming links, switching from spaces to hyphens in order to idiot-proof them.)

My Patreon profile has increased to $700.


As before, I did minimal active promotion of my content. Referrals:

  1. HN: 26,112
  2. Reddit: 10,715
  3. Marginal Revolution: 8,064
  4. Twitter: 7,111
  5. Slate Star Codex: 5,906
  6. Facebook: 2228
  7. LessWrong: 1,547
  8. The Browser: 1,057 (this would have been higher if The Browser had not carelessly broken its URL due to the space-escaping issue, which prompted my site-wide renaming)
  9. Duck Duck Go: 782
  10. Longecity: 777

My monthly mailing list has reached 1786 subscribers, up ~367.

January 2018–July 2018

“People will have their miracles, their stories, their heroes and heroines and saints and martyrs and divinities to exercise their gifts of affection, admiration, wonder, and worship, and their Judases and devils to enable them to be angry and yet feel that they do well to be angry. Every one of these legends is the common heritage of the human race; and there is only one inexorable condition attached to their healthy enjoyment, which is that no one shall believe them literally. The reading of stories and delighting in them made Don Quixote a gentleman: the believing them literally made him a madman who slew lambs instead of feeding them.”

George Bernard Shaw (A Touchstone For Dogma)

Activity graph for this period (4 January 2018–3 July 2018):

Plot of patch creations (y-axis) versus date (x-axis): January 2018–July 2018
Plot of patch creations (y-axis) versus date (x-axis): January 2018–July 2018


Analytics traffic records 329,057 page-views by 148,481 unique visitors over the 181 days, or 1818 page-views per day for a daily increase of 42 pageviews a day compared to the previous six months. (Lifetime total: 6,236,345 page-views by 2,945,757 unique visitors.)

Plot of page-views (y-axis) versus date (x-axis), early 2018
Plot of page-views (y-axis) versus date (x-axis), early 2018

Page rankings:

  1. index: 43,535
  2. “Laws of Tech: Commoditize Your Complement”: 29,755
  3. Modafinil: 16,339
  4. DNB FAQ: 15,922
  5. Spaced repetition: 15,599
  6. “Danbooru2017: A Large-Scale Crowdsourced and Tagged Anime Illustration Dataset”: 10,531
  7. Nootropics: 8,020
  8. Links: 7,105
  9. DNM arrests: 6,707
  10. Death Note Anonymity: 6,464

“Laws of Tech: Commoditize Your Complement” is based on an old Joel Spolsky essay I found insightful and to explain a lot of puzzling behaviors in Silicon Valley, so I was surprised when, during my SF trip, no one who I mentioned it to in discussions of technology had heard of it; I returned resolved to try to promote it some more by re-emphasizing it and collating a list of examples so no one could fail to see the pattern, and I believe I succeeded. The launch of Danbooru2017 was less satisfying. I kept running into difficulties and delays, most associated with how to format the metadata (provided in a not-too-useful JSON format)—indeed, I had expected to launch it in April 2017 and build my deep learning workstation in December 2017. Instead, it wasn’t until February 2018 that I gave up on the metadata, reasoning that I was letting the perfect be the enemy of better, and until July 2018 that I finished building my workstation with 2 GPUs (after 2 dead motherboards, among other problems). There have been somewhere around 200 downloads of the dataset as far as I can extrapolate from the torrent client upload & rsync logs, but thus far I only know of 1 serious use of it (a CycleGAN paper). I remain hopeful that uses will gradually increase over the next few years, the way they did for my darknet market dataset; in any case, I’ve begun working on some uses of Danbooru2017 myself.

My Patreon profile decreased to $670.



  1. HN: 13,533
  2. Reddit: 6,686
  3. Twitter: 6,529
  4. Slate Star Codex: 6,246
  5. Facebook: 2,830
  6. adme.ru: 2,414
  7. Android Google Search: 788
  8. The Browser: 562
  9. 4chan: 523
  10. Duck Duck Go: 521

My monthly mailing list has reached 2183 subscribers, up ~307. By crossing the 2000 subscriber mark, MailChimp disabled my account and began charging ~$30 a month to send out a single newsletter; as it’s a little hard for me to justify spending anywhere up to $400/year on my newsletter, which I don’t directly make any money on, I had to stop using MailChimp. after looking around some, I switched to TinyLetter, which oddly enough is also run by MailChimp but doesn’t have that arbitrary subscriber limit (though it lacks many features of MailChimp proper).

July 2018–January 2019

“Is not a patron, my lord, one who looks with unconcern on a man struggling for life in the water, and when he has reached ground, encumbers him with help?”

Samuel Johnson (Letter to Chesterfield, 1755)

Activity graph for this period (4 July 2018–3 January 2019):

Plot of patch creations (y-axis) versus date (x-axis): July 2018–January 2019
Plot of patch creations (y-axis) versus date (x-axis): July 2018–January 2019


Analytics traffic records 407,395 page-views by 185,141 unique visitors over the 184 days, or 2214 page-views per day for an increase of 396 pageviews a day compared to the previous six months. (Lifetime total: 6,638,722 page-views by 3,113,460 unique visitors.)

Plot of page-views (y-axis) versus date (x-axis), late 2018
Plot of page-views (y-axis) versus date (x-axis), late 2018

Page rankings:

  1. index: 50,849
  2. “What Is The Morning Writing Effect?”: 27,241
  3. Turing-complete: 26,662
  4. Notes: (for open questions/Littlewood’s Law): 17,706
  5. Spaced repetition: 14,471
  6. Modafinil: 13,958
  7. “Bitcoin Is Worse Is Better”: 13,169
  8. DNB FAQ: 10,344
  9. Links: 9,729
  10. Drug heuristics/Algernon’s Law: 9,724
  11. And honorable mentions for the new pages “Internet Search Tips” which almost made it to the top 10 (#13) despite being released mid-way December 2018, and Cat Sense, Bradshaw 2013: Are We Good Owners? (on cat psychology, genetics, domestication, & dysgenics) at #21.

I was disappointed that two of my new ones had little or no interest (My Little Pony: Immanetizing The Equestrian” & “Multi-Stage Selection Bean Machine JS Demo”), and that Danbooru2017, despite its use in the impressive style2paints v4, saw little uptake. Maybe Danbooru2018 will do better.

My Patreon profile increased to $835.


Referrals saw some unusual changes:

  1. HN: 37,603
  2. Twitter: 10,909
  3. Reddit: 6,046
  4. Slate Star Codex: 4,479
  5. Facebook: 1,389
  6. Github: 1,209
  7. CoinMarketCap: 1,115
  8. TinyLetter: 1,089
  9. Alexey Guzey: 795
  10. LessWrong: 782

My monthly mailing list has reached 2753 subscribers, up ~570.

January 2019–July 2019

“You think you know when you learn, are more sure when you can write, even more when you can teach, but certain when you can program.”

Alan Perlis (“Epigrams in Programming” 1982)

Activity graph for this period (4 January 2019–2 July 2019):

Plot of patch creations (y-axis) versus date (x-axis): January–July 2019
Plot of patch creations (y-axis) versus date (x-axis): January–July 2019


Analytics traffic records 653,876 page-views by 325,842 unique visitors over the 181 days, or 3612 page-views per day for an increase of 1398 pageviews a day compared to the previous six months. (Lifetime total: 7,286,760 page-views by 3,431,759 unique visitors.)

Plot of page-views (y-axis) versus date (x-axis), early 2019
Plot of page-views (y-axis) versus date (x-axis), early 2019

Page rankings:

  1. This Waifu Does Not Exist site writeup: 95,635
  2. index: 97,767
  3. Danbooru2018: 30,936
  4. Complement: 28,165
  5. Spaced-repetition: 26,846
  6. CO2/ventilation sleep experiment: 20,368
  7. Making Anime Faces With StyleGAN tutorial: 17,616
  8. Links: 14,594
  9. Modafinil: 14,488
  10. About: 14,057
  11. Everything Is Correlated: 11,738

The main event in early 2019 was the release of the StyleGAN codebase, which I immediately began using to generate anime faces; it worked even better than I hoped, creating amusing 512px anime faces in a fraction of the time of ProGAN. For a lark, in February I created a simple little static website, ThisWaifuDoesNotExist.net (TWDNE) to show off random face samples, which then went viral, quickly receiving over a million visits (mostly from China). I enhanced TWDNE with additional StyleGAN improvements and anime plot summaries generated by GPT-2, and wrote up the StyleGAN training and the website details as tutorials, which benefit from the still-considerable traffic and have been useful to many people training their own StyleGANs.

My Patreon profile was steady at $838.


Referrals changed dramatically, driven mostly by TWDNE:

  1. thiswaifudoesnotexist.net: 75,707
  2. news.ycombinator.com: 34,761
  3. t.co: 12,744
  4. reddit.com: 7,993
  5. slatestarcodex.com: 4,754
  6. m.facebook.com: 2,476
  7. link.zhihu.com: 1,886
  8. github.com: 1,746
  9. mail01.tinyletterapp.com: 1,342
  10. old.reddit.com: 1,164

My monthly mailing list has reached 3559 subscribers, up ~806.



gwern.net is served by Amazon S3 through the CloudFlare CDN. (Amazon charges less for bandwidth and disk space than NFSN, although one loses all the capabilities offered by Apache’s .htaccess, and Gzip compression is difficult so must be handled by CloudFlare; total costs may turn out to be a wash and I will consider the switch to Amazon S3 a success if it can bring my monthly bill to <$10 or <$120 a year.)

From October 2010 to June 2012, the site was hosted on NearlyFreeSpeech.net, an old hosting company; its specific niche is controversial material and activist-friendly pricing. Its libertarian owners cast a jaundiced eye on takedown requests, and pricing is pay-as-you-go. I like the former aspect, but the latter sold me on NFSN. Before I stumbled on NFSN (someone mentioned it offhandedly while chatting), I was getting ready to pay $10–15 a month ($120 yearly) to Linode. Linode’s offerings are overkill since I do not run dynamic websites or something like Haskell.org (with wikis and mailing lists and darcs repositories), but I didn’t know a good alternative. NFSN’s pricing meant that I paid for usage rather than large flat fees. I put in $32 to cover registering gwern.net until 2014, and then another $10 to cover bandwidth & storage price. DNS aside, I was billed $8.27 for October-December 2010; DNS included, January-April 2011 cost $10.09. $10 covered months of gwern.net for what I would have paid Linode in 1 month! In total, my 2010 costs were $39.44 (bill archive); my 2011 costs were $118.32 ($9.86 a month; archive); and my 2012 costs through June were $112.54 ($21 a month; archive); sum total: $270.3.

The switch to Amazon S3 hosting is complicated by my simultaneous addition of CloudFlare as a CDN; my total June 2012 Amazon bill is $1.62, with $0.19 for storage. CloudFlare claims it covered 17.5GB of 24.9GB total bandwidth, so the $1.41 represents 30% of my total bandwidth; multiply 1.41 by 3 is 4.30, and my hypothetical non-CloudFlare S3 bill is ~$4.5. Even at $10, this was well below the $21 monthly cost at NFSN. (The traffic graph indicates that June 2012 was a relatively quiet period, but I don’t think this eliminates the factor of 5.) From July 2012 to June 2013, my Amazon bills totaled $60, which is reasonable except for the steady increase ($1.62/$3.27/$2.43/$2.45/$2.88/$3.43/$4.12/$5.36/$5.65/$5.49/$4.88/$8.48/$9.26), being primarily driven by out-bound bandwidth (in June 2013, the $9.26 was largely due to the 75GB transferred—and that was after CloudFlare dealt with 82GB); $9.26 is much higher than I would prefer since that would be >$110 annually. This was probably due to all the graphics I included in the “Google shutdowns” analysis, since it returned to a more reasonable $5.14 on 42GB of traffic in August. September, October, November and December 2013 saw high levels maintained at $7.63/$12.11/$5.49/$8.75, so it’s probably a new normal. 2014 entailed new costs related to EC2 instances & S3 bandwidth spikes due to hosting a multi-gigabyte scientific dataset, so bills ran $8.51/$7.40/$7.32/$9.15/$26.63/$14.75/$7.79/$7.98/$8.98/$7.71/$7/$5.94. 2015 & 2016 were similar: $5.94/$7.30/$8.21/$9.00/$8.00/$8.30/$10.00/$9.68/$14.74/$7.10/$7.39/$8.03/$8.20/$8.31/$8.25/$9.04/$7.60/$7.93/$7.96/$9.98/$9.22/$11.80/$9.01/$8.87. 2017 saw costs increase due to one of my side-projects, aggressively increasing fulltexting of gwern.net by providing more papers & scanning cited books, only partially offset by changes like lossy optimization of images & converting GIFs to WebMs: $12.49/$10.68/$11.02/$12.53/$11.05/$10.63/$9.04/$11.03/$14.67/$15.52/$13.12/$12.23 (total: $144.01). In 2018, I continued fulltexting: $13.08/$14.85/$14.14/$18.73/$18.88/$15.92/$15.64/$15.27/$16.66/$22.56/$23.59/$25.91/(total: $213).


The revision history is kept in git; individual page sources can be read by appending .page to their URL.


As of 4 January 2019, the source of gwern.net is composed of >321 text files with >3,208,000 words or 23MB; this includes my writings & documents I have transcribed into Markdown, but excludes images, PDFs, archives, infrastructure, and the revision history. With those included and everything compiled to the static41 HTML, the site is >14.2GB. The source repository contains >11,801 patches (this is an under-count as the creation of the repository in 26 September 2008 included already-written material).

Benford’s law

In March 2013 I wondered, upon seeing a mention of Benford’s law: “if I extracted all the numbers from everything I’ve written on gwern.net, would it satisfy Benford’s law?” It seems the answer is… almost. I generate the list of numbers by running a Haskell program to parse digits, commas, and periods; and then I process it with shell utilities.42 This can then be read in R to run a chi-squared test confirming lack of fit (p=~0) and generate this comparison of the data & Benford’s law43:

Histogram/barplot of parsed numbers vs predicted
Histogram/barplot of parsed numbers vs predicted

There’s a clear resemblance for everything but the digit ‘2’, which then blows the fit to heck. I have no idea why 2 is overrepresented—it may be due to all the citations to recent academic papers which would involve numbers starting with ‘2’ (2002, 2010, 2013…) and cause a double-count in both the citation and filename, since if I look in the docs/ fulltext folder, I see 160 files starting with ‘1’ but 326 starting with ‘2’. But this can’t be the entire explanation since ‘2’ has 20.3k entries while to fit Benford, it needs to be just 11.5k—leaving a gap of ~10k numbers unexplained. A mystery.


Software tools & libraries used in the site as a whole:

  • The source files are written in Pandoc Markdown

  • math is written in LaTeX compiled to MathML

  • the site is compiled with the Hakyll (v4) static site generator, used to generate gwern.net, written in Haskell; for the gory details, see hakyll.hs which implements the compilation, RSS feed generation, & parsing of interwiki links as well.

    My preferred method of use is to browse & edit locally using Emacs, and then distribute using Hakyll. To use Hakyll, you cd into your repository and runhaskell hakyll.hs build (with hakyll.hs having whatever options you like). Hakyll will build a static HTML/CSS hierarchy inside _site/; you can then do something like firefox _static/index. (Because HTML extensions are not specified in the interest of cool URIs, you cannot use the Hakyll watch webserver as of January 2014.) Hakyll’s main advantage for me is relatively straightforward integration with the Pandoc Markdown libraries; Hakyll is not that easy to use, and so I do not recommend use of Hakyll as a general static site generator unless one is already adept with Haskell.

  • the CSS is borrowed from a motley of sources but stems primarily from the Hakyll homepage & Gitit; for specifics, see the unminified static/css/default.css in the source repository.

  • JavaScript:

  • Book affiliate links are through an Amazon Affiliates tag appended in the hakyll.hs

These tools encourage a minimalist site; I believe that minimalism helps one focus on the content. Anything besides the content is distraction and not design. ‘Attention!’, as Ikkyu would say44.


This site is licensed under the Creative Commons public domain (CC-0) license.

I believe the public domain license reduces FUD and dead-weight loss45, encourages copying (LOCKSS), gives back (however little) to Free Software/Free Content, and costs me nothing46.

  1. Gennadi Sosonko, pg 19 of Russian Silhouettes, on why he wrote his book of biographical sketches of great Soviet chess players. (As Richardson asks (Vectors 1.0, 2001): “25. Why would we write if we’d already heard what we wanted to hear?”)↩︎

  2. “It is only the attempt to write down your ideas that enables them to develop.” –Wittgenstein (pg 109, Recollections of Wittgenstein); “I thought a little [while in the isolation tank], and then I stopped thinking altogether…incredible how idleness of body leads to idleness of mind. After 2 days, I’d turned into an idiot. That’s the reason why, during a flight, astronauts are always kept busy.” –Oriana Fallaci, quoted in Rocket Men: The Epic Story of the First Men on the Moon by Craig Nelson.↩︎

  3. One danger of such an approach is that you will simply engage in confirmation bias, and build up an impressive-looking wall of citations that is completely wrong but effective in brainwashing yourself. The only solution is to be diligent to include criticism—so even if you do not escape brainwashing, at least your readers have a chance. The Autobiography of Charles Darwin, 1902:

    I had, also, during many years followed a golden rule, namely, that whenever a published fact, a new observation or thought came across me, which was opposed to my general results, to make a memorandum of it without fail and at once; for I had found by experience that such facts and thoughts were far more apt to escape from the memory than favourable ones. Owing to this habit, very few objections were raised against my views which I had not at least noticed and attempted to answer.

  4. Such as Larry Niven’s Known Space universe; consider the introduction to the chronologically last story in that setting, “Safe at Any Speed” (Tales of Known Space).↩︎

  5. Erich Fromm:

    “If the individual lived five hundred or one thousand years, this clash (between his interests and those of society) might not exist or at least might be considerably reduced. He then might live and harvest with joy what he sowed in sorrow; the suffering of one historical period which will bear fruit in the next one could bear fruit for him too.”

  6. From Richard Posner’s Aging and Old Age:

    One way to distinguish empirically between aging effects and proximity-to-death effects would be to compare, with respect to choice of occupation, investment, education, leisure activities, and other activities, elderly people on the one hand with young or middle-aged people who have truncated life expectancies but are in apparent good health, on the other. For example, a person newly infected with the AIDS virus (HIV) has roughly the same life expectancy as a 65-year-old and is unlikely to have, as yet, [major] symptoms. The conventional human-capital model implies that, after correction for differences in income and for other differences between such persons and elderly persons who have the same life expectancy (a big difference is that the former will not have pension entitlements to fall back upon), the behavior of the two groups will be similar. It does appear to be similar, so far as investing in human capital is concerned; the truncation of the payback period causes disinvestment. And there is a high suicide rate among HIV-infected persons (even before they have reached the point in the progression of the disease at which they are classified as persons with AIDS), just as there is, as we shall see in chapter 6, among elderly persons.

  7. John F. Kennedy, 1962:

    I am reminded of the story of the great French Marshal Lyautey, who once asked his gardener to plant a tree. The gardener objected that the tree was slow-growing and would not reach maturity for a hundred years. The Marshal replied, “In that case, there is no time to lose, plant it this afternoon.”

  8. Jorge Luis Borges, Dreamtigers Epilogue↩︎

  9. Mark Pilgrim, “Freedom 0”:

    In the long run, the utility of all non-Free software approaches zero. All non-Free software is a dead end.

  10. These dependencies can be subtle. Computer archivist Jason Scott writes of URL shortening services that:

    URL shorteners may be one of the worst ideas, one of the most backward ideas, to come out of the last five years. In very recent times, per-site shorteners, where a website registers a smaller version of its hostname and provides a single small link for a more complicated piece of content within it… those are fine. But these general-purpose URL shorteners, with their shady or fragile setups and utter dependence upon them, well. If we lose TinyURL or bit.ly, millions of weblogs, essays, and non-archived tweets lose their meaning. Instantly. To someone in the future, it’ll be like everyone from a certain era of history, say ten years of the 18th century, started speaking in a one-time pad of cryptographic pass phrases. We’re doing our best to stop it. Some of the shorteners have been helpful, others have been hostile. A number have died. We’re going to release torrents on a regular basis of these spreadsheets, these code breaking spreadsheets, and we hope others do too.

  11. Joshua Schachter remarks (and the comments provide even more examples) further on URL shorteners:

    But the biggest burden falls on the clicker, the person who follows the links. The extra layer of indirection slows down browsing with additional DNS lookups and server hits. A new and potentially unreliable middleman now sits between the link and its destination. And the long-term archivability of the hyperlink now depends on the health of a third party. The shortener may decide a link is a Terms Of Service violation and delete it. If the shortener accidentally erases a database, forgets to renew its domain, or just disappears, the link will break. If a top-level domain changes its policy on commercial use, the link will break. If the shortener gets hacked, every link becomes a potential phishing attack.

  12. A static text-source site has many advantages for Long Content that I consider use almost a no-brainer.

    • By nature, they compile most content down to flat standalone textual files, which allow recovery of content even if the original site software has bit-rotted or the source files have been lost or the compiled versions cannot be directly used in new site software: one can parse them with XML tools or with quick hacks or by eye.
    • Site compilers generally require dependencies to be declared up front, and the approach makes explicitness and content easy, but dynamic interdependent components difficult, all of which discourages creeping complexity and hidden state.
    • A static site can be archived into a tarball of files which will be readable as long as web browsers exist (or afterwards if the HTML is reasonably clean), but it could be difficult to archive a CMS like WordPress or Blogspot (the latter doesn’t even provide the content in HTML—it only provides a rat’s-nest of inscrutable JavaScript files which then download the content from somewhere and display it somehow; indeed, I’m not sure how I would automate archiving of such a site if I had to; I would need some sort of headless browser to run the JS and serialize the final resulting DOM, possibly with some scripting of mouse/keyboard actions).
    • The content is often not available locally, or is stored in opaque binary formats rather than text (if one is lucky, it will at least be a database), both of which make it difficult to port content to other website software; you won’t have the necessary pieces, or they will be in wildly incompatible formats.
    • Static sites are usually written in a reasonably standardized markup language such as Markdown or Latex, in distinction to blogs which force one through WYSIWYG editors or invent their own markup conventions, which is yet another barrier: parsing a possibly ill-defined language.
    • The lowered sysadmin efforts (who wants to be constantly cleaning up spam or hacks on their WordPress blog?) are a final advantage: lower running costs make it more likely that a site will stay up rather than cease to be worth the hassle.

    Static sites are not appropriate for many kinds of websites, but they are appropriate for websites which are content-oriented, do not need interactivity, expect to migrate website software several times over coming decades, want to enable archiving by oneself or third parties (“lots of copies keeps stuff safe”), and to gracefully degrade after loss or bitrot.↩︎

  13. Such as burning the occasional copy onto read-only media like DVDs.↩︎

  14. One can’t be sure; the IA is fed by Alexa, and Alexa doesn’t guarantee pages will be spidered & preserved if one goes through their request form.↩︎

  15. I am diligent in backing up my files, in periodically copying my content from the cloud, and in preserving viewed Internet content; why do I do all this? Because I want to believe that my memories are precious, that the things I saw and said are valuable; “I want to meet them again, because I believe my feelings at that time were real.” My past is not trash to me, used up & discarded.↩︎

  16. Examples of such blogs:

    1. Eliezer Yudkowsky’s contributions to LessWrong were the rough draft of a philosophy book (or two)
    2. John Robb’s Global Guerrillas lead to his Brave New War: The Next Stage of Terrorism and the End of Globalization
    3. Kevin Kelly’s Technium was turned into What Technology Wants.

    An example of how not to do it would be Robin Hanson‘s Overcoming Bias blog; it is stuffed with fascinating citations & sketches of ideas, but they never go anywhere with the exception of his mind emulation economy posts which were eventually published in 2016 as The Age of Em. Just his posts on medicine would make a fascinating essay or just list—but he has never made one. (“Showing That You Care: The Evolution of Health Altruism” would be a natural home for many of his posts’ contents, but will never be updated.)↩︎

  17. “Kevin Kelly Answers Your Questions”, 6 September 2011:

    [Question:] “One purpose of the Long Now Clock is to encourage long-term thinking. Aside from the Clock, though, what do you think people can do in their everyday lives to adopt or promote long-term thinking?”

    KK: “The 10,000-year Clock we are building in the hills of west Texas is meant to remind us to think long-term, but learning how to do that as in individual is difficult. Part of the difficulty is that as individuals we constrained to short lives, and are inherently not long-term. So part of the skill in thinking long-term is to place our values and energies in ways that transcend the individual—either in generational projects, or in social enterprises.”

    “As a start I recommend engaging in a project that will not be complete in your lifetime. Another way is to require that your current projects exhibit some payoff that is not immediate; perhaps some small portion of it pays off in the future. A third way is to create things that get better, or run up in time, rather than one that decays and runs down in time. For instance a seedling grows into a tree, which has seedlings of its own. A program like Heifer Project which gives breeding pairs of animals to poor farmers, who in turn must give one breeding pair away themselves, is an exotropic scheme, growing up over time.”

  18. ‘Princess Irulan’, Frank Herbert, Dune↩︎

  19. GiveWell reports in “A good volunteer is hard to find” that of volunteers motivated enough to email them asking to help, something like <20% will complete the GiveWell test assignment and render meaningful help. Such persons would have been well-advised to have simply donated some money. I have long noted that many of the most popular pages on gwern.net could have been written by anyone and drew on no unique talents of mine; I have on several occasions received offers to help with the DNB FAQ—none of which have resulted in actual help.↩︎

  20. An old sentiment; consider “A drop hollows out the stone” (Ovid, Epistles) or Thomas Carlyle’s “The weakest living creature, by concentrating his powers on a single object, can accomplish something. The strongest, by dispensing his over many, may fail to accomplish anything. The drop, by continually falling, bores its passage through the hardest rock. The hasty torrent rushes over it with hideous uproar, and leaves no trace behind.” (The life of Friedrich Schiller, 1825)↩︎

  21. “Ten Lessons I wish I had been Taught”, Gian-Carlo Rota:

    Richard Feynman was fond of giving the following advice on how to be a genius. You have to keep a dozen of your favorite problems constantly present in your mind, although by and large they will lay in a dormant state. Every time you hear or read a new trick or a new result, test it against each of your twelve problems to see whether it helps. Every once in a while there will be a hit, and people will say: ‘How did he do it? He must be a genius!’

  22. IQ is sometimes used as a proxy for health, like height, because it sometimes seems like any health problem will damage IQ. Didn’t get much protein as a kid? Congratulations, your nerves will lack myelination and you will literally think slower. Missing some iodine? Say good bye to <10 points! If you’re anemic or iron-deficient, that might increase to <15 points. Have tapeworms? There go some more points, and maybe centimeters off your adult height, thanks to the worms stealing nutrients from you. Have a rough birth and suffer a spot of hypoxia before you began breathing on your own? Tough luck, old bean. It is very easy to lower IQ; you can do it with a baseball bat. It’s the other way around that’s nearly impossible.↩︎

  23. And America has tried pretty hard over the past 60 years to affect IQ. The whole nature/nurture black-white IQ debate would be moot if there were some nutrient or educational system which could add even 10 points on average, because then we would use it on all the blacks. But it seems that I’m constantly reading about programs like Headstart which boost IQ for a little while… and do nothing in the long run.↩︎

  24. For details on the many valuable correlates of the Conscientiousness personality factor, see Conscientiousness and online education.↩︎

  25. 25 episodes, 6 movies, >11 manga volumes—just to stick to the core works.↩︎

  26. Mibu no Tadamine, KKS XII: 609:

    More than my life
    What I most regret
    A dream unfinished
    And awakening.
  27. As with Cloud Nine; I accidentally erased everything on a routine basis while messing around with Windows.↩︎

  28. For example, I notice I am no longer deeply interested in the occult. Hopefully this is because I have grown mentally and recognize it as rubbish; I would be embarrassed if when I died it turned out my youthful self had a better grasp on the real world.↩︎

  29. Some pages don’t have any connection to predictions. It’s possible to make predictions for some border cases like the terrorism essays (death tolls, achievements of particular groups’ policy goals), but what about the short stories or poems? My imagination fails there.↩︎

  30. Thinking of predictions is good mental discipline; we should always be able to cash out our beliefs in terms of the real world, or know why we cannot. Unfortunately, humans being humans, we need to actually track our predictions—all of them—lest our predicting degenerate into entertainment like political punditry.↩︎

  31. Dozens of theories have been put forth. I have been collecting & making predictions; and am up to 219. It will be interesting to see how the movies turn out.↩︎

  32. I have 2 predictions registered about the thesis on PB.com: 1 reviewer will accept my theory by 2016 and the light novels will finish by 2015.↩︎

  33. See Robin Hanson, “If Uploads Come First”↩︎

  34. I originally used last file modification time but this turned out to be confusing to readers, because I so regularly add or update links or add new formatting features that the file modification time was usually quite recent, and so it was meaningless.↩︎

  35. Reactive archiving is inadequate because such links may die before my crawler gets to them, may not be archivable, or will just expose readers to dead links for an unacceptably long time before I’d normally get around to them.↩︎

  36. Anecdotally, the rankings seem correct. When I went to a LessWrong meetup in California, many knew of or had read the DNB FAQ, some had read or used my modafinil price-chart, and very few remembered reading anything else.↩︎

  37. I am sometimes reminded of another waka, by Ikkyu:

    To write something and leave it behind us,
    Is but a dream.
    When we awake we know
    There is not even anyone to read it.

  38. They spend an average of 27 seconds; in comparison, my second largest source of traffic, LessWrongers, average 3 minutes and 29 seconds; even my third largest traffic source, Redditers, manage almost 2 minutes. Even random people coming from Google manage to spend 44 seconds on their visit!↩︎

  39. Since I converted the repo to use git, darcs-graph was no longer available, so I wrote my own script in R using splines rather than a 7-day moving average:

  40. I rewrote the script a little to use the nicer plots of ggplot2 and switches to LOESS for the smoothed curve:

  41. I like the static site approach to things; it tends to be harder to use and more restrictive, but in exchange it yields better performance & leads to fewer hassles or runtime issues.↩︎

  42. We write a short Haskell program as part of a pipeline:

  43. Graph then test:

  44. Paraphrased from Dialogues of the Zen Masters as quoted in pg 11 of the Editor’s Introduction to Three Pillars of Zen:

    One day a man of the people said to Master Ikkyu: “Master, will you please write for me maxims of the highest wisdom?” Ikkyu immediately brushed out the word ‘Attention’. “Is that all? Will you not write some more?” Ikkyu then brushed out twice: ‘Attention. Attention.’ The man remarked irritably that there wasn’t much depth or subtlety to that. Then Ikkyu wrote the same word 3 times running: ‘Attention. Attention. Attention.’ Half-angered, the man demanded: “What does ‘Attention’ mean anyway?” And Ikkyu answered gently: “Attention means attention.”

  45. PD increases economic efficiency through—if nothing else—making works easier to find. Tim O’Reilly says that “Obscurity is a far greater threat to authors and creative artists than piracy.” If that is so, then that means that difficulty of finding works reduces the welfare of artists and consumers, because both forgo a beneficial trade (the artist loses any revenue and the consumer loses any enjoyment). Even small increases in inconvenience make big differences.↩︎

  46. Not that I could sell anything on this wiki; and if I could, I would polish it as much as possible, giving me fresh copyright.↩︎