Internet Search Tips

A description of tips and tricks for effective Internet search of papers/books for research.
topics: archiving, technology, shell, Google, tutorial
source; created: 11 Dec 2018; modified: 1 Jan 2020; status: finished; confidence: certain; importance: 4

Over time, I developed a certain google-fu and expertise in finding references, papers, and books online. Some of these tricks are not well-known, like checking the Internet Archive (IA) for books. I try to write down my search workflow, and give general advice about finding and hosting documents.

Google-fu search skill is something I’ve prided myself ever since elementary school, when the librarian challenged the class to find things in the almanac; not infrequently, I’d win. The Internet is the greatest almanac of all, and to the curious, a never-ending cornucopia, so I am sad to see many fail to find things after a cursory search—or not look at all. For most people, if it’s not the first hit in Google/Google Scholar, it doesn’t exist. Below, I reveal my best Internet search tricks and try to provide a rough flowchart of how to go about an online search, explaining the subtle tricks and intuition of search-fu.



Human flesh search engine. Last resort: if none of this works, there are a few places online you can request a copy (however, they will usually fail if you have exhausted all previous avenues):

Finally, you can always try to contact the author. This only occasionally works for the papers I have the hardest time with, since they tend to be old ones where the author is dead or unreachable—any author publishing a paper since 1990 will usually have been digitized somewhere—but it’s easy to try.


After finding a fulltext copy, you should find a reliable long-term link/place to store it and make it more findable (remember—if it’s not in Google/Google Scholar, it doesn’t exist!):

  • never link unreliable hosts:

    • LG/SH: Always operate under the assumption they could be gone tomorrow. (As my uncle found out with shortly after paying for a lifetime membership!) There are no guarantees either one will be around for long under their legal assaults, and no guarantee that they are being properly mirrored or will be restored elsewhere. Download anything you need and keep a copy of it yourself and, ideally, host it publicly.
    • NBER: never rely on a or URL, as they are temporary. (SSRN is also undesirable due to making it increasingly difficult to download, but it is at least reliable.)
    • Scribd: never link Scribd—they are a scummy website which impede downloads, and anything on Scribd usually first appeared elsewhere anyway.
    • RG: avoid linking to (compromised by new ownership & PDFs get deleted routinely, apparently often by authors) or (the URLs are one-time and break)
    • high-impact journals: be careful linking to or Cell (if a paper is not explicitly marked as Open Access, even if it’s available, it may disappear in a few months!); similarly, watch out for,,,,, &, who pull similar shenanigans.
    • ~/: be careful linking to academic personal directories on university websites (often noticeable by the Unix convention .edu/~user/); they have short half-lives.
  • check & improve metadata.

    Adding metadata to papers/books is a good idea because it makes the file findable in G/GS (if it’s not online, does it really exist?) and helps you if you decide to use bibliographic software like in the future. Many academic publishers & LG are terrible about metadata, and will not include even title/author/DOI/year.

    PDFs can be easily annotated with metadata using : : exiftool -All prints all metadata, and the metadata can be set individually using similar fields.

    For papers hidden inside volumes or other files, you should extract the relevant page range to create a single relevant file. (For extraction of PDF page-ranges, I use , eg: pdftk 2010-davidson-wellplayed10-videogamesvaluemeaning.pdf cat 180-196 output 2009-fortugno.pdf.)

    I try to set at least title/author/DOI/year/subject, and stuff any additional topics & bibliographic information into the “Keywords” field. Example of setting metadata:

  • PDF editing: if a scan, it may be worth editing the PDF to crop the edges, threshold to binarize it (which, for a bad grayscale or color scan, can drastically reduce filesize while increasing readability), and OCRing it. I use but there are alternatives worth checking out.

  • public hosting: if possible, host a public copy; especially if it was very difficult to find, even if it was useless, it should be hosted. The life you save may be your own.

  • link on WP/social media: for bonus points, link it in appropriate places on Wikipedia or Reddit or Twitter; this makes people aware of the copy being available, and also supercharges visibility in search engines.


Aside from the (highly-recommended) use of hotkeys and Booleans for searches, there are a few useful tools for the researcher, which while expensive initially, can pay off in the long-term:

  • : automatically archive your web browsing and/or links from arbitrary websites to forestall linkrot; particularly useful for detecting & recovering from dead PDF links

  • subscriptions like & GS search alerts: set up alerts for a specific search query, or for new citations of a specific paper. ( is not as useful as it seems.)

    1. PubMed has straightforward conversion of search queries into alerts: “Create alert” below the search bar. (Given the volume of PubMed indexing, I recommend carefully tailoring your search to be as narrow as possible, or else your alerts may overwhelm you.)
    2. To create generic GS search query alert, simply use the “Create alert” on the sidebar for any search. To follow citations of a key paper, you must: 1. bring up the paper in GS; 2. click on “Cited by X”; 3. then use “Create alert” on the sidebar.
  • GCSE: a is a specialized search queries limited to whitelisted pages/domains etc (eg my ).

    A GCSE can be thought of as a saved search query on steroids. If you find yourself regularly including scores of the same domains in multiple searches search, or constantly blacklisting domains with -site: or using many negations to filter out common false positives, it may be time to set up a GCSE which does all that by default.

  • clippings: like /: regularly making and keeping excerpts creates a personalized search engine, in effect.

    This can be vital for refinding old things you read where the search terms are hopelessly generic or you can’t remember an exact quote or reference; it is one thing to search a keyword like “autism” in a few score thousand clippings, and another thing to search that in the entire Internet! (One can also reorganize or edit the notes to add in the keywords one is thinking of, to help with refinding.) I make heavy use of Evernote clipping and it is key to refinding my references.

  • Crawling websites: sometimes having copies of whole websites might be useful, either for more flexible searching or for ensuring you have anything you might need in the future. (example: ).

    Useful tools to know about: , , ; Firefox plugins: , , , , cookie exporting.

    Short of downloading a website, it might also be useful to pre-emptively archive it by using linkchecker to crawl it, compile a list of all external & internal links, and store them for processing by another archival program (see for examples). In certain rare circumstances, security tools like can be useful to examine a mysterious server in more detail: what web server and services does it run, what else might be on it (sometimes interesting things like old anonymous FTP servers turn up), has a website moved between IPs or servers, etc.

Web pages

With proper use of pre-emptive archiving tools like archiver-bot, fixing linkrot in one’s own pages is much easier, but that leaves other references. Searching for lost web pages is similar to searching for papers:

  • just search the title: if the page title is given, search for the title.

    It is a good idea to include page titles in one’s own pages, as well as the URL, to help with future searches, since the URL may be meaningless gibberish on its own, and pre-emptive archiving can fail. HTML supports both alt and title parameters in link tags, and, in cases where displaying a title is not desirable (because the link is being used inline as part of normal hypertextual writing), titles can be included cleanly in Markdown documents like this: [inline text description](URL "Title").

  • clean URLs: check the URL for weirdness or trailing garbage like ?rss=1 or ?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+blogspot%2FgJZg+%28Google+AI+Blog%29? Or a variant domain, like a URL? Those are all less likely to be findable or archived than the canonical version.

  • domain search: restrict G search to the original domain with site:, or to related domains

  • time search: restrict G search to the original date-range/years

  • switch engines: try a different search engine: corpuses can vary, and in some cases G tries to be too smart for its own good when you need a literal search; and are usable alternatives (especially if one of DuckDuckGo’s ‘bang’ special searches is what one needs)

  • check archives: if nowhere on the clearnet, try the Internet Archive (IA) or the :

    IA is the default backup for a dead URL. If IA doesn’t Just Work, there may be other versions in it:

    • misleading redirects: did the IA ‘helpfully’ redirect you to a much-later-in-time error page? Kill the redirect and check the earliest stored version for the exact URL rather than the redirect. Did the page initially load but then error out/redirect? Disable JS with NoScript and reload.

    • within-domain archives: IA lets you list all URLs with any archived versions, by searching for URL/*; the list of available URLs may reveal an alternate newer/older URL. It can also be useful to filter by filetype or substring.

      For example, one might list all URLs in a domain, and if the list is too long and filled with garbage URLs, then using the “Filter results” incremental-search widget to search for “uploads/” on a WordPress blog.9

      Screenshot of an oft-overlooked feature of the Internet Archive: displaying all available/archived URLs for a specific domain, filtered down to a subset matching a string like *uploads/*.
      Screenshot of an oft-overlooked feature of the Internet Archive: displaying all available/archived URLs for a specific domain, filtered down to a subset matching a string like *uploads/*.
      • (not to be confused with the which provides a CLI interface to uploading files) is a Ruby tool which lets you download whole domains from IA, which can be useful for running a local fulltext search using regexps (a good grep query is often enough), in cases where just looking at the URLs via URL/* is not helpful. (An alternative which might work is .)


    • did the domain change, eg from to or Entirely different as far as IA is concerned.

    • is this a Blogspot blog? Blogspot is uniquely horrible in that it has versions of each blog for every country domain: a blog could be under any of,,, foo.blogspot.jp10

    • did the website provide RSS feeds?

      A little known fact is that (GR; October 2005–July 2013) stored all RSS items it crawled, so if a website’s RSS feed was configured to include full items, the RSS feed history was an alternate mirror of the whole website, and since GR never removed RSS items, it was possible to retrieve pages or whole websites from it. GR has since closed down, sadly, but before it closed, a large fraction of GR’s historical RSS feeds, and those archives are . The catch is that they are stored in mega-, which, for all their archival virtues, are not the most user-friendly format. The raw GR mega-WARCs are difficult enough to work with that I defer an example to the appendix.

    • : an IA-like mirror

    • any local archives, such as those made with my

    • Google Cache (GC): GC works, sometimes, but the copies are usually the worst around, ephemeral & cannot be relied upon. Google also appears to have been steadily deprecating GC over the years, as GC shows up less & less in search results. A last resort.



E-books are rarer and harder to get than papers, although the situation has improved vastly since the early 2000s. To search for books online:


Expensive but feasible. Books are something of a double-edged sword compared to papers/theses. On the one hand, books are much more often unavailable online, and must be bought offline, but at least you almost always can buy used books offline without much trouble (and often for <$10 total); on the other hand, while paper/theses are often available online, when one is not unavailable, it’s usually very unavailable, and you’re stuck (unless you have a university ILL department backing you up or are willing to travel to the few or only universities with paper or microfilm copies).

Purchasing from used book sellers:

  • Sellers:

    • used book search engines: Google Books/: a good starting point for seller links; if buying from a marketplace like AbeBooks/Amazon/Barnes & Noble, it’s worth searching the seller to see if they have their own website, which is potentially much cheaper. They may also have multiple editions in stock.

    • bad: eBay & Amazon are often bad, due to high-minimum-order+S&H and sellers on Amazon seem to assume Amazon buyers are easily rooked; but can be useful in providing metadata like page count or ISBN or variations on the title

    • good: , , , , .

      Note: on AbeBooks, international orders can be useful (especially for behavioral genetics or psychology books) but be careful of international orders with your credit card—many debit/credit cards will fail on international orders and trigger a fraud alert, and PayPal is not accepted.

  • subscriptions: if a book is not available or too expensive, set price watches: AbeBooks supports email alerts on stored searches, and Amazon can be monitored via (remember the CCC price alert you want is on the used third-party category, as new books are more expensive, less available, and unnecessary).


  • destructive vs non-destructive: the fundamental dilemma of book scanning—destructively debinding books with a razor or guillotine cutter works much better & is much less time-consuming than spreading them on a flatbed scanner to scan one-by-one12, because it allows use of a sheet-fed scanner instead, which is easily 5x faster and will give higher-quality scans (because the sheets will be flat, scanned edge-to-edge, and much more closely aligned), but does, of course, require effectively destroying the book.

  • Tools:

    • cutting: For simple debinding of a few books a year, an X-acto knife/razor is good (avoid the ‘triangle’ blades, get curved blades intended for large cuts instead of detail work)

      once you start doing more than one a month, it’s time to upgrade to a guillotine blade paper cutter (a fancier swinging-arm paper cutter, which uses a two-joint system to clamp down and cut uniformly).

      A guillotine blade can cut chunks of 200 pages easily without much slippage, so for books with more pages, I use both: an X-acto to cut along the spine and turn it into several 200-page chunks for the guillotine cutter.

    • scanning: at some point, it may make sense to switch to a scanning service like (1DS has acceptable quality for the black-white scans I have used them for thus far, but watch out for their nickel-and-diming fees for OCR or “setting the PDF title”; these can be done in no time yourself using gscan2pdf/exiftool/ocrmypdf and will save a lot of money as they, amazingly, bill by 100-page units). Books can be sent directly to 1DS, reducing logistical hassles.

  • clean up: after scanning, crop/threshold/OCR/add metadata

    • Adding metadata: same principles as papers. While more elaborate metadata can be added, like bookmarks, I have not experimented with those yet.
  • File format: PDF.

    In the past, I used for documents I produce myself, as it produces much smaller scans than gscan2pdf’s default PDF settings (at least half the size, sometimes one-tenth the size), making them more easily hosted & a superior browsing experience.

    The downsides of DjVu are that not all PDF viewers can handle DjVu files, and it appears that G/GS ignore all DjVu files (despite the format being 20 years old), rendering them completely unfindable online. In addition, DjVu is an increasingly obscure format and has, for example, been dropped by the IA as of 2016. The former is a relatively small issue, but the latter is fatal—being consigned to oblivion by search engines largely defeats the point of scanning! (“If it’s not in Google, it doesn’t exist.”) Hence, despite being a worse format, I now recommend PDF and have stopped using DjVu for new scans13 and have converted my old DjVu files to PDF.

  • Uploading: to LibGen, usually. For backups, filelockers like Dropbox, Mega, MediaFire, or Google Drive are good. I usually upload 3 copies including LG. I rotate accounts once a year, to avoid putting too many files into a single account.

  • Hosting: hosting papers is easy but books come with risk:

    Books can be dangerous; in deciding whether to host a book, my rule of thumb is host only books pre-2000 and which do not have Kindle editions or other signs of active exploitation and is effectively an ‘’.

    As of 23 October 2019, hosting 4090 files over 9 years (very roughly, assuming linear growth, <6.7 million document-days of hosting: ), I’ve received 4 takedown orders: a behavioral genetics textbook (2013), The Handbook of Psychopathy (2005), a recent meta-analysis paper (Roberts et al 2016), and a CUP DMCA takedown order for 27 files. I broke my rule of thumb to host the 2 books (my mistake), which leaves only the 1 paper, which I think was a fluke. So, as long as one avoids relatively recent books, the risk should be minimal.

Case Studies

Below are case studies of difficult to find resources and how I went about locating them, demonstrating the various techniques and how to think about searches.

  • Does anybody know where the online appendix to Nordhaus’ “Two Centuries of Productivity Growth in Computing” is hiding?

    I look up the title in Google Scholar; seeing a friendly PDF link (CiteSeerx), I click. The paper says “The data used in this study are provided in a background spreadsheet available at Sadly, this is a lie. (Sandberg would of course have tried that.)

    I immediately check the URL in the IA—nothing. The IA didn’t catch it at all. Maybe the has it? Nope, it references the same URL, and doesn’t provide a copy as an appendix or supplement. (What do we pay these publishers such enormous sums of money for, exactly?) So I back off to checking, to check Nordhaus’s personal website for a newer link. The Yale personal website is empty and appears to’ve been replaced by a Google Sites personal page. It links nothing useful, so I check a more thorough index, Google, by searching Nothing there either (and it appears almost empty, so Nordhaus has allowed most of his stuff to be deleted and bitrot). I try a broader Google: nordhaus appendix.xls. This turns up some spreadsheets, but still nothing.

    Easier approaches having been exhausted, I return to the IA and I pull up all URLs archived for his original personal website:*/* This pulls up way too many URLs to manually review, so I filter results for xls, which reduces to a more manageable 60 hits; reading through the hits, I spot from 10 Oct 2014; this sounds right, albeit substantially later in time than expected (either 2010 or 2012, judging from the filename).

    Downloading it, opening it up and cross-referencing with the paper, it has the same spreadsheet ‘sheets’ as mentioned, like “Manual” or “Capital_Deep”, and seems to be either the original file in question or an updated version thereof (which may be even better). The spreadsheet metadata indicates it was created “04/09/2001, 23:20:43, ITS Academic Media & Technology”, and modified “12/22/2010, 02:40:20”, so it seems to be the latter—it’s the original spreadsheet Nordhaus created when he began work several years prior to the formal 2007 publication (6 years seems reasonable given all the delays in such a process), and then was updated 3 years afterwards. Close enough.

  • :

    I was in a consignment type store once and picked up a book called “Eat fat, get thin”. Giving it a quick scan through, it was basically the same stuff as Atkins but this book was from the 50s or 60s. I wish I’d have bought it. I think I found a reference to it once online but it’s been drowned out since someone else released a book with the same name (and it wasn’t Barry Groves either).

    The easiest way to find a book given a corrupted title, a date range, and the information there are many similar titles drowning out a naive search engine query, is to skip to a specialized search engine with clean metadata (ie. a library database).

    Searching in WorldCat for 1950s–1970s, “Eat fat, get thin” turns up nothing relevant. This is unsurprising, as he was unlikely to’ve remembered the title exactly, and this title doesn’t quite sound right for the era anyway (a little too punchy and ungrammatical, and ‘thin’ wasn’t a desirable word back then compared to words like ‘slim’ or ‘sleek’ or ‘svelte’). People often oversimplify titles, so I dropped back to just “Eat fat”.

    This immediately turned up the book: ‘s 1958 Eat Fat and Grow Slim—note that it is almost the same title, with a comma serving as conjunction and ’slim’ rather than the more contemporary ‘thin’, but just different enough to screw up an overly-literal search.

    With the same trick in mind, we could also have found it in a regular Google search query by adding additional terms to hint to Google that we want old books, not recent ones: both "Eat Fat" 1950s or "Eat Fat" 1960s would have turned it up in the first 5 search results. If we didn’t use quotes, the searches get harder because broader hits get pulled in. For example, Eat fat, get thin 1950s -Hyman excludes the recent book mentioned, but you still have to go down 15 hits before finding Mackarness, and Eat fat, get thin -Hyman requires going down 18 hits.

  • , on the phenomenon of quotes striking transcripts from a major example of a disappearing crystal, when ~1998 Abbott suddenly became unable to manufacture the anti-retroviral drug (Norvir™) due to a rival (and less effective) crystal form spontaneously infecting all its plants, threatening many AIDS patients, but notes:

    The transcripts were originally published on the website42 of the International Association of Physicians in AIDS Care [IAPAC], but no longer appear there.

    A search using the quotes confirms that the originals have long since vanished from the open Internet, turning up only quotes of the quotations. Unfortunately, no URL is given. The Internet Archive has comprehensive mirrors of the IAPAC, but too many to easily search through. Using the filter feature, I keyword searched for “ritonavir”, but while this turned up a number of pages from roughly the right time period, they do not mention it and none of the quotes appear. The key turned out to be to use the trademark name instead which pulls up many more pages, and after checking a few, the IAPAC turned out to have organized all the Norvir material into a single subdirectory with a convenient index.html; the articles/transcripts, in turn, were indexed under the linked “Description of the Problem” index page.

    I then pulled the Norvir subdirectory with a ~/.gem/ruby/2.5.0/bin/wayback_machine_downloader wayback_machine_downloader '' command and hosted a mirror to make it visible in Google.

See also


Searching the Google Reader archives

A tutorial on how to do manual searches of the 2013 archives on the . Google Reader provides fulltext mirrors of many websites which are long gone and not otherwise available even in the IA; however, the Archive Team archives are extremely user-unfriendly and challenging to work with even for programmers. I explain how to find & extract specific websites.

A little known way to ‘undelete’ a blog or website is to use Google Reader (GR). Unusual archive: Google Reader. GR crawled regularly almost all blogs’ RSS feeds; RSS feeds often contain the fulltext of articles. If a blog author writes an article, the fulltext is included in the RSS feed, GR downloads it, and then the author changes their mind and edits or deletes it, GR would redownload the new version but it would continue to show the version the old version as well (you would see two versions, chronologically). If the author blogged regularly and so GR had learned to check regularly, it could hypothetically grab many different edited versions, even, not just ones with weeks or months in between. Assuming that GR did not, as it sometimes did for inscrutable reasons, stop displaying the historical archives and only showed the last 90 days or so to readers; I was never able to figure out why this happened or if indeed it really did happen and was not some sort of UI problem. Regardless, if all went well, this let you undelete an article, albeit perhaps with messed up formatting or something. Sadly, GR was closed back in 2013 and you cannot simply log in and look for blogs.

Archive Team mirrored Google Reader. However, before it was closed, launched a major effort to download as much of GR as possible. So in that dump, there may be archives of all of a random blog’s posts. Specifically: if a GR user subscribed to it; if Archive Team knew about it; if they requested it in time before closure; and if GR did keep full archives stretching back to the first posting.

AT mirror is raw binary data. Downside: the Archive Team dump is not in an easily browsed format, and merely figuring out what it might have is difficult. In fact, it’s so difficult that before researching Craig Wright in November–December 2015, I never had an urgent enough reason to figure out how to get anything out of it before, and I’m not sure I’ve ever seen anyone actually use it before; Archive Team takes the attitude that it’s better to preserve the data somehow and let posterity worry about using it. (There is a site which claimed to be a frontend to the dump but when I tried to use it, & still is in December 2018.)


Find the right archive. The 9TB of data is stored in ~69 opaque compressed WARC archives. 9TB is a bit much to download and uncompress to look for one or two files, so to find out which WARC you need, you have to download the ~69 CDX indexes which record the contents of their respective WARC, and search them for the URLs you are interested in. (They are plain text so you can just grep them.)


In this example, we will look at the main blog of Craig Wright, (Another blog,, appears to have been too obscure to be crawled by GR.) To locate the WARC with the Wright RSS feeds, download the . To search:

for file in *.gz; do echo $file; zcat $file | fgrep -e 'gse-compliance' -e 'security-doctor'; done
# com,google/reader/api/0/stream/contents/feed/http:/\
# archiveteam&comments=true&hl=en&likes=true&n=1000&r=n 20130602001238\
# api/0/stream/contents/feed/\
# likes=true&comments=true&client=ArchiveTeam unk - 4GZ4KXJISATWOFEZXMNB4Q5L3JVVPJPM - - 1316181\
# 19808229791 archiveteam_greader_20130604001315/greader_20130604001315.megawarc.warc.gz
# com,google/reader/api/0/stream/contents/feed/http:/\
# alt=rss?client=archiveteam&comments=true&hl=en&likes=true&n=1000&r=n 20130602001249\
# com/reader/api/0/stream/contents/feed/\
# %3Falt%3Drss?r=n&n=1000&hl=en&likes=true&comments=true&client=ArchiveTeam unk - HOYKQ63N2D6UJ4TOIXMOTUD4IY7MP5HM\
# - - 1326824 19810951910 archiveteam_greader_20130604001315/greader_20130604001315.megawarc.warc.gz
# com,google/reader/api/0/stream/contents/feed/http:/\
# client=archiveteam&comments=true&hl=en&likes=true&n=1000&r=n 20130602001244\
# reader/api/0/stream/contents/feed/\
# r=n&n=1000&hl=en&likes=true&comments=true&client=ArchiveTeam unk - XXISZYMRUZWD3L6WEEEQQ7KY7KA5BD2X - - \
# 1404934 19809546472 archiveteam_greader_20130604001315/greader_20130604001315.megawarc.warc.gz
# com,google/reader/api/0/stream/contents/feed/http:/\
# &comments=true&hl=en&likes=true&n=1000&r=n 20130602001253\
# /feed/\
# &client=ArchiveTeam text/html 404 AJSJWHNSRBYIASRYY544HJMKLDBBKRMO - - 9467 19812279226 \
# archiveteam_greader_20130604001315/greader_20130604001315.megawarc.warc.gz

Understanding the output: the format is defined by the first line, which then can be :

  • the format string is: CDX N b a m s k r M S V g; which means here:

    • N: massaged url
    • b: date
    • a: original url
    • m: mime type of original document
    • s: response code
    • k: new style checksum
    • r: redirect
    • M: meta tags (AIF)
    • S: ?
    • V: compressed arc file offset
    • g: file name


Converts to:

  • massaged URL: (com,google)/reader/api/0/stream/contents/feed/ http:/ client=archiveteam&comments=true&hl=en&likes=true&n=1000&r=n
  • date: 20130602001238
  • original URL: r=n&n=1000&hl=en&likes=true&comments=true&client=ArchiveTeam
  • MIME type: unk [unknown?]
  • response code:—[none?]
  • new-style checksum: 4GZ4KXJISATWOFEZXMNB4Q5L3JVVPJPM
  • redirect:—[none?]
  • meta tags:—[none?]
  • S [? maybe length?]: 1316181
  • compressed arc file offset: 19808229791 (19,808,229,791; so somewhere around 19.8GB into the mega-WARC)
  • filename: archiveteam_greader_20130604001315/greader_20130604001315.megawarc.warc.gz

Knowing the offset theoretically makes it possible to extract directly from the IA copy without having to download and decompress the entire thing… The S & offsets for gse-compliance are:

  1. 1316181/19808229791
  2. 1326824/19810951910
  3. 1404934/19809546472
  4. 9467/19812279226

So we found hits pointing towards archiveteam_greader_20130604001315 & archiveteam_greader_20130614211457 which we then need to download (25GB each):

Once downloaded, how do we get the feeds? There are a number of ; I contacted the original GR archiver, ivan, but that wasn’t too helpful.


I tried using to unpack the entire WARC archive into individual files, and then delete everything which was not relevant:

But this was too slow, and crashed partway through before finishing.

Bug reports:

A more recent alternative library, which I haven’t tried, is , which may be able to find the byte ranges & extract them.


If we are feeling brave, we can use the offset and presumed length to have directly extract byte ranges:


Success: raw HTML. My dd extraction was successful, and the resulting HTML/RSS could then be browsed with a command like cat *.warc | fold --spaces -width=200 | less. They can probably also be converted to a local form and browsed, although they won’t include any of the site assets like images or CSS/JS, since the original RSS feed assumes you can load any references from the original website and didn’t do any kind of or mirroring (not, after all, having been intended for archive purposes in the first place…)

  1. For example, the info: operator is entirely useless. The link: operator, in almost a decade of me trying it once in a great while, has never returned remotely as many links to my website as Google Webmaster Tools returns for inbound links, and seems to have been disabled entirely at some point.↩︎

  2. WP is increasingly out of date & unrepresentative due to increasingly narrow policies about sourcing & preprints, part of its overall , so it’s not a good place to look for references. It is a good place to look for key terminology, though.↩︎

  3. This probably explains part of why no one cites that paper, and those who cite it clearly have not actually read it, even though it invented racial admixture analysis, which, since reinvented by others, has become a major method in medical genetics.↩︎

  4. University ILL privileges are one of the most underrated fringe benefits of being a student, if you do any kind of research or hobbyist reading—you can request almost anything you can find in , whether it’s an ultra-obscure book or a master’s thesis from 1950! Why wouldn’t you make regular use of it‽ Of things I miss from being a student, ILL is near the top.↩︎

  5. The complaint and indictment are not necessarily the same thing. An indictment frequently will leave out many details and confine itself to listing what the defendant is accused of. Complaints tend to be much richer in detail. However, sometimes there will be only one and not the other, perhaps because the more detailed complaint has been sealed (possibly precisely because it is more detailed).↩︎

  6. Trial testimony can run to hundreds of pages and blow through your remaining PACER budget, so one must be careful. In particular, testimony operates under an interesting & system related to how report—who are not necessarily paid employees but may be contractors or freelancers—intended to ensure covering transcription costs: the transcript initially may cost hundreds of dollars, intended to extract full value from those who need the trial transcript immediately, such as lawyers or journalists, but then a while later, PACER drops the price to something more reasonable. That is, the first “original” fee costs a fortune, but then “copy” fees are cheaper. So for , the “original”, when ordered within hours of the testimony, will cost <$7.25/page but then the second person ordering the same transcript pays only <$1.20/page & everyone subsequently <$0.90/page, and as further time passes, that drops to <$0.60 (and I believe after a few months, PACER will then charge only the standard $0.10). So, when it comes to trial transcript on PACER, patience pays off.↩︎

  7. I’ve heard that LexisNexis terminals are sometimes available for public use in places like federal libraries or courthouses, but I have never tried this myself.↩︎

  8. I advise prepending, like instead of appending, like because the former is slightly easier to type but more importantly, Sci-Hub does not have SSL certificates set up properly (I assume they’re missing a wildcard) and so appending the Sci-Hub domain will fail to work in many web browsers due to HTTPS errors! However, if prepended, it’ll always work correctly.↩︎

  9. To further illustrate this IA feature: if one was looking for Alex St. John’s , a 2013 account of organizing the wild 1996 Doom tournament thrown by Microsoft, but one didn’t have the URL handy, one could search the entire domain by going to*/* and using the filter with “judgment”, or if one at least remembered it was in 2013, one could narrow it down further to*/* and then filter or search by hand.↩︎

  10. If any Blogspot employee is reading this, for god’s sake stop this insanity!↩︎

  11. Uploading is not as hard as it may seem. There is a web interface (user/password: “genesis”/“upload”). Uploading large files can fail, so I usually use the FTP server: curl -T "$FILE" ↩︎

  12. Although flatbed scanning is sometimes destructive too—I’ve cracked the spine of books while pressing them flat into a flatbed scanner.↩︎

  13. My workaround is to export from gscan2pdf as DjVu, which avoids the bug, then convert the DjVu files with ddjvu -format=pdf; this strips any OCR, so I add OCR with and metadata with exiftool.↩︎