Torrent

The images have been downloaded using a curl script & the Danbooru API, and losslessly optimized using optipng/jpegoptim²; the metadata has been exported from the Danbooru BigQuery mirror.³

Torrents are the preferred download method as they stress the seed server less, can potentially be faster due to many peers, are resilient to server downtime, and have built-in ECC.

Due to the number of files, the torrent has been broken up into 10 separate torrents, each covering a range of IDs modulo 1000. They are available as an XZ-compressed tarball (full archive) (19MB) and the SFW 512px downscaled subset torrent (11MB); download & unpack into one’s torrent directory.

The torrents appear to work with rTorrent on Linux & Transmission on Linux/Windows; it reportedly doesn’t work on Qbittorrent 3.3-4.0.4 (but may on >=4.0.5⁴), Deluge, or most Windows torrent clients.

Rsync

Due to torrent compatibility & network issues, I provide an alternate download route via a public anonymous rsync server. To list all available files:

rsync --verbose --recursive rsync://78.46.86.149:873/danbooru2018/ ./danbooru2018/

For a single file (eg the metadata tarball), one can download like thus:

rsync --verbose rsync://78.46.86.149:873/danbooru2018/metadata.json.tar.xz ./

For a specific subset, like the SFW 512px subset:

rsync --recursive --times --verbose rsync://78.46.86.149:873/danbooru2018/512px/ ./danbooru2018/512px/
rsync --recursive --times --verbose rsync://78.46.86.149:873/danbooru2018/original/ ./danbooru2018/original/

And for the full dataset (metadata+original+512px):

rsync --recursive --times --verbose rsync://78.46.86.149:873/danbooru2018 ./danbooru2018/

Kaggle

A combination of a n=300k subset of the 512px SFW subset of Danbooru2017 and Nagadomi’s moeimouto face dataset are available as a Kaggle-hosted dataset: “Tagged Anime Illustrations” (36GB).

Kaggle also hosts the metadata of Safebooru up to 20 November 2016: “Safebooru—Anime Image Metadata”.

Updating

If there is interest, the dataset will be updated at regular annual intervals (“Danbooru2019”, “Danbooru2020” etc).

Updates exploits the ECC capability of BitTorrent by updating the images/metadata and creating a new .torrent file; users download the new .torrent, overwrite the old .torrent, and after rehashing files to discover which ones have changed/are missing, the new ones are downloaded. (This method has been successfully used by other periodically-updated large torrents, such as the Touhou Lossless Music Torrent, at ~1.75tb after 19 versions.)

Turnover in BitTorrent swarms means that earlier versions of the torrent will quickly disappear, so for easier reproducibility, the metadata files can be archived into subdirectories (images generally will not change, so reproducibility is less of a concern—to reproduce the subset for an earlier release, one simply filters on upload date or takes the file list from the old metadata).

Size and metadata

Image classification has been supercharged by work on ImageNet, but ImageNet itself is limited by its small set of classes, many of which are debatable, and which encompass only a limited set. Compounding these limits, tagging/classification datasets are notoriously undiverse & have imbalance problems or are small:

• ImageNet: dog breeds (memorably brought out by DeepDream)

  • WebVision (Li et al 2017a; Li et al 2017b; Guo et al 2018): 2.4m images noisily classified via search engine/Flickr queries into the ImageNet 1k categories

• Youtube-BB: toilets/giraffes

• MS COCO: bathrooms and African savannah animals; 328k images, 80 categories, short 1-sentence descriptions

• bird/flowers: a few score of each kind (eg no eagles in the birds dataset)

• Visual Relationship Detection (VRD) dataset: 5k images

• Pascal VOC: 11k images

• Visual Genome: 108k images

• nico-opendata: 400k, but SFW & restricted to approved researchers

• Open Images V4: released 2018, 30.1m tags for 9.2m images and 15.4m bounding-boxes, with high label quality; a major advantage of this dataset is that it uses CC-BY-licensed Flickr photographs/images, and so it should be freely distributable,

• BAM! (Wilber et al 2017): 65m raw images, 393k? tags for 2.5m? tagged images (semi-supervised), restricted access?

The external validity of classifiers trained on these datasets is somewhat questionable as the learned discriminative models may collapse or simplify in undesirable ways, and overfit on the datasets’ individual biases (Torralba & Efros 2011). For example, ImageNet classifiers sometimes appear to ‘cheat’ by relying on localized textures in a “bag-of-words”-style approach and simplistic outlines/shapes—recognizing leopards only by the color texture of the fur, or believing barbells are extensions of arms. CNNs by default appear to rely almost entirely on texture and ignore shapes/outlines, unlike human vision, rendering them fragile to transforms; training which emphasizes shape/outline data augmentation can improve accuracy & robustness (Geirhos et al 2018), making anime images a challenging testbed (and this texture-bias possibly explaining poor performance of anime-targeted NNs in the past). The dataset is simply not large enough, or richly annotated enough, to train classifiers or tagger better than that, or, with residual networks reaching human parity, reveal differences between the best algorithms and the merely good. (Dataset biases have also been issues on question-answering datasets.) As well, the datasets are static, not accepting any additions, better metadata, or corrections. Like MNIST before it, ImageNet is verging on ‘solved’ (the ILSVRC organizers ended it after the 2017 competition) and further progress may simply be overfitting to idiosyncrasies of the datapoints and errors; even if lowered error rates are not overfitting, the low error rates compress the differences between algorithm, giving a misleading view of progress and understating the benefits of better architectures, as improvements become comparable in size to simple chance in initializations/training/validation-set choice. As Dong et al 2017 note:

It is an open issue of text-to-image mapping that the distribution of images conditioned on a sentence is highly multi-modal. In the past few years, we’ve witnessed a breakthrough in the application of recurrent neural networks (RNN) to generating textual descriptions conditioned on images [1, 2], with Xu et al. showing that the multi-modality problem can be decomposed sequentially [3]. However, the lack of datasets with diversity descriptions of images limits the performance of text-to-image synthesis on multi-categories dataset like MSCOCO [4]. Therefore, the problem of text-to-image synthesis is still far from being solved

In contrast, the Danbooru dataset is larger than ImageNet as a whole and larger than the most widely-used multi-description dataset, MS COCO, with far richer metadata than the ‘subject verb object’ sentence summary that is dominant in MS COCO or the birds dataset (sentences which could be adequately summarized in perhaps 5 tags, if even that⁷). While the Danbooru community does focus heavily on female anime characters, they are placed in a wide variety of circumstances with numerous surrounding tagged objects or actions, and the sheer size implies that many more miscellaneous images will be included. It is unlikely that the performance ceiling will be reached anytime soon, and advanced techniques such as attention will likely be required to get anywhere near the ceiling. And Danbooru is constantly expanding and can be easily updated by anyone anywhere, allowing for regular releases of improved annotations.

Danbooru and the image boorus have been only minimally used in previous machine learning work; principally, in “Illustration2Vec: A Semantic Vector Representation of Images”, Saito & Matsui 2015 (project), which used 1.287m images to train a finetuned VGG-based CNN to detect 1,539 tags (drawn from the 512 most frequent tags of general/copyright/character each) with an overall precision of 32.2%, or “Symbolic Understanding of Anime Using Deep Learning”, Li 2018 But the datasets for past research are typically not distributed and there has been little followup.

Non-photographic

Anime images and illustrations, on the other hand, as compared to photographs, differ in many ways—for example, illustrations are frequently black-and-white rather than color, line art rather than photographs, and even color illustrations tend to rely far less on textures and far more on lines (with textures omitted or filled in with standard repetitive patterns), working on a higher level of abstraction—a leopard would not be as trivially recognized by simple pattern-matching on yellow and black dots—with irrelevant details that a discriminator might cheaply classify based on typically suppressed in favor of global gestalt, and often heavily stylized (eg frequent use of “Dutch angles”). With the exception of MNIST & Omniglot, almost all commonly-used deep learning-related image datasets are photographic.

Humans can still easily perceive a black-white line drawing of a leopard as being a leopard—but can a standard ImageNet classifier? Likewise, the difficulty face detectors encounter on anime images suggests that other detectors like nudity or pornographic detectors may fail; but surely moderation tasks require detection of penises, whether they are drawn or photographed? The attempts to apply CNNs to GANs, image generation, image inpainting, or style transfer have sometimes thrown up artifacts which don’t seem to be issues when using the same architecture on photographic material; for example, in GAN image generation & style transfer, I almost always note, in my own or others’ attempts, what I call the “watercolor effect”, where instead of producing the usual abstracted regions of whitespace, monotone coloring, or simple color gradients, the CNN instead consistently produces noisy transition textures which look like watercolor paintings—which can be beautiful, and an interesting style in its own right (eg the style2paints samples), but means the CNNs are failing to some degree. This watercolor effect appears to not be a problem in photographic applications, but on the other hand, photos are filled with noisy transition textures and watching a GAN train, you can see that the learning process generates textures first and only gradually learns to build edges and regions and transitions from the blurred texts; is this anime-specific problem due to simply insufficient data/training, or is there something more fundamentally the issue with current convolutions?

Because illustrations are produced by an entirely different process and focus only on salient details while abstracting the rest, they offer a way to test external validity and the extent to which taggers are tapping into higher-level semantic perception.

As well, many ML researchers are anime fans and might enjoy working on such a dataset—training NNs to generate anime images can be amusing. It is, at least, more interesting than photos of street signs or storefronts. (“There are few sources of energy so powerful as a procrastinating grad student.”)

Community value

A full dataset is of immediate value to the Danbooru community as an archival snapshot of Danbooru which can be downloaded in lieu of hammering the main site and downloading terabytes of data; backups are occasionally requested on the Danbooru forum but the need is currently not met.

There is potential for a symbiosis between the Danbooru community & ML researchers: in a virtuous circle, the community provides curation and expansion of a rich dataset, while ML researchers can contribute back tools from their research on it which help improve the dataset. The Danbooru community is relatively large and would likely welcome the development of tools like taggers to support semi-automatic (or eventually, fully automatic) image tagging, as use of a tagger could offer orders of magnitude improvement in speed and accuracy compared to their existing manual methods, as well as being newbie-friendly⁸ They are also a pre-existing audience which would be interested in new research results.

Image Metadata

The metadata is available as a XZ-compressed tarball of JSON files as exported from the Danbooru BigQuery database mirror (metadata.json.tar.xz). Each line is an individual JSON object for a single image; ad hoc queries can be run easily by piping into jq.

Danbooru2017

The first release, Danbooru2017, contained ~1.9tb of 2.94m images with 77.5m tag instances (of 333k defined tags, ~26.3/image) covering Danbooru from 24 May 2005 through 31 December 2017 (final ID: #2,973,532).

Danbooru2018 added 0.413TB/392,557 images/15,208,974 tags/31,698 new unique tags.

To reconstruct Danbooru2017, download Danbooru2018, and take the image subset ID #1-2973532 as the image dataset, and the JSON metadata in the subdirectory metadata/2017/ as the metadata. That should give you Danbooru2017 as released on 2018-02-13.

Model zoo

If possible, additional models and derived metadata may be supplied as part of a “model zoo”. Particularly desirable would be:

• “s”/“q”/“e” classifier
• top-10,000-tag tagger
• text embedding RNN, and pre-computed text embeddings for all images’ tags

Metadata Quality Improvement via Active Learning

How high quality is the Danbooru metadata quality? As with ImageNet, it is critical that the tags are extremely accurate or else this will lowerbound the error rates and impede the learning of taggers, especially on rarer tags where a low error may still cause false negatives to outweigh the true positives.

I would say that the Danbooru tag data is quite high but imbalanced: almost all tags on images are correct, but the absence of a tag is often wrong—that is, many tags are missing on Danbooru (there are so many possible tags that no user could possibly know them all). So the absence of a tag isn’t as informative as the presence of a tag—eyeballing images and some rarer tags, I would guess that tags are present <10% of the time they should be.

This suggests leveraging an active learning (Settles 2010) form of training: train a tagger, have a human review the errors, update the metadata when it was not an error, and retrain.

More specifically: train the tagger; run the tagger on the entire dataset, recording the outputs and errors; a human examines the errors interactively by comparing the supposed error with the image; and for false negatives, the tag can be added to the Danbooru source using the Danbooru API and added to the local image metadata database, and for false positives, the ‘negative tag’ can be added to the local database; train a new model (possibly initializing from the last checkpoint). Since there will probably be thousands of errors, one would go through them by magnitude of error: for a false positive, start with tagging probabilities of 1.0 and go down, and for false negatives, 0.0 and go up. This would be equivalent to the active learning strategy “uncertainty sampling”, which is simple, easy to implement, and effective (albeit not necessarily optimal for active learning as the worst errors will tend to be highly correlated/redundant and the set of corrections overkill). Once all errors have been hand-checked, the training weight on absent tags can be increased, as any missing tags should have shown up as false positives.

Over multiple iterations of active learning + retraining, the procedure should be able to ferret out errors in the dataset and boost its quality while also increasing its performance.

Based on my experiences with semi-automatic editing on Wikipedia (using pywikipediabot for solving disambiguation wikilinks), I would estimate that given an appropriate terminal interface, a human should be able to check at least 1 error per second and so checking ~30,000 errors per day is possible (albeit extremely tedious). Fixing the top million errors should offer a noticeable increase in performance.

There are many open questions about how best to optimize tagging performance: is it better to refine tags on the existing set of images or would adding more only-partially-tagged images be more useful?

Shell queries for statistics

# count number of images/files in Danbooru2018
find /media/gwern/My\ Book/danbooru2018/original/ -type f | wc --lines
2941205
# count total filesize of images in Danbooru2018
du -sch /media/gwern/My\ Book/danbooru2018/original/
1.9TB

# on JSON files concatenated together:
## number of unique tags
cat all.json | jq '.tags | .[] | .name' > tags.txt
sort -u tags.txt  | wc --lines
# 333333
## number of total tags
wc --lines tags.txt
# 77565442
## Average tag count per image:
R
# R> 77565442 / 2941205
# [1] 26.37199447
## Most popular tags:
sort tags.txt  | uniq -c | sort -g | tac | head -19
# 2060363 "1girl"
# 1710762 "solo"
# 1318516 "long_hair"
# 1018512 "highres"
#  900500 "breasts"
#  870086 "blush"
#  813241 "short_hair"
#  800738 "smile"
#  662570 "multiple_girls"
#  651028 "open_mouth"
#  631508 "looking_at_viewer"
#  585282 "blue_eyes"
#  580097 "blonde_hair"
#  573047 "touhou"
#  541144 "brown_hair"
#  508437 "skirt"
#  478805 "hat"
#  466367 "thighhighs"
#  450133 "black_hair"

## count Danbooru images by rating
cat all.json  | jq '.rating' > ratings.txt
sort ratings.txt  | uniq -c | sort -g
#  257967 "e"
#  443750 "q"
# 2262211 "s"
wc --lines ratings.txt
## 2963928 ratings.txt
R
# R> c(257967,  443750, 2262211) / 2963928
# [1] 0.08703551503 0.14971686222 0.76324762275

# earliest upload:
cat all.json | jq '.created_at' | fgrep '2005' > uploaded.txt
sort -g uploaded.txt | head -1
# "2005-05-24 03:35:31 UTC"