---
title: Making Anime Faces With StyleGAN
description: A tutorial explaining how to train and generate high-quality anime faces with StyleGAN 1+2 neural networks, and tips/scripts for effective StyleGAN use.
thumbnail: /doc/ai/nn/gan/stylegan/anime/gwern-stylegan-asuka-face-sample.png
thumbnail-text: "A hand-selected StyleGAN sample from my Asuka-finetuned TWDNE StyleGAN: A blond-haired blue-eyed anime face looking at the viewer based on the Neon Genesis Evangelion character, Asuka Souryuu Langley."
created: 2019-02-04
modified: 2022-10-19
status: finished
confidence: highly likely
importance: 5
css-extension: dropcaps-kanzlei
...
> Generative neural networks, such as GANs, have [struggled for years](#why-dont-gans-work) to generate decent-quality anime faces, despite their great success with photographic imagery such as real human faces. The task has now been effectively solved, for anime faces as well as many other domains, by the development of a new generative adversarial network, [**StyleGAN**](https://arxiv.org/abs/1812.04948#nvidia "'A Style-Based Generator Architecture for Generative Adversarial Networks', Karras et al 2018"), whose [source code](https://github.com/NVlabs/stylegan) was released in February 2019.
>
> I [show off](#examples) my StyleGAN 1+2 CC-0-licensed anime faces & videos, provide downloads for the final models & [anime portrait face dataset](/crop#danbooru2019-portraits "'Anime Crop Datasets: Faces, Figures, & Hands § Danbooru2019 Portraits', Branwen et al 2020"), provide the 'missing manual' & explain how I trained them based on [Danbooru2017/2018](/danbooru2021#danbooru2018) with source code for the [data preprocessing](#data-preparation), document [installation](#installation) & [configuration](#configuration) & [training tricks](#running).
>
> For application, I document various scripts for generating [images & videos](#sampling), briefly [describe the website](#twdne) ["This Waifu Does Not Exist"](https://www.thiswaifudoesnotexist.net/ "'ThisWaifuDoesNotExist.net', Gwern 2019") [I set up](/twdne "'This Waifu Does Not Exist', Gwern 2019") as a public demo & [its followup](#extended-stylegan2-danbooru2019-aydao) [This Anime Does Not Exist.ai (TADNE)](https://thisanimedoesnotexist.ai/) (see also [Artbreeder](https://www.artbreeder.com/)), discuss how the trained models can be [used for transfer learning](#transfer-learning) such as generating high-quality faces of anime characters with small datasets (eg. [Holo](#holo) or [Asuka Souryuu Langley](#asuka)), and touch on [more advanced StyleGAN applications](#reversing-stylegan-to-control-modify-images) like encoders & controllable generation.
>
> The [anime face graveyard](/face-graveyard "'Anime Neural Net Graveyard', Gwern 2019") gives samples of my failures with earlier GANs for anime face generation, and I provide samples & model from a relatively large-scale [BigGAN](/biggan "‘Making Anime With BigGAN’, Gwern 2019") training run suggesting that [BigGAN](https://arxiv.org/abs/1809.11096#deepmind "‘Large Scale GAN Training for High Fidelity Natural Image Synthesis’, Brock et al 2018") may be the next step forward to generating full-scale anime images.
>
> A minute of reading could save an hour of debugging!
>
>
>
StyleGAN2 Is Obsolete
> As of 2022, StyleGAN2 is obsolete for anime generation. Alternatives include (newest-first): [Pony Diffusion v6 XL](https://civitai.com/models/257749), [niji • journey](https://nijijourney.com/en/), [Holara](https://holara.ai/holara/), [Waifu-Diffusion](https://huggingface.co/hakurei/waifu-diffusion), [NovelAI](https://www.reddit.com/r/NovelAi/comments/xu8xpg/novelai_image_generation_launch_announcement/), [Waifu Labs](https://waifulabs.com/), or [Crypko](https://crypko.ai/).
>
When Ian Goodfellow's first [GAN](!W "Generative adversarial network") paper [came out in 2014](https://arxiv.org/abs/1406.2661 "'Generative Adversarial Networks', Goodfellow et al 2014"), with its blurry 64px grayscale faces, I said to myself, "given the rate at which GPUs & NN architectures improve, in a few years, we'll probably be able to throw a few GPUs at some anime collection like Danbooru and the results will be *hilarious*."
There is something intrinsically amusing about trying to make computers draw anime, and it would be much more fun than working with yet more celebrity headshots or ImageNet samples; further, anime/illustrations/drawings are so different from the exclusively-photographic datasets always (over)used in contemporary ML research that I was curious how it would work on anime---better, worse, faster, or different failure modes?
Even more amusing---if random images become doable, then text → images would not be far behind.
So when GANs hit [128px color images on ImageNet](https://arxiv.org/abs/1606.03498#openai "'Improved Techniques for Training GANs', Salimans et al 2016"), and could do somewhat passable [CelebA face samples](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html "'Large-scale CelebFaces Attributes (CelebA) Dataset', Liu et al 2015") around 2015, along with my [char-RNN experiments](/rnn-metadata "Teaching a text-generating char-RNN to automatically imitate many different authors by labeling the input text by author; additional experiments include imitating Geocities and retraining GPT-2 on a large Project Gutenberg poetry corpus."), I began experimenting with [Soumith Chintala's implementation](https://github.com/soumith/dcgan.torch) of [DCGAN](https://arxiv.org/abs/1511.06434 "'Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks', Radford et al 2015"), restricting myself to faces of single anime characters where I could easily scrape up ~5--10k faces.
(I did a lot of [Asuka Souryuu Langley](!W) from _[Neon Genesis Evangelion](!W)_ because she has a color-centric design which made it easy to tell if a GAN run was making any progress: blonde-red hair, blue eyes, and red hair ornaments.)
It did not work.
Despite many runs on my laptop & a borrowed desktop, DCGAN never got remotely near to the level of the CelebA face samples, typically topping out at reddish blobs before diverging or outright crashing.^[Turns out that when training goes *really* wrong, you can crash many GAN implementations with either a segfault, integer overflow, or division by zero error.]
Thinking perhaps the problem was too-small datasets & I needed to train on *all* the faces, I began creating the Danbooru2017 version of ["Danbooru2018: A Large-Scale Crowdsourced & Tagged Anime Illustration Dataset"](/danbooru2021#danbooru2020 "Danbooru2018 is a large-scale anime image database with 3.33m+ images annotated with 99.7m+ tags; it can be useful for machine learning purposes such as image recognition and generation").
Armed with a large dataset, I subsequently began working through particularly promising members of the [GAN zoo](https://github.com/hindupuravinash/the-gan-zoo), emphasizing SOTA & open implementations.
Among others, [I have](https://x.com/gwern/status/828311639472611328) [tried](https://x.com/gwern/status/828718629181075466) [StackGAN](https://arxiv.org/abs/1612.03242 "'StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks', Zhang et al 2016")/[StackGAN++](https://arxiv.org/abs/1710.10916 "'StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks', Zhang et al 2017") & [Pixel\*NN\*](https://towardsdatascience.com/auto-regressive-generative-models-pixelrnn-pixelcnn-32d192911173 "Auto-Regressive Generative Models (PixelRNN, PixelCNN++)") (failed to get running)^[StackGAN/StackGAN++/PixelCNN et al are difficult to run as they require an unique image embedding which could only be computed in the unmaintained Torch framework using Reed's prior work on a joint text+image embedding which however doesn't run on anything but the Birds & Flowers datasets, and so no one has ever, as far as I am aware, run those implementations on anything else---certainly I never managed to despite quite a few hours trying to reverse-engineer the embedding & various implementations.], [WGAN-GP](/face-graveyard#wgan), [Glow](/face-graveyard#glow), [GAN-QP](/face-graveyard#gan-qp), [MSG-GAN](/face-graveyard#msg-gan), [SAGAN](/face-graveyard#self-attention-gan-tensorflow), [[IntroVAE](/face-graveyard#introvae)]{#introvae} [VGAN](/face-graveyard#vgan), [PokeGAN](/face-graveyard#pokegan), [BigGAN](/face-graveyard#biggan-unofficial)^[Be sure to check out [Artbreeder](https://www.artbreeder.com/).], [ProGAN](/face-graveyard#progan){#progan}, & StyleGAN.
These architectures vary widely in their design & core algorithms and which of the many stabilization tricks ([Wiatrak & Albrecht 2019](https://arxiv.org/abs/1910.00927 "Stabilizing Generative Adversarial Network Training: A Survey")) they use, but they were more similar in their results: dismal.
Glow & BigGAN had promising results reported on CelebA & ImageNet respectively, but unfortunately their training requirements were out of the question.^[Glow's reported results [required >40 GPU-weeks](https://github.com/openai/glow/issues/37); BigGAN's total compute is unclear as it was trained on a TPUv3 Google cluster but it would appear that a 128px BigGAN might be ~4 GPU-months assuming hardware like an 8-GPU machine, 256px ~8 GPU-months, and 512px ≫8 GPU-months, with VRAM being the main limiting factor for larger models (although progressive growing might be able to cut those estimates).]
(As interesting as [SPIRAL](https://arxiv.org/abs/1804.01118#deepmind "'SPIRAL: Synthesizing Programs for Images using Reinforced Adversarial Learning', Ganin et al 2018") and [CAN](https://arxiv.org/abs/1706.07068 "'CAN: Creative Adversarial Networks, Generating 'Art' by Learning About Styles and Deviating from Style Norms', Elgammal et al 2017") are, no source was released and I couldn't even attempt them.)
While some remarkable tools like [PaintsTransfer/style2paints](https://github.com/lllyasviel/style2paints "'Style2Paints GitHub repository', Zhang et al 2018") were created, and there were the occasional semi-successful anime face GANs like [IllustrationGAN](https://github.com/tdrussell/IllustrationGAN), the most notable attempt at anime face generation was [Make Girls.moe](https://make.girls.moe/) ([Jin et al 2017](https://arxiv.org/abs/1708.05509 "Towards the Automatic Anime Characters Creation with Generative Adversarial Networks")).
MGM could, interestingly, do *in-browser* 256px anime face generation using tiny GANs, but that is a dead end.
MGM accomplished that much by making the problem easier: they added some light supervision in the form of a crude tag embedding[^illustration2vec], and then simplifying the problem drastically to _n_ = 42k faces cropped from professional video game character artwork, which I regarded as not an acceptable solution---the faces were small & boring, and it was unclear if this data-cleaning approach could scale to anime faces in general, much less anime images in general.
They are recognizably anime faces but the resolution is low and the quality is not great:
[^illustration2vec]: [`illustration2vec`](/doc/ai/anime/2015-saito.pdf "'Illustration2Vec: A Semantic Vector Representation of Illustrations', Saito & Matsui") is an old & small CNN trained to predict a few -booru tags on anime images, and so provides an embedding---but not a good one. The lack of a good embedding is the major limitation for anime deep learning as of February 2019. ([DeepDanbooru](https://www.reddit.com/r/MachineLearning/comments/akbc11/p_tag_estimation_for_animestyle_girl_image/), while performing well apparently, has not yet been used for embeddings.) An embedding is necessary for text → image GANs, image searches & nearest-neighbor checks of overfitting, FID errors for objectively comparing GANs, [minibatch discrimination](https://arxiv.org/pdf/1606.03498#page=3&org=openai){#minibatch-discrimination-1} to help the D/provide an auxiliary loss to stabilize learning, anime style transfer (both for its own sake & for creating a 'StyleDanbooru2018' to reduce texture cheating), encoding into GAN latent spaces for manipulation, data cleaning (to detect anomalous datapoints like failed face crops), perceptual losses for encoders or as an additional auxiliary loss/pretraining (like ["NoGAN"](https://www.fast.ai/posts/2019-05-03-decrappify.html "Decrappification, DeOldification, and Super Resolution"), which trains a Generator on a perceptual loss and does GAN training only for finetuning) etc. A good tagger is also a good starting point for doing pixel-level semantic segmentation (via "weak supervision", with Transformers working [especially well](https://arxiv.org/abs/2104.14294#facebook "'DINO: Emerging Properties in Self-Supervised Vision Transformers', Caron et al 2021")), which metadata is key for training something like Nvidia's [GauGAN](https://blogs.nvidia.com/blog/gaugan-photorealistic-landscapes-nvidia-research/) successor to pix2pix ([Park et al 2019](https://arxiv.org/abs/1903.07291 "Semantic Image Synthesis with Spatially-Adaptive Normalization"); [source](https://github.com/NVlabs/SPADE)).
Typically, a GAN would diverge after a day or two of training, or it would collapse to producing a limited range of faces (or a single face), or if it was stable, simply converge to a low level of quality with a lot of fuzziness; perhaps the most typical failure mode was heterochromia (which is [common in anime](https://safebooru.org/index.php?page=post&s=list&tags=heterochromia) but not *that* common)---mismatched eye colors (each color individually plausible), from the Generator apparently being unable to coordinate with itself to pick consistently.
With more recent architectures like VGAN or SAGAN, which carefully weaken the Discriminator or which add extremely-powerful components like self-attention layers, I could reach fuzzy 128px faces.
Given the miserable failure of all the prior NNs I had tried, I had begun to seriously wonder if there was something about non-photographs which made them intrinsically unable to be easily modeled by convolutional neural networks (the common ingredient to them all).
Did convolutions render it unable to generate sharp lines or flat regions of color?
Did [regular GANs work only](#why-dont-gans-work) because photographs were made almost entirely of blurry textures?
But BigGAN demonstrated that a large cutting-edge GAN architecture could scale, given enough training, to *all of ImageNet* at even 512px.
And [ProGAN](/face-graveyard#progan) demonstrated that regular CNNs *could* learn to generate sharp clear anime images with only *somewhat* infeasible amounts of training.
[ProGAN](https://arxiv.org/abs/1710.10196#nvidia "'ProGAN: Progressive Growing of GANs for Improved Quality, Stability, and Variation', Karras et al 2017") ([source](https://github.com/tkarras/progressive_growing_of_gans); [video](https://www.youtube.com/watch?v=XOxxPcy5Gr4)), while expensive and requiring >6 GPU-weeks^[Technical note: I typically train NNs using my workstation with 2×1080ti GPUs. For easier comparison, I convert all my times to single-GPU equivalent (ie. "6 GPU-weeks" means 3 realtime/wallclock weeks on my 2 GPUs).], did *work* and was even powerful enough to overfit single-character face datasets; I didn't have enough GPU time to train on unrestricted face datasets, much less anime images in general, but merely getting this far was exciting.
Because, a common sequence in DL/DRL (unlike many areas of AI) is that a problem seems intractable for long periods, until someone modifies a scalable architecture slightly, produces somewhat-credible (not necessarily human or even near-human) results, and then throws a ton of compute/data at it and, since the architecture scales, it rapidly exceeds SOTA and approaches human levels (and potentially exceeds human-level).
Now I just needed a faster GAN architecture which I could train a much bigger model with on a much bigger dataset.
StyleGAN was the final breakthrough in providing ProGAN-level capabilities but fast: by switching to a radically different architecture, it minimized the need for the slow progressive growing (perhaps eliminating it entirely^[[Kynkäänniemi et al 2019](https://arxiv.org/abs/1904.06991 "Improved Precision and Recall Metric for Assessing Generative Models") observes (§4 "Using precision and recall to analyze and improve StyleGAN") that StyleGAN with progressive growing disabled *does* work but at some cost to precision/recall quality metrics; whether this reflects inferior performance on a given training budget or an inherent limit---BigGAN and other self-attention-using GANs do not use progressive growing at all, suggesting it is not *truly* necessary---is not investigated. In December 2019, StyleGAN 2 successfully dropped progressive growing entirely at modest performance cost.]), and learned efficiently at multiple levels of resolution, with bonuses in providing much more control of the generated images with its "style transfer" metaphor.
# Examples
First, some demonstrations of what is possible with StyleGAN on anime faces:
 StyleGAN](/doc/ai/nn/gan/stylegan/anime/gwern-stylegan-asuka-face-sample.png "A blond-haired blue-eyed anime face looking at the viewer based on the _Neon Genesis Evangelion_ character, Asuka Souryuu Langley.")
{.width-full}
{.width-full}
Even a quick look at the MGM & StyleGAN samples demonstrates the latter to be superior in resolution, fine details, and overall appearance (although the MGM faces admittedly have fewer global mistakes).
It is also superior to my 2018 ProGAN faces.
Perhaps the most striking fact about these faces, which should be emphasized for those fortunate enough not to have spent as much time looking at awful GAN samples as I have, is not that the individual faces are good, but rather that the faces are so *diverse*, particularly when I look through face samples with [Ψ ≥ 1](#psitruncation-trick)---it is not just the hair/eye color or head orientation or fine details that differ, but the overall style ranges from CG to cartoon sketch, and even the 'media' differ, I could swear many of these are trying to imitate watercolors, charcoal sketching, or oil painting rather than digital drawings, and some come off as recognizably '90s-anime-style vs '00s-anime-style.
(I could look through samples all day despite the global errors because so many are interesting, which is not something I could say of the MGM model whose novelty is quickly exhausted, and it appears that users of my TWDNE website feel similarly as the average length of each visit is 1m:55s.)
```{=HTML}
Interpolation video of the 2019-02-11 face StyleGAN demonstrating generalization.
```
[^viral], in happier times.](/doc/ai/nn/gan/stylegan/anime/2019-02-13-twitter-_ryobot-styleganelonmusklike.jpg)
[^viral]: This has confused some people, so to clarify the sequence of events: I trained my anime face StyleGAN and posted notes on Twitter, releasing an early model; [roadrunner01](https://x.com/ak92501) generated an interpolation video using said model (but a different random seed, of course); this interpolation video was retweeted by the Japanese Twitter user [\_Ryobot](https://x.com/_Ryobot), upon which it went viral and was 'liked' by Elon Musk, further driving virality (19k reshares, 65k likes, 1.29m watches as of 2019-03-22).
```{=HTML}
Later interpolation video (2019-03-08 face StyleGAN)
```
# Background
)](/doc/ai/nn/gan/stylegan/anime/2020-06-30-shawnpresser-stylegan-ffhq-imagepyramid.jpg){.float-left}
StyleGAN was published in 2018 as ["A Style-Based Generator Architecture for Generative Adversarial Networks", Karras et al 2018](https://arxiv.org/abs/1812.04948#nvidia "'A Style-Based Generator Architecture for Generative Adversarial Networks', Karras et al 2018") ([source code](https://github.com/NVlabs/stylegan); [demo video](https://www.youtube.com/watch?v=kSLJriaOumA)/[algorithmic review video](https://www.youtube.com/watch?v=SPI5uGCnxlc "[StyleGAN] A Style-Based Generator Architecture for GANs, part 1 (algorithm review) | TDLS [2019-01-24]")/[results & discussions video](https://www.youtube.com/watch?v=_bh3U9HB-kg "[StyleGAN] A Style-Based Generator Architecture for GANs, part 2 (results and discussion) | TDLS [2019-01-24]"); [Colab notebook](https://github.com/ak9250/stylegan-art/blob/master/styleganportraits.ipynb)[^Colab-limitations]; [GenForce PyTorch reimplementation](https://github.com/genforce/genforce "GenForce: an efficient PyTorch library for deep generative modeling (StyleGAN v1v2, PGGAN, etc.)") with model zoo/[Keras](https://github.com/manicman1999/StyleGAN-Keras); explainers: [Skymind.ai](https://yippy.ai/skymind)/[Lyrn.ai](https://www.lyrn.ai/2018/12/26/a-style-based-generator-architecture-for-generative-adversarial-networks/)/[Two Minute Papers video](https://www.youtube.com/watch?v=1ct_P3IZow0)).
StyleGAN takes the standard GAN architecture embodied by ProGAN (whose source code it reuses) and, like the similar GAN architecture [Chen et al 2018](https://arxiv.org/abs/1810.01365 "On Self Modulation for Generative Adversarial Networks"), draws inspiration from the field of "style transfer" (essentially invented by [Gatys et al 2014](https://arxiv.org/abs/1508.06576 "'A Neural Algorithm of Artistic Style', Gatys et al 2015")), by changing the Generator (G) which creates the image by repeatedly upscaling its resolution to take, at each level of resolution from 8px → 16px → 32px → 64px → 128px etc. a random input or "style noise", which is combined with [AdaIN](https://arxiv.org/abs/1703.06868 "'AdaIN: Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization', Huang & Belongie 2017") and is used to tell the Generator how to 'style' the image at that resolution by changing the hair or changing the skin texture and so on.
'Style noise' at a low resolution like 32px affects the image relatively globally, perhaps determining the hair length or color, while style noise at a higher level like 256px might affect how frizzy individual strands of hair are.
In contrast, ProGAN and almost all other GANs inject noise into the G as well, but only at the beginning, which appears to work not nearly as well (perhaps because it is difficult to propagate that randomness 'upwards' along with the upscaled image itself to the later layers to enable them to make consistent choices?).
To put it simply, by systematically providing a bit of randomness at each step in the process of generating the image, StyleGAN can 'choose' variations effectively.
[^Colab-limitations]: Google Colab is a free service includes free GPU time (up to 12 hours on a small GPU). Especially for people who do not have a reasonably capable GPU on their personal computers (such as all Apple users) or do not want to engage in the admitted hassle of renting a real cloud GPU instance, Colab can be a great way to play with a pretrained model, like generating GPT-2-117M text completions or StyleGAN interpolation videos, or prototype on tiny problems.
However, it is a bad idea to try to train real models, like 512--1024px StyleGANs, on a Colab instance as the GPUs are low VRAM, far slower (6 hours per StyleGAN tick!), unwieldy to work with (as one must save snapshots constantly to restart when the session runs out), doesn't have a real command-line, etc. Colab is just barely adequate for perhaps 1 or 2 ticks of transfer learning, but not more. If you harbor greater ambitions but still refuse to spend any money (rather than time), Kaggle has a similar service with P100 GPU slices rather than K80s. Otherwise, one needs to get access to real GPUs.
![Karras et al 2018, StyleGAN vs ProGAN architecture: "Figure 1. While a traditional generator [\[29\]](https://arxiv.org/abs/1710.10196#nvidia "'ProGAN: Progressive Growing of GANs for Improved Quality, Stability, and Variation', Karras et al 2017") feeds the latent code [_z_] though the input layer only, we first map the input to an intermediate latent space _W_, which then controls the generator through adaptive instance normalization (AdaIN) at each convolution layer. Gaussian noise is added after each convolution, before evaluating the nonlinearity. Here "A" stands for a learned affine transform, and "B" applies learned per-channel scaling factors to the noise input. The mapping network _f_ consists of 8 layers and the synthesis network _g_ consists of 18 layers---two for each resolution (4^2^-−1024^2^). The output of the last layer is converted to RGB using a separate 1×1 convolution, similar to Karras et al. [29]. Our generator has a total of 26.2M trainable parameters, compared to 23.1M in the traditional generator."](/doc/ai/nn/gan/stylegan/2018-karras-stylegan-figure1-styleganarchitecture.png){.invert}
StyleGAN makes a number of additional improvements, but they appear to be less important: for example, it introduces a new ["FFHQ"](https://github.com/NVlabs/ffhq-dataset "Flickr-Faces-HQ Dataset (FFHQ): a high-quality image dataset of human faces, originally created as a benchmark for generative adversarial networks (GAN)") face/portrait dataset with 1024px images in order to show that StyleGAN convincingly improves on ProGAN in final image quality; switches to a loss which is more well-behaved than the usual logistic-style losses; and architecture-wise, it makes unusually heavy use of fully-connected (FC) layers to process an initial random input, no less than 8 layers of 512 neurons, where most GANs use 1 or 2 FC layers.^[Curiously, the benefit of many more FC layers than usual may have been stumbled across before: IllustrationGAN found that adding some FC layers seemed to help their DCGAN generate anime faces, and when I & [FeepingCreature](https://github.com/FeepingCreature) experimented with adding 2--4 FC layers to WGAN-GP along IllustrationGAN's lines, it *did* help our lackluster results, and at the time [I speculated](https://github.com/martinarjovsky/WassersteinGAN/issues/2#issuecomment-278710552) that "the fully-connected layers are transforming the latent-_z_/noise into a sort of global template which the subsequent convolution layers can then fill in more locally." But we never dreamed of going as deep as 8!]
More striking is that it omits techniques that other GANs have found critical for being able to train at 512px--1024px scale: it does not use newer losses like the [relativistic loss](https://arxiv.org/abs/1807.00734 "'The relativistic discriminator: a key element missing from standard GAN', Jolicoeur-Martineau 2018"), SAGAN-style self-attention layers in either G/D, [VGAN-style](https://arxiv.org/abs/1810.00821 "'Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow', Peng et al 2018") variational Discriminator bottlenecks, conditioning on a tag or category embedding^[The ProGAN/StyleGAN codebase [reportedly does work](https://x.com/davidstap/status/1120667403837423616) with conditioning, but none of the papers report on this functionality and I have not used it myself.], BigGAN-style large minibatches, different noise distributions[^Bernoulli], advanced regularization like [spectral](https://arxiv.org/abs/1705.10941 "'Spectral norm regularization for improving the generalizability of deep learning', Yoshida & Miyato 2017") [normalization](https://arxiv.org/abs/1802.05957 "'Spectral Normalization for Generative Adversarial Networks', Miyato et al 2018"), etc.^[Which raises the question: if you added any or all of those features, would StyleGAN become that much better? Unfortunately, while theorists & practitioners have had many ideas, so far theory has proven more fecund than fatidical and the large-scale GAN experiments necessary to truly test the suggestions are too expensive for most. Half of these suggestions are great ideas---but which half?]
One possible reason for StyleGAN's success is the way it combines outputs from the multiple layers into a single final image rather than repeatedly upscaling; when we visualize the output of each layer as an RGB image in anime StyleGANs, there is a striking division of labor between layers---some layers focus on monochrome outlines, while others fill in textured regions of color, and they sum up into an image with sharp lines and good color gradients while maintaining details like eyes.
[^Bernoulli]: The latent embedding _z_ is usually generated in about the simplest possible way: draws from the Normal distribution, 𝒩(0,1). A Uniform(−1,1) is sometimes used instead. There is no good justification for this and some reason to think this can be bad (how does a GAN easily map a discrete or binary latent factor, such as the [presence or absence of the left ear](#discretized-latent-space), onto a Normal variable?).
The [BigGAN paper explores alternatives](https://arxiv.org/pdf/1809.11096.pdf#page=26 "Appendix E: Choosing Latent Spaces"), finding improvements in training time and/or final quality from using instead (in ascending order): a Normal + binary Bernoulli (_p_ = 0.5; personal communication, Brock) variable, a binary (Bernoulli), and a [Rectified Gaussian](!W "Rectified Gaussian distribution") (sometimes called a "censored normal" even though that sounds like a [truncated normal distribution](!W) rather than the rectified one). The rectified Gaussian distribution "outperforms 𝒩(0,1)(in terms of IS) by 15--20% and tends to require fewer iterations."
The downside is that the ["truncation trick"](#psitruncation-trick), which yields even larger average improvements in image quality (at the expense of diversity) doesn't quite apply, and the rectified Gaussian sans truncation produced similar results as the Normal+truncation, so BigGAN reverted to the default Normal distribution+truncation (Brock 2020, personal communication).
The truncation trick either directly applies to some of the other distributions, particularly the Rectified Gaussian, or could easily be adapted---possibly yielding an improvement over either approach. The Rectified Gaussian can be truncated just like the default Normals can. And for the Bernoulli, one could decrease _p_ during the generation, or what is probably equivalent, re-sample whenever the variance (ie. squared sum) of all the Bernoulli latent variables exceeds a certain constant. (With _p_ = 0.5, a latent vector of 512 Bernouillis would on average all sum up to simply 0.5 × 512 = 256, with the 2.5%--97.5% quantiles being 234--278, so a 'truncation trick' here might be throwing out every vector with a sum above, say, the 80% quantile of 266.)
One also wonders about vectors which draw from multiple distributions rather than just one. Could the StyleGAN 8-FC-layer learned-latent-variable [be reverse-engineered](/idea#better-initializations)? Perhaps the first layer or two merely converts the normal input into a more useful distribution & parameters/training could be saved or insight gained by imitating that.
Aside from the FCs and style noise & normalization, it is a vanilla architecture.
(One oddity is the use of only 3×3 convolutions & so few layers in each upscaling block; a more conventional upscaling block than StyleGAN's 3×3 → 3×3 would be [something like BigGAN](https://arxiv.org/pdf/1809.11096.pdf#page=18 "Figure 16: (a) A typical architectural layout for BigGAN-deep's _G_") which does 1×1 → 3×3 → 3×3 → 1×1. It's not clear if this is a good idea as it limits the spatial influence of each pixel by providing limited receptive fields[^convolution-arithmetic].)
Thus, if one has some familiarity with training a ProGAN or another GAN, one can immediately work with StyleGAN with no trouble: the training dynamics are similar and the hyperparameters have their usual meaning, and the codebase is much the same as the original ProGAN (with the main exception being that `config.py` has been renamed `train.py` (or `run_training.py` in S2) and the original `train.py`, which stores the critical configuration parameters, has been moved to `training/training_loop.py`; there is still no support for command-line options and StyleGAN must be controlled by editing `train.py`/`training_loop.py` by hand).
[^convolution-arithmetic]: For more on the choice of convolution layers/kernel sizes, see Karpathy's 2015 notes for ["CS231n: Convolutional Neural Networks for Visual Recognition"](https://cs231n.github.io/convolutional-networks/), or take a look at these [Convolution animations](https://github.com/vdumoulin/conv_arithmetic) & Yang's interactive ["Convolution Visualizer"](https://ezyang.github.io/convolution-visualizer/index.html).
## Applications
Because of its speed and stability, when the source code was released on 2019-02-04 (a date that will long be noted in the ANNals of GANime), the Nvidia models & sample dumps were quickly perused & new StyleGANs trained on a wide variety of image types, yielding, in addition to the original faces/carts/cats of Karras et al 2018:
- ["This Person Does Not Exist"](https://web.archive.org/web/20230101012202/https://thispersondoesnotexist.com/) (random samples from the 1024px FFHQ face StyleGAN)
- quizzes: ["Which Face is Real"](https://www.whichfaceisreal.com/)/["Real or Fake?"](https://blurrd.ai/realorfake/)
- voting: ["Judge Fake People"](https://thecleverest.com/judgefakepeople/main.php?sort=highest)
- [Instagram portraits](https://www.youtube.com/watch?v=VZT29WnYib4 "StyleGAN Generates Instagram Portraits, Tafseer Ahmed (ai.imagine) [2019-08-27]")
- **cats**: ["These Cats Do Not Exist"](https://thesecatsdonotexist.com/), ["This Cat Does Not Exist"](https://thiscatdoesnotexist.com/) ([cat failure modes](https://www.aiweirdness.com/gancats-19-02-07/ "GANcats"); [interpolation/style-transfer](https://x.com/genekogan/status/1093180351437029376))/[corgies](https://x.com/MichaelFriese10/status/1151236302559305728)
- hotel **rooms** (with char-RNN-generated text descriptions): ["This Rental Does Not Exist"](https://thisrentaldoesnotexist.com/)
- [kitchen/dining room/living room/bedroom](https://x.com/crschmidt/status/1099562911960350720) (using transfer learning)
- [satellite imagery](https://x.com/xsteenbrugge/status/1096820308164661248); [Gothic cathedrals](https://x.com/crschmidt/status/1097200249779769344); [Frank Gehry buildings](https://x.com/refikanadol/status/1106798493299949568); [cityscapes](https://x.com/roadrunning01/status/1109488507591028740); [floor plans](https://x.com/erikswahn/status/1123951017148788738)
- ["This Waifu Does Not Exist"](https://www.thiswaifudoesnotexist.net/ "'ThisWaifuDoesNotExist.net', Gwern 2019") ([background/implementation](/twdne "'This Waifu Does Not Exist', Gwern 2019"))
- [Artbreeder](https://www.artbreeder.com/)
large upgrade over TWDNE: random generation, exploration, image attribute editing, saving to a gallery, and crossbreeding portraits
- [Waifu Labs](https://waifulabs.com/)
interactive waifu generator (improved results, inspired by TWDNE & using Danbooru2018 as a dataset)
- [`seeprettyface-ganerator-dongman`](https://github.com/a312863063/seeprettyface-ganerator-dongman)
- [photo-booth selfies](https://x.com/highqualitysh1t/status/1095699293011435520)
- [ramen](https://x.com/knjcode/status/1102771002222637056)
- [Western portraits](https://x.com/kikko_fr/status/1094685986691399681) ([2](https://imgur.com/a/8nkMmeB))
- [Pokemon (successful)](https://x.com/roadrunning01/status/1111686125431783424) ([2](https://x.com/MichaelFriese10/status/1127614400750346240), [3](https://github.com/t04glovern/stylegan-pokemon); [human illustration](https://iguanamouth.tumblr.com/post/158982472537/pokemon-generated-by-neural-network))[^pixel-art]
- **fonts**: [1](https://x.com/kikko_fr/status/1095603397179396098)/[2](https://www.machinelearningfont.com/ "A Machine Learning Font"); [Unicode characters](https://towardsdatascience.com/creating-new-scripts-with-stylegan-c16473a50fd0); [kanji](https://x.com/kintopp/status/1218795800400101376)
- [Arabic calligraphy](https://x.com/zaidalyafeai/status/1346841324461416458)
- [graffiti samples](https://x.com/PINguAR/status/1097130957163937792)
- [wine labels](https://x.com/drose101/status/1108104217577832449)
- [Large Logo Dataset](https://x.com/mattjarviswall/status/1110548997729452035); [Cedric Oeldorf's Conditional StyleGAN](https://github.com/cedricoeldorf/ConditionalStyleGAN)
- [_Doom_ textures](https://www.youtube.com/watch?v=jjmu5_ewCIE); [_Doom_ characters](https://futurism.com/neural-network-draw-doom-guy-high-res "Someone Used a Neural Network to Draw Doom Guy in High-Res: A series of algorithms turned the famous pixelated face into an HD portrait")
- [flowers](https://www.reddit.com/r/computervision/comments/bfcnbj/p_stylegan_on_oxford_visual_geometry_group/)
- [Donald Trump](https://medium.com/gradientcrescent/this-president-does-not-exist-generating-artistic-portraits-of-donald-trump-using-stylegan-a97a17902dd4 "This President Does Not Exist: Generating Artistic Portraits of Donald Trump using StyleGAN Transfer Learning: Theory and Implementation in Tensorflow")
- [_Magic: The Gathering_ cards](https://iforcedabot.com/i-have-no-mana-and-i-must-tap/ "I Have No Mana And I Must Tap")
- [Tom Cruise faces](https://x.com/ionicdevil/status/1122756808991330304)
- [Hockey player faces](https://github.com/colinrsmall/This-Hockey-Player-Does-Not-Exist "Eastside Hockey Manager faces, Colin R. Small")
- [music album covers](https://www.reddit.com/r/MachineLearning/comments/bkrn3i/p_stylegan_trained_on_album_covers/); [book covers](https://booksby.ai/)
- [eyes](https://web.archive.org/web/20230604002332/https://thiseyedoesnotexist.com/story/) (ProGAN)
- [Face pareidolia](https://x.com/realmeatyhuman/status/1233084317032681483 "Curated output from a StyleGAN 2 model trained on images that trigger pareidolia in the viewer - scraped from the `#iseefaces` and `#pareidolia` hashtags on Instagram.")
- [emojis](https://x.com/MichaelFriese10/status/1130604229372997632)
- [Wikihow illustrations](https://x.com/MichaelFriese10/status/1132777932802236417)
- [vases](https://www.derekau.net/this-vessel-does-not-exist)
- watches: [1](https://digital-thinking.de/watchgan-advancing-generated-watch-images-with-stylegans/)/[2](https://evigio.com/post/generating-new-watch-designs-with-stylegan "Generating New Watch Designs With StyleGAN")
- [T-shirts](https://github.com/EvgenyKashin/ganarts "This T-shirt does not exist")
- [butterflies](https://www.reddit.com/r/MediaSynthesis/comments/ea5qoy/butterflies_generated_with_stylegan/ "I trained a StyleGAN on images of butterflies from the Natural History Museum in London.")
- [Generating creepy images](https://svilentodorov.xyz/blog/stylegan-for-evil/ "StyleGAN for Evil: Trypophobia and Clockwork Oranging") ([trypophobia](!W))
- [maps](/doc/ai/nn/gan/stylegan/2020-05-05-tjukanov-mapdreameraicartography.html "Mapdreamer - AI cartography")
- [Chinese landscape art](https://arxiv.org/abs/2011.05552 "'End-to-End Chinese Landscape Painting Creation Using Generative Adversarial Networks', Xue 2020")
 visualization of the wide range of visual subjects StyleGAN has been applied to](/doc/ai/nn/gan/stylegan/anime/2019-03-19-gwern-stylegan-imagequilt-allstyleganexamples.jpg)
[^pixel-art]: One thing people keep trying GANs on, but which never works well, is pixel art, such as Pokemon. The results are always bad: the individual data points will be memorized and generated fine, but the variations and interpolations will be poor, and the GANs are unable to generate meaningfully 'new' pixel art, whereas with photographs or art, there is a much greater diversity and genuinely novel compositions.
The problem is that 'pixel art' is underspecified.
Pixel art is by design an ultra-impoverished representation of art or the real world: under the extreme constraints of a palette enabling a few colors at a time or objects which might max out at 8x8 tiles, it is only enough pixels, carefully reduced to a parody or caricature or abstraction---*just* enough to trigger the association in the human viewer.
Perhaps if one had a *lot* of the relevant art, that could brute force the problem, but so far no one has convincingly done so.
Think about [Picasso's famous drawing series of 'a bull'](https://en.wikipedia.org/wiki/Le_Taureau), going from the realistic drawing to a few lines which are not so much a bull but evoking 'bullness' - how could you possibly learn what a bull is, with all its possible movements in 3D space, from even a dozen 'bullness' drawings?
Pixel-art anything is derivative of a photorealistic world. If you look at 8-bit art and standard block sizes like Mario in NES _Mario_, if you were not already intimately familiar with the distribution of human faces, and had to learn starting with a completely blank slate like a GAN would, how would you ever understand what a human face was such that you could imagine correct variations like Princess Peach or Luigi?
Or if you wanted to generate Pokemon, which are all based heavily on real animals, how would the model know anything about horses or zebras or hamsters or butterflies and be able to generate a sprite of Butterfree independently? If you look at the Pokemon GAN failure cases closer and compare them to 'real' Pokemon, you start to realize to what an extent each Pokemon is derivative of several real-world animals or objects---Pokemon in some ways do not exist in their own right, they are only shorthand or mnemonics of *other* things. Pikachu is the "electric mouse": but if you had never seen any electricity iconography like 'lightning bolts' *or* any rodents like hamsters or [chinchilla](!W) or [jerboa](!W), how could you ever understand an image of a 'Pikachu' or generate a plausible rodent variation of it? If you could, you'd need a lot more Pikachu training data, that's for sure.
(One is reminded of the joke about the mathematicians telling jokes. They knew all each other's favorites, you see, so they only needed to call out the number. "#9." Sensible chuckles. "#81." Laughter. "#17!" Chortling all around. The new grad student ventures his own joke: "...#112?" Outright hysteria! You see, they had never heard that one before...)
Since this was the case, it would never be possible to train a generative model from scratch, from a pure blank slate, to generate convincing new Pokemon, or to do really good pixel art. The single-domain approach, which 100% dominated generative models before 2021, could never work. You have to bring in extensive real-world knowledge from somewhere. There's no free lunch.
Successful applications to pixel art tend to inject real-world knowledge, such as through models pretrained on FFHQ, or focus on tasks involving 'throwing away' information rather than generating it, such as style transfer of pixel art styles.
Thus, if I wanted to make a Pokemon GAN, I would not attempt to train on solely pixel art scraped from a few games.
I would instead start with a large dataset of animals, perhaps from ImageNet or iNaturalist or Wikipedia, real or fictional, and grab all Pokemon art of any kind from anywhere, including dumping individual frames from the _Pokemon_ anime and exploiting CGI models of animals/Pokemon to densely sample all possible images, and would focus on generating as high-quality and diverse a distribution of fantastic beasts as possible; and when that succeeded, treat '_Pokemon_-style pixelization' as a second phase, to be applied to the high-quality high-resolution photographic fantasy animals generated by the first model.
(This is why I was pushing in Tensorfork for training a [single big BigGAN](/gan#tensorfork-chaos-runs "‘GANs Didn’t Fail, They Were Abandoned § Tensorfork Chaos Runs’, Gwern 2022") on all the datasets we had, because I knew that a single universal model would beat all of the specialized GANs everyone was doing, and would also likely unlock capabilities that simply could not be trained in isolation, like Pokemon.)
It is noteworthy that the first really good pixel art neural generative model, [CLIPIT PixelDraw](https://x.com/dribnet/status/1427613617973653505 "Here's a link to my Colab if you'd like to give it a go yourself. This codebase builds off of previous work from many people including @advadnoun @RiversHaveWings @NerdyRodent as well as ClipDraw from @kvfrans @crosslabstokyo @err_more and @okw") (and later [pixray](https://github.com/dribnet/pixray)), relies entirely on the pretrained [CLIP](https://openai.com/index/clip/ "'CLIP: Connecting Text and Images: We’re introducing a neural network called CLIP which efficiently learns visual concepts from natural language supervision. CLIP can be applied to any visual classification benchmark by simply providing the names of the visual categories to be recognized, similar to the “zero-shot” capabilities of GPT-2 and GPT-3', Radford et al 2021") model which was trained on _n_ = 400m Internet images. Similarly, [Projected GAN's Pokemon](https://arxiv.org/pdf/2111.01007#page=9) work, but because it is drawing on ImageNet knowledge through the enriched features.
Neural networks show us that sometimes, the [hard things are easy & easy hard](!W "Moravec's paradox"), because we don't understand how we think or learn.
## Why Don't GANs Work?
Why does StyleGAN work so well on anime images while other GANs worked not at all or slowly at best?
The lesson I took from ["Are GANs Created Equal? A Large-Scale Study"](https://arxiv.org/abs/1711.10337#google), Lucic et al 2017, is that CelebA/CIFAR-10 are too easy, as almost all evaluated GAN architectures were capable of occasionally achieving good FID if one simply did enough iterations & hyperparameter tuning.
Interestingly, I consistently observe in training *all* GANs on anime that clear lines & sharpness & cel-like smooth gradients appear only toward the end of training, after typically initially blurry textures have coalesced.
This suggests an inherent bias of CNNs: color images work because they provide some degree of textures to start with, but lineart/monochrome stuff fails because the GAN optimization dynamics flail around.
This is consistent with [Geirhos et al 2018's findings](https://arxiv.org/abs/1811.12231 "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness")---which uses style transfer to construct a data-augmented/transformed "Stylized-ImageNet"---showing that ImageNet CNNs are lazy and, because the tasks can be achieved to some degree with texture-only classification (as demonstrated by several of Geirhos et al 2018's authors via ["BagNets"](https://openreview.net/forum?id=SkfMWhAqYQ "'Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet', Brendel & Bethge 2019")), focus on textures unless otherwise forced; and by [ResNeXt](https://arxiv.org/abs/1907.07640 "'Robustness properties of Facebook's ResNeXt WSL models', Orhan 2019") & [Hermann & Kornblith 2019](https://arxiv.org/abs/1911.09071 "Exploring the Origins and Prevalence of Texture Bias in Convolutional Neural Networks"), who find that although CNNs are perfectly capable of emphasizing shape over texture, lower-performing models tend to rely more heavily on texture and that many kinds of training (including [BigBiGAN](https://arxiv.org/abs/1907.02544 "'Large Scale Adversarial Representation Learning', Donahue & Simonyan 2019")) will induce a texture focus, suggesting texture tends to be lower-hanging fruit.
So while CNNs *can* learn sharp lines & shapes rather than textures, the typical GAN architecture & training algorithm do not make it easy.
Since CIFAR-10/CelebA can be fairly described as being just as heavy on textures as ImageNet (which is not true of anime images), it is not surprising that GANs train easily on them starting with textures and gradually refining into good samples but then struggle on anime.
This raises a question of whether the StyleGAN architecture is necessary and whether many GANs might work, if only one had good style transfer for anime images and could, to defeat the texture bias, generate many versions of each anime image which kept the shape while changing the color palette? (Current style transfer methods like the [AdaIN PyTorch implementation](https://github.com/naoto0804/pytorch-AdaIN "naoto0804/pytorch-AdaIN: Unofficial pytorch implementation of 'Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization' [Huang+, ICCV2017]") used by Geirhos et al 2018, do not work well on anime images, ironically enough, because they are trained on photographic images, typically using the old VGG model.)
# FAQ
> ...Its social accountability seems sort of like that of designers of military weapons: unculpable right up until they get a little too good at their job.
>
> [David Foster Wallace](!W), ["_E unibus pluram_: Television and U.S. Fiction"](https://www.thefreelibrary.com/E+unibus+pluram%3A+television+and+U.S.+fiction.-a013952319)
To address some common questions people have after seeing generated samples:
- **Overfitting**: "Aren't StyleGAN (or BigGAN) just overfitting & memorizing data?"
Amusingly, this is not a question anyone really bothered to ask of earlier GAN architectures, which is a sign of progress. Overfitting is a better problem to have than underfitting, because overfitting means you can use a smaller model or more data or more aggressive regularization techniques, while underfitting means your approach just isn't working.
In any case, while there is currently no way to conclusively prove that cutting-edge GANs are not 100% memorizing (because they should be memorizing to a considerable extent in order to learn image generation, and evaluating generative models is hard in general, and for GANs in particular, because they don't provide standard metrics like likelihoods which could be used on held-out samples), there are several reasons to think that they are not *just* memorizing:^[This observations apply only to the Generator in GANs (which is what we primarily care about); curiously, there's some reason to think that GAN Discriminators are in fact mostly memorizing ([see later](#discriminator-ranking)).]
#. [Sample/Dataset Overlap]{.smallcaps}: a standard check for overfitting is to compare generated images to their closest matches using [nearest-neighbors](!W "Nearest neighbor search") (where distance is defined by features like a CNN embedding) lookup; an example of this are [StackGAN's Figure 6](https://openaccess.thecvf.com/content_ICCV_2017/papers/Zhang_StackGAN_Text_to_ICCV_2017_paper.pdf#page=7) & BigGAN's [Figures 10--14](https://arxiv.org/pdf/1809.11096.pdf#page=14) & [Dhariwal & Nichol 2021](https://arxiv.org/pdf/2105.05233.pdf#page=20), where the photorealistic samples are nevertheless completely different from the most similar ImageNet datapoints. (It's worth noting that [Clearview AI's](!W "Clearview AI") facial recognition [reportedly does not return Flickr matches](https://x.com/kashhill/status/1218542846694871040) for random FFHQ StyleGAN faces, suggesting the generated faces genuinely look like new faces rather than any of the original Flickr faces.)
One intriguing [observation about GANs made by the BigGAN paper](#discriminator-ranking) is that the criticisms of Generators memorizing datapoints may be precisely the opposite of reality: GANs may work primarily by the *Discriminator* (adaptively) overfitting to datapoints, thereby repelling the Generator away from real datapoints and forcing it to learn nearby possible images which collectively span the image distribution. (With enough data, this creates generalization because "neural nets are lazy" and only learn to generalize when easier strategies fail.)
#. [Semantic Understanding]{.smallcaps}: GANs appear to learn meaningful concepts like individual objects, as demonstrated by "latent space addition" or research tools like [GANdissection](https://web.archive.org/web/20200209041559/https://blog.acolyer.org/2019/02/27/gan-dissection-visualizing-and-understanding-generative-adversarial-networks/)/[Suzuki et al 2018](https://arxiv.org/abs/1811.10153 "Spatially Controllable Image Synthesis with Internal Representation Collaging"); [regression tasks](https://arxiv.org/abs/2107.11186 "'LARGE: Latent-Based Regression through GAN Semantics', Nitzan et al 2021"), [depth maps](https://arxiv.org/abs/2311.17137 "‘Generative Models: What do they know? Do they know things? Let’s find out!’, Du et al 2023"), or image edits like object deletions/additions ([Bau et al 2020](https://arxiv.org/abs/2007.15646 "Rewriting a Deep Generative Model")) or segmenting objects like dogs from their backgrounds ([Voynov & Babenko 2020](https://arxiv.org/abs/2002.03754 "Unsupervised Discovery of Interpretable Directions in the GAN Latent Space")/[Voynov et al 2020](https://arxiv.org/abs/2006.04988 "Big GANs Are Watching You: Towards Unsupervised Object Segmentation with Off-the-Shelf Generative Models"), [Tritong et al 2021](https://arxiv.org/abs/2103.04379 "Repurposing GANs for One-shot Semantic Part Segmentation"), [Abdal et al 2021](https://arxiv.org/abs/2103.14968 "Labels4Free: Unsupervised Segmentation using StyleGAN"), [Zhang et al 2021](https://arxiv.org/abs/2104.06490 "DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort"), [Li et al 2022](https://arxiv.org/abs/2201.04684 "BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations")---as do [diffusion models](https://arxiv.org/abs/2306.05720 "‘Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model’, Chen et al 2023")) are difficult to explain without some genuine understanding of images.
In the case of StyleGAN anime faces, there are [encoders and controllable face generation](#reversing-stylegan-to-control-modify-images) now which demonstrate that the latent variables do map onto meaningful factors of variation & the model must have genuinely learned about creating images rather than merely memorizing real images or image patches. Similarly, when we use the ["truncation trick"/ψ](#psitruncation-trick) to sample from relatively extreme unlikely images and we look at the distortions, they show how generated images break down in semantically-relevant ways, which would not be the case if it was just plagiarism. (A particularly extreme example of the power of the learned StyleGAN primitives is [Abdal et al 2019's](/doc/ai/nn/gan/stylegan/2019-abdal.pdf "Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?") demonstration that Karras et al's FFHQ faces StyleGAN can be used to generate fairly realistic images of cats/dogs/cars.)
#. [Latent Space Smoothness]{.smallcaps}: in general, interpolation in the latent space (_z_) shows smooth changes of images and logical transformations or variations of face features; if StyleGAN were merely memorizing individual datapoints, the interpolation would be expected to be low quality, yield many terrible faces, and exhibit 'jumps' in between points corresponding to real, memorized, datapoints. The StyleGAN anime face models do not exhibit this. (In contrast, the Holo ProGAN, which overfit badly, does show severe problems in its latent space interpolation videos.)
Which is not to say that GANs do not have issues: "mode dropping" seems to still be an issue for BigGAN despite the expensive large-minibatch training, which is overfitting to some degree, and StyleGAN presumably suffers from it too.
#. [[Transfer Learning](#transfer-learning)]{.smallcaps}: GANs have been used for semi-supervised learning (eg. generating plausible 'labeled' samples to train a classifier on), imitation learning like [GAIL](https://arxiv.org/abs/1606.03476 "'Generative Adversarial Imitation Learning', Ho & Ermon 2016"), and retraining on further datasets; if the G is merely memorizing, it is difficult to explain how any of this would work.
- **Compute Requirements**: "Doesn't StyleGAN take too long to train?"
StyleGAN is remarkably fast-training for a GAN. With the anime faces, I got better results after 1--3 days of StyleGAN training than I'd gotten with >3 weeks of ProGAN training. The training times quoted by the StyleGAN repo may sound scary, but they are, in practice, a steep overestimate of what you actually need, for several reasons:
- [Lower Resolution]{.smallcaps}: the largest figures are for 1024px images but you may not need them to be that large or even *have* a big dataset of 1024px images. For anime faces, 1024px-sized faces are relatively rare, and training at 512px & upscaling 2× to 1,024 with `waifu2x`^[A possible alternative is [ESRGAN](https://github.com/xinntao/ESRGAN) ([Wang et al 2018](https://arxiv.org/abs/1809.00219 "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks")); [Joey Ballantine's PyTorch anime models](https://github.com/JoeyBallentine/ESRGAN) like `4x_NMKD-YandereNeoXL_200k.pth` work much better than waifu2x does (as of 2021-05-28).] works fine & is much faster. Since upscaling is relatively simple & easy, another strategy is to change the progressive-growing schedule: instead of proceeding to the final resolution as fast as possible, instead adjust the schedule to stop at a more feasible resolution & spend the bulk of training time *there* instead and then do just enough training at the final resolution to learn to upscale (eg. spend 10% of training growing to 512px, then 80% of training time at 512px, then 10% at 1024px).
- [Diminishing Returns]{.smallcaps}: the largest gains in image quality are seen in the first few days or weeks of training with the remaining training being not that useful as they focus on improving small details (so just a few days may be more than adequate for your purposes, especially if you're willing to select a little more aggressively from samples)
- [Transfer Learning]{.smallcaps} from a related model can save days or weeks of training, as there is no need to train from scratch; with the anime face StyleGAN, one can train a character-specific StyleGAN with a few hours or days at most, and certainly do not need to spend multiple weeks training from scratch! (assuming that wouldn't just cause overfitting) Similarly, if one wants to train on some 1024px face dataset, why start from scratch, taking ~1000 GPU-hours, when you can start from Nvidia's FFHQ face model which is already fully trained, and can converge in a fraction of the from-scratch time?
For 1024px, you could use a super-resolution GAN like to upscale? Alternately, you could change the image progression budget to spend most of your time at 512px and then at the tail end try 1024px.
- [One-Time Costs]{.smallcaps}: the upfront cost of a few hundred dollars of GPU-time (at inflated AWS prices) may seem steep, but should be kept in perspective. As with almost all NNs, training 1 StyleGAN model can be literally tens of millions of times more expensive than simply running the Generator to produce 1 image; but it also need be paid only once by only one person, and the total price need not even be paid by the same person, given transfer learning, but can be amortized across various datasets. Indeed, given how fast running the Generator is, the trained model doesn't even need to be run on a GPU. (The rule of thumb is that a GPU is 20--30× faster than the same thing on CPU, with rare instances when overhead dominates of the CPU being as fast or faster, so since generating 1 image takes on the order of ~0.1s on GPU, a CPU can do it in ~3s, which is adequate for many purposes.)
- **Copyright Infringement**: "Who owns StyleGAN images?"
#. The Nvidia [Source Code & Released Models]{.smallcaps} for StyleGAN 1 are under a [CC](!W "Creative Commons license")-[BY-NC](https://creativecommons.org/licenses/by-nc/4.0/ "‘CC BY-NC 4.0 Deed Attribution-NonCommercial 4.0 International’, Commons 2013") license, and you cannot edit them or produce "derivative works" such as retraining their FFHQ, cat, or cat StyleGAN models. (StyleGAN 2 is under a new ["Nvidia Source Code License-NC"](https://nvlabs.github.io/stylegan2/license.html), which appears to be effectively the same as the CC-BY-NC with the addition of a [patent retaliation clause](!W "Software patents and free software#Patent retaliation").)
If a model is trained from scratch, then that does not apply as the source code is simply another tool used to create the model and nothing about the CC-BY-NC license forces you to donate the copyright to Nvidia.
(It would be odd if such a thing did happen---if your word processor claimed to transfer the copyrights of everything written in it to Microsoft!)
For those concerned by the CC-BY-NC license, a 512px FFHQ config-f StyleGAN 2 has been trained & [released into the public domain](https://x.com/AydaoGMan/status/1269690778324013061) by Aydao, and is available [for download](/doc/ai/nn/gan/stylegan/2020-06-07-aydao-stylegan2-configf-ffhq-512-avg-tpurun1.pkl "ffhq-512-avg-tpurun1.pkl (348MB)")
#. [Models]{.smallcaps} in general are generally considered ["transformative works"](!W "Transformativeness") and the copyright owners of whatever data the model was trained on have no copyright on the model. (The fact that the datasets or inputs are copyrighted is irrelevant, as training on them is universally considered [fair use](https://cdn.openai.com/policy-submissions/OpenAI+Comments+on+Intellectual+Property+Protection+for+Artificial+Intelligence+Innovation.pdf "Comment Regarding Request for Comments on Intellectual Property Protection for Artificial Intelligence Innovation") and transformative, similar to artists or search engines; see the further reading.) The model is copyrighted to whomever created it. Hence, Nvidia has copyright on the models it created but I have copyright under the models I trained (which I release under CC-0).
#. [Samples]{.smallcaps} are trickier. The usual widely-stated legal interpretation is that the standard copyright law position is that only [human authors can earn a copyright](https://www.copyright.gov/comp3/chap300/ch300-copyrightable-authorship.pdf#Compendium%20300.indd%3A.122046%3A96431) and that machines, animals, inanimate objects or most famously, [monkeys](!W "Monkey selfie copyright dispute"), cannot. The US Copyright Office states clearly that regardless of whether we regard a GAN as a machine or a something more intelligent like an animal, either way, it doesn't count:
> A work of authorship must possess "some minimal degree of creativity" to sustain a copyright claim. _Feist_, 499 U.S. at 358, 362 (citation omitted). "[T]he requisite level of creativity is extremely low." Even a "slight amount" of creative expression will suffice. "The vast majority of works make the grade quite easily, as they possess some creative spark, 'no matter how crude, humble or obvious it might be.'" _Id_. at 346 (citation omitted).
>
> ... To qualify as a work of "authorship" a work must be created by a human being. See _Burrow-Giles Lithographic Co._, 111 U.S. at 58. Works that do not satisfy this requirement are not copyrightable. The Office will not register works produced by nature, animals, or plants.
>
> *Examples*:
>
> - A photograph taken by a monkey.
> - A mural painted by an elephant.
>
> ...the Office will not register works produced by a machine or mere mechanical process that operates randomly or automatically without any creative input or intervention from a human author.
A dump of random samples such as the Nvidia samples or TWDNE therefore has no copyright & by definition is in the public domain.
A new copyright can be created, however, if a human author is sufficiently 'in the loop', so to speak, as to exert a _de minimis_ amount of creative effort, even if that 'creative effort' is simply selecting a single image out of a dump of thousands of them or twiddling knobs (eg. on Make Girls.Moe). [Crypko](https://crypko.ai/), for example, [take this position.](https://www.itmedia.co.jp/news/articles/1711/28/news020.html "萌えキャラ生成AI、学習データを'ネットの海'からゲッチュするのはアリか？ (1/5)")
Further reading on computer-generated art copyrights:
- ["Copyrights In Computer-Generated Works: Whom, If Anyone, Do We Reward?"](https://scholarship.law.duke.edu/cgi/viewcontent.cgi?article=1023&context=dltr#pdf), Glasser 2001
- ["Ex Machina: Copyright Protection For Computer-Generated Works"](https://www.rutgerslawreview.com/wp-content/uploads/2017/07/Robert-Denicola-Ex-Machina-69-Rutgers-UL-Rev-251-2016.pdf), Denicola 2016
- ["Computer Generated Works and Copyright: Selfies, Traps, Robots, AI and Machine Learning"](https://files.osf.io/v1/resources/np2jd/providers/osfstorage/59614dec594d9002288271b6?action=download&version=1&direct#pdf), Lambert 2017
- ["Who holds the Copyright in AI Created Art?"](https://journal.atp.art/the-next-rembrandt-who-holds-the-copyright-in-computer-generated-art/), Steve Schlackman (2018)
- ["The Machine as Author"](/doc/ai/poetry/2019-gervais.pdf "'The Machine As Author', Gervais 2019"), Gervais 2019
- ["Why Is AI Art Copyright So Complicated?"](https://www.artnome.com/news/2019/3/27/why-is-ai-art-copyright-so-complicated), Jason Bailey (2019)
- ["We've been warned about AI and music for over 50 years, but no one's prepared: 'This road is literally being paved as we're walking on it'"](https://www.theverge.com/2019/4/17/18299563/ai-algorithm-music-law-copyright-human) (_The Verge_ 2019)
## Copyright
Per the copyright point above, all my generated videos and samples and models are released under the [CC-0](https://creativecommons.org/public-domain/cc0/ "‘CC-0: Creative Commons public domain license’, Commons 2002") (public domain equivalent) license.
Source code listed may be derivative works of Nvidia's CC-BY-NC-licensed StyleGAN code, and may be CC-BY-NC.
# Training requirements
## Data
> The road of excess leads to the palace of wisdom \
> ...If the fool would persist in his folly he would become wise \
> ...You never know what is enough unless you know what is more than enough. \
> ...If others had not been foolish, we should be so.
>
> William Blake, "Proverbs of Hell", [_The Marriage of Heaven and Hell_](!W)
The necessary size for a dataset depends on the complexity of the domain and whether transfer learning is being used.
StyleGAN's default settings yield a 1024px Generator with 26.2M parameters, which is a large model and can soak up potentially millions of images, so there is no such thing as too much.
For learning decent-quality anime faces from scratch, a minimum of 5000 appears to be necessary in practice; for learning a specific character when using the anime face StyleGAN, potentially as little as ~500 (especially with data augmentation) can give good results.
For domains as complicated as "any cat photo" like Karras et al 2018's cat StyleGAN which is trained on the [LSUN](https://github.com/fyu/lsun) CATS category of ~1.8M^[Based on eyeballing the 'cat' bar graph in Figure 3 of [Yu et al 2015](https://arxiv.org/abs/1506.03365 "LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop").] cat photos, that appears to either not be enough or StyleGAN was not trained to convergence; Karras et al 2018 note that "CATS continues to be a difficult dataset due to the high intrinsic variation in poses, zoom levels, and backgrounds."^[CATS offer an amusing instance of the dangers of data augmentation: ProGAN used horizontal flipping/mirroring for everything, because why not? This led to strange Cyrillic text captions showing up in the generated cat images. Why not Latin alphabet captions? Because every cat image was being shown *mirrored* as well as normally! For StyleGAN, mirroring was disabled, so now the lolcat captions are recognizably Latin alphabetical, and even *almost* English words. This demonstrates that even datasets where left/right doesn't seem to matter, like cat photos, can surprise you.]
## Compute
To fit reasonable minibatch sizes, one will want GPUs with >11GB VRAM.
At 512px, that will only train _n_ = 4, and going below that means it'll be even slower (and you may have to reduce learning rates to avoid unstable training).
So, Nvidia 1080ti & up would be good.
(Reportedly, AMD/OpenCL works [for running StyleGAN models](https://x.com/syoyo/status/1093526177891770369), and there is one report of successful training with "Radeon VII with `tensorflow-rocm` 1.13.2 and `rocm` 2.3.14".)
The StyleGAN repo provide the following estimated training times for 1--8 GPU systems (which I convert to total GPU-hours & provide a worst-case AWS-based cost estimate):
>
>
> | GPUs | 1024^2^ | 512^2^ | 256^2^ | [March 2019 AWS Costs[^estimate]] |
> | ---: | :-------------- | :------------ | :------------ | --------------------- |
> | 1 | 41 days 4 hours [988 GPU-hours] | 24 days 21 hours [597 GPU-hours] | 14 days 22 hours [358 GPU-hours] | [[$320]($2019), [$194]($2019), [$115]($2019)]
> | 2 | 21 days 22 hours [1,052] | 13 days 7 hours [638] | 9 days 5 hours [442] | [NA]
> | 4 | 11 days 8 hours [1,088] | 7 days 0 hours [672] | 4 days 21 hours [468] | [NA]
> | 8 | 6 days 14 hours [1,264] | 4 days 10 hours [848] | 3 days 8 hours [640] | [[$2,730]($2019), [$1,831]($2019), [$1,382]($2019)]
>
> Table: Estimated StyleGAN wallclock training times for various resolutions & GPU-clusters (source: StyleGAN repo)
>
>
[^estimate]: I estimated the total cost using AWS EC2 preemptible hourly costs on 2019-03-15 as follows:
- 1 GPU: `p2.xlarge` instance in `us-east-2a`, Half of a K80 (12GB VRAM): [$0.3235]($2019)/hour
- 2 GPUs: NA---there is no [P2 instance](https://aws.amazon.com/ec2/instance-types/p2/) with 2 GPUs, only 1/8/16
- 8 GPUs: `p2.8xlarge` in `us-east-2a`, 8 halves of K80s (12GB VRAM each): [$2.160]($2019)/hour
As usual, there is sublinear scaling, and larger instances cost disproportionately more, because one is paying for faster wallclock training (time is valuable) and for not having to create a distributed infrastructure which can exploit the cheap single-GPU instances.
This cost estimate does not count additional costs like hard drive space. In addition to the dataset size (the StyleGAN data encoding is ~18× larger than the raw data size, so a 10GB folder of images → 200GB of `.tfrecords`), you would need at least 100GB HDD (50GB for the OS, and 50GB for checkpoints/images/etc to avoid crashes from running out of space).
AWS GPU instances are some of the most expensive ways to train a NN and provide an upper bound (compare [Vast.ai](https://vast.ai/)); 512px is often an acceptable (or necessary) resolution; and in practice, the full quoted training time is not really necessary---with my anime face StyleGAN, the faces themselves were high quality within 48 GPU-hours, and what training it for ~1000 additional GPU-hours accomplished was primarily to improve details like the shoulders & backgrounds.
(ProGAN/StyleGAN particularly struggle with backgrounds & edges of images because those are cut off, obscured, and highly-varied compared to the faces, whether anime or FFHQ.
I hypothesize that the telltale blurry backgrounds are due to the impoverishment of the backgrounds/edges in cropped face photos, and they could be fixed by transfer-learning or pretraining on a more generic dataset like ImageNet, so it learns what the backgrounds even *are* in the first place; then in face training, it merely has to remember them & defocus a bit to generate correct blurry backgrounds.)
](/doc/ai/nn/gan/stylegan/anime/2019-02-10-gwern-stylegan-animefaces-1dayvs6daystraining.jpg){.width-full}
Training on a desktop is a bit painful because GPU use is not controlled by `nice`/`ionice` priorities and any training will badly degrade the usability/latency of your GUI, which is a frustration that builds up over month-long runtimes.
If you have multiple GPUs, you can offload training to the spare GPUs; if you have only 1 GPU, you can try using [`lazy`: a tool for running processes in idle time](https://github.com/nshepperd/lazy "'Lazy, a tool for running things in idle time', nshepperd 2021")---it will cost you some throughput as it pauses training while you are actively using your computer, but it'll be worth it.
# Data Preparation
The most difficult part of running StyleGAN is preparing the dataset properly.
StyleGAN does not, unlike most GAN implementations (particularly PyTorch ones), support reading a directory of files as input; it can only read its unique `.tfrecord` format which stores each image as raw arrays at every relevant resolution.^[I regard this as a flaw in StyleGAN & TF in general. Computers are more than fast enough to load & process images asynchronously using a few worker threads, and working with a directory of images (rather than a special binary format 10--20× larger) avoids imposing serious burdens on the user & hard drive. PyTorch GANs almost always avoid this mistake, and are much more pleasant to work with as one can freely modify the dataset between (and even during) runs.]
Thus, input files must be perfectly uniform, (slowly) converted to the `.tfrecord` format by the special `dataset_tool.py` tool, and will take up ~19× more disk space.^[For example, my Danbooru2018 anime portrait dataset is 16GB, but the StyleGAN encoded dataset is 296GB.]
A StyleGAN dataset must consist of images all formatted *exactly* the same way.
Images must be precisely 512×512px or 1,024×1,024px etc. (any eg. 512×513px images will kill the entire run), they must all be the *same* colorspace (you cannot have sRGB and Grayscale JPGs---and I doubt other color spaces work at all), they must not be transparent, the filetype must be the same as the model you intend to (re)train (ie. you cannot retrain a PNG-trained model on a JPG dataset, StyleGAN will crash every time with inscrutable convolution/channel-related errors)^[This may be why some people report that StyleGAN just crashes for them & they can't figure out why. They should try changing their dataset JPG ↔ PNG.], and there must be no subtle errors like CRC checksum errors which image viewers or libraries like ImageMagick often ignore.
## Faces preparation
My workflow:
#. Download raw images from [Danbooru2021](/danbooru2021#download) if necessary
#. Extract from the JSON Danbooru2018 metadata all the IDs of a subset of images if a specific Danbooru tag (such as a single character) is desired, using `jq` and shell scripting
#. Crop square anime faces from raw images using [Nagadomi's `lbpcascade_animeface`](https://github.com/nagadomi/lbpcascade_animeface) (regular face-detection methods do not work on anime images)
#. Delete empty files, monochrome or grayscale files, & exact-duplicate files
#. Convert to JPG
#. Upscale below the target resolution (512px) images with [`waifu2x`](https://github.com/nagadomi/waifu2x)
#. Convert all images to *exactly* 512×512 resolution sRGB JPG images
#. If feasible, improve data quality by checking for low-quality images by hand, removing near-duplicates images found by `findimagedupes`, and filtering with a pretrained GAN's Discriminator
#. Convert to StyleGAN format using `dataset_tool.py`
The goal is to turn this:
into this:
Below I use shell scripting to prepare the dataset.
A possible alternative is [`danbooru-utility`](https://github.com/reidsanders/danbooru-utility), which aims to help "explore the dataset, filter by tags, rating, and score, detect faces, and resize the images".
### Cropping
The [Danbooru download](/danbooru2021#download) can be done via rsync, which provides a JSON metadata tarball which unpacks into `metadata/2*` & a folder structure of `{original,512px}/{0-999}/$ID.{png,jpg,...}`.
For training on SFW whole images, the `512px/` version of Danbooru2018 would work, but it is not a great idea for *faces* because by scaling images down to 512px, a lot of face detail has been lost, and getting high-quality faces is a challenge.
The SFW IDs can be extracted from the filenames in `512px/` directly or from the metadata by extracting the `id` & `rating` fields (and saving to a file):
~~~{.Bash}
find ./512px/ -type f | sed -e 's/.*\/\([[:digit:]]*\)\.jpg/\1/'
# 967769
# 1853769
# 2729769
# 704769
# 1799769
# ...
tar xf metadata.json.tar.xz
cat metadata/20180000000000* | jq '[.id, .rating]' -c | grep -F '"s"' | cut -d '"' -f 2 # "
# ...
~~~
After installing and testing [Nagadomi's `lbpcascade_animeface`](https://github.com/nagadomi/lbpcascade_animeface) to make sure it & [OpenCV](!W) works, one can use [a simple script](https://github.com/nagadomi/lbpcascade_animeface/issues/1#issue-205363706) which crops the face(s) from a single input image.
The accuracy on Danbooru images is fairly good, perhaps 90% excellent faces, 5% low-quality faces (genuine but either awful art or tiny little faces on the order of 64px which useless), and 5% outright errors---non-faces like armpits or elbows (oddly enough).
It can be improved by making the script more restrictive, such as requiring 250×250px regions, which eliminates most of the low-quality faces & mistakes.
(There is an alternative more-difficult-to-run library by Nakatomi which offers a face-cropping script, [animeface-2009's](https://github.com/nagadomi/animeface-2009) [`face_collector.rb`](https://github.com/nagadomi/animeface-2009/blob/master/animeface-ruby/face_collector.rb), which Nakatomi says is better at cropping faces, but I was not impressed when I tried it out.)
`crop.py`:
~~~{.Python .collapse}
import cv2
import sys
import os.path
def detect(cascade_file, filename, outputname):
if not os.path.isfile(cascade_file):
raise RuntimeError("%s: not found" % cascade_file)
cascade = cv2.CascadeClassifier(cascade_file)
image = cv2.imread(filename)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
gray = cv2.equalizeHist(gray)
## NOTE: Suggested modification: increase minSize to '(250,250)' px,
## increasing proportion of high-quality faces & reducing
## false positives. Faces which are only 50×50px are useless
## and often not faces at all.
## FOr my StyleGANs, I use 250 or 300px boxes
faces = cascade.detectMultiScale(gray,
# detector options
scaleFactor = 1.1,
minNeighbors = 5,
minSize = (50, 50))
i=0
for (x, y, w, h) in faces:
cropped = image[y: y + h, x: x + w]
cv2.imwrite(outputname+str(i)+".png", cropped)
i=i+1
if len(sys.argv) != 4:
sys.stderr.write("usage: detect.py