Applications

Be­cause of its speed and sta­bil­i­ty, when the source code was re­leased on 2019-02-04 (a date that will long be noted in the ANNals of GANime), the Nvidia mod­els & sam­ple dumps were quickly pe­rused & new StyleGANs trained on a wide va­ri­ety of im­age types, yield­ing, in ad­di­tion to the orig­i­nal faces/­cart­s/­cats of Kar­ras et al 2018:

Why Don’t GANs Work?

Why does StyleGAN work so well on anime im­ages while other GANs worked not at all or slowly at best?

The les­son I took from “Are GANs Cre­ated Equal? A Large-S­cale Study”, Lu­cic et al 2017, is that CelebA/CIFAR10 are too easy, as al­most all eval­u­ated GAN ar­chi­tec­tures were ca­pa­ble of oc­ca­sion­ally achiev­ing good FID if one sim­ply did enough it­er­a­tions & hy­per­pa­ra­me­ter tun­ing.

In­ter­est­ing­ly, I con­sis­tently ob­serve in train­ing all GANs on anime that clear lines & sharp­ness & cel-like smooth gra­di­ents ap­pear only to­ward the end of train­ing, after typ­i­cally ini­tially blurry tex­tures have co­a­lesced. This sug­gests an in­her­ent bias of CNNs: color im­ages work be­cause they pro­vide some de­gree of tex­tures to start with, but lin­eart/­mono­chrome stuff fails be­cause the GAN op­ti­miza­tion dy­nam­ics flail around. This is con­sis­tent with Geirhos et al 2018’s find­ings—which uses style trans­fer to con­struct a data-aug­ment­ed/­trans­formed “Styl­ized-Im­a­geNet”—show­ing that Im­a­geNet CNNs are lazy and, be­cause the tasks can be achieved to some de­gree with tex­ture-only clas­si­fi­ca­tion (as demon­strated by sev­eral of Geirhos et al 2018’s au­thors via “Bag­Nets”), fo­cus on tex­tures un­less oth­er­wise forced; and by ResNeXt & Her­mann & Ko­rn­blith 2019, who find that al­though CNNs are per­fectly ca­pa­ble of em­pha­siz­ing shape over tex­ture, low­er-per­form­ing mod­els tend to rely more heav­ily on tex­ture and that many kinds of train­ing (in­clud­ing BigBiGAN) will in­duce a tex­ture fo­cus, sug­gest­ing tex­ture tends to be low­er-hang­ing fruit. So while CNNs can learn sharp lines & shapes rather than tex­tures, the typ­i­cal GAN ar­chi­tec­ture & train­ing al­go­rithm do not make it easy. Since CIFAR10/CelebA can be fairly de­scribed as be­ing just as heavy on tex­tures as Im­a­geNet (which is not true of anime im­ages), it is not sur­pris­ing that GANs train eas­ily on them start­ing with tex­tures and grad­u­ally re­fin­ing into good sam­ples but then strug­gle on ani­me.

This raises a ques­tion of whether the StyleGAN ar­chi­tec­ture is nec­es­sary and whether many GANs might work, if only one had good style trans­fer for anime im­ages and could, to de­feat the tex­ture bi­as, gen­er­ate many ver­sions of each anime im­age which kept the shape while chang­ing the color palet­te? (Cur­rent style trans­fer meth­ods like the AdaIN Py­Torch im­ple­men­ta­tion used by Geirhos et al 2018, do not work well on anime im­ages, iron­i­cally enough, be­cause they are trained on pho­to­graphic im­ages, typ­i­cally us­ing the old VGG mod­el.)

Copyright

Per the copy­right point above, all my gen­er­ated videos and sam­ples and mod­els are re­leased un­der the CC-0 (pub­lic do­main equiv­a­lent) li­cense. Source code listed may be de­riv­a­tive works of Nvidi­a’s CC-BY-NC-li­censed StyleGAN code, and may be CC-BY-NC.

Data

The nec­es­sary size for a dataset de­pends on the com­plex­ity of the do­main and whether trans­fer learn­ing is be­ing used. StyleGAN’s de­fault set­tings yield a 1024px Gen­er­a­tor with 26.2M pa­ra­me­ters, which is a large model and can soak up po­ten­tially mil­lions of im­ages, so there is no such thing as too much.

For learn­ing de­cen­t-qual­ity anime faces from scratch, a min­i­mum of 5000 ap­pears to be nec­es­sary in prac­tice; for learn­ing a spe­cific char­ac­ter when us­ing the anime face StyleGAN, po­ten­tially as lit­tle as ~500 (e­spe­cially with data aug­men­ta­tion) can give good re­sults. For do­mains as com­pli­cated as “any cat photo” like Kar­ras et al 2018’s cat StyleGAN which is trained on the LSUN CATS cat­e­gory of ~1.8M¹⁷ cat pho­tos, that ap­pears to ei­ther not be enough or StyleGAN was not trained to con­ver­gence; Kar­ras et al 2018 note that “CATS con­tin­ues to be a diffi­cult dataset due to the high in­trin­sic vari­a­tion in pos­es, zoom lev­els, and back­grounds.”¹⁸

Compute

To fit rea­son­able mini­batch sizes, one will want GPUs with >11GB VRAM. At 512px, that will only train n = 4, and go­ing be­low that means it’ll be even slower (and you may have to re­duce learn­ing rates to avoid un­sta­ble train­ing). So, Nvidia 1080ti & up would be good. (Re­port­ed­ly, AMD/OpenCL works for run­ning StyleGAN mod­els, and there is one re­port of suc­cess­ful train­ing with “Radeon VII with tensorflow-rocm 1.13.2 and rocm 2.3.14”.)

The StyleGAN repo pro­vide the fol­low­ing es­ti­mated train­ing times for 1–8 GPU sys­tems (which I con­vert to to­tal GPU-hours & pro­vide a worst-case AWS-based cost es­ti­mate):

Es­ti­mated StyleGAN wall­clock train­ing times for var­i­ous res­o­lu­tions & GPU-clusters (source: StyleGAN re­po)

GPUs	10242	5122	2562	[March 2019 AWS Costs19]
1	41 days 4 hours [988 GPU-hours]	24 days 21 hours [597 GPU-hours]	14 days 22 hours [358 GPU-hours]	[$320, $194, $115]
2	21 days 22 hours [1,052]	13 days 7 hours [638]	9 days 5 hours [442]	[NA]
4	11 days 8 hours [1,088]	7 days 0 hours [672]	4 days 21 hours [468]	[NA]
8	6 days 14 hours [1,264]	4 days 10 hours [848]	3 days 8 hours [640]	[$2,730, $1,831, $1,382]

AWS GPU in­stances are some of the most ex­pen­sive ways to train a NN and pro­vide an up­per bound (com­pare Vast.ai); 512px is often an ac­cept­able (or nec­es­sary) res­o­lu­tion; and in prac­tice, the full quoted train­ing time is not re­ally nec­es­sary—with my anime face StyleGAN, the faces them­selves were high qual­ity within 48 GPU-hours, and what train­ing it for ~1000 ad­di­tional GPU-hours ac­com­plished was pri­mar­ily to im­prove de­tails like the shoul­ders & back­grounds. (ProGAN/StyleGAN par­tic­u­larly strug­gle with back­grounds & edges of im­ages be­cause those are cut off, ob­scured, and high­ly-var­ied com­pared to the faces, whether anime or FFHQ. I hy­poth­e­size that the tell­tale blurry back­grounds are due to the im­pov­er­ish­ment of the back­ground­s/edges in cropped face pho­tos, and they could be fixed by trans­fer­-learn­ing or pre­train­ing on a more generic dataset like Im­a­geNet, so it learns what the back­grounds even are in the first place; then in face train­ing, it merely has to re­mem­ber them & de­fo­cus a bit to gen­er­ate cor­rect blurry back­ground­s.)

Faces preparation

My work­flow:

1. Down­load raw im­ages from Dan­booru2018 if nec­es­sary
2. Ex­tract from the JSON Dan­booru2018 meta­data all the IDs of a sub­set of im­ages if a spe­cific Dan­booru tag (such as a sin­gle char­ac­ter) is de­sired, us­ing jq and shell script­ing
3. Crop square anime faces from raw im­ages us­ing Na­gadomi’s lbpcascade_animeface (reg­u­lar face-de­tec­tion meth­ods do not work on anime im­ages)
4. Delete empty files, mono­chrome or grayscale files, & ex­ac­t-du­pli­cate files
5. Con­vert to JPG
6. Up­scale be­low the tar­get res­o­lu­tion (512px) im­ages with waifu2x
7. Con­vert all im­ages to ex­actly 512×512 res­o­lu­tion sRGB JPG im­ages
8. If fea­si­ble, im­prove data qual­ity by check­ing for low-qual­ity im­ages by hand, re­mov­ing near-du­pli­cates im­ages found by findimagedupes, and fil­ter­ing with a pre­trained GAN’s Dis­crim­i­na­tor
9. Con­vert to StyleGAN for­mat us­ing dataset_tool.py

The goal is to turn this:

into this:

Be­low I use shell script­ing to pre­pare the dataset. A pos­si­ble al­ter­na­tive is danbooru-utility, which aims to help “ex­plore the dataset, fil­ter by tags, rat­ing, and score, de­tect faces, and re­size the im­ages”.

find ./512px/ -type f | sed -e 's/.*\/\([[:digit:]]*\)\.jpg/\1/'
# 967769
# 1853769
# 2729769
# 704769
# 1799769
# ...
tar xf metadata.json.tar.xz
cat metadata/20180000000000* | jq '[.id, .rating]' -c | fgrep '"s"' | cut -d '"' -f 2 # "
# ...

import cv2
import sys
import os.path

def detect(cascade_file, filename, outputname):
    if not os.path.isfile(cascade_file):
        raise RuntimeError("%s: not found" % cascade_file)

    cascade = cv2.CascadeClassifier(cascade_file)
    image = cv2.imread(filename)
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    gray = cv2.equalizeHist(gray)

    ## NOTE: Suggested modification: increase minSize to '(250,250)' px,
    ## increasing proportion of high-quality faces & reducing
    ## false positives. Faces which are only 50×50px are useless
    ## and often not faces at all.
    ## FOr my StyleGANs, I use 250 or 300px boxes

    faces = cascade.detectMultiScale(gray,
                                     # detector options
                                     scaleFactor = 1.1,
                                     minNeighbors = 5,
                                     minSize = (50, 50))
    i=0
    for (x, y, w, h) in faces:
        cropped = image[y: y + h, x: x + w]
        cv2.imwrite(outputname+str(i)+".png", cropped)
        i=i+1

if len(sys.argv) != 4:
    sys.stderr.write("usage: detect.py <animeface.xml file>  <input> <output prefix>\n")
    sys.exit(-1)

detect(sys.argv[1], sys.argv[2], sys.argv[3])

cropFaces() {
    BUCKET=$(printf "%04d" $(( $@ % 1000 )) )
    ID="$@"
    CUDA_VISIBLE_DEVICES="" nice python ~/src/lbpcascade_animeface/examples/crop.py  \
     ~/src/lbpcascade_animeface/lbpcascade_animeface.xml \
     ./original/$BUCKET/$ID.* "./faces/$ID"
}
export -f cropFaces

mkdir ./faces/
cat sfw-ids.txt | parallel --progress cropFaces

# NOTE: because of the possibility of multiple crops from an image, the script appends a N counter;
# remove that to get back the original ID & filepath: eg
#
## original/0196/933196.jpg  → portrait/9331961.jpg
## original/0669/1712669.png → portrait/17126690.jpg
## original/0997/3093997.jpg → portrait/30939970.jpg

## Delete failed/empty files
find faces/ -size 0    -type f -delete

## Delete 'too small' files which is indicative of low quality:
find faces/ -size -40k -type f -delete

## Delete exact duplicates:
fdupes --delete --omitfirst --noprompt faces/

## Delete monochrome or minimally-colored images:
### the heuristic of <257 unique colors is imperfect but better than anything else I tried
deleteBW() { if [[ `identify -format "%k" "$@"` -lt 257 ]];
             then rm "$@"; fi; }
export -f deleteBW
find faces -type f | parallel --progress deleteBW

import pickle
import numpy as np
import cv2
import dnnlib.tflib as tflib
import random
import argparse
import PIL.Image
from training.misc import adjust_dynamic_range


def preprocess(file_path):
    # print(file_path)
    img = np.asarray(PIL.Image.open(file_path))

    # Preprocessing from dataset_tool.create_from_images
    img = img.transpose([2, 0, 1])  # HWC => CHW
    # img = np.expand_dims(img, axis=0)
    img = img.reshape((1, 3, 512, 512))

    # Preprocessing from training_loop.process_reals
    img = adjust_dynamic_range(data=img, drange_in=[0, 255], drange_out=[-1.0, 1.0])
    return img


def main(args):
    random.seed(args.random_seed)
    minibatch_size = args.minibatch_size
    input_shape = (minibatch_size, 3, 512, 512)
    # print(args.images)
    images = args.images
    images.sort()

    tflib.init_tf()
    _G, D, _Gs = pickle.load(open(args.model, "rb"))
    # D.print_layers()

    image_score_all = [(image, []) for image in images]

    # Shuffle the images and process each image in multiple minibatches.
    # Note: networks.stylegan2.minibatch_stddev_layer
    # calculates the standard deviation of a minibatch group as a feature channel,
    # which means that the output of the discriminator actually depends
    # on the companion images in the same minibatch.
    for i_shuffle in range(args.num_shuffles):
        # print('shuffle: {}'.format(i_shuffle))
        random.shuffle(image_score_all)
        for idx_1st_img in range(0, len(image_score_all), minibatch_size):
            idx_img_minibatch = []
            images_minibatch = []
            input_minibatch = np.zeros(input_shape)
            for i in range(minibatch_size):
                idx_img = (idx_1st_img + i) % len(image_score_all)
                idx_img_minibatch.append(idx_img)
                image = image_score_all[idx_img][0]
                images_minibatch.append(image)
                img = preprocess(image)
                input_minibatch[i, :] = img
            output = D.run(input_minibatch, None, resolution=512)
            print('shuffle: {}, indices: {}, images: {}'
                  .format(i_shuffle, idx_img_minibatch, images_minibatch))
            print('Output: {}'.format(output))
            for i in range(minibatch_size):
                idx_img = idx_img_minibatch[i]
                image_score_all[idx_img][1].append(output[i][0])

    with open(args.output, 'a') as fout:
        for image, score_list in image_score_all:
            print('Image: {}, score_list: {}'.format(image, score_list))
            avg_score = sum(score_list)/len(score_list)
            fout.write(image + ' ' + str(avg_score) + '\n')


def parse_arguments():
    parser = argparse.ArgumentParser()
    parser.add_argument('--model', type=str, required=True,
                        help='.pkl model')
    parser.add_argument('--images', nargs='+')
    parser.add_argument('--output', type=str, default='rank.txt')
    parser.add_argument('--minibatch_size', type=int, default=4)
    parser.add_argument('--num_shuffles', type=int, default=5)
    parser.add_argument('--random_seed', type=int, default=0)
    return parser.parse_args()


if __name__ == '__main__':
    main(parse_arguments())

. ~/src/torch/install/bin/torch-activate
upscaleWaifu2x() {
    SIZE1=$(identify -format "%h" "$@")
    SIZE2=$(identify -format "%w" "$@");

    if (( $SIZE1 < 512 && $SIZE2 < 512  )); then
        echo "$@" $SIZE
        TMP=$(mktemp "/tmp/XXXXXX.png")
        CUDA_VISIBLE_DEVICES="$((RANDOM % 2 < 1))" nice th ~/src/waifu2x/waifu2x.lua -model_dir \
            ~/src/waifu2x/models/upconv_7/art -tta 1 -m scale -scale 2 \
            -i "$@" -o "$TMP"
        convert "$TMP" "$@"
        rm "$TMP"
    fi;  }

export -f upscaleWaifu2x
find faces/ -type f | parallel --progress --jobs 9 upscaleWaifu2x

dataAugment () {
    image="$@"
    target=$(basename "$@")
    suffix="png"
    convert -deskew 50                     "$image" "$target".deskew."$suffix"
    convert -resize 110%x100%              "$image" "$target".horizstretch."$suffix"
    convert -resize 100%x110%              "$image" "$target".vertstretch."$suffix"
    convert -blue-shift 1.1                "$image" "$target".midnight."$suffix"
    convert -fill red -colorize 5%         "$image" "$target".red."$suffix"
    convert -fill orange -colorize 5%      "$image" "$target".orange."$suffix"
    convert -fill yellow -colorize 5%      "$image" "$target".yellow."$suffix"
    convert -fill green -colorize 5%       "$image" "$target".green."$suffix"
    convert -fill blue -colorize 5%        "$image" "$target".blue."$suffix"
    convert -fill purple -colorize 5%      "$image" "$target".purple."$suffix"
    convert -adaptive-blur 3x2             "$image" "$target".blur."$suffix"
    convert -adaptive-sharpen 4x2          "$image" "$target".sharpen."$suffix"
    convert -brightness-contrast 10        "$image" "$target".brighter."$suffix"
    convert -brightness-contrast 10x10     "$image" "$target".brightercontraster."$suffix"
    convert -brightness-contrast -10       "$image" "$target".darker."$suffix"
    convert -brightness-contrast -10x10    "$image" "$target".darkerlesscontrast."$suffix"
    convert +level 5%                      "$image" "$target".contraster."$suffix"
    convert -level 5%\!                    "$image" "$target".lesscontrast."$suffix"
  }
export -f dataAugment
find faces/ -type f | parallel --progress dataAugment

convertPNGToJPG() { convert -quality 33 "$@" "$@".jpg && rm "$@"; }
export -f convertPNGToJPG
find faces/ -type f -name "*.png" | parallel --progress convertPNGToJPG

find faces/ -type f | xargs --max-procs=16 -n 9000 \
    mogrify -resize 512x512\> -extent 512x512\> -gravity center -background black

find faces/ -type f | xargs --max-procs=16 -n 9000 identify | \
    # remember the warning: images must be identical, square, and sRGB/grayscale:
    fgrep -v " JPEG 512x512 512x512+0+0 8-bit sRGB"| cut -d ' ' -f 1 | \
    xargs --max-procs=16 -n 10000 rm

diff --git a/dataset_tool.py b/dataset_tool.py
index 4ddfe44..e64e40b 100755
--- a/dataset_tool.py
+++ b/dataset_tool.py
@@ -519,6 +519,7 @@ def create_from_images(tfrecord_dir, image_dir, shuffle):
     with TFRecordExporter(tfrecord_dir, len(image_filenames)) as tfr:
         order = tfr.choose_shuffled_order() if shuffle else np.arange(len(image_filenames))
         for idx in range(order.size):
+            print(image_filenames[order[idx]])
             img = np.asarray(PIL.Image.open(image_filenames[order[idx]]))
             if channels == 1:
                 img = img[np.newaxis, :, :] # HW => CHW

source activate MY_TENSORFLOW_ENVIRONMENT
python dataset_tool.py create_from_images datasets/faces /media/gwern/Data/danbooru2018/faces/

Installation

I as­sume you have CUDA in­stalled & func­tion­ing. If not, good luck. (On my Ubuntu Bionic 18.04.2 LTS OS, I have suc­cess­fully used the Nvidia dri­ver ver­sion #410.104, CUDA 10.1, and Ten­sor­Flow 1.13.1.)

A Python ≥3.6²⁷ vir­tual en­vi­ron­ment can be set up for StyleGAN to keep de­pen­den­cies tidy, Ten­sor­Flow & StyleGAN de­pen­den­cies in­stalled:

conda create -n stylegan pip python=3.6
source activate stylegan

## TF:
pip install tensorflow-gpu
## Test install:
python -c "import tensorflow as tf; tf.enable_eager_execution(); \
    print(tf.reduce_sum(tf.random_normal([1000, 1000])))"
pip install tensorboard

## StyleGAN:
## Install pre-requisites:
pip install pillow numpy moviepy scipy opencv-python lmdb # requests?
## Download:
git clone 'https://github.com/NVlabs/stylegan.git' && cd ./stylegan/
## Test install:
python pretrained_example.py
## ./results/example.png should be a photograph of a middle-aged man

StyleGAN can also be trained on the in­ter­ac­tive Google Co­lab ser­vice, which pro­vides free slices of K80 GPUs 12-GPU-hour chunks, us­ing this Co­lab note­book. Co­lab is much slower than train­ing on a lo­cal ma­chine & the free in­stances are not enough to train the best StyleGANs, but this might be a use­ful op­tion for peo­ple who sim­ply want to try it a lit­tle or who are do­ing some­thing quick like ex­tremely low-res­o­lu­tion train­ing or trans­fer­-learn­ing where a few GPU-hours on a slow small GPU might be enough.

Configuration

StyleGAN does­n’t ship with any sup­port for CLI op­tions; in­stead, one must edit train.py and train/training_loop.py:

1. train/training_loop.py

   The core con­fig­u­ra­tion is done in the func­tion de­faults to training_loop be­gin­ning line 112.

   The key ar­gu­ments are G_smoothing_kimg & D_repeats (affects the learn­ing dy­nam­ic­s), network_snapshot_ticks (how often to save the pickle snap­shot­s—­more fre­quent means less progress lost in crash­es, but as each one weighs 300M­B+, can quickly use up gi­ga­bytes of space), resume_run_id (set to "latest"), and resume_kimg.

   Don’t Erase Your Model

   resume_kimg gov­erns where in the over­all pro­gres­sive-grow­ing train­ing sched­ule StyleGAN starts from. If it is set to 0, train­ing be­gins at the be­gin­ning of the pro­gres­sive-grow­ing sched­ule, at the low­est res­o­lu­tion, re­gard­less of how much train­ing has been pre­vi­ously done. It is vi­tally im­por­tant when do­ing trans­fer learn­ing that it is set to a suffi­ciently high num­ber (eg 10000) that train­ing be­gins at the high­est de­sired res­o­lu­tion like 512px, as it ap­pears that lay­ers are erased when added dur­ing pro­gres­sive-grow­ing. (resume_kimg may also need to be set to a high value to make it skip straight to train­ing at the high­est res­o­lu­tion if you are train­ing on small datasets of small im­ages, where there’s risk of it over­fit­ting un­der the nor­mal train­ing sched­ule and never reach­ing the high­est res­o­lu­tion.) This trick is un­nec­es­sary in StyleGAN 2, which is sim­pler in not us­ing pro­gres­sive grow­ing.

   More ex­per­i­men­tal­ly, I sug­gest set­ting minibatch_repeats = 1 in­stead of minibatch_repeats = 5; in line with the sus­pi­cious­ness of the gra­di­en­t-ac­cu­mu­la­tion im­ple­men­ta­tion in ProGAN/StyleGAN, this ap­pears to make train­ing both sta­bler & faster.

   Note that some of these vari­ables, like learn­ing rates, are over­rid­den in train.py. It’s bet­ter to set those there or else you may con­fuse your­self badly (like I did in won­der­ing why ProGAN & StyleGAN seemed ex­tra­or­di­nar­ily ro­bust to large changes in the learn­ing rates…).

2. train.py (pre­vi­ously config.py in ProGAN; re­named run_training.py in StyleGAN 2)

   Here we set the num­ber of GPUs, im­age res­o­lu­tion, dataset, learn­ing rates, hor­i­zon­tal flip­ping/mir­ror­ing data aug­men­ta­tion, and mini­batch sizes. (This file in­cludes set­tings in­tended ProGAN—watch out that you don’t ac­ci­den­tally turn on ProGAN in­stead of StyleGAN & con­fuse your­self.) Learn­ing rate & mini­batch should gen­er­ally be left alone (ex­cept to­wards the end of train­ing when one wants to lower the learn­ing rate to pro­mote con­ver­gence or re­bal­ance the G/D), but the im­age res­o­lu­tion/­dataset/mir­ror­ing do need to be set, like thus:

   desc += '-faces';     dataset = EasyDict(tfrecord_dir='faces', resolution=512);              train.mirror_augment = True

   This sets up the 512px face dataset which was pre­vi­ously cre­ated in dataset/faces, turns on mir­ror­ing (be­cause while there may be writ­ing in the back­ground, we don’t care about it for face gen­er­a­tion), and sets a ti­tle for the check­points/logs, which will now ap­pear in results/ with the ‘-faces’ string.

   As­sum­ing you do not have 8 GPUs (as you prob­a­bly do not), you must change the -preset to match your num­ber of GPUs, StyleGAN will not au­to­mat­i­cally choose the cor­rect num­ber of GPUs. If you fail to set it cor­rectly to the ap­pro­pri­ate pre­set, StyleGAN will at­tempt to use GPUs which do not ex­ist and will crash with the opaque er­ror mes­sage (note that CUDA uses ze­ro-in­dex­ing so GPU:0 refers to the first GPU, GPU:1 refers to my sec­ond GPU, and thus /device:GPU:2 refers to my—nonex­is­ten­t—third GPU):

   tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation \
       G_synthesis_3/lod: {{node G_synthesis_3/lod}}was explicitly assigned to /device:GPU:2 but available \
       devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0, \
       /job:localhost/replica:0/task:0/device:GPU:1, /job:localhost/replica:0/task:0/device:XLA_CPU:0, \
       /job:localhost/replica:0/task:0/device:XLA_GPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:1 ]. \
       Make sure the device specification refers to a valid device.
        [[{{node G_synthesis_3/lod}}]]

   For my 2×1080ti I’d set:

   desc += '-preset-v2-2gpus'; submit_config.num_gpus = 2; sched.minibatch_base = 8; sched.minibatch_dict = \
       {4: 256, 8: 256, 16: 128, 32: 64, 64: 32, 128: 16, 256: 8}; sched.G_lrate_dict = {512: 0.0015, 1024: 0.002}; \
       sched.D_lrate_dict = EasyDict(sched.G_lrate_dict); train.total_kimg = 99000

   So my re­sults get saved to results/00001-sgan-faces-2gpu etc (the run ID in­cre­ments, ‘sgan’ be­cause StyleGAN rather than ProGAN, ‘-faces’ as the dataset be­ing trained on, and ‘2gpu’ be­cause it’s multi-GPU).

Running

I typ­i­cally run StyleGAN in a screen ses­sion which can be de­tached and keeps mul­ti­ple shells or­ga­nized: 1 ter­mi­nal/shell for the StyleGAN run, 1 ter­mi­nal/shell for Ten­sor­Board, and 1 for Emacs.

With Emacs, I keep the two key Python files open (train.py and train/training_loop.py) for ref­er­ence & easy edit­ing.

With the “lat­est” patch, StyleGAN can be thrown into a while-loop to keep run­ning after crash­es, like:

while true; do nice py train.py ; date; (xmessage "alert: StyleGAN crashed" &); sleep 10s; done

Ten­sor­Board is a log­ging util­ity which dis­plays lit­tle time-series of recorded vari­ables which one views in a web browser, eg:

tensorboard --logdir results/02022-sgan-faces-2gpu/
# TensorBoard 1.13.0 at http://127.0.0.1:6006 (Press CTRL+C to quit)

Note that Ten­sor­Board can be back­ground­ed, but needs to be up­dated every time a new run is started as the re­sults will then be in a differ­ent fold­er.

Train­ing StyleGAN is much eas­ier & more re­li­able than other GANs, but it is still more of an art than a sci­ence. (We put up with it be­cause while GANs suck, every­thing else sucks more.) Notes on train­ing:

• Crash­proofing:

  The ini­tial re­lease of StyleGAN was prone to crash­ing when I ran it, seg­fault­ing at ran­dom. Up­dat­ing Ten­sor­Flow ap­peared to re­duce this but the root cause is still un­known. Seg­fault­ing or crash­ing is also re­port­edly com­mon if run­ning on mixed GPUs (eg a 1080ti + Ti­tan V).

  Un­for­tu­nate­ly, StyleGAN has no set­ting for sim­ply re­sum­ing from the lat­est snap­shot after crash­ing/ex­it­ing (which is what one usu­ally wants), and one must man­u­ally edit the resume_run_id line in training_loop.py to set it to the lat­est run ID. This is te­dious and er­ror-prone—at one point I re­al­ized I had wasted 6 GPU-days of train­ing by restart­ing from a 3-day-old snap­shot be­cause I had not up­dated the resume_run_id after a seg­fault!

  If you are do­ing any runs longer than a few wall­clock hours, I strongly ad­vise use of nshep­perd’s patch to au­to­mat­i­cally restart from the lat­est snap­shot by set­ting resume_run_id = "latest":

  diff --git a/training/misc.py b/training/misc.py
  index 50ae51c..d906a2d 100755
  --- a/training/misc.py
  +++ b/training/misc.py
  @@ -119,6 +119,14 @@ def list_network_pkls(run_id_or_run_dir, include_final=True):
           del pkls[0]
       return pkls
  
  +def locate_latest_pkl():
  +    allpickles = sorted(glob.glob(os.path.join(config.result_dir, '0*', 'network-*.pkl')))
  +    latest_pickle = allpickles[-1]
  +    resume_run_id = os.path.basename(os.path.dirname(latest_pickle))
  +    RE_KIMG = re.compile('network-snapshot-(\d+).pkl')
  +    kimg = int(RE_KIMG.match(os.path.basename(latest_pickle)).group(1))
  +    return (locate_network_pkl(resume_run_id), float(kimg))
  +
   def locate_network_pkl(run_id_or_run_dir_or_network_pkl, snapshot_or_network_pkl=None):
       for candidate in [snapshot_or_network_pkl, run_id_or_run_dir_or_network_pkl]:
           if isinstance(candidate, str):
  diff --git a/training/training_loop.py b/training/training_loop.py
  index 78d6fe1..20966d9 100755
  --- a/training/training_loop.py
  +++ b/training/training_loop.py
  @@ -148,7 +148,10 @@ def training_loop(
       # Construct networks.
       with tf.device('/gpu:0'):
           if resume_run_id is not None:
  -            network_pkl = misc.locate_network_pkl(resume_run_id, resume_snapshot)
  +            if resume_run_id == 'latest':
  +                network_pkl, resume_kimg = misc.locate_latest_pkl()
  +            else:
  +                network_pkl = misc.locate_network_pkl(resume_run_id, resume_snapshot)
               print('Loading networks from "%s"...' % network_pkl)
               G, D, Gs = misc.load_pkl(network_pkl)
           else:

  (The diff can be edited by hand, or copied into the repo as a file like latest.patch & then ap­plied with git apply latest.patch.)

• Tun­ing Learn­ing Rates

  The LR is one of the most crit­i­cal hy­per­pa­ra­me­ters: too-large up­dates based on too-s­mall mini­batches are dev­as­tat­ing to GAN sta­bil­ity & fi­nal qual­i­ty. The LR also seems to in­ter­act with the in­trin­sic diffi­culty or di­ver­sity of an im­age do­main; Kar­ras et al 2019 use 0.003 G/D LRs on their FFHQ dataset (which has been care­fully cu­rated and the faces aligned to put land­marks like eye­s/­mouth in the same lo­ca­tions in every im­age) when train­ing on 8-GPU ma­chines with mini­batches of n = 32, but I find lower to be bet­ter on my anime face/­por­trait datasets where I can only do n = 8. From look­ing at train­ing videos of whole-Dan­booru2018 StyleGAN runs, I sus­pect that the nec­es­sary LRs would be lower still. Learn­ing rates are closely re­lated to mini­batch size (a com­mon rule of thumb in su­per­vised learn­ing of CNNs is that the re­la­tion­ship of biggest us­able LR fol­lows a square-root curve in mini­batch size) and the BigGAN re­search ar­gues that mini­batch size it­self strongly in­flu­ences how bad mode drop­ping is, which sug­gests that smaller LRs may be more nec­es­sary the more di­verse/d­iffi­cult a dataset is.

• Bal­anc­ing G/D:

  

  Later in train­ing, if the G is not mak­ing good progress to­wards the ul­ti­mate goal of a 0.5 loss (and the D’s loss grad­u­ally de­creas­ing to­wards 0.5), and has a loss stub­bornly stuck around −1 or some­thing, it may be nec­es­sary to change the bal­ance of G/D. This can be done sev­eral ways but the eas­i­est is to ad­just the LRs in train.py, sched.G_lrate_dict & sched.D_lrate_dict.

  One needs to keep an eye on the G/D losses and also the per­cep­tual qual­ity of the faces (s­ince we don’t have any good FID equiv­a­lent yet for anime faces, which re­quires a good open-source Dan­booru tag­ger to cre­ate em­bed­dings), and re­duce both LRs (or usu­ally just the D’s LR) based on the face qual­ity and whether the G/D losses are ex­plod­ing or oth­er­wise look im­bal­anced. What you want, I think, is for the G/D losses to be sta­ble at a cer­tain ab­solute amount for a long time while the qual­ity vis­i­bly im­proves, re­duc­ing D’s LR as nec­es­sary to keep it bal­anced with G; and then once you’ve run out of time/­pa­tience or ar­ti­facts are show­ing up, then you can de­crease both LRs to con­verge onto a lo­cal op­ti­ma.

  I find the de­fault of 0.003 can be too high once qual­ity reaches a high level with both faces & por­traits, and it helps to re­duce it by a third to 0.001 or a tenth to 0.0003. If there still is­n’t con­ver­gence, the D may be too strong and it can be turned down sep­a­rate­ly, to a tenth or a fifti­eth even. (Given the sto­chas­tic­ity of train­ing & the rel­a­tiv­ity of the loss­es, one should wait sev­eral wall­clock hours or days after each mod­i­fi­ca­tion to see if it made a differ­ence.)

• Skip­ping FID met­rics:

  Some met­rics are com­puted for log­ging/re­port­ing. The FID met­rics are cal­cu­lated us­ing an old Im­a­geNet CNN; what is re­al­is­tic on Im­a­geNet may have lit­tle to do with your par­tic­u­lar do­main and while a large FID like 100 is con­cern­ing, FIDs like 20 or even in­creas­ing are not nec­es­sar­ily a prob­lem or use­ful guid­ance com­pared to just look­ing at the gen­er­ated sam­ples or the loss curves. Given that com­put­ing FID met­rics is not free & po­ten­tially ir­rel­e­vant or mis­lead­ing on many im­age do­mains, I sug­gest dis­abling them en­tire­ly. (They are not used in the train­ing for any­thing, and dis­abling them is safe.)

  They can be edited out of the main train­ing loop by com­ment­ing out the call to metrics.run like so:

  @@ -261,7 +265,7 @@ def training_loop()
          if cur_tick % network_snapshot_ticks == 0 or done or cur_tick == 1:
              pkl = os.path.join(submit_config.run_dir, 'network-snapshot-%06d.pkl' % (cur_nimg // 1000))
              misc.save_pkl((G, D, Gs), pkl)
              # metrics.run(pkl, run_dir=submit_config.run_dir, num_gpus=submit_config.num_gpus, tf_config=tf_config)

• ‘Blob’ & ‘Crack’ Ar­ti­facts:

  Dur­ing train­ing, ‘blobs’ often show up or move around. These blobs ap­pear even late in train­ing on oth­er­wise high­-qual­ity im­ages and are unique to StyleGAN (at least, I’ve never seen an­other GAN whose train­ing ar­ti­facts look like the blob­s). That they are so large & glar­ing sug­gests a weak­ness in StyleGAN some­where. The source of the blobs was un­clear. If you watch train­ing videos, these blobs seem to grad­u­ally morph into new fea­tures such as eyes or hair or glass­es. I sus­pect they are part of how StyleGAN ‘cre­ates’ new fea­tures, start­ing with a fea­ture-less blob su­per­im­posed at ap­prox­i­mately the right lo­ca­tion, and grad­u­ally re­fined into some­thing use­ful. The StyleGAN 2 pa­per in­ves­ti­gated the blob ar­ti­facts & found it to be due to the Gen­er­a­tor work­ing around a flaw in StyleGAN’s use of AdaIN nor­mal­iza­tion. Kar­ras et al 2019 note that im­ages with­out a blob some­where are se­verely cor­rupt­ed; be­cause the blobs are in fact do­ing some­thing use­ful, it is un­sur­pris­ing that the Dis­crim­i­na­tor does­n’t fix the Gen­er­a­tor. StyleGAN 2 changes the AdaIN nor­mal­iza­tion to elim­i­nate this prob­lem, im­prov­ing over­all qual­i­ty.²⁸

  If blobs are ap­pear­ing too often or one wants a fi­nal model with­out any new in­tru­sive blobs, it may help to lower the LR to try to con­verge to a lo­cal op­tima where the nec­es­sary blob is hid­den away some­where un­ob­tru­sive.

  In train­ing anime faces, I have seen ad­di­tional ar­ti­facts, which look like ‘cracks’ or ‘waves’ or ele­phant skin wrin­kles or the sort of fine craz­ing seen in old paint­ings or ce­ram­ics, which ap­pear to­ward the end of train­ing on pri­mar­ily skin or ar­eas of flat col­or; they hap­pen par­tic­u­larly fast when trans­fer learn­ing on a small dataset. The only so­lu­tion I have found so far is to ei­ther stop train­ing or get more da­ta. In con­trast to the blob ar­ti­facts (i­den­ti­fied as an ar­chi­tec­tural prob­lem & fixed in StyleGAN 2), I cur­rently sus­pect the cracks are a sign of over­fit­ting rather than a pe­cu­liar­ity of nor­mal StyleGAN train­ing, where the G has started try­ing to mem­o­rize noise in the fine de­tail of pix­e­la­tion/­li­nes, and so these are a kind of over­fit­ting/­mode col­lapse. (More spec­u­la­tive­ly: an­other pos­si­ble ex­pla­na­tion is that the cracks are caused by the StyleGAN D be­ing sin­gle-s­cale rather than mul­ti­-s­cale—as in MSG-GAN and a num­ber of oth­er­s—and the ‘cracks’ are ac­tu­ally high­-fre­quency noise cre­ated by the G in spe­cific patches as ad­ver­sar­ial ex­am­ples to fool the D. They re­port­edly do not ap­pear in MSG-GAN or StyleGAN 2, which both use mul­ti­-s­cale Ds.)

• Gra­di­ent Ac­cu­mu­la­tion:

  ProGAN/StyleGAN’s code­base claims to sup­port gra­di­ent ac­cu­mu­la­tion, which is a way to fake large mini­batch train­ing (eg n = 2048) by not do­ing the back­prop­a­ga­tion up­date every mini­batch, but in­stead sum­ming the gra­di­ents over many mini­batches and ap­ply­ing them all at once. This is a use­ful trick for sta­bi­liz­ing train­ing, and large mini­batch NN train­ing can differ qual­i­ta­tively from small mini­batch NN training—BigGAN per­for­mance in­creased with in­creas­ingly large mini­batches (n = 2048) and the au­thors spec­u­late that this is be­cause such large mini­batches mean that the full di­ver­sity of the dataset is rep­re­sented in each ‘mini­batch’ so the BigGAN mod­els can­not sim­ply ‘for­get’ rarer dat­a­points which would oth­er­wise not ap­pear for many mini­batches in a row, re­sult­ing in the GAN pathol­ogy of ‘mode drop­ping’ where some kinds of data just get ig­nored by both G/D.

  How­ev­er, the ProGAN/StyleGAN im­ple­men­ta­tion of gra­di­ent ac­cu­mu­la­tion does not re­sem­ble that of any other im­ple­men­ta­tion I’ve seen in Ten­sor­Flow or Py­Torch, and in my own ex­per­i­ments with up to n = 4096, I did­n’t ob­serve any sta­bi­liza­tion or qual­i­ta­tive differ­ences, so I am sus­pi­cious the im­ple­men­ta­tion is wrong.

Here is what a suc­cess­ful train­ing pro­gres­sion looks like for the anime face StyleGAN:

The anime face model as of 2019-03-08, trained for 21,980 it­er­a­tions or ~21m im­ages or ~38 GPU-days, is avail­able for down­load. (It is still not ful­ly-con­verged, but the qual­ity is good.)

Psi/“truncation trick”

The Ψ/“trun­ca­tion trick” (BigGAN dis­cus­sion, StyleGAN dis­cus­sion; ap­par­ently first in­tro­duced by March­esi 2017) is the most im­por­tant hy­per­pa­ra­me­ter for all StyleGAN gen­er­a­tion.

The trun­ca­tion trick is used at sam­ple gen­er­a­tion time but not train­ing time. The idea is to edit the la­tent vec­tor z, which is a vec­tor of 𝒩(0,1), to re­move any vari­ables which are above a cer­tain size like 0.5 or 1.0, and re­sam­ple those.²⁹ This seems to help by avoid­ing ‘ex­treme’ la­tent val­ues or com­bi­na­tions of la­tent val­ues which the G is not as good at—a G will not have gen­er­ated many data points with each la­tent vari­able at, say, +1.5SD. The trade­off is that those are still le­git­i­mate ar­eas of the over­all la­tent space which were be­ing used dur­ing train­ing to cover parts of the data dis­tri­b­u­tion; so while the la­tent vari­ables close to the mean of 0 may be the most ac­cu­rately mod­eled, they are also only a small part of the space of all pos­si­ble im­ages. So one can gen­er­ate la­tent vari­ables from the full un­re­stricted 𝒩(0,1) dis­tri­b­u­tion for each one, or one can trun­cate them at some­thing like +1SD or +0.7SD. (Like the dis­cus­sion of the best dis­tri­b­u­tion for the orig­i­nal la­tent dis­tri­b­u­tion, there’s no good rea­son to think that this is an op­ti­mal method of do­ing trun­ca­tion; there are many al­ter­na­tives, such as ones pe­nal­iz­ing the sum of the vari­ables, ei­ther re­ject­ing them or scal­ing them down, and some ap­pear to work much bet­ter than the cur­rent trun­ca­tion trick.)

At Ψ = 0, di­ver­sity is nil and all faces are a sin­gle global av­er­age face (a brown-eyed brown-haired school­girl, un­sur­pris­ing­ly); at ±0.5 you have a broad range of faces, and by ±1.2, you’ll see tremen­dous di­ver­sity in faces/styles/­con­sis­tency but also tremen­dous ar­ti­fact­ing & dis­tor­tion. Where you set your Ψ will heav­ily in­flu­ence how ‘orig­i­nal’ out­puts look. At Ψ = 1.2, they are tremen­dously orig­i­nal but ex­tremely hit or miss. At Ψ = 0.5 they are con­sis­tent but bor­ing. For most of my sam­pling, I set Ψ = 0.7 which strikes the best bal­ance be­tween crazi­ness/ar­ti­fact­ing and qual­i­ty/­di­ver­si­ty. (Per­son­al­ly, I pre­fer to look at Ψ = 1.2 sam­ples be­cause they are so much more in­ter­est­ing, but if I re­leased those sam­ples, it would give a mis­lead­ing im­pres­sion to read­er­s.)

Random Samples

The StyleGAN repo has a sim­ple script pretrained_example.py to down­load & gen­er­ate a sin­gle face; in the in­ter­ests of re­pro­ducibil­i­ty, it hard­wires the model and the RNG seed so it will only gen­er­ate 1 par­tic­u­lar face. How­ev­er, it can be eas­ily adapted to use a lo­cal model and (s­lowly³⁰) gen­er­ate, say, 1000 sam­ple im­ages with the hy­per­pa­ra­me­ter Ψ = 0.6 (which gives high­-qual­ity but not high­ly-di­verse im­ages) which are saved to results/example-{0-999}.png:

import os
import pickle
import numpy as np
import PIL.Image
import dnnlib
import dnnlib.tflib as tflib
import config

def main():
    tflib.init_tf()
    _G, _D, Gs = pickle.load(open("results/02051-sgan-faces-2gpu/network-snapshot-021980.pkl", "rb"))
    Gs.print_layers()

    for i in range(0,1000):
        rnd = np.random.RandomState(None)
        latents = rnd.randn(1, Gs.input_shape[1])
        fmt = dict(func=tflib.convert_images_to_uint8, nchw_to_nhwc=True)
        images = Gs.run(latents, None, truncation_psi=0.6, randomize_noise=True, output_transform=fmt)
        os.makedirs(config.result_dir, exist_ok=True)
        png_filename = os.path.join(config.result_dir, 'example-'+str(i)+'.png')
        PIL.Image.fromarray(images[0], 'RGB').save(png_filename)

if __name__ == "__main__":
    main()

Karras et al 2018 Figures

The fig­ures in Kar­ras et al 2018, demon­strat­ing ran­dom sam­ples and as­pects of the style noise us­ing the 1024px FFHQ face model (as well as the oth­er­s), were gen­er­ated by generate_figures.py. This script needs ex­ten­sive mod­i­fi­ca­tions to work with my 512px anime face; go­ing through the file:

• the code uses Ψ = 1 trun­ca­tion, but faces look bet­ter with Ψ = 0.7 (sev­eral of the func­tions have truncation_psi= set­tings but, trick­i­ly, the Fig­ure 3 draw_style_mixing_figure has its Ψ set­ting hid­den away in the synthesis_kwargs global vari­able)
• the loaded model needs to be switched to the anime face mod­el, of course
• di­men­sions must be re­duced 1024→512 as ap­pro­pri­ate; some ranges are hard­coded and must be re­duced for 512px im­ages as well
• the trun­ca­tion trick fig­ure 8 does­n’t show enough faces to give in­sight into what the la­tent space is do­ing so it needs to be ex­panded to show both more ran­dom seed­s/­faces, and more Ψ val­ues
• the bed­room/­car/­cat sam­ples should be dis­abled

The changes I make are as fol­lows:

diff --git a/generate_figures.py b/generate_figures.py
index 45b68b8..f27af9d 100755
--- a/generate_figures.py
+++ b/generate_figures.py
@@ -24,16 +24,13 @@ url_bedrooms    = 'https://drive.google.com/uc?id=1MOSKeGF0FJcivpBI7s63V9YHloUTO
 url_cars        = 'https://drive.google.com/uc?id=1MJ6iCfNtMIRicihwRorsM3b7mmtmK9c3' # karras2019stylegan-cars-512x384.pkl
 url_cats        = 'https://drive.google.com/uc?id=1MQywl0FNt6lHu8E_EUqnRbviagS7fbiJ' # karras2019stylegan-cats-256x256.pkl

-synthesis_kwargs = dict(output_transform=dict(func=tflib.convert_images_to_uint8, nchw_to_nhwc=True), minibatch_size=8)
+synthesis_kwargs = dict(output_transform=dict(func=tflib.convert_images_to_uint8, nchw_to_nhwc=True), minibatch_size=8, truncation_psi=0.7)

 _Gs_cache = dict()

 def load_Gs(url):
-    if url not in _Gs_cache:
-        with dnnlib.util.open_url(url, cache_dir=config.cache_dir) as f:
-            _G, _D, Gs = pickle.load(f)
-        _Gs_cache[url] = Gs
-    return _Gs_cache[url]
+    _G, _D, Gs = pickle.load(open("results/02051-sgan-faces-2gpu/network-snapshot-021980.pkl", "rb"))
+    return Gs

 #----------------------------------------------------------------------------
 # Figures 2, 3, 10, 11, 12: Multi-resolution grid of uncurated result images.
@@ -85,7 +82,7 @@ def draw_noise_detail_figure(png, Gs, w, h, num_samples, seeds):
     canvas = PIL.Image.new('RGB', (w * 3, h * len(seeds)), 'white')
     for row, seed in enumerate(seeds):
         latents = np.stack([np.random.RandomState(seed).randn(Gs.input_shape[1])] * num_samples)
-        images = Gs.run(latents, None, truncation_psi=1, **synthesis_kwargs)
+        images = Gs.run(latents, None, **synthesis_kwargs)
         canvas.paste(PIL.Image.fromarray(images[0], 'RGB'), (0, row * h))
         for i in range(4):
             crop = PIL.Image.fromarray(images[i + 1], 'RGB')
@@ -109,7 +106,7 @@ def draw_noise_components_figure(png, Gs, w, h, seeds, noise_ranges, flips):
     all_images = []
     for noise_range in noise_ranges:
         tflib.set_vars({var: val * (1 if i in noise_range else 0) for i, (var, val) in enumerate(noise_pairs)})
-        range_images = Gsc.run(latents, None, truncation_psi=1, randomize_noise=False, **synthesis_kwargs)
+        range_images = Gsc.run(latents, None, randomize_noise=False, **synthesis_kwargs)
         range_images[flips, :, :] = range_images[flips, :, ::-1]
         all_images.append(list(range_images))

@@ -144,14 +141,11 @@ def draw_truncation_trick_figure(png, Gs, w, h, seeds, psis):
 def main():
     tflib.init_tf()
     os.makedirs(config.result_dir, exist_ok=True)
-    draw_uncurated_result_figure(os.path.join(config.result_dir, 'figure02-uncurated-ffhq.png'), load_Gs(url_ffhq), cx=0, cy=0, cw=1024, ch=1024, rows=3, lods=[0,1,2,2,3,3], seed=5)
-    draw_style_mixing_figure(os.path.join(config.result_dir, 'figure03-style-mixing.png'), load_Gs(url_ffhq), w=1024, h=1024, src_seeds=[639,701,687,615,2268], dst_seeds=[888,829,1898,1733,1614,845], style_ranges=[range(0,4)]*3+[range(4,8)]*2+[range(8,18)])
-    draw_noise_detail_figure(os.path.join(config.result_dir, 'figure04-noise-detail.png'), load_Gs(url_ffhq), w=1024, h=1024, num_samples=100, seeds=[1157,1012])
-    draw_noise_components_figure(os.path.join(config.result_dir, 'figure05-noise-components.png'), load_Gs(url_ffhq), w=1024, h=1024, seeds=[1967,1555], noise_ranges=[range(0, 18), range(0, 0), range(8, 18), range(0, 8)], flips=[1])
-    draw_truncation_trick_figure(os.path.join(config.result_dir, 'figure08-truncation-trick.png'), load_Gs(url_ffhq), w=1024, h=1024, seeds=[91,388], psis=[1, 0.7, 0.5, 0, -0.5, -1])
-    draw_uncurated_result_figure(os.path.join(config.result_dir, 'figure10-uncurated-bedrooms.png'), load_Gs(url_bedrooms), cx=0, cy=0, cw=256, ch=256, rows=5, lods=[0,0,1,1,2,2,2], seed=0)
-    draw_uncurated_result_figure(os.path.join(config.result_dir, 'figure11-uncurated-cars.png'), load_Gs(url_cars), cx=0, cy=64, cw=512, ch=384, rows=4, lods=[0,1,2,2,3,3], seed=2)
-    draw_uncurated_result_figure(os.path.join(config.result_dir, 'figure12-uncurated-cats.png'), load_Gs(url_cats), cx=0, cy=0, cw=256, ch=256, rows=5, lods=[0,0,1,1,2,2,2], seed=1)
+    draw_uncurated_result_figure(os.path.join(config.result_dir, 'figure02-uncurated-ffhq.png'), load_Gs(url_ffhq), cx=0, cy=0, cw=512, ch=512, rows=3, lods=[0,1,2,2,3,3], seed=5)
+    draw_style_mixing_figure(os.path.join(config.result_dir, 'figure03-style-mixing.png'), load_Gs(url_ffhq), w=512, h=512, src_seeds=[639,701,687,615,2268], dst_seeds=[888,829,1898,1733,1614,845], style_ranges=[range(0,4)]*3+[range(4,8)]*2+[range(8,16)])
+    draw_noise_detail_figure(os.path.join(config.result_dir, 'figure04-noise-detail.png'), load_Gs(url_ffhq), w=512, h=512, num_samples=100, seeds=[1157,1012])
+    draw_noise_components_figure(os.path.join(config.result_dir, 'figure05-noise-components.png'), load_Gs(url_ffhq), w=512, h=512, seeds=[1967,1555], noise_ranges=[range(0, 18), range(0, 0), range(8, 18), range(0, 8)], flips=[1])
+    draw_truncation_trick_figure(os.path.join(config.result_dir, 'figure08-truncation-trick.png'), load_Gs(url_ffhq), w=512, h=512, seeds=[91,388, 389, 390, 391, 392, 393, 394, 395, 396], psis=[1, 0.7, 0.5, 0.25, 0, -0.25, -0.5, -1])

All this done, we get some fun anime face sam­ples to par­al­lel Kar­ras et al 2018’s fig­ures:

Videos

cat $(ls ./results/*faces*/fakes*.png | sort --numeric-sort) | ffmpeg -framerate 10 \ # show 10 inputs per second
    -i - # stdin
    -r 25 # output frame-rate; frames will be duplicated to pad out to 25FPS
    -c:v libx264 # x264 for compatibility
    -pix_fmt yuv420p # force ffmpeg to use a standard colorspace - otherwise PNG colorspace is kept, breaking browsers (!)
    -crf 33 # adequate high quality
    -vf "scale=iw/2:ih/2" \ # shrink the image by 2×, the full detail is not necessary & saves space
    -preset veryslow -tune animation \ # aim for smallest binary possible with animation-tuned settings
    ./stylegan-facestraining.mp4

import dnnlib.tflib as tflib
import math
import moviepy.editor
from numpy import linalg
import numpy as np
import pickle

def main():
    tflib.init_tf()
    _G, _D, Gs = pickle.load(open("results/02051-sgan-faces-2gpu/network-snapshot-021980.pkl", "rb"))

    rnd = np.random
    latents_a = rnd.randn(1, Gs.input_shape[1])
    latents_b = rnd.randn(1, Gs.input_shape[1])
    latents_c = rnd.randn(1, Gs.input_shape[1])

    def circ_generator(latents_interpolate):
        radius = 40.0

        latents_axis_x = (latents_a - latents_b).flatten() / linalg.norm(latents_a - latents_b)
        latents_axis_y = (latents_a - latents_c).flatten() / linalg.norm(latents_a - latents_c)

        latents_x = math.sin(math.pi * 2.0 * latents_interpolate) * radius
        latents_y = math.cos(math.pi * 2.0 * latents_interpolate) * radius

        latents = latents_a + latents_x * latents_axis_x + latents_y * latents_axis_y
        return latents

    def mse(x, y):
        return (np.square(x - y)).mean()

    def generate_from_generator_adaptive(gen_func):
        max_step = 1.0
        current_pos = 0.0

        change_min = 10.0
        change_max = 11.0

        fmt = dict(func=tflib.convert_images_to_uint8, nchw_to_nhwc=True)

        current_latent = gen_func(current_pos)
        current_image = Gs.run(current_latent, None, truncation_psi=0.7, randomize_noise=False, output_transform=fmt)[0]
        array_list = []

        video_length = 1.0
        while(current_pos < video_length):
            array_list.append(current_image)

            lower = current_pos
            upper = current_pos + max_step
            current_pos = (upper + lower) / 2.0

            current_latent = gen_func(current_pos)
            current_image = images = Gs.run(current_latent, None, truncation_psi=0.7, randomize_noise=False, output_transform=fmt)[0]
            current_mse = mse(array_list[-1], current_image)

            while current_mse < change_min or current_mse > change_max:
                if current_mse < change_min:
                    lower = current_pos
                    current_pos = (upper + lower) / 2.0

                if current_mse > change_max:
                    upper = current_pos
                    current_pos = (upper + lower) / 2.0


                current_latent = gen_func(current_pos)
                current_image = images = Gs.run(current_latent, None, truncation_psi=0.7, randomize_noise=False, output_transform=fmt)[0]
                current_mse = mse(array_list[-1], current_image)
            print(current_pos, current_mse)
        return array_list

    frames = generate_from_generator_adaptive(circ_generator)
    frames = moviepy.editor.ImageSequenceClip(frames, fps=30)

    # Generate video.
    mp4_file = 'results/circular.mp4'
    mp4_codec = 'libx264'
    mp4_bitrate = '3M'
    mp4_fps = 20

    frames.write_videofile(mp4_file, fps=mp4_fps, codec=mp4_codec, bitrate=mp4_bitrate)

if __name__ == "__main__":
    main()

Anime Faces

The pri­mary model I’ve trained, the anime face model is de­scribed in the data pro­cess­ing & train­ing sec­tion. It is a 512px StyleGAN model trained on n = 218,794 faces cropped from all of Dan­booru2017, cleaned, & up­scaled, and trained for 21,980 it­er­a­tions or ~21m im­ages or ~38 GPU-days.

Down­loads (I rec­om­mend us­ing the more-re­cent por­trait StyleGAN un­less cropped faces are specifi­cally de­sired):

• ran­dom sam­ples gen­er­ated on 2091-02-14 with an ex­treme Ψ = 1.2 (165MB, JPG)

• the StyleGAN model used for TWDNEv1 sam­ples as of 2019-02-26 (294MB, .pkl)

  • all TWDNE faces via rsync

• the anime face StyleGAN model, fur­ther trained, as of 2019-03-8

  The anime face model is ob­so­leted by the StyleGAN 2 por­trait model.

Anime Bodies

Aaron Gokaslan ex­per­i­mented with a cus­tom 256px anime game im­age dataset which has in­di­vid­ual char­ac­ters posed in whole-per­son im­ages to see how StyleGAN coped with more com­plex geome­tries. Progress re­quired ad­di­tional data clean­ing and low­er­ing the learn­ing rate but, trained on a 4-GPU sys­tem for week or two, the re­sults are promis­ing (even down to re­pro­duc­ing the copy­right state­ments in the im­ages), pro­vid­ing pre­lim­i­nary ev­i­dence that StyleGAN can scale:

Conditional Anime Faces, Arfafax

In March 2020, Ar­fafax trained a con­di­tional anime face StyleGAN: it takes a list of tags (a sub­set of Dan­booru2019 tags rel­e­vant to faces), processes them via doc2vec into a fixed-size in­put, and feeds them into a con­di­tional StyleGAN.³¹ (While al­most all uses of StyleGAN are un­con­di­tion­al—you just dump in a big pile of im­ages and it learns to gen­er­ate ran­dom sam­ples—the code ac­tu­ally sup­ports con­di­tional use where a cat­e­gory or set of vari­ables is turned into an out­put im­age, and a few other peo­ple have ex­per­i­mented with it.)

>>> face_tags
['angry', 'anger_vein', 'annoyed', 'clenched_teeth', 'blush', 'blush_stickers', 'embarrassed', 'bored',
    'closed_eyes', 'confused', 'crazy', 'disdain', 'disgust', 'drunk', 'envy', 'expressionless', 'evil', 'facepalm',
    'flustered', 'frustrated', 'grimace', 'guilt', 'happy', 'kubrick_stare', 'lonely', 'nervous', 'nosebleed',
    'one_eye_closed', 'open_mouth', 'closed_mouth', 'parted_lips', 'pain', 'pout', 'raised_eyebrow', 'rape_face',
    'rolling_eyes', 'sad', 'depressed', 'frown', 'gloom_(expression)', 'tears', 'horns', 'scared', 'panicking',
    'worried', 'serious', 'sigh', 'sleepy', 'tired', 'sulking', 'thinking', 'pensive', 'wince', 'afterglow',
    'ahegao', 'fucked_silly', 'naughty_face', 'torogao', 'smile', 'crazy_smile', 'evil_smile', 'fingersmile',
    'forced_smile', 'glasgow_smile', 'grin', 'fang', 'evil_grin', 'light_smile', 'sad_smile', 'seductive_smile',
    'stifled_laugh', 'smug', 'doyagao', 'smirk', 'smug', 'troll_face', 'surprised', 'scared', '/\\/\\/\\',
    'color_drain', 'horror_(expression)', 'screaming', 'turn_pale', 'trembling', 'wavy_mouth', ';)', ':d',
    ';d', 'xd', 'd:', ':}', ':{', ':3', ';3', 'x3', '3:', 'uwu', '=.w.=', ':p', ';p', ':q', ';q', '>:)', '>:(',
    ':t', ':i', ':/', ':x', ':c', 'c:', ':<', ';<', ':<>', ':>', ':>=', ';>=', ':o', ';o', '=', '=)', '=d',
    '=o', '=v', '|3', '|d', '|o', 'o3o', '(-3-)', '>3<', 'o_o', '0_0', '._.', '•_•', 'solid_circle_eyes',
    '♥_♥', 'heart_eyes', '^_^', '^o^', '\\(^o^)/', '└(^o^)┐≡', '^q^', '>_<', 'xd', 'x3', '>o<', '<_>', ';_;',
    '@_@', '>_@', '+_+', '+_-', '-_-', '\\_/', '=_=', '=^=', '=v=', '<o>_<o>', 'constricted_pupils', 'cross_eyed',
    'rectangular_mouth', 'sideways_mouth', 'no_nose', 'no_mouth', 'wavy_mouth', 'wide-eyed', 'mouth_drool',
    'awesome_face', 'foodgasm', 'henohenomoheji', 'nonowa', 'portrait', 'profile', 'smiley_face', 'uso_da',
    'food_awe', 'breast_awe', 'penis_awe']
>>> eye_tags
['aqua_eyes', 'black_eyes', 'blue_eyes', 'brown_eyes', 'green_eyes', 'grey_eyes', 'orange_eyes', 'lavender_eyes',
    'pink_eyes', 'purple_eyes', 'red_eyes', 'silver_eyes', 'white_eyes', 'yellow_eyes', 'heterochromia', 'multicolored_eyes',
    'al_bhed_eyes', 'pac-man_eyes', 'ringed_eyes', 'constricted_pupils', 'dilated_pupils', 'horizontal_pupils',
    'no_pupils', 'slit_pupils', 'symbol-shaped_pupils', '+_+', 'heart-shaped_pupils', 'star-shaped_pupils',
    'blue_sclera', 'black_sclera', 'blank_eyes', 'bloodshot_eyes', 'green_sclera', 'mismatched_sclera', 'orange_sclera',
    'red_sclera', 'yellow_sclera', 'bags_under_eyes', 'bruised_eye', 'flaming_eyes', 'glowing_eyes', 'glowing_eye',
    'mako_eyes', 'amphibian_eyes', 'button_eyes', 'cephalopod_eyes', 'compound_eyes', 'frog_eyes', 'crazy_eyes',
    'empty_eyes', 'heart_eyes', 'nonowa', 'solid_circle_eyes', 'o_o', '0_0', 'jitome', 'tareme', 'tsurime',
    'sanpaku', 'sharingan', 'mangekyou_sharingan', 'eye_reflection', 'text_in_eyes', 'missing_eye', 'one-eyed',
    'third_eye', 'extra_eyes', 'no_eyes']
>>> eye_expressions
['>_<', 'x3', 'xd', 'o_o', '0_0', '3_3', '6_9', '@_@', '^_^', '^o^', '9848', '26237', '=_=', '+_+', '._.',
    '<o>_<o>', 'blinking', 'closed_eyes', 'wince', 'one_eye_closed', ';<', ';>', ';p']
>>> eye_other
['covering_eyes', 'hair_over_eyes', 'hair_over_one_eye', 'bandage_over_one_eye', 'blindfold', 'hat_over_eyes',
    'eyepatch', 'eyelashes', 'colored_eyelashes', 'fake_eyelashes', 'eyes_visible_through_hair', 'glasses',
    'makeup', 'eyeliner', 'eyeshadow', 'mascara', 'eye_contact', 'looking_afar', 'looking_at_another', 'looking_at_breasts',
    'looking_at_hand', 'looking_at_mirror', 'looking_at_phone', 'looking_at_viewer', 'looking_away', 'looking_back',
    'looking_down', 'looking_out_window', 'looking_over_glasses', 'looking_through_legs', 'looking_to_the_side',
    'looking_up', 'akanbe', 'blind', 'cross-eyed', 'drawn_on_eyes', 'eyeball', 'eye_beam', 'eye_poke', 'eye_pop',
    'persona_eyes', 'shading_eyes', 'squinting', 'staring', 'uneven_eyes', 'upturned_eyes', 'wall-eyed', 'wide-eyed', 'wince']
>>> ears_tags
['animal_ears', 'bear_ears', 'bunny_ears', 'cat_ears', 'dog_ears', 'fake_animal_ears', 'fox_ears', 'horse_ears',
    'kemonomimi_mode', 'lion_ears', 'monkey_ears', 'mouse_ears', 'raccoon_ears', 'sheep_ears', 'tiger_ears',
    'wolf_ears', 'pointy_ears', 'robot_ears', 'extra_ears', 'ear_piercing', 'ear_protection', 'earrings',
    'single_earring', 'headphones', 'covering_ears', 'ear_biting', 'ear_licking', 'ear_grab']
>>> hair_tags
['heartbreak_haircut', 'hand_in_hair', 'adjusting_hair', 'bunching_hair', 'hair_flip', 'hair_grab', 'hair_pull',
    'hair_tucking', 'hair_tousle', 'hair_twirling', 'hair_sex', 'hair_brush', 'hair_dryer', 'shampoo', 'bun_cover',
    'hairpods', 'chopsticks', 'comb', 'hair_ornament', 'hair_bell', 'hair_bobbles', 'hair_bow', 'hair_ribbon',
    'hairclip', 'hairpin', 'hair_flower', 'hair_tubes', 'kanzashi', 'hair_tie', 'hairband', 'hair_weapon',
    'headband', 'scrunchie', 'wig', 'facial_hair', 'beard', 'bearded_girl', 'goatee', 'mustache', 'fake_mustache',
    'stubble', 'fiery_hair', 'prehensile_hair', 'helicopter_hair', 'tentacle_hair', 'living_hair', 'detached_hair',
    'severed_hair', 'floating_hair', 'hair_spread_out', 'wet_hair']
>>> hair_color_tags
['aqua_hair', 'black_hair', 'blonde_hair', 'blue_hair', 'light_blue_hair', 'brown_hair', 'light_brown_hair',
    'green_hair', 'grey_hair', 'magenta_hair', 'orange_hair', 'pink_hair', 'purple_hair', 'lavender_hair',
    'red_hair', 'auburn_hair', 'maroon_hair', 'silver_hair', 'white_hair', 'multicolored_hair', 'colored_inner_hair',
    'gradient_hair', 'rainbow_hair', 'streaked_hair', 'two-tone_hair', 'highlights', 'colored_tips', 'alternate_hair_color']
>>> hair_style_tags
['very_short_hair', 'short_hair', 'medium_hair', 'long_hair', 'very_long_hair', 'absurdly_long_hair',
    'big_hair', 'bald', 'bald_girl', 'alternate_hairstyle', 'hair_down', 'hair_up', 'curly_hair', 'drill_hair',
    'twin_drills', 'flipped_hair', 'hair_flaps', 'messy_hair', 'pointy_hair', 'ringlets', 'spiked_hair', 'wavy_hair',
    'bangs', 'asymmetrical_bangs', 'blunt_bangs', 'hair_over_eyes', 'hair_over_one_eye', 'parted_bangs', 'swept_bangs',
    'hair_between_eyes', 'hair_intakes', 'sidelocks', "widow's_peak", 'ahoge', 'heart_ahoge', 'huge_ahoge',
    'antenna_hair', 'comb_over', 'hair_pulled_back', 'hair_slicked_back', 'mohawk', 'hair_bikini', 'hair_censor',
    'hair_in_mouth', 'hair_over_breasts', 'hair_over_one_breast', 'hair_over_crotch', 'hair_over_shoulder',
    'hair_scarf', 'bow_by_hair', 'braid', 'braided_bangs', 'front_braid', 'side_braid', 'french_braid', 'crown_braid',
    'single_braid', 'multiple_braids', 'twin_braids', 'tri_braids', 'quad_braids', 'hair_bun', 'braided_bun',
    'double_bun', 'triple_bun', 'hair_rings', 'half_updo', 'one_side_up', 'two_side_up', 'low-braided_long_hair',
    'low-tied_long_hair', 'mizura', 'multi-tied_hair', 'nihongami', 'ponytail', 'folded_ponytail', 'front_ponytail',
    'high_ponytail', 'short_ponytail', 'side_ponytail', 'split_ponytail', 'topknot', 'twintails', 'low_twintails',
    'short_twintails', 'uneven_twintails', 'tri_tails', 'quad_tails', 'quin_tails', 'bob_cut', 'bowl_cut',
    'buzz_cut', 'chonmage', 'crew_cut', 'flattop', 'pixie_cut', 'undercut', 'cornrows', 'hairlocs', 'hime_cut',
    'mullet', 'afro', 'huge_afro', 'beehive_hairdo', 'pompadour', 'quiff', 'shouten_pegasus_mix_mori']
>>> skin_color_tags
['dark_skin', 'pale_skin', 'tan', 'tanlines', 'sun_tattoo', 'black_skin', 'blue_skin', 'green_skin', 'grey_skin',
    'orange_skin', 'pink_skin', 'purple_skin', 'red_skin', 'white_skin', 'yellow_skin', 'shiny_skin']
>>> headwear_tags
['crown', 'hat', 'helmet', 'black_headwear', 'blue_headwear', 'brown_headwear', 'green_headwear', 'grey_headwear',
    'orange_headwear', 'pink_headwear', 'purple_headwear', 'red_headwear', 'white_headwear', 'yellow_headwear',
    'ajirogasa', 'animal_hat', 'cat_hat', 'penguin_hat', 'baseball_cap', 'beanie', 'beret', 'bicorne', 'boater_hat',
    'bowl_hat', 'bowler_hat', 'bucket_hat', 'cabbie_hat', 'chef_hat', 'toque_blanche', 'flat_top_chef_hat',
    'cloche_hat', 'cowboy_hat', 'deerstalker', 'deviruchi_hat', 'dixie_cup_hat', 'eggshell_hat', 'fedora',
    'female_service_cap', 'flat_cap', 'fur_hat', 'garrison_cap', 'jester_cap', 'kepi', 'mian_guan', 'mitre',
    'mob_cap', 'mortarboard', 'nightcap', 'nurse_cap', 'party_hat', 'peaked_cap', 'pillow_hat', 'pirate_hat',
    'porkpie_hat', 'pumpkin_hat', 'rice_hat', 'robe_and_wizard_hat', 'sailor_hat', 'santa_hat', 'mini_santa_hat',
    'shako_cap', 'shampoo_hat', 'sombrero', 'sun_hat', "tam_o'_shanter", 'tate_eboshi', 'tokin_hat', 'top_hat',
    'mini_top_hat', 'tricorne', 'ushanka', 'witch_hat', 'mini_witch_hat', 'wizard_hat', 'veil', 'zun_hat',
    'baseball_helmet', 'bicycle_helmet', 'brodie_helmet', 'diving_helmet', 'football_helmet', 'hardhat', 'horned_helmet',
    'helm', 'kabuto', 'motorcycle_helmet', 'pickelhaube', 'pith_helmet', 'stahlhelm', 'tank_helmet', 'winged_helmet',
    'circlet', 'diadem', 'mini_crown', 'saishi', 'tiara', 'aviator_cap', 'bandana', 'bonnet', 'dalachi_(headdress)',
    'habit', 'hijab', 'keffiyeh', 'shower_cap', 'visor_cap', 'checkered_hat', 'frilled_hat', 'military_hat',
    'mini_hat', 'multicolored_hat', 'police_hat', 'print_hat', 'school_hat', 'straw_hat', 'adjusting_hat',
    'hand_on_headwear', 'hands_on_headwear', 'hat_basket', 'hat_loss', 'hat_on_chest', 'hat_over_eyes', 'hat_over_one_eye',
    'hat_removed', 'hat_tip', 'holding_hat', 'torn_hat', 'no_hat', 'hat_bow', 'hat_feather', 'hat_flower',
    'hat_ribbon', 'hat_with_ears', 'adjusting_hat', 'backwards_hat', 'hat_removed', 'holding_hat', 'torn_hat',
    'hair_bow', 'hair_ribbon', 'hairband', 'headband', 'forehead_protector', 'sweatband', 'hachimaki', 'nejiri_hachimaki',
    'mongkhon', 'headdress', 'maid_headdress', 'veil', 'hood']
>>> eyewear_tags
['glasses', 'monocle', 'sunglasses', 'aqua-framed_eyewear', 'black-framed_eyewear', 'blue-framed_eyewear',
    'brown-framed_eyewear', 'green-framed_eyewear', 'grey-framed_eyewear', 'orange-framed_eyewear', 'pink-framed_eyewear',
    'purple-framed_eyewear', 'red-framed_eyewear', 'white-framed_eyewear', 'yellow-framed_eyewear', 'blue-tinted_eyewear',
    'brown-tinted_eyewear', 'green-tinted_eyewear', 'orange-tinted_eyewear', 'pink-tinted_eyewear', 'purple-tinted_eyewear',
    'red-tinted_eyewear', 'yellow-tinted_eyewear', 'heart-shaped_eyewear', 'round_eyewear', 'over-rim_eyewear',
    'rimless_eyewear', 'semi-rimless_eyewear', 'under-rim_eyewear', 'adjusting_eyewear', 'eyewear_on_head',
    'eyewear_removed', 'eyewear_hang', 'eyewear_in_mouth', 'holding_eyewear', 'eyewear_strap', 'eyewear_switch',
    'looking_over_eyewear', 'no_eyewear', '3d_glasses', 'coke-bottle_glasses', 'diving_mask', 'fancy_glasses',
    'heart-shaped_eyewear', 'funny_glasses', 'goggles', 'nodoka_glasses', 'opaque_glasses', 'pince-nez', 'safety_glasses',
    'shooting_glasses', 'ski_goggles', 'x-ray_glasses', 'bespectacled', 'kamina_shades', 'star_shades']
>>> piercings_tags
['ear_piercing', 'eyebrow_piercing', 'anti-eyebrow_piercing', 'eyelid_piercing', 'lip_piercing', 'labret_piercing',
    'nose_piercing', 'bridge_piercing', 'tongue_piercing']
>>> format_tags
['3d', 'animated', 'animated_png', 'flash', 'music_video', 'song', 'video', 'animated_gif', 'non-looping_animation',
    'archived_file', 'artbook', 'bmp', 'calendar_(medium)', 'card_(medium)', 'comic', '2koma', '3koma', '4koma',
    'multiple_4koma', '5koma', 'borderless_panels', 'doujinshi', 'eromanga', 'left-to-right_manga', 'right-to-left_comic',
    'silent_comic', 'corrupted_image', 'cover', 'album_cover', 'character_single', 'cover_page', 'doujin_cover',
    'dvd_cover', 'fake_cover', 'game_cover', 'magazine_cover', 'manga_cover', 'fake_screenshot', 'game_cg',
    'gyotaku_(medium)', 'highres', 'absurdres', 'incredibly_absurdres', 'lowres', 'thumbnail', 'huge_filesize',
    'icon', 'logo', 'kirigami', 'lineart', 'no_lineart', 'outline', 'long_image', 'tall_image', 'wide_image',
    'mosaic_art', 'photomosaic', 'oekaki', 'official_art', 'phonecard', 'photo', 'papercraft', 'paper_child',
    'paper_cutout', 'pixel_art', 'postcard', 'poster', 'revision', 'bad_revision', 'artifacted_revision',
    'censored_revision', 'corrupted_revision', 'lossy_revision', 'watermarked_revision', 'scan', 'screencap',
    'shitajiki', 'tegaki', 'transparent_background', 'triptych_(art)', 'vector_trace', 'wallpaper', 'dual_monitor',
    'ios_wallpaper', 'official_wallpaper', 'phone_wallpaper', 'psp_wallpaper', 'tileable', 'wallpaper_forced', 'widescreen']
>>> style_tags
['abstract', 'art_deco', 'art_nouveau', 'fine_art_parody', 'flame_painter', 'impressionism', 'nihonga',
    'sumi-e', 'ukiyo-e', 'minimalism', 'realistic', 'photorealistic', 'sketch', 'style_parody', 'list_of_style_parodies',
    'surreal', 'traditional_media', 'faux_traditional_media', 'work_in_progress', 'backlighting', 'blending',
    'bloom', 'bokeh', 'caustics', 'chiaroscuro', 'chromatic_aberration', 'chromatic_aberration_abuse', 'diffraction_spikes',
    'depth_of_field', 'dithering', 'drop_shadow', 'emphasis_lines', 'foreshortening', 'gradient', 'halftone',
    'lens_flare', 'lens_flare_abuse', 'motion_blur', 'motion_lines', 'multiple_monochrome', 'optical_illusion',
    'anaglyph', 'exif_thumbnail_surprise', 'open_in_internet_explorer', 'open_in_winamp', 'stereogram', 'scanlines',
    'silhouette', 'speed_lines', 'vignetting']

Extended StyleGAN2 Danbooru2019, Aydao

Ay­dao (Twit­ter), in par­al­lel with the BigGAN ex­per­i­ments, worked on grad­u­ally ex­tend­ing StyleGAN2’s mod­el­ing pow­ers to cover Dan­booru2019 SFW. His re­sults were ex­cel­lent and power nearcyan’s (Twit­ter) This Anime Does Not Ex­ist.ai (TADNE). Sam­ples:

Anime Faces → Character Faces

Holo

The first trans­fer learn­ing was done with Holo of Spice & Wolf. It used a 512px Holo face dataset cre­ated with Na­gadomi’s crop­per from all of Dan­booru2017, up­scaled with waifu2x, cleaned by hand, and then data-aug­mented from n = 3900 to n = 12600; mir­ror­ing was en­abled since Holo is sym­met­ri­cal. I then used the anime face model as of 2019-02-09—it was not fully con­verged, in­deed, would­n’t con­verge with weeks more train­ing, but the qual­ity was so good I was too cu­ri­ous as to how well re­train­ing would work so I switched gears.

It’s worth men­tion­ing that this dataset was used pre­vi­ously with ProGAN, where after weeks of train­ing, ProGAN over­fit badly as demon­strated by the sam­ples & in­ter­po­la­tion videos.

Train­ing hap­pened re­mark­ably quick­ly, with all the faces con­verted to rec­og­niz­ably Holo faces within a few hun­dred it­er­a­tions:

The best sam­ples were con­vinc­ing with­out ex­hibit­ing the fail­ures of the ProGAN:

The StyleGAN was much more suc­cess­ful, de­spite a few fail­ure la­tent points car­ried over from the anime faces. In­deed, after a few hun­dred it­er­a­tions, it was start­ing to over­fit with the ‘crack’ ar­ti­facts & smear­ing in the in­ter­po­la­tions. The lat­est I was will­ing to use was it­er­a­tion #11370, and I think it is still some­what over­fit any­way. I thought that with its to­tal n (after data aug­men­ta­tion), Holo would be able to train longer (be­ing 1⁄7^th the size of FFHQ), but ap­par­ently not. Per­haps the data aug­men­ta­tion is con­sid­er­ably less valu­able than 1-for-1, ei­ther be­cause the in­vari­ants en­coded in aren’t that use­ful (sug­gest­ing that Geirhos et al 2018-like style trans­fer data aug­men­ta­tion is what’s nec­es­sary) or that they would be but the anime face StyleGAN has al­ready learned them all as part of the pre­vi­ous train­ing & needs more real data to bet­ter un­der­stand Holo-like faces. It’s also pos­si­ble that the re­sults could be im­proved by us­ing one of the later anime face StyleGANs since they did im­prove when I trained them fur­ther after my 2 Holo/A­suka trans­fer ex­per­i­ments.

Nev­er­the­less, im­pressed, I could­n’t help but won­der if they had reached hu­man-levels of verisimil­i­tude: would an un­wary viewer as­sume they were hand­made?

So I se­lected ~100 of the best sam­ples (24MB; Imgur mir­ror) from a dump of 2000, cropped about 5% from the left­/right sides to hide the back­ground ar­ti­facts a lit­tle bit, and sub­mit­ted them on 2019-02-11 to /r/Spice­and­Wolf un­der an alt ac­count. I made the mis­take of sort­ing by file­size & thus lead­ing with a face that was par­tic­u­larly sus­pi­cious (streaky hair) so one Red­di­tor voiced the sus­pi­cion they were from MGM (ab­surd yet not en­tirely wrong) but all the other com­menters took the faces in stride or prais­ing them, and the sub­mis­sion re­ceived +248 votes (99% pos­i­tive) by March. A Red­di­tor then turned them all into a GIF video which earned +192 (100%) and many pos­i­tive com­ments with no fur­ther sus­pi­cions un­til I ex­plained. Not bad in­deed.

The #11370 Holo StyleGAN model is avail­able for down­load.

Asuka

After the Holo train­ing & link sub­mis­sion went so well, I knew I had to try my other char­ac­ter dataset, Asuka, us­ing n = 5300 data-aug­mented to n = 58,000.³⁶ Keep­ing in mind how data seemed to limit the Holo qual­i­ty, I left mir­ror­ing en­abled for Asuka, even though she is not sym­met­ri­cal due to her 3.0 eye­patch over her left eye (as purists will no doubt note).

In­ter­est­ing­ly, while Holo trained within 4 GPU-hours, Asuka proved much more diffi­cult and did not seem to be fin­ished train­ing or show­ing the cracks de­spite train­ing twice as long. Is this due to hav­ing ~35% more real data, hav­ing 10× rather than 3× data aug­men­ta­tion, or some in­her­ent differ­ence like Asuka be­ing more com­plex (eg be­cause of more vari­a­tions in her ap­pear­ance like the eye­patches or plug­suit­s)?

I gen­er­ated 1000 ran­dom sam­ples with Ψ = 1.2 be­cause they were par­tic­u­larly in­ter­est­ing to look at. As with Holo, I picked out the best 100 (13MB; Imgur mir­ror) from ~2000:

And I sub­mit­ted to the /r/E­van­ge­lion sub­red­dit, where it also did well (+109, 98%); there were no spec­u­la­tions about the faces be­ing NN-gen­er­ated be­fore I re­vealed it, merely re­quests for more. Be­tween the two, it ap­pears that with ad­e­quate data (n > 3000) and mod­er­ate cu­ra­tion, a sim­ple kind of art Tur­ing test can be passed.

The #7903 Asuka StyleGAN model is avail­able for down­load.

Zuihou

In early Feb­ru­ary 2019, us­ing the then-re­leased mod­el, Red­di­tor End­ing_­Cred­its tried trans­fer learn­ing to n = 500 faces of the Kan­tai Col­lec­tion Zui­hou for ~1 tick (~60k it­er­a­tions).

The sam­ples & in­ter­po­la­tions have many ar­ti­facts, but the sam­ple size is tiny and I’d con­sider this good fine­tun­ing from a model never in­tended for few-shot learn­ing:

Prob­a­bly it could be made bet­ter by start­ing from the lat­est anime face StyleGAN mod­el, and us­ing ag­gres­sive data aug­men­ta­tion. An­other op­tion would be to try to find as many char­ac­ters which look sim­i­lar to Zui­hou (match­ing on hair color might work) and train on a joint dataset—un­con­di­tional sam­ples would then need to be fil­tered for just Zui­hou faces, but per­haps that draw­back could be avoided by a third stage of Zui­hou-only train­ing?

Ganso

Akizuki

An­other Kan­colle char­ac­ter, Ak­izuki, was trained in April 2019 by Gan­so.

Ptilopsis

In Jan­u­ary 2020, Ganso trained a StyleGAN 2 model from the S2 por­trait model on a tiny cor­pus of Ptilop­sis im­ages, a char­ac­ter from Arknights, a 2017 Chi­nese tower de­fense RPG mo­bile game.

Ptilop­sis are owls, and her char­ac­ter de­sign shows promi­nent ears; de­spite the few im­ages to work with (just 21 on Dan­booru as of 2020-01-19), the in­ter­po­la­tion shows smooth ad­just­ments of the ears in all po­si­tions & align­ments, demon­strat­ing the power of trans­fer learn­ing:

Lexington

A Chi­nese user 3D_DLW (S2 write­up/­tu­to­ri­al: 1/2) in Feb­ru­ary 2020 did trans­fer­-learn­ing from the S2 por­trait model to Pixiv art­work of the char­ac­ter Lex­ing­ton from War­ship Girls. He used a sim­i­lar work­flow: crop­ping faces with lbpcascade_animeface, up­scal­ing with wai­fu2x, and clean­ing with ranker.py (us­ing the orig­i­nal S2 mod­el’s Dis­crim­i­na­tor & pro­duc­ing datasets of vary­ing clean­li­ness at n = 302–1659). Sam­ples:

Emilia (Re:Zero)

Anime Faces → Anime Headshots

Twit­ter user Sunk did trans­fer learn­ing to an im­age cor­pus of a spe­cific artist, Kure­hito Mis­aki (深崎暮人), n≅1000. His im­ages work well and the in­ter­po­la­tion looks nice:

Anime Faces → Portrait

TWDNE was a huge suc­cess and pop­u­lar­ized the anime face StyleGAN. It was not per­fect, though, and flaws were not­ed.

Portrait Results

After re­train­ing the fi­nal face StyleGAN 2019-03-08–2019-04-30 on the new im­proved por­traits dataset, the re­sults im­proved:

This S1 anime por­trait model is ob­so­leted by the StyleGAN 2 por­trait model.

The fi­nal model from 2019-04-30 is avail­able for down­load.

I used this model at 𝛙=0.5 to gen­er­ate 100,000 new por­traits for TWDNE (#100,000–199,999), bal­anc­ing the pre­vi­ous faces.

I was sur­prised how diffi­cult up­grad­ing to por­traits seemed to be; I spent al­most two months train­ing it be­fore giv­ing up on fur­ther im­prove­ments, while I had been ex­pect­ing more like a week or two. The por­trait re­sults are in­deed bet­ter than the faces (I was right that not crop­ping off the top of the head adds verisimil­i­tude), but the up­grade did­n’t im­press me as much as the orig­i­nal faces did com­pared to ear­lier GANs. And our other ex­per­i­men­tal runs on whole-Dan­booru2018 im­ages never pro­gressed be­yond sug­ges­tive blobs dur­ing this pe­ri­od.

I sus­pect that StyleGAN—at least, on its de­fault ar­chi­tec­ture & hy­per­pa­ra­me­ters, with­out a great deal more com­pute—is reach­ing its lim­its here, and that changes may be nec­es­sary to scale to richer im­ages. (Self-at­ten­tion is prob­a­bly the eas­i­est to add since it should be easy to plug in ad­di­tional lay­ers to the con­vo­lu­tion code.)

Anime Faces → Male Faces

A few peo­ple have ob­served that it would be nice to have an anime face GAN for male char­ac­ters in­stead of al­ways gen­er­at­ing fe­male ones. The anime face StyleGAN does in fact have male faces in its dataset as I did no fil­ter­ing—it’s merely that fe­male faces are over­whelm­ingly fre­quent (and it may also be that male anime faces are rel­a­tively an­drog­y­nous/fem­i­nized any­way so it’s hard to tell any differ­ence be­tween a fe­male with short hair & a guy³⁷).

Train­ing a male-only anime face StyleGAN would be an­other good ap­pli­ca­tion of trans­fer learn­ing.

The faces can be eas­ily ex­tracted out of Dan­booru2018 by query­ing for "male_focus", which will pick up ~150k im­ages. More nar­row­ly, one could search "1boy" & "solo", to en­sure that the only face in the im­age is a male face (as op­posed to, say, 1boy 1girl, where a fe­male face might be cropped out as well). This pro­vides n = 99k raw hits. It would be good to also fil­ter out ‘trap’ or over­ly-fe­male-look­ing faces (else what’s the point?), by fil­ter­ing on tags like cat ears or par­tic­u­larly pop­u­lar ‘trap’ char­ac­ters like Fate/­Grand Or­der’s As­tol­fo. A more com­pli­cated query to pick up scenes with mul­ti­ple males could be to search for both "1boy" & "multiple_boys" and then fil­ter out "1girl" & "multiple_girls", in or­der to se­lect all im­ages with 1 or more males and then re­move all im­ages with 1 or more fe­males; this dou­bles the raw hits to n = 198k. (A down­side is that the face-crop­ping will often un­avoid­ably yield crops with two faces, a pri­mary face and an over­lap­ping face, which is bad and in­tro­duces ar­ti­fact­ing when I tried this with all faces.)

Com­bined with trans­fer learn­ing from the gen­eral anime face StyleGAN, the re­sults should be as good as the gen­eral (fe­male) faces.

I set­tled for "1boy" & "solo", and did con­sid­er­able clean­ing by hand. The raw count of im­ages turned out to be highly mis­lead­ing, and many faces are un­us­able for a male anime face StyleGAN: many are so highly styl­ized (such as ac­tion sce­nes) as to be dam­ag­ing to a GAN, or they are al­most in­dis­tin­guish­able from fe­male faces (be­cause they are bis­honen or trap or just an­drog­y­nous), which would be point­less to in­clude (the reg­u­lar por­trait StyleGAN cov­ers those al­ready). After hand clean­ing & use of ranker.py, I was left with n~3k, so I used heavy data aug­men­ta­tion to bring it up to n~57k, and I ini­tial­ized from the fi­nal por­trait StyleGAN for the high­est qual­i­ty.

It did not over­fit after ~4 days of train­ing, but the re­sults were not no­tice­ably im­prov­ing, so I stopped (in or­der to start train­ing the GPT-2-345M, which Ope­nAI had just re­leased, on po­etry). There are hints in the in­ter­po­la­tion videos, I think, that it is in­deed slightly over­fit­ting, in the form of ‘glitches’ where the im­age abruptly jumps slight­ly, pre­sum­ably to an­other mod­e/­face/char­ac­ter of the orig­i­nal data; nev­er­the­less, the male face StyleGAN mostly works.

The male face StyleGAN model is avail­able for down­load, as is 1000 ran­dom faces with 𝛙=0.7 (mir­ror; par­tial Imgur al­bum).

Anime Faces → Ukiyo-e Faces

In Jan­u­ary 2020, Justin (@Bunt­wor­thy) used 5000 ukiy­o-e faces cropped with Ama­zon Rekog­ni­tion from Ukiy­o-e Search to do trans­fer learn­ing. After ~24h train­ing:

Anime Faces → Western Portrait Faces

In 2019, ay­dao ex­per­i­mented with trans­fer learn­ing to Eu­ro­pean por­trait faces drawn from WikiArt; the trans­fer learn­ing was done via Nathan Ship­ley’s abuse of SWA where two mod­els are sim­ply av­er­aged to­geth­er, pa­ra­me­ter by pa­ra­me­ter and layer by lay­er, to yield a new mod­el. (Sur­pris­ing­ly, this work­s—as long as the mod­els aren’t too differ­ent; if they are, the av­er­aged model will gen­er­ate only col­or­ful blob­s.) The re­sults were amus­ing. From early in train­ing:

Lat­er:

Anime Faces → Danbooru2018

nshep­perd be­gan a train­ing run us­ing an early anime face StyleGAN model on the 512px SFW Dan­booru2018 sub­set; after ~3–5 weeks (with many in­ter­rup­tions) on 1 GPU, as of 2019-03-22, the train­ing sam­ples look like this:

The StyleGAN is able to pick up global struc­ture and there are rec­og­niz­ably anime fig­ures, de­spite the sheer di­ver­sity of im­ages, which is promis­ing. The fine de­tails are se­ri­ously lack­ing, and train­ing, to my eye, is wan­der­ing around with­out any steady im­prove­ment or sharp de­tails (ex­cept per­haps the faces which are in­her­ited from the pre­vi­ous mod­el). I sus­pect that the learn­ing rate is still too high and, es­pe­cially with only 1 GPU/n = 4, such small mini­batches don’t cover enough modes to en­able steady im­prove­ment. If so, the LR will need to be set much lower (or gra­di­ent ac­cu­mu­la­tion used in or­der to fake hav­ing large mini­batches where large LRs are sta­ble) & train­ing time ex­tended to mul­ti­ple months. An­other pos­si­bil­ity would be to restart with added self­-at­ten­tion lay­ers, which I have no­ticed seem to par­tic­u­larly help with com­pli­cated de­tails & sharp­ness; the style noise ap­proach may be ad­e­quate for the job but just a few vanilla con­vo­lu­tion lay­ers may be too few (pace the BigGAN re­sults on the ben­e­fits of in­creas­ing depth while de­creas­ing pa­ra­me­ter coun­t).

FFHQ Variations

Anime Faces → FFHQ Faces

If StyleGAN can smoothly warp anime faces among each other and ex­press global trans­forms like hair length­+­color with Ψ, could Ψ be a quick way to gain con­trol over a sin­gle large-s­cale vari­able? For ex­am­ple, male vs fe­male faces, or… anime ↔︎ real faces? (Given a par­tic­u­lar im­age/la­tent vec­tor, one would sim­ply flip the sign to con­vert it to the op­po­site; this would give the op­po­site ver­sion of each ran­dom face, and if one had an en­coder, one could do au­to­mat­i­cally ani­me-fy or re­al-fy an ar­bi­trary face by en­cod­ing it into the la­tent vec­tor which cre­ates it, and then flip­ping.³⁸)

Since Kar­ras et al 2801 pro­vide a nice FFHQ down­load script (al­beit slower than I’d like once Google Drive rate-lim­its you a wall­clock hour into the full down­load) for the ful­l-res­o­lu­tion PNGs, it would be easy to down­scale to 512px and cre­ate a 512px FFHQ dataset to train on, or even cre­ate a com­bined anime+FFHQ dataset.

The first and fastest thing was to do trans­fer learn­ing from the anime faces to FFHQ real faces. It was un­likely that the model would re­tain much anime knowl­edge & be able to do mor­ph­ing, but it was worth a try.

The ini­tial re­sults early in train­ing are hi­lar­i­ous and look like zom­bies:

After 97 ticks, the model has con­verged to a bor­ingly nor­mal ap­pear­ance, with the only hint of its ori­gins be­ing per­haps some ex­ces­sive­ly-fab­u­lous hair in the train­ing sam­ples:

Anime Faces → Anime Faces + FFHQ Faces

So, that was a bust. The next step is to try train­ing on anime & FFHQ faces si­mul­ta­ne­ous­ly; given the stark differ­ence be­tween the datasets, would pos­i­tive vs neg­a­tive Ψ wind up split­ting into real vs anime and pro­vide a cheap & easy way of con­vert­ing ar­bi­trary faces?

This sim­ply merged the 512px FFHQ faces with the 512px anime faces and re­sumed train­ing from the pre­vi­ous FFHQ model (I rea­soned that some of the ani­me-ness should still be in the mod­el, so it would be slightly faster than restart­ing from the orig­i­nal anime face mod­el). I trained it for 812 it­er­a­tions, #11,359–12,171 (some­what over 2 GPU-days), at which point it was mostly done.

It did man­age to learn both kinds of faces quite well, sep­a­rat­ing them clearly in ran­dom sam­ples:

How­ev­er, the style trans­fer & Ψ sam­ples were dis­ap­point­ments. The style mix­ing shows lim­ited abil­ity to mod­ify faces cross-do­main or con­vert them, and the trun­ca­tion trick chart shows no clear dis­en­tan­gle­ment of the de­sired fac­tor (in­deed, the var­i­ous halves of Ψ cor­re­spond to noth­ing clear):

The in­ter­po­la­tion video does show that it learned to in­ter­po­late slightly be­tween real & anime faces, giv­ing half-anime/half-real faces, but it looks like it only hap­pens some­times—­mostly with young fe­male faces³⁹:

They’re hard to spot in the in­ter­po­la­tion video be­cause the tran­si­tion hap­pens abrupt­ly, so I gen­er­ated sam­ples & se­lected some of the more in­ter­est­ing ani­me-ish faces:

Sim­i­lar­ly, Alexan­der Reben trained a StyleGAN on FFHQ+Western por­trait il­lus­tra­tions, and the in­ter­po­la­tion video is much smoother & more mixed, sug­gest­ing that more re­al­is­tic & more var­ied il­lus­tra­tions are eas­ier for StyleGAN to in­ter­po­late be­tween.

Editing Rare Attributes

A strat­egy of hand-edit­ing or us­ing a tag­ger to clas­sify at­trib­utes works for com­mon ones which will be well-rep­re­sented in a sam­ple of a few thou­sand since the clas­si­fier needs a few hun­dred cases to work with, but what about rarer at­trib­utes which might ap­pear only on one in a thou­sand ran­dom sam­ples, or at­trib­utes too rare in the dataset for StyleGAN to have learned, or at­trib­utes which may not be in the dataset at all? Edit­ing “red eyes” should be easy, but what about some­thing like “bunny ears”? It would be amus­ing to be able to edit por­traits to add bunny ears, but there aren’t that many bunny ear sam­ples (although cat ears might be much more com­mon); is one doomed to gen­er­ate & clas­sify hun­dreds of thou­sands of sam­ples to en­able bunny ear edit­ing? That would be in­fea­si­ble for hand la­bel­ing, and diffi­cult even with a tag­ger.

One sug­ges­tion I have for this use-case would be to briefly train an­other StyleGAN model on an en­riched or boosted dataset, like a dataset of 50:50 bunny ear im­ages & nor­mal im­ages. If one can ob­tain a few thou­sand bunny ear im­ages, then this is ad­e­quate for trans­fer learn­ing (com­bined with a few thou­sand ran­dom nor­mal im­ages from the orig­i­nal dataset), and one can re­train the StyleGAN on an equal bal­ance of im­ages. The high pres­ence of bunny ears will en­sure that the StyleGAN quickly learns all about those, while the nor­mal im­ages pre­vent it from over­fit­ting or cat­a­strophic for­get­ting of the full range of im­ages.

This new bun­ny-ear StyleGAN will then pro­duce bun­ny-ear sam­ples half the time, cir­cum­vent­ing the rare base rate is­sue (or fail­ure to learn, or nonex­is­tence in dataset), and en­abling effi­cient train­ing of a clas­si­fi­er. And since nor­mal faces were used to pre­serve its gen­eral face knowl­edge de­spite the trans­fer learn­ing po­ten­tially de­grad­ing it, it will re­main able to en­code & op­ti­mize nor­mal faces. (The orig­i­nal clas­si­fiers may even be reusable on this, de­pend­ing on how ex­treme the new at­tribute is, as the la­tent space z might not be too affected by the new at­tribute and the var­i­ous other at­trib­utes ap­prox­i­mately main­tain the orig­i­nal re­la­tion­ship with z as be­fore the re­train­ing.)

Running S2

Be­cause of the op­ti­miza­tions, which re­quires cus­tom lo­cal com­pi­la­tion of CUDA code for max­i­mum effi­cien­cy, get­ting S2 run­ning can be more chal­leng­ing than get­ting S1 run­ning.

• No Ten­sor­Flow 2 com­pat­i­bil­i­ty: the TF ver­sion must be 1.14/1.15. Try­ing to run with TF 2 will give er­rors like: TypeError: int() argument must be a string, a bytes-like object or a number, not 'Tensor'.

  I ran into cuDNN com­pat­i­bil­ity prob­lems with TF 1.15 (which re­quires cuDNN >7.6.0, 2019-05-20, for CUDA 10.0), which gave er­rors like this:

  ...[2020-01-11 23:10:35.234784: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library:
     7.4.2 but source was compiled with: 7.6.0.  CuDNN library major and minor version needs to match or have higher
     minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.
     If building from sources, make sure the library loaded at runtime is compatible with the version specified
     during compile configuration...

  But then with 1.14, the tpu-estimator li­brary was not found! (I ul­ti­mately took the risk of up­grad­ing my in­stal­la­tion with libcudnn7_7.6.0.64-1+cuda10.0_amd64.deb, and thank­ful­ly, that worked and did not seem to break any­thing else.)

• Get­ting the en­tire pipeline to com­pile the cus­tom ops in a Conda en­vi­ron­ment was an­noy­ing so Gokaslan tweaked it to use 1.14 on Lin­ux, used cudatoolkit-dev from Conda Forge, and changed the build script to use gcc-7 (s­ince gcc-8 was un­sup­port­ed)

• one is­sue with Ten­sor­Flow 1.14 is you need to force allow_growth or it will er­ror out on Nvidia 2080tis

• con­fig name change: train.py has been re­named (a­gain) to run_training.py

• buggy learn­ing rates: S2 (but not S1) ac­ci­den­tally uses the same LR for both G & D; ei­ther fix this or keep it in mind when do­ing LR tun­ing—changes to D_lrate do noth­ing!

• n = 1 mini­batch prob­lems: S2 is not a large NN so it can be trained on low-end GPUs; how­ev­er, the S2 code make an un­nec­es­sary as­sump­tion that n≥2; to fix this in training/loss.py (fixed in Shawn Presser’s TPU/self-attention ori­ented fork):

  @@ -157,9 +157,8 @@ def G_logistic_ns_pathreg(G, D, opt, training_set, minibatch_size, pl_minibatch_
      with tf.name_scope('PathReg'):
  
          # Evaluate the regularization term using a smaller minibatch to conserve memory.
          if pl_minibatch_shrink > 1 and minibatch_size > 1:
              assert minibatch_size % pl_minibatch_shrink == 0
              pl_minibatch = minibatch_size // pl_minibatch_shrink
          if pl_minibatch_shrink > 1:
              pl_minibatch = tf.maximum(1, minibatch_size // pl_minibatch_shrink)
              pl_latents = tf.random_normal([pl_minibatch] + G.input_shapes[0][1:])
              pl_labels = training_set.get_random_labels_tf(pl_minibatch)
              fake_images_out, fake_dlatents_out = G.get_output_for(pl_latents, pl_labels, is_training=True, return_dlatents=True)

• S2 has some sort of mem­ory leak, pos­si­bly re­lated to the FID eval­u­a­tions, re­quir­ing reg­u­lar restarts, like putting it into a loop

Once S2 was run­ning, Gokaslan trained the S2 por­trait model with gen­er­ally de­fault hy­per­pa­ra­me­ters.

ImageNet StyleGAN

As part of ex­per­i­ments in scal­ing up StyleGAN 2, us­ing TFRC re­search cred­its, we ran StyleGAN on large-s­cale datasets in­clud­ing Dan­booru2019, Im­a­geNet, and sub­sets of the Flickr YFCC100M dataset. De­spite run­ning for mil­lions of im­ages, no S2 run ever achieved re­motely the re­al­ism of S2 on FFHQ or BigGAN on Im­a­geNet: while the tex­tures could be sur­pris­ingly good, the se­man­tic global struc­ture never came to­geth­er, with glar­ing flaws—there would be too many heads, or they would be de­tached from bod­ies, etc.

Aaron Gokaslan took the time to com­pute the FID on Im­a­geNet, es­ti­mat­ing a ter­ri­ble score of FID ~120. (High­er=­worse; for com­par­ison, BigGAN with EvoNorm can be as good as FID ~7, and reg­u­lar BigGAN typ­i­cally sur­passes FID 120 within a few thou­sand it­er­a­tions.) Even ex­per­i­ments in in­creas­ing the S2 model size up to ~1GB (by in­creas­ing the fea­ture map mul­ti­pli­er) im­proved qual­ity rel­a­tively mod­est­ly, and showed no signs of ever ap­proach­ing BigGAN-level qual­i­ty. We con­cluded that StyleGAN is in fact fun­da­men­tally lim­ited as a GAN, trad­ing off sta­bil­ity for power, and switched over to BigGAN work.

For those in­ter­est­ed, we pro­vide our 512px Im­a­geNet S2 (step 1,394,688):

rsync --verbose rsync://78.46.86.149:873/biggan/2020-04-07-shawwn-stylegan-imagenet-512px-run52-1394688.pkl.xz ./