GPT-2 Neural Network Poetry

Demonstration tutorial of retraining OpenAI’s GPT-2 (a text-generating Transformer neural network) on large poetry corpuses to generate high-quality English verse.
statistics, NN, fiction, shell, GPT, tutorial, poetry
2019-03-032019-10-29 finished certainty: likely importance: 7

In Feb­ru­ary 2019, fol­low­ing up on my , I exper­i­ment with the cut­ting-edge Trans­former NN archi­tec­ture for lan­guage mod­el­ing & text gen­er­a­tion. Using Ope­nAI’s GPT-2-117M (117M) model pre-­trained on a large Inter­net cor­pus and nshep­perd’s fine­tun­ing code, I retrain GPT-2-117M on a large (117MB) Project Guten­berg poetry cor­pus. I demon­strate how to train 2 vari­ants: “GPT-2-poetry”, trained on the poems as a con­tin­u­ous stream of text, and “GPT-2-poetry-prefix”, with each line pre­fixed with the meta­data of the PG book it came from. In May 2019, I trained the nex­t-largest GPT-2, GPT-2-345M, sim­i­lar­ly, for a fur­ther qual­ity boost in gen­er­ated poems. In Octo­ber 2019, I retrained GPT-2-117M on a Project Guten­berg cor­pus with improved for­mat­ting, and com­bined it with a con­tem­po­rary poem dataset based on web­site. .> With just a few GPU-days on 1080ti GPUs, GPT-2-117M fine­tun­ing can pro­duce high­-qual­ity poetry which is more the­mat­i­cally con­sis­tent than my char-RNN poems, capa­ble of mod­el­ing sub­tle fea­tures like rhyming, and some­times even a plea­sure to read. I list the many pos­si­ble ways to improve poem gen­er­a­tion and fur­ther approach human-level poems. For the high­est-qual­ity AI poetry to date, see my fol­lowup page, .

For anime plot sum­maries, see TWDNE; for gen­er­at­ing ABC-formatted folk music, see & ; for play­ing chess, see ; for the Red­dit com­ment gen­er­a­tor, see ; for fan­fic­tion, the Ao3; and for video games, the walk­through model. For Ope­nAI’s GPT-3 fol­lowup, see .

Ope­nAI announced in Feb­ru­ary 2019 in their cre­ation of “GPT-2-1.5b”, a Trans­former1 neural net­work 10× larger trained (like a char-RNN with a pre­dic­tive loss) by unsu­per­vised learn­ing on 40GB of high­-qual­ity text curated by Red­di­tors. GPT-2-1.5b led to large improve­ments over GPT-1’s nat­ural lan­guage gen­er­a­tion, is close to or , and demon­strated high per­for­mance on untrained NLP tasks (see the paper for more details: , Rad­ford et al 2019). By large improve­ments, one means that the best sam­ples like the ones included in the OA announce­ment have started to reach an uncanny val­ley of text, capa­ble of telling entire semi­-­co­her­ent sto­ries which can almost fool a sloppy reader—cer­tain­ly, the verisimil­i­tude is bet­ter than any char-RNN out­put I’ve seen. (A dump of many more sam­ples is avail­able on GitHub. There is also an inter­ac­tive word-by-­word .) The full GPT-2-1.5b model was not released, but a much smaller one a tenth the size, GPT-2-117M was released in Feb­ru­ary 2019, which I call “GPT-2-117M” to avoid con­fu­sion.

GPT-2-117M was used in most ini­tial exper­i­ments with GPT-2-based text gen­er­a­tion. OA’s next largest mod­els, GPT-2-355M & were released in May & August 2019, and the final, largest, GPT-2-1.5b model was released in (too late to be used in most of these exper­i­ments); 355M–774M turn out to just barely be train­able on com­mod­ity GPUs.2 Also worth not­ing is the release of Gokaslan & Cohen 2019’s , which pro­duces good sam­ples if per­haps not quite as good as the Ope­nAI GPT-2-1.5b, but which was not train­able at the time on desk­top GPUs3 although it does still at least run (al­low­ing for sampling/prompting if not train­ing). Ope­nAI notes that “Since Feb­ru­ary, we’ve spo­ken with more than five groups who have repli­cated GPT-2”, and some have or gone fur­ther than GPT-2-1.5b: replications/extensions include Gokaslan & Cohen 2019, , Hug­ging Face (a NLP star­tup), , Nvidi­a’s , Google’s (fine­tun­ing Colab), and MS’s .

GPT-2-117M: Generating Poetry

“‘I don’t speak’, Bijaz said. ‘I oper­ate a machine called lan­guage. It creaks and groans, but is mine own.’”


Nat­u­ral­ly, peo­ple imme­di­ately used GPT-2-117M for all sorts of things, and I applied it myself to gen­er­ate sur­real anime plot sum­maries & dia­logue for .4 Even more nat­u­ral­ly, just as with char-RNNs, GPT-2 mod­els, even unfine­tuned, work well for poet­ry:

Poetry is a nat­ural fit for machine gen­er­a­tion because we don’t nec­es­sar­ily expect it to make sense or have stan­dard syntax/grammar/vocabulary, and because it is often as much about the sound as the sense. Humans may find even mediocre poetry quite hard to write, but and can gen­er­ate many sam­ples to select from, so the final results can be pretty decent.

The qual­ity of the results is lim­ited by some­times only hav­ing access to smaller mod­els and dif­fi­culty in run­ning larger mod­els at all; that can’t be fixed (yet). But qual­ity is also reduced by GPT-2-117M being trained on all kinds of text, not just poet­ry, which means sam­pling may quickly diverge into prose (as seems to hap­pen par­tic­u­larly eas­ily if given only a sin­gle open­ing line, which pre­sum­ably makes it hard for it to infer that it’s sup­posed to gen­er­ate poetry rather than much more com­mon pros­e), and it may not have learned poetry as well as it could have, as poetry pre­sum­ably made up a minute frac­tion of its cor­pus (Red­di­tors not being par­tic­u­larly fond of as unpop­u­lar a genre these days as poet­ry). Fine­tun­ing or retrain­ing the released GPT-2-117M model on a large poetry cor­pus would solve the lat­ter two prob­lems.

The poetry sam­ples above did not exploit fine­tun­ing because Ope­nAI did not pro­vide any code to do so and declined to pro­vide any when asked. For­tu­nate, nshep­perd wrote a sim­ple fine­tun­ing train­ing imple­men­ta­tion, which I could use for adding more inter­est­ing sam­ples to my TWDNE and for retrain­ing on poetry cor­puses to com­pare with my pre­vi­ous char-RNN poetry attempts back in 2015–2016 (see the top of this page). An alter­na­tive GPT-2 train­ing imple­men­ta­tion with sup­port for train­ing on GCP TPUs has been cre­ated by Con­nor Leahy (), who .

Training GPT-2-117M To Generate Poetry

Data: The Project Gutenberg Poetry Corpus

“My heart, why come you here alone?
The wild thing of my heart is grown
To be a thing,
Fairy, and wild, and fair, and whole”


For the poetry cor­pus, Alli­son Par­rish’s pub­lic domain “A Guten­berg Poetry Cor­pus” (“approx­i­mately three mil­lion lines of poetry extracted from hun­dreds of books from Project Guten­berg”) will serve admirably. A few other pos­si­bil­i­ties sur­face in Google Dataset Search, like “Poems from, but noth­ing par­tic­u­larly com­pelling.

As far as the text for­mat­ting goes, GPT-2-117M is flex­i­ble, you can dump in pretty much any text into a text file to use as the cor­pus, but some text for­mats are bet­ter than oth­ers. You want some­thing which is as reg­u­lar as pos­si­ble (in both syn­tax & seman­tic­s), but also one which is as close to the kind of text you want gen­er­at­ed, but also which wastes as few sym­bols as pos­si­ble. Reg­u­lar­ity makes learn­ing eas­ier, and you don’t want to have to mas­sage the out­put too much, but on the other hand, GPT-2-117M has a nar­row ‘win­dow’ and no mem­ory what­so­ev­er, so if each line is padded out with a lot of for­mat­ting or even just white­space, one would expect that to con­sid­er­ably dam­age out­put coher­ence—as most of the fixed ‘win­dow’ is wasted on mean­ing­less repet­i­tive white­space, while other changes like replac­ing new­lines with the poetic con­ven­tion of ’ / ’ are worse than noth­ing (since new­line is 1 char­ac­ter vs 3 and max­i­mally dense). Min­i­miz­ing for­mat­ting also makes the cross-en­tropy loss eas­ier to inter­pret or com­pare across datasets/runs: if there is a lot of for­mat­ting which is easy to pre­dict or lit­tle for­mat­ting, the loss can look mis­lead­ingly good (or bad) as it eas­ily pre­dicts the for­mat­ting but strug­gles on more mean­ing­ful con­tent.6

The PG cor­pus has a strange for­mat: each line is a sep­a­rate JSON object, con­sist­ing of one line of poetry and a numeric ID for the work it’s from. For­tu­nate­ly, the file as a whole is in order (if the lines were out of order, train­ing on them would destroy the long-range lan­guage mod­el­ing which is the Trans­former’s rai­son d’être!), so to turn it into a clean text file for train­ing on, we can sim­ply query it with jq and strip out the remain­ing for­mat­ting. This pro­vides a pretty good for­mat over all: the new­lines are mean­ing­ful, no sym­bols are wasted on lead­ing or trail­ing white­space, and it looks like what we want. It is imper­fect in that metadata/formatting we would like to be there, such as author or poem title, is not there, and things we would pre­fer not to be there, like the prose pref­aces of books or anno­ta­tions, are, but hard to see how to fix those eas­i­ly. Another flaw I learned about only after­wards is that the PG cor­pus has been cen­sored ad usum Del­phini to remove “egre­giously offen­sive con­tent” such as “racist/sexist/ableist” words like “Pak­istan” or “homoge­nous” (while, of course, per­mit­ting words like “shit” or “fuck”). This cor­pus is unsuited for any seri­ous aca­d­e­mic work, but it should be fine for play­ing around with gen­er­at­ing poems.

Set­ting up the GPT-2-117M train­ing envi­ron­ment & obtain­ing the poetry cor­pus:

git clone ''
cd gpt-2
source activate $MY_TENSORFLOW_ENVIRONMENT # set up your particular virtualenv
sh 117M # download original OA model

wget ''
gunzip gutenberg-poetry-v001.ndjson.gz
cat gutenberg-poetry-v001.ndjson | jq .s | sed -e 's/^.//' -e 's/.$//' -e 's/\\//g' \
    >> gutenberg-poetry-v001.txt ## delete JSON quoting

shuf gutenberg-poetry-v001.txt | head ## random poetry lines:
# For the black bat, night, has flown,
# The other's fate, Gaville, still dost rue.
# That make me mad. Oh, save me from those eyes!
# eyes, "w'y don't you talk straight out from the
# Make all there is in love so true.
# But p'r'aps I couldn't.
# And the soft ground turned to gravel,
# "'We will fight in bloody scuffle.'"
# For the fever'd dreams on thy rest that throng!"
# It is strange--my heart is heavy,
du -h gutenberg-poetry-v001.txt; wc gutenberg-poetry-v001.txt
# 117M  gutenberg-poetry-v001.txt
# 3085117  21959786 121730091 gutenberg-poetry-v001.txt

There is an addi­tional step before begin­ning train­ing. GPT-2-117M works with text in a “byte-­pair encod­ing”, which is some­where in between a char­ac­ter embed­ding & a word embed­ding. The point of this BPE encod­ing is that it is some­what more effi­cient than raw char­ac­ters, because it can chunk more com­mon sub­-­words or phrases & this gets more com­plete words or phrases into the Trans­former’s fixed ‘win­dow’ of n sym­bols, but BPE still assigns sym­bols to indi­vid­ual let­ters, and thus arbi­trary out­puts can be gen­er­at­ed, unlike word-level NNs which are more com­pact but trade this off by hav­ing a restricted vocab­u­lary of m words seen in the train­ing cor­pus and must treat every­thing else as the unknown token <UNK> (espe­cially bad for rare words like proper names or vari­ants of words like plu­ral­iza­tion or tens­es). The train­ing code will encode the text cor­pus at startup if nec­es­sary, but for 117MB of text this is so slow that it is worth the extra work to run the encod­ing process in advance & store the results before train­ing on it:

PYTHONPATH=src ./ gutenberg-poetry-v001.txt gutenberg-poetry-v001.txt.npz

Training GPT-2-poetry

The temp­ta­tion of CPU train­ing.

I assume you have a ful­ly-­work­ing Nvidia CUDA & GPU-enabled Ten­sor­flow instal­la­tion, and have either run other DL code suc­cess­fully or run the TF instal­la­tion check­list’s MNIST toy exam­ple to ver­ify that you have work­ing GPU train­ing. If you do not, I can give you no advice other than “good luck”. Debug­ging CUDA prob­lems are the worst, and once you get a work­ing setup, you should stick with it. If you just can’t solve the inscrutable crash­es, you should look into using the free Google Colab GPU/TPU note­books, or rent­ing a cloud VM. (Train­ing in the cloud is not as hard or com­pli­cated as it may seem, since you can pick a VM OS image which comes with CUDA/Tensorflow pre­in­stalled and which will always work with the avail­able GPUs.) You may be tempted to train on CPU to avoid the GPU-support mess entire­ly, but I advise against this: it will be at least 20× slower even if you have a many-­core CPU like Thread­rip­per, and it’ll save you time in the short run to switch to Colab or cloud or some alter­na­tive.

Then train­ing proper can begin; my Nvidia 1080ti7 can fit a mini­batch size of 2 (GPT-2-117M is still a large mod­el), and I’d rather not see too much out­put so I reduce the fre­quency of check­point­ing & ran­dom text gen­er­a­tion:

PYTHONPATH=src ./ --model_name 117M --dataset gutenberg-poetry-v001.txt.npz \
    --batch_size 2 --save_every 10000 --sample_every 1000
Check your CLI options

The Python library “fire” used in the OA GPT-2 code is treach­er­ous—it will not error out or even warn you if you typo a com­mand-­line option! Dou­ble or triple-check any new options you set against the avail­able argu­ments defined by train_main in, and keep this gotcha in mind if set­ting an option does­n’t appear to be doing any­thing.8 While nshep­perd has removed use of “fire” in favor of saner CLI options, watch out for this if you are using the orig­i­nal OA code or other deriv­a­tives.

Some hyper­pa­ra­me­ters could use tweak­ing:

  1. run­time, Tem­per­a­ture:

    ‘Tem­per­a­ture’ (0–∞) is used in sam­pling: the top-k most likely words are gen­er­at­ed, and then selected ran­domly from; at 0, the most likely word is always cho­sen, while 1 means each is selected accord­ing to its like­li­hood, and it degen­er­ates to a uni­form 1 in k prob­a­bil­ity with higher val­ues. In other words, the higher the tem­per­a­ture, the more chaotic or unlikely the gen­er­ated sequences will be.

    In the orig­i­nal nshep­perd code release, the default tem­per­a­ture set­ting for the sam­ples dur­ing train­ing, 1.0, is not the usual 0.7 every­one uses for GPT-2 prose sam­pling—although it turns out for poetry we don’t want it at 0.7 as that forces too many repeated lines & 0.9–1 turns out to be much bet­ter, so use tem­per­a­ture in that range when gen­er­at­ing sam­ples. (Higher still may be bet­ter but I have not exper­i­mented with >1.)

    If you are sam­pling after 2019-05-15, it may be a bet­ter idea to use a new sam­pling strat­e­gy, (which essen­tially sets a dif­fer­ent k at each step to avoid sam­pling extremely unlikely words and greatly reduces the rep­e­ti­tion prob­lem), which can be enabled like --top_p 0.9. (An inter­est­ing but untested sam­pling strat­egy is “tail free sam­pling”.)

  2. train time, Learn­ing Rate (LR):

    A key NN hyper­pa­ra­me­ter as always.

    In nshep­perd’s code, the Adam SGD learn­ing rate is left at its Ten­sor­Flow default of 0.001, which works ini­tial­ly, but appears to be much too high for this pur­pose (per­haps because the mini­batch is so tiny on 1 GPU). After train­ing overnight, the loss was not decreas­ing below 2.5, so I decayed it man­u­ally to 0.0001 & resumed train­ing (edit­ing line 136 of to read tf.train.AdamOptimizer(learning_rate = 0.001*0.10)), even­tu­ally decay­ing it again (to 0.001*0.0001) to get it down to a loss of ~1.95. (nshep­perd has since added a --learning_rate option so man­ual edit­ing of the source is no longer nec­es­sary.)

GPT-2-poetry Samples

After train­ing GPT-2-117M an hour or two, a sam­ple

# ...[14999 | 3855.62] loss=3.77 avg=3.62
# ======== SAMPLE 1 ========
#  in her place before
# We could not see her smile.
# If, the dear one there are,
# That thou dost hide thy face--
# In thy own eyes, too, I shall see--
# A gentle smile, and smile.
# And I shall see thee, even as in that place,
# There be whom death shall be.   By some, at least in some,
# Not to them I shall say; wherefore
# I have, to whom I will not say.
# For though I should be sad,
# 'Tis not to see them--no,
# No voice I can convey
# Or voice to me or voice--no,
# In a single case my own.
# They who, in all this state,
# Celestial are.  The world
# Of heaven must have its own:
# And when that thou canst tell--
# That which I cannot tell,
# No matter whose.
# If I should tell thee then--
# That I should tell thee then--
# This man--but this man!
# That I should say to thee,
# That thou shouldst say to thee,
# It cannot be without,
# But that I shall speak to thee--
# Thou knowest not the way.
# Whence no speech!  No tongue!
# In that I am not weak in thy power.
# No, no, I will speak to thee--
# I will speak for thee:
# For if it still do need
# I shall speak a word or an act.
# In that place, far away,
# On this sea, and that place!
# I will tell thee, and I will speak to thee.
# ...

Overnight sam­ples dur­ing train­ing:

# ...[115999 | 2375.96] loss=1.10 avg=2.31
# ======== SAMPLE 1 ========
# The sun is gone, and the night is late,
# The lights are few, and the chimneys close,
# And yet the moon hangs lonely
# Above the town, where the tides divide,
# On the shores of the silent tide.
# The tide is still, and the tide is at rest,
# And the sands are many, and many a breast
# And breast of the tide is weary,
# And far away the tide's last foam
# Stands silent under the waves;
# But the restless tide creeps slowly,
# And the tide is strong, and the tide is deep,
# And the sea, like a white-faced phantom,
# Floats motionless over the deep.
# The tide is past, and the tide is at rest,
# And the sands are many, and far apart,
# And the sands are many, and far apart,
# And the sea, like a white-faced phantom,
# Floats motionless over the sea.
# 'Tis evening, and he comes to me,
# With scent of flowers, from the village lane;
# And the wind is blowing faint and low,
# And the dew, like tears, on the hill-tops pale,
# Is drifting faint and cold from the eastern sky,
# As he floats adown the western wave;
# And the trees that bend their branches sigh,
# And the winds that waft a sigh,
# Are fading dimly away,
# As the tide goes wandering,
# Tossing waves of sand.
# And the cloud that floats and swells
# Has passed, like the wave on the bay,
# And the moon is slowly sinking low,
# As he sinks adown the western bay.
# There are waters blue and bright
# On the breast of the silent sea;
# And the foam is like a bright
# And lingering pearl of the sea.
# Oh, I would my heart might still
# Ever be where the waves are free.
# Where the foam-flakes fall and swell,
# Falling into the silent sea;
# Where the foam-flakes fall and swell,
# Falling into the silent sea;
# Where the foam-flakes fall and swell
# Falling into the silent sea.
# Oh, I would my heart might still
# Ever be where the waves are free.
# Far away from the sea and the tide,
# In a quiet little bay,
# Where the foam-flakes fall and swell,
# And the rocks are telling of a day
# When I would my heart could stay;
# Where the foam-flakes fall and swell
# Falling into the silent sea;
# Where the foam-flakes fall and swell
# Falling into the silent sea.
# Oh, I would my heart might still
# Ever be where the waves are free.
# With a love that never will,
# With a passion never still,
# With a memory never still
# For a love that never will,
# With a sorrow never still
# For a love that never will,
# With a passion never still,
# With a passion never still,
# With a passionate love that never will,
# With a sorrow never still
# For a love that never will.
# Oh, I would my heart might still
# Ever be where the waves are free.
# With a love that never will,
# With a passion never still,
# With a sorrow never still,
# With a passion never will,
# With a passion never will,
# With a sorrow never still,
# With a love that never will.
# With a love that never will,
# With a passion never will,
# With a sorrow never still,
# With a love that never will,
# With a sorrow never will,
# With a sorrow never still,
# With a love that never will,
# With a grief that never will,
# With a sorrow never will,
# With a love that never will.
# Oh, I would my heart might still
# Ever be where the waves are free.
# With a love that never will,
# With a sorrow never still,
# With a sorrow never still,
# With a sorrow never will,
# With a love that never will,
# With a sorrow never will,
# With a sorrow never will.
# With a love that never will,
# With a sorrow never will,
# With a sorrow never still,
# With a sorrow never will,
# With a sorrow never will,
# With a love that never will.
# With a love that never will,
# With a sorrow never will,
# With a sorrow never still,
# With a sorrow never will,
# With a sorrow never will,
# With a sorrow never will,
# With a sorrow never will,
# With a sorrow never will,
# With a love that never will,
# With a sorrow never will.
# With a love that never will,
# With a sorrow never will,
# With a sorrow never still,
# With a sorrow never will,
# With a sorrow never will,
# With a grief that never
# [116000 | 2385.67] loss=3.33 avg=2.32
# ...
# [133999 | 5919.61] loss=2.89 avg=2.21
# ======== SAMPLE 1 ========
# "When the night is gone and the sun goes out,
# "There goes my bride and my lover:
# "Can she never come to her wedding-ring
# "With the light of the seven candles?
# "Oh! to bed with the bridegroom and bride,
# "Or to lie with my lover!
# "He shall not come to my bed at night
# "For a little while longer.
# "She shall not come to my chamber
# "For a little longer."
# The night is spent, the sun has gone;
# The bridal of the bride hath done;
# The bridal of the bridegroom comes
# To wed the bridegroom's attour.
# I come to see the wedding-feast.
# He cries to me--"Wilt seek the bride,
# "Wilt seek the bridal?"
# "Oh! I have sought her mother's bower
# "And never found her! oh! my flower!
# "Would we should love as brides should do!
# "I shall not find her!"
# The bridegroom at the bridegroom's door
# Gave his bride a ring and a prayer.
# "Now, bridegroom, sing a bridegroom loud!
# "I shall not find her!"
# The bridegroom at the bridegroom's door
# Gave his bride a ring and a prayer.
# What means the bridegroom or the bride?
# The bridegroom's bridegroom waits to ride.
# And looks with wonder at the bride;
# "And does she dream?"
# "Oh! I have dreamed!
# "She dreams of my youth,
# "As one that hears--it cannot be -
# "The story of a marriage vow!"
# And a voice answers:
# "The story of a marriage vow!"
# And the words reach the bridegroom's door
# As the bridegroom at the bridegroom's door
# Kisses the ring with a bridegroom's kiss on his cheek.
# They are wed! They are wedded!
# Each is in his bridegroom's bower;
# Each hath his bride in his bosom now!
# And each hath his bride in his heart.
# She is wedded!
# With each is the bridegroom's bride!
# And love is a bridegroom's bride!
# She is wedded!
# With each is the bridegroom's bride!
# "They are wedded!
# They are wedded!
# They are wedded!"
# A man from a fair tower, where the birds of the air
# Fairer and fainter and fairer the flowers and the trees,
# When the sweet light fades from the gardens of day he found
# Came he in quest of a maiden, whose form was wildly wild,
# So she came at the summons. He came, and stood by her side;
# And he gazed on the dream of her marvellous face, and smiled;
# And he said:  "I came by the river's side, when the day was still
# And I think of my bridegroom--he--whom ye all have a will!
# She is wedded!
# And he loves me not, my bride!
# And he looks at her eyes with a love that seems to him divine,
# Then he pines, and thinks of her eyes with an inward passionate shine
# They are wed! They are wedded!
# By the river's side they are, bridegroom, led!"
# He hath summoned her maiden with the will, and she answers
# That he needs her for his bridegroom still.
# A man from a fair tower, where the birds of the air
# Fairer and fainter and fairer the flowers and the trees,
# What is the meaning of that--no--is it, Lord, where ye bear
# All the wonders of the world in this fair maiden's hair?
# What doth the meaning of that--no--nor what doth she know,
# She is wedded!
# What doth the meaning of that--no--she hath chosen so?
# She is wedded!
# She is wedded!
# She is wedded!
# She is wedded!
# She is wedded!
# She is wedded!
# She is wedded!
# And the will is good,
# In the light of days,
# In the heat and stress,
# It is wise and wise
# To be wedded!
# She is wedded!
# And the love is good,
# In the light of days,
# It is hard to live
# And bear the wreath!
# And he girdeth her to his bosom, and thinks of her
# As she stands, all white, in the open air,
# With the full moon facing her face; and her breath
# Cries and whispers and wails and cries
# ...[134000 | 5929.46] loss=2.12 avg=2.21
# ...[135998 | 6934.07] loss=1.96 avg=2.16
# [135999 | 6934.58] loss=3.06 avg=2.17
# ======== SAMPLE 1 ========
# re
# Their love and beauty are as one in dream,
# A visible sign of the things that were.
# But I have seen these things by all men's eyes,
# Felt them as kindred of man's earthly life,
# And with the instinct of the unseen spirit
# Been caught by likeness of the thing he is.
# My love is but the wind, and I have blown
# From earth to where I am, and I have seen
# The things that no man dreamed of; yet at last
# I know by my soul's sense of a sense of things
# That are not, and may be, but the things that were;
# And yet I know these things are not, but are
# As earth and heaven, if earth and heaven and hell
# Are but the same things that it seems. Yea, then
# I am the wind. God knows the ways of men,
# He knows the insensate secrets of delight,
# And they are mysteries, if there be any to be seen."
# But with that word the wind in wonder strode.
# He heard the rustle of the leaves, and saw
# The shadows move about him, and he leaned
# Against the doorway like a god, and knew
# The inner meanings of the leaves and streams.
# There where the trees lie down at their root-holes,
# There where the wind smells of the blossoming boughs,
# He saw the grass, and felt the green blades come,
# As if it were the buds and boughs upon air,
# And heard the green birds sing. He saw the fields,
# The trees, the rivers, and the flowers within,
# The birds, the grasses, and the living things,
# And the strange river on the shore that rolls
# Through all its quiet marge into the sky.
# There let him live till time should come, and then
# Let love be like the heaven, and we be one
# To love, and not be one, being all in all.
# And if he had not done me the good work
# Had it been well not I. The things that he said
# Should never be fulfilled by simple sense;
# For all must have a meaning in themselves.
# But he that works out of his mind is one
# With whom the things that are and are not are,
# And makes them meet and good. 'T were a good thing
# For him to work and win for me, and so
# If he were not I would have it all.'
# But he that lives and not lives in the world
# Was not more worthy of the hand of Fate,
# And knows life's meaning, and would seek for it
# Through failure, and in death's despite. For him,
# Who hath been stricken with me through the brain,
# Forget to tell me how his brother, he
# Whom he had saved and murdered--so let it be
# By some great memory left.
# But at last,
# As I said this, he saw me, and he said
# To one, whose face was grey with tears in me,
# "What is it? let me tell you who I am.
# Do you see the things that you have seen before?
# What is it?"
# "They are more wise
# Than wise men think of wisdom and good will,"
# Replied the other. "What I deem is good.
# The gods are good to mortals as they are,
# And they know well whereby we are born: but they
# Who have loved God and died to him the most
# Of all the gods are fallen into ill things:
# For God we know is good, and hath not been,
# And therefore must be, so it be, with men
# Who love, and love because we loved them not.
# Alas, I do not think that God alone
# Hath power over the earth to let the gods
# Face to face with the world. I hate at times
# The gods that made them: the gods that knew
# Their names are our own gods, and would not know
# One other reason, for I have the power,
# And all the gods are fallen into ill things."
# Then she said to me, "What may have been
# To have known, before I came into this land
# To find you in some other place and knew you,
# And know you, seeing so many and strange,
# And knowing such a godlike way to go
# Among the gods and suffer such long-sought.
# I can take my crown of gold and wear a garland,
# Take some crown for my sake, and the happy crown
# And let it be for all the years long held
# That I have known, and felt so like a god
# Some few suns live. My heart is all in all
# To live again, my life upon earth dead."
# So I said to the god that loved me well
# And longed to have him come back into my prayers,
# [136000 | 6944.39] loss=3.15 avg=2.18
# ...

The loss here is the usual cross-en­tropy we often see in archi­tec­tures like a char-RNN. Typ­i­cal­ly, the best text gen­er­a­tion results come when the model has trained down to a cross-en­tropy of <1, and 2–4 tend to be inco­her­ent gib­ber­ish. (For exam­ple, in .) That loss is per char­ac­ter, while GPT-2 oper­ates on BPEs, which usu­ally encode mul­ti­ple char­ac­ters, so are harder to pre­dict; it seems to me that the con­ver­sion fac­tor is ~2–3, so a GPT-2 model should aim for a loss of <2 if a good char-RNN would reach losses like <1. In this case, GPT-2-117M’s orig­i­nal poetry mod­el­ing capa­bil­ity is not too shabby (as demon­strated by the var­i­ous prompted sam­ples), and it shows decent poetry sam­ples start­ing ~3.5. (Gib­ber­ish seems to set in at losses >6.) Given how large & pow­er­ful GPT-2-117M is, even with this much poetry to work with, over­fit­ting remains a con­cern—mem­o­riz­ing poetry is not amus­ing, we want cre­ative extrap­o­la­tion or mashups.

For this model & dataset, I trained for 519,407 steps to a final loss of ~2 in 72 GPU-hours; almost all of the learn­ing was achieved in the first ~16 GPU-hours, and train­ing it addi­tional days did not do any appar­ent good in terms of the loss itself.9 This sug­gests that GPT-2-poetry was under­fit­ting the poetry cor­pus & would ben­e­fit from an even larger model size.


Before sam­pling from any new fine­tuned ver­sion of GPT-2-117M, remem­ber to copy encoder.json/hparams.json/vocab.bpe from the GPT-2-117MB model direc­tory into the new mod­el’s direc­to­ry. I find higher tem­per­a­ture set­tings work bet­ter for poetry (per­haps because poetry is inher­ently more repet­i­tive than pros­e), and top-k appears to work fine at OA’s top-40. So uncon­di­tional sam­pling can be done like this to gen­er­ate 2 sam­ples:

python src/ --top_k 40 --temperature 0.9 --nsamples 2 --seed 0 \
    --model_name 2019-03-06-gwern-gpt2-poetry-projectgutenberg-network519407
# ======================================== SAMPLE 1 ========================================
# --
# And it must be decided.
# It must be decided,
# And it must be decided.
# It must be decided,
# And it must be considered.
# It will be decided,
# Though the hill be steep,
# And the dale and forest
# Hold the land of sheep.
# And it must be decided,
# There's a jolt above,
# And its paths are narrow,
# And its paths are long.
# Yes, it is decided,
# And it is completely.
# All the hills are covered
# With grey snowdrifts,
# Shaded with a shimmer of misty veils,
# And the hills have a shimmer of hills between,
# And the valleys are covered with misty veils,
# And there lie a vast, grey land, like a queen,
# And they are not, in truth, but many and many streams,
# O'er the purple-grey sea whose waves are white
# As the limbs of a child of ten.  And there
# The river stands, like a garden-fair
# In the valleys of the north, the valleys of the west,
# Blue and green in the summer, and runneth softly forth
# To the blue far upland beyond the sea;
# And over the high white upland far away
# Floats a white and tender water, and wearily
# Through the trees the rosiest water-lilies play
# In the sun, and rise and fall--the purple and red
# Of the streams.  The waters are hidden in their bed
# By the stone o'er the darkling hills.  The waters run
# Like a ringlet under the stone.  The water flows
# Through the rocks like a river, and the stream
# Is a ribbon of gold spun by the sun.  It gleams
# Like a gold sunbeam shining through the gleam
# Of a sudden silver, and silently falls
# On the pool, and is lost in the darkling deeps--
# Sink, sink in the shadows, ere it flee
# Into the darkling depths.  And the waters sleep
# In the light of the moon and the silver of dawn,
# And silently float past the mountains of heaven.
# As we gazed the city fades into the clouds
# Of the sky, and we are above the roofs.
# And suddenly as the moon, flurrying,
# Dazzles the sea with her swan-throated song,
# And there is a faint far singing of birds,
# And a sound from the land, as of swarming seas,
# The grey sea, and the land that hideth rest,
# And the sky that hides the lovely green of God.
# So we are caught, like the moving sea,
# That calleth unto its sleeping
# Soft and still, like the moon that calleth
# In the twilight depths vast and hoary--
# Till we see the City changing toward the dark,
# And its changing towers in the distance darken.
# In the city is a calm and quiet street,
# Full of sunlight, and a smell of rain,
# That falls from unseen towers like soft white feet
# On sleeping city's rue and misty pane.
# There is peace, and a vague peace over death,
# And a far-off singing in the city's breath.
# And all fair cities must go to dust,
# And every body be one tomb--
# And all white houses dwindle and grow dull,
# And the city's breath is a dull death-blow.
# But this place is a place of peace and trust,
# And it is but a little street,
# Whose idle heads and sunken faces
# Are bright with light that makes them bright.
# Then it is not alone fair Town that lies,
# With open pillared streets beneath a sun,
# And many a weary world and dusty town,
# And a sunflowers and a great tide onward run
# In the blue of the heavens that are not gray,
# But only blue and pale, like tender wings
# Sailing with wide-spread, languid, luminous eyes.
# This place is the very heart of it,
# Whose quiet hours with its peace throng
# The silent nights and the perpetual sea.
# The City slept with her silent towers,
# A stream that ran in an idle stream,
# And a mist hung at the windows of the tower.
# And it was a street--a sunlit dream,
# A dream of a world that lay
# Open in the summer morning,
# And in its heart a joy all gay.
# For its sunshines and palaces were there,
# Till a wind came softly here.
# And it was a new, new city,
# A city that arose in the early morning;
# That opened its gates on June morning,
# With a sunset and a moonrise sweet.
# The city was a cathedral;
# And out of the sound of the bells and t
# ======================================== SAMPLE 2 ========================================
#  of the world
# The best, that, when once dead, is found again.
# And what is this?  Where can we find a place,
# Save in the solitude, where he may be
# The friend of all beneath the sun, and be
# An unseen presence, if the traveller's eye
# Can follow where he cannot:  there he stands
# Dark in majestic pomp, like those whom owls
# Could once have told down with a lion's maw.
# His form is like his fathers, and the crown
# Of all his race:  the very colours are
# As his to-day, which we must see and bear;
# The only parent is the creature's he.
# His face, where we have marked it, is but veiled
# In twilight, when we see, and he appears
# Himself in all his nature--where, if man
# Can recollect, he saw it in the frame:
# 'Tis clay wherever found--and so is called,
# When nature gives him back her clay.  It means
# That clay was form'd; but clay is form'd elsewhere;
# He needs must feel through all this frame, and, lo,
# The horse he rears, is human in his mind.
# So too, his nature is a thing apart
# From the great Nature, which has made him thus
# A likeness of himself:  and he beholds
# The creatures that he knows, and not intends
# To visit them, and only in their hearts
# Deserts them; and if they come indeed,
# And if the sea doth bring them, then the man
# Is still a child of theirs.  He can recall
# His mother's features and the father's look.
# And often he has said that he foresaw
# The sea, the winds, that he may all at will
# Be sea.  In short, the man is all he sees.
# He fears the sea may hurt him.
# Lashed to the helm,
# The ship was in the sea, and, on its moor
# And the sails furled, in silence sat the maid
# Motionless, like a star; no sound was heard
# Save of the distant ocean's fitful hum;
# The sounds of tempest came to him, his ears
# Mercurially listless, and his heart
# Disturbed like a distempered sea; he stood,
# And gazed from heaven in an unblest thought;
# He had not heard his mother's voice; he gazed;
# The mother's look was of a loftier mood;
# He had not heard his own; he had not heard
# What ever was, where his own heart has been;
# He had not understood the very thought
# Of his own heart, where life could find no shore.
# The sea beats on:  the vessel's bell strikes six:
# Dive down, O death! to earth, to heaven! to heaven!
# And it is sweet thus to be two souls alone:
# Dive down for home, and to the air renounce
# The galling bonds of everlasting life
# In some lone bark, that, dying, to the last
# Are still as death without her:  so to him,
# The mother's voice, still sweeter, spoke of home;
# And as the young man fell upon her breast,
# The mother's oracle, the words of death,
# Even as he spoke, a living death arose:
# He feels his heart rise, and ascend the sky.
# The wreck shall surely reach the sea; he dies,
# A mortal change, as earth, in which it was;
# And God, though dead, had still a dying man.
# But when they parted, he can never die.
# There are thousands, yes, there are thousands who,
# Without a mother, could not die unheard
# Of by a hand unseen:  yet some are sad,
# Lonely and wretched here, without a mate;
# Or if the grave touch, the great hearts' light
# Have no soft touch, even of a brother's grief
# Scarce suffered, they shall each a new life yield;
# And one, once more on earth, to heaven, or God,
# Shall meet his father's face, or bless his grave.
# Not vainly on these mocking thoughts he breathes;
# They sink to nothing when he sinks to rise:
# The tears of fatherly compassion reach
# The mother's eyelids, her, but not her eyes.
# And now a voice was heard by the wild bird,
# With words of comfort from the infant boy.
# Oh, had it stayed the angel's birth, and then
# Those tresses streaming, would have felt the strain
# For the bright star, and for a glorious man.
# It is a noble deed:  and, through the world,
# Doth woman triumph, though she suffer loss
# And poverty and pain, and,

Not bad.

Cleaning Project Gutenberg & Contemporary Poetry

“In the dark the sun doth gleam,
And in the dark the moon doth seem
But now the evening is begun—
Gone is the sun upon the earth!
The sil­ver moon doth like a cup
Of blood­-red wine, and as that cup
Is drained of life, doth quench no drop.
What man will drink such wine?”


Shawn Presser noted the issues with the Project Guten­berg cor­pus and, as book-level tran­si­tions are solved, sug­gested a heuris­tic for recon­struct­ing the blank lines denot­ing (pre­sum­ably) stan­zas: use the num­bers (GIDs) from the orig­i­nal JSON for book-level tran­si­tions, and look for lines which might be tran­si­tions to insert new­lines. (Im­per­fect, but bet­ter than noth­ing!) Since stan­zas are still con­nect­ed, <|endoftext|> is used for the book-level tran­si­tions, and a blank line is used for the stan­za­-level, pre­serv­ing as much of the seman­tics as pos­si­ble.

Presser’s Python script:

import json
import os
import sys

def split_by_book(poems):
  prev_gid = -1
  r = []
  for entry in poems:
    gid = entry['gid']
    if gid != prev_gid:
      if prev_gid != -1 and len(r) > 0:
        yield r
        r = []
      prev_gid = gid
    r += [entry]
  if len(r) > 0:
    yield r

def split_by_transition(poems):
  prev_line = -2
  r = []
  for entry in poems:
    line = entry['line']
    if line != prev_line + 1:
      if prev_line != -2 and len(r) > 0:
        yield r
        r = []
    prev_line = line
    r += [entry]
  if len(r) > 0:
    yield r

def extract(entries):
  return '\n'.join(list(map(lambda x: x['s'], entries)))

def final(poems):
  for book in split_by_book(poems):
    yield '<|endoftext|>'
    for stanza in split_by_transition(book):
      if len(stanza) > 5:
        yield '\n'
        for entry in stanza:
          line = entry['s'] # {
          line = line.rstrip('} ')
          yield line

def output(poems, fname):
  with open(fname, "w") as f:
    for line in final(poems):
      f.write(line + '\n')

def main():
  print('Loading poems...')
  with open("gutenberg-poetry-v2.ndjson") as f:
    poems=[json.loads(line) for line in f]
  output(poems, "corpus.txt")

if __name__ == "__main__":

This is con­verted to the NPZ and trained as usu­al. I retrained the pre­vi­ous non-pre­fix GPT-2-117M PG poetry model for ~30k steps (>16h?) down to a loss of ~1.73. (I used GPT-2-117M instead of GPT-2-345M for com­pat­i­bil­ity with my con­cur­rent exper­i­ment in pref­er­ence-learn­ing train­ing.)

The results are quite nice, and com­pet­i­tive even with 345M. Some selected sam­ples:

The gods are they who came to earth
And set the seas ablaze with gold.
There is a breeze upon the sea,
A sea of summer in its folds,
A salt, enchanted breeze that mocks
The scents of life, from far away
Comes slumbrous, sad, and quaint, and quaint.
The mother of the gods, that day,
With mortal feet and sweet voice speaks,
And smiles, and speaks to men:  "My Sweet,
I shall not weary of thy pain."

...Let me drink of the wine of pain
And think upon the agonies of hope,
And of the blessed Giver of all good things;
For, man for man, mine is the deepest love
That sorrow takes upon the humblest soul;
But who hath learned how sorrow turns to gall
The places where my feet have trod before.

...And 'stead of light, o'er earth, o'er rocky mountains,
A slowly falling star,
Its pointed pointed splendor far uplifting,
Heaven's flowery path bore down;
Each cranny of the air a gracious feeling,
It waved divinely round,
It called us hence, "Come what wouldst thou here?"--
Sweet mountain, that I love,
With that bright tint of heaven above,
'Twould make me still to see
One like to thee,
As fades the light that seeks the wandering eye.

...The skies are smiling sweetly on,
And summer's fairest hours are gone.
Oh, blessed Mercy! how the blest
Taste life itself can truly taste.
Thy morn of days, with all its past,
May on life's tempest paint the last.

...When you come to die,
Every nerve and bone
Soon lulled in sleep,
Secure and free,
Sleep will seize on you.
When you come to die,
Every nerve and bone
Soon lulled in sleep,
Sleep will seize on you.
When you come to die,
Every nerve and bone
Soon lulled in sleep,
We'll still be free,
And you'll never escape from our woe!

...I would be all that I can do
And this to carry with me
Along with me, O brother,
And bid my lagging days relent
For every worthy deed done,
And glorious though the world be,
They never will repent me,
But in God's name endureth ever,
Whose blessed hope my soul abides
For refuge through the awful doors of death.

...We are old men, who pass
On the sands with gaze
Out of the narrow world of fashion;
We are old men, who stay
On a river's flow
And a common day
Where the life of youth is waiting,
And a longing grows
For the world of youth and beauty
Where the old man goes.

...When I am dead, my dearest,
Sing no sad songs for me;
Plant thou no roses at my head,
That by that token may grow cold.
My dirge shall be a muffled noise,
My trentals stiff with dread,
For he who once his faith hath won
Will never know it read.

...O beautiful, golden-bosomed ships!
O sunburned ships on the sea; O ship which breams
Above the waves and beams; O songs of love
Sent from the wide West, that shall sing us songs
In our hearts afar, as a summer star.

While train­ing that, I recalled that my other major beef with the PG cor­pus was its absence of more con­tem­po­rary poet­ry. I did­n’t really feel like try­ing to scrape the Poetry Foun­da­tion (PF) web­site myself, but I gave Google Dataset Search another try and to my sur­prise, dis­cov­ered a scrape had sur­faced on Kag­gle. Aside from being large, it comes with inter­est­ing meta­data: the title and author, but also a some­what eccen­tric set of ‘tags’ describ­ing the poem. They would be nice to add inline, allow­ing some degree of con­trol­la­bil­ity (like my use of author or book ID pre­fix­es, or CTRL’s use of sub­red­dit­s).

The Kag­gle Poetry Foun­da­tion scrape has numer­ous issues I had to fix before its for­mat was accept­able for GPT-2:

  • I replaced pre­fixed white­space, trim­ming leading/trailing white­space in all fields
  • replace 3+ spaces with new­lines
  • deleted all 2+ spaces
  • dropped poems with <100 char­ac­ters (gen­er­ally a scrape error)
  • remove Uni­code junk
  • seri­al­ize it as title+au­thor+­tags (if any) / poem / <|endoftext|> (ie the inline meta­data trick, allow­ing for poten­tially bet­ter learn­ing and a small degree of con­trol in con­di­tional gen­er­a­tion)
  • the final for­mat­ted cor­pus is avail­able for down­load

Once the cleaned PG was done, I then swapped out the PG for PF dataset and began fine­tun­ing. (I could train on the com­bined dataset, but the PF dataset is only 20MB and at 1⁄6 the size of PG, train­ing on that would take a long time to pick up on PF’s detail­s.) Sur­pris­ing­ly, the PF dataset trained down to ~0.60 loss after ~10k steps, as com­pared to PG’s ~1.73, a decrease much greater than I had expected from pro­vid­ing some meta­data, sug­gest­ing that mod­ern poet­ry, being so pros­e-­like, is much eas­ier for GPT-2 to mod­el—which does­n’t strike me as a com­pli­ment.

Some selected sam­ples from the PF-fine­tun­ing:

"The New York Movie", by William Collins

for Billie Holiday

Billie Holiday is dead. I am unhappy,
standing there like a gray stone,
standing there in my sweater,
thinking I can't kill you: I would like to be a big, red dog
with snow. I wish you a good night
with your face, your cheek and your legs, your legs.
I want to throw away this falling Sunday
with your face, my ugly body, my ugly body.
Nothing to give my body to,
nothing to give my heart to.

..."To a History Store", by Yehuda Amichai [Living, Death, Time & Brevity, Nature, Fall]

How hard it is for the living to die
holding only memory, and then to study the minutes
how like a trumpet of the future it sounds.

Gentlest of birds, patient as a body, cautious,
clear as the tongues of previous vervain birds.

Perhaps you are close to it, beautiful world.

..."Zagajas", by Robertizing [Religion, Christianity, Faith & Doubt, God & the Divine]

Our Lord who bore a child upon his knee.
We'll ask no more.
As sandpaper for what it wears.

Autumn, a new day.
Remember November, a new day.
Remember November, a new day.
Think of the silver
smiling into the barracud

Already at the edge of day.
Think of the hardson of earth
crumbling into something new

Into every tingling ring
to the full edge. Think of what

New, bright, revolutionary clouds

scattered out of the blue:
the clouds that dissolved them,
soundless, self-edesfilled.
Think of the cycles in and around of this

crescent myriads of ants starting
to collect beneath each other,

their seeds suddenly

burning each to the other, each

moving and flashing.

..."The Bean Eaters", by Rudyard Kipling [Relationships, Home Life, Pets, Nature, Animals, Landscapes & Pastorals, Winter]

The fairies were wonderful.
They trod the snow, chasing
the catkins to the north.

Frosty violin-skins were flying
and they began to sing,
leaving an echo of singing.
Then, as the she-torches rang,
a second spring
flowed up from the fur brush.
It was the strangest sight
all through the wintry night.

It was the woods, falling in long grass,
and I was thinking of you, Little Brother,

in the sweet marsh,
that I might recognize, Little Brother,

as I think of you.

..."In Golden Gate Park", by James Jenny Xie [Living, Coming of Age, Time & Brevity, Activities, Jobs & Working, Philosophy]

In Golden Gate Park's the day is breaking, only
the timeless moments of the night sketch the sky's
high promenade of flying goldenness now
and never a late, dissolving splinter of black glass.
But in Golden Gate Park's the morning breaks. The sidewalks
bask to me like cars at a funeral or the stars
like blind lights waiting on cars long since gone.
There, to the streaming windowpane, the little birds
scarve to get ready to swoop, and the sky's yellow
and gold. It is the end of hunger that slays the bird.

..."To Theodore", by Kenneth Slessor

Death may forgive, but love is better.

He that loves the rose
Whose pale cheek glows
With one hand swift and close,
Whose fingers move
The gold hair of the rose,
Gone to pass.
Where his lips draw breath
The bitter thong
Sigh as if Death had
No part with them,
He hears the song,
Hears the shout,
Saying me,
As I must.
Love is better, they say,
Than the loss they know;
Dreaming is worse, they say,
Love must hate so.
As his torch I carry the air;
He shakes my wings;
He speaks no word;
Saying me,
As I reach,
As he calls me,
Call him, O dear,
Call him, oh dear.
Love has been my constant care.

The con­tem­po­rary PF sam­ples prop­erly mimic all the meta­data and for­mat­ting, and are good, for what they are. (If you doubt this, read through a ran­dom selec­tion of PF poem­s.) The ones I liked seemed like they ben­e­fited greatly from the PG pre­train­ing. There are still a lot of flaws in the unclean PF data: run-on lines are par­tic­u­larly irri­tat­ing, and appear to be flaws in the Kag­gle scrape dataset rather than the orig­i­nal PF web­site. I have brought up the prob­lems on Kag­gle, but I doubt they’ll be fixed soon.

With PF done, I com­bined it with PG and trained on the com­bi­na­tion dataset for another ~20,000 steps, yield­ing a final loss of 1.61. The com­bined model appears able to do both datasets well (the weighted aver­age of a dataset with a loss of 0.6 and another dataset 6 times larger and a loss of 1.7 would be ~1.55, close to the mod­el’s ~1.6), and the sam­ples don’t appear to dif­fer much, so I don’t excerpt any. But the com­bined model should make a great start­ing point for RL pref­er­ence-learn­ing train­ing.

Ran­dom sam­ples & model down­loads:

Training GPT-2-poetry-prefix

The first ver­sion of the PG data for GPT-2-poetry just runs all the lines togeth­er, eras­ing the meta­data about what book each line comes from. A good model should nev­er­the­less grad­u­ally learn about the tran­si­tions between poems & whole books, but that is hard and there may not be enough tran­si­tions in the data to learn effec­tive­ly.

Much like the char-RNN exper­i­ments on this page, there is no rea­son one can’t inject that meta­data in a struc­tured way to see if the model can learn to exploit the meta­data; even if it can­not, the added meta­data should­n’t hurt that much because it is so reg­u­lar & repet­i­tive. Insert­ing the meta­data also allows for some degree of con­trol in con­di­tional gen­er­a­tion; one should be able to put in the book ID for, say, Home­r’s Iliad as a prompt and get out a long block of con­sis­tent Home­ric pas­tiche.10

Ide­al­ly, there would be unique IDs for every author, poem, and book and these would appear at the begin­ning of every poem and the end of the poem would be delim­ited with the <|endoftext|> sym­bol that OA’s GPT-2 mod­els were trained with, but unfor­tu­nately only the book ID is avail­able in this par­tic­u­lar dataset. (Pro­ject Guten­berg ebooks do not include any meta­data or for­mat­ting which would cleanly split each dis­crete poem from each oth­er.) Like before with authors, the book ID meta­data can be for­mat­ted as a pre­fix on every line with a delim­iter like the pipe char­ac­ter.

Rather than start over with GPT-2-117M again, GPT-2-poetry can just be fur­ther fine­tuned on this new pre­fixed ver­sion of the PG cor­pus to pro­duce what I call “GPT-2-poetry-prefix”:

cat gutenberg-poetry-v001.ndjson | jq .gid | tr -d '"'                                 > id.txt # "
cat gutenberg-poetry-v001.ndjson | jq .s   | sed -e 's/^.//' -e 's/.$//' -e 's/\\//g' >> poetry.txt
paste --delimiters='|' id.txt poetry.txt > gutenberg.txt

shuf gutenberg.txt | head
# 14869|Beware of the brand of the fiery Frank!
# 1727|and they have great power among the Argives. I am flying to
# 38550|Shows heaven in page of living book;
# 22421|First, for effusions due unto the dead, I. 26.
# 26275|blossomed beneath their temples, and covered their chins with
# 1745|What happiness, who can enjoy alone,
# 1645|When first he won the fairy clime.
# 4332|And out of these molten flowers,
# 36916|What!  Never more go gladly back?
# 2507|Raged for hours the heady fight,

PYTHONPATH=src ./ gutenberg-poetry-v001-delimited.txt gutenberg-poetry-v001-delimited.txt.npz

The loss of GPT-2-poetry-prefix will be much lower than GPT-2-poetry because the pre­fix is so pre­dictable, but it will hope­fully learn inter­est­ing things beyond that.

In other sam­ples, the gen­er­ated IDs switch in the first two lines, and while that’s not much to judge from, GPT-2-poetry-prefix seems to ignore key­words from the first line when the IDs change, and does­n’t repeat them in the rest of the sam­ple or attempt to rhyme off them, which is fur­ther evi­dence it is suc­cess­fully asso­ci­at­ing & learn­ing to mod­e-switch.

Like GPT-2-poetry, GPT-2-poetry-prefix con­verged quickly to a final loss of ~1.6 after 224,474 steps tak­ing 31 GPU-hours, not improv­ing much after the first ~8 GPU-hours despite decreas­ing the learn­ing rate. (Di­min­ish­ing returns appear to set in quickly for fine­tun­ing GPT-2-117M even if one has a rel­a­tively large new cor­pus.)

GPT-2-poetry-prefix Samples

Training samples

# ...[33999 | 4308.76] loss=2.45 avg=1.86
# ======== SAMPLE 1 ========
# 11|But I have had a day that was gone when I saw you in a dream,
# 1322|A year ago, a year ago.... I'm going back
# 1322|Oh, the years are many and the years begin!
# 1322|You and I have traveled far and wide,
# 1322|You and I have wandered far and wide
# 1322|Through the land I love so well,
# 1322|Where the beautiful land of long ago
# 1322|Died and ebbed and glowed,
# 1322|In the little green land of long ago,
# 1322|In the little green land of long ago,
# 1322|In the little green land of long ago.
# 1322|You and I have traveled far and wide
# 1322|Through the land of long ago;
# 1322|You and we have traveled far and wide
# 1322|O'er the land we love so well,
# 1322|In the little green land of long ago,
# 1322|In the little green land of long ago.
# 1322|You and I have traveled far and wide
# 1322|Through the land of long ago;
# 1322|And the years have slowly swept us on
# 1322|Till the dust has left the sod,
# 1322|And we all must rue the little day
# 1322|When we all are gone from God....
# 1322|And we all must rue the little day
# 1322|When we all are gone from God....
# 1322|I know a little garden very old,
# 1322|Built in a nook near the western sea,
# 1322|With doors and windows closely locked and barred
# 1322|And steps that made no sound but the world outside,
# 1322|And a wall of white-walled flowers all in a row
# 1322|Of violet bloom that just began to blow,
# 1322|And a dial that looked like a golden globe,
# 1322|Cool, square, and green, with the leaves and the grass,
# 1322|And a dial that looked like a needle of bronze;
# 1322|And I wonder if these petals are lost to-day,
# 1322|These petals will be forgotten to-morrow.
# 1322|"A rose would be a rose,"
# 1322|I wonder much at it;
# 1322|It does not matter what or who,
# 1322|So very, very much it matters:
# 1322|It is so nice to think
# 1322|The world is such a place for petals,
# 1322|And so much like a pink.
# 1322|"A purple fox-hawk lived down by the sea
# 1322|And danced and slept in the white-walled flowers,
# 1322|And I thought about a tree,
# 1322|A rose that dropped as an apple falls
# 1322|Into the water, and fell in showers."
# 1322|And I wondered what people said,
# 1322|And wherefore took the trouble,
# 1322|And wherefore came my little red bird
# 1322|To look at me and wave its wings
# 1322|For joy of self and for its red,
# 1322|And why I cannot tell;
# 1322|For it is so nice to think
# 1322|The world is such a place for petals.
# 1322|And so I sat and played,
# 1322|And the old clock set the chimney-top
# 1322|To ring time to itself and stop
# 1322|The time with a slow, frost-ridden drone,
# 1322|And a rustle as of bees.
# 1322|The house was old, and had grown lean
# 1322|As if it were but a leaf,
# 1322|But I found there was only one room
# 1322|And I slept on my stem at the end of the porch
# 1322|Like a child asleep.
# 1322|The door was old, but very young,
# 1322|And very, very young;
# 1322|And a weary smile on its satin lips
# 1322|Hung like a crown upon the night,
# 1322|And the roses crept into the room
# 1322|By the little red door in the wall,
# 1322|And I woke, and felt as a king is,
# 1322|And thought of all the trouble and pain
# 1322|Of a little life and a little love
# 1322|When only a boy is born.
# 1322|For my heart was old, and restless with the years,
# 1322|And my face with no trace of the things I had
# 1322|And no remembrance when I was twenty-five,
# 1322|And
# [34000 | 4318.77] loss=2.15 avg=1.86
# ...
# [47998 | 11504.75] loss=1.91 avg=1.84
# [47999 | 11505.25] loss=2.37 avg=1.85
# ======== SAMPLE 1 ========
# |So when the sun has touched the hill
# 30599|And made the earth a little heave
# 30599|Of gold-dust on the firmament.
# 30599|The sea is a molten pearl
# 30599|Silvering the yellow moon.
# 30599|Gold is the poppies of purple plush,
# 30599|Which make a star within a pool
# 30599|Of purple, blue, and green and blue.
# 30599|The sea is a molten pearl,
# 30599|Whose petals burn as light
# 30599|In the rose-yarn of the sea.
# 30599|The gold star beckons, sparkles, shines.
# 30599|The water gleams and glows.
# 30599|The poppies turn to a golden rim,
# 30599|And a thousand petals, one by one,
# 30599|Come dancing through the purple waves
# 30599|Of purple.
# 30599|My soul goes up to the sun.
# 30599|The sun, all gold, is gone.
# 30599|The sun is gone, the sun
# 30599|Is not more gold than my soul.
# 30599|What is the rainbow, my soul?
# 30599|The rain is falling in the tree
# 30599|In dewdrops falling, that are wet
# 30599|With dewdrops falling.
# 30599|I have heard music in the woods
# 30599|Under a great sky.
# 30599|Their notes, on a hundred harps,
# 30599|Dance by a liquid, falling star.
# 30599|The song of the whole universe
# 30599|Rings like a loon's tune.
# 30599|The rain is falling, my soul,
# 30599|In the tree-tops, the rain is falling,
# 30599|And the rain is not more musical.
# 30599|The trees, like great globed fruit in a garden of heaven,
# 30599|Are as full of little shining blossoms
# 30599|As the face of a child of ten minutes.
# 30599|The wind is playing a soft tune
# 30599|Like the silver notes of a bell.
# 30599|The grass is a dance for a child,
# 30599|And the sun is going down.
# 30599|The rain is playing a soft tune
# 30599|Like the golden notes of a bell.
# 30599|The rain is playing asleep in the meadows
# 30599|Like a sea of dreams,
# 30599|And the wind is playing a soft tune.
# 30599|The leaves are wet with the tears:
# 30599|There is only the wind that comes.
# 30599|The leaves are wet with the tears.
# 30599|The trees have bent to the wind,
# 30599|Like heavy blossoms, and nothing stays.
# 30599|The earth is flooded with tears
# 30599|Like small white stars in the sky.
# 30599|The wind's song is marching by
# 30599|Like a song in a wind from a string.
# 30599|It comes from nothing, it comes not from the years;
# 30599|An atom of dust and a tide,
# 30599|The ceaseless rain falls heavily over the world,
# 30599|The summer flowers are red
# 30599|And one like a gold cup, in a crimson cup,
# 30599|Flows out from nothing, and goes
# 30599|Without speech or motion nor sound.
# 30599|The yellow flowers all slip,
# 30599|All their leaves are wet,
# 30599|And their crimson petals are wet.
# 30599|The rain has driven the sun outside,
# 30599|The wind has driven the rain outside,
# 30599|The moon is going out and in,
# 30599|With the stars on the roof like snow.
# 30599|The rain has driven the rain outside,
# 30599|The moon is driving the clouds in the air
# 30599|Like white, wool flakes in a snow.
# 30599|The trees are wet with the tears,
# 30599|They weep in their night-black tatters,
# 30599|They moan like mortals who lost their way
# 30599|Because they were glad of the rain.
# 30599|The rain is falling, my soul,
# 30599|It has torn away the flowers,
# 30599|They are wet with the tears of the sun.
# 30599|In the black ooze and the snow,
# 30599|The cold winds, shuddering, are blowing,
# 30599|The leaves are blown like blown corn,
# 30599|There is only the wind that comes.
# 30599|Black and long!  The night is
# [48000 | 11515.52] loss=2.14 avg=1.85
# ...
# [166998 | 5752.38] loss=2.17 avg=1.52
# [166999 | 5752.88] loss=2.10 avg=1.53
# ======== SAMPLE 1 ========
# 26|His golden radiance is no more."
# 16452|"I shall not think of men in Argos more
# 16452|Than they are now, who many a bloody deed
# 16452|Wrought on the Greeks, nor yet of Ilium's king
# 16452|In arts and arts like these can speak the rest.
# 16452|But they--their kings--the Trojans and their sons
# 16452|Have fallen. Their deaths, the Grecians and their friends
# 16452|Have fallen in battle, from whom little hope
# 16452|To escape the battle, but the steadfast hearts
# 16452|Of heroes and of Trojans have become
# 16452|Inglorious still. The immemorial strife
# 16452|Shall rise for ever in a glorious day,
# 16452|When wars are waged between us and the Greeks.
# 16452|The battle shall be theirs, the mirth, the song;
# 16452|The mirth which all the Olympian people share,
# 16452|Shall bless the younger warriors with a joy
# 16452|So great, so glorious, and a greater fame,
# 16452|That all the Greeks shall learn, that in the van
# 16452|Ye stand yourselves, and they will praise your deeds.
# 16452|But I beseech you, if indeed by mine
# 16452|Unknown dishonour you be wrested hence,
# 16452|That with your lusts, illustrious and august,
# 16452|All others ye may vanquish. Now, my friend,
# 16452|Behold this prize to crown your father's pride.
# 16452|He said, and shaking both his palms, assent
# 16452|That I should also wish it. Thou art brave;
# 16452|Thou know'st how Menoetiades the swift
# 16452|Was dragged, of Hector and the fierce compeers
# 16452|And Phrygian warriors. So, we will dispatch
# 16452|Your bodies, then, yourselves to burn the ships
# 16452|In sacrifice; with torches and with bells
# 16452|To burn them, and with oxen to replace
# 16452|Your gallant friends for ever. But I wish
# 16452|That no man living has so long endured
# 16452|The onset of his foes, as I have power
# 16452|To burn or storm; for mighty Hector erst
# 16452|Was slain, and now returns his safe return
# 16452|To the Thesprotians, and each other's wives
# 16452|And tender children, and all other babes
# 16452|Assemble round me now, for ye have more
# 16452|To suffer than they know. Go then--the rest
# 16452|Will bear you safely; if ye dare to use
# 16452|The Grecians with your ships, leave not the flames
# 16452|Or fire, or slay yourselves and leave the rest,
# 16452|Ye may, at least, escape, but Hector's hands
# 16452|Will set you free; the vultures on the hills
# 16452|Held forth, and I will pierce those hollow shades
# 16452|Adown the trench, and ye shall find them there
# 16452|Scaped unaware. If, then, the Gods above
# 16452|Should bring thee to the fane of Peleus' son,
# 16452|In his despite, thou wouldst be left alone,
# 16452|Beside the yoke unwearied, there to stand,
# 16452|And from the Trojans scatter all thy dead.
# 16452|So spake the son of Atreus, and retired.
# 16452|Then, leaving Hector's body in the tent
# 16452|Of Periphas, King Agamemnon seized,
# 16452|And in his royal hand his spear adorned
# 16452|With silver, and with gold, studded thick all o'er
# 16452|With glittering tin, the weapon of Mars thus toil'd.
# 16452|Brave as he is, and of unwearied arm,
# 16452|So shall be thy revenge, if thou shalt make
# 16452|Fierce contest for the Danaï. But fly
# 16452|With shout and clamour fill'd throughout the host.
# 16452|So shall ye perish, slain by Hector's hands,
# 16452|Each living soul, and, flying into fire,
# 16452|Sink under Troy's wide gulfs. Be this the work
# 16452|Of your brave sires, and all the parents' hearts
# 16452|Shall prosper, one and all. But ye in haste
# 16452|(For ye are valiant) shall
# [167000 | 5762.79] loss=1.32 avg=1.53
# [167001 | 5763.29] loss=1.60 avg=1.53

One train­ing sam­ple is worth remark­ing on:

# [49999 | 12526.44] loss=1.83 avg=1.77
# ======== SAMPLE 1 ========
# ._
# 19221|The curfew tolls the knell of parting day,
# 19221|The lowing herd winds slowly o'er the lea,
# 19221|The ploughman homeward plods his weary way,
# 19221|And leaves the world to darkness and to me.
# 19221|Now fades the glimmering landscape on the sight,
# 19221|And all the air a solemn stillness holds,
# 19221|Save where the beetle wheels his droning flight,
# 19221|And drowsy tinklings lull the distant folds.
# 19221|Save that from yonder ivy-mantled tower
# 19221|The moping owl does to the moon complain
# 19221|Of such as, wandering near her secret bower,
# 19221|Molest her ancient solitary reign.
# 19221|Methinks I see her in her blissful dreams:
# 19221|--Or, fancy-like, in some mirage she lies,
# 19221|Majestic yet majestic, and of seems
# 19221|The image of the unconquerable skies.
# 19221|Methinks I see her in her blissful dreams:
# 19221|--Or, fancy-like, in some majestic cell,
# 19221|Where lordly seraphs strew their balmy dreams
# 19221|On the still night, or in their golden shell.
# 19221|There, in the calm of some Platonic dream,
# 19221|Sits she, and views the unclouded moon arise
# 19221|Like a fair lady full of realms divine;
# 19221|And, all at once, a stony face and bright
# 19221|Glittering in moonlight, like the noon-tints of a night.
# 19221|Methinks I see her in her blissful dreams:
# 19221|--Or, fancy-like, among the moonlight pale,
# 19221|A heavenly lark twitters with every gleam
# 19221|Of the rapt rapt beauty that she craves to hail.
# 19221|Or in the moonlight gleams she seems to see
# 19221|A virgin glory, and a power divine,
# 19221|And every motion of the looking-glass
# 19221|Is full of love and divine gentleness.
# 19221|Methinks I hear her in the twilight dim:
# 19221|--Or is it fancy-free, and musical,
# 19221|And full of music? Or some holy hymn
# 19221|Over some water-lily's long-lost lute?
# 19221|O no! it is not fancy-free, I see:
# 19221|Nay! fain would I thy spirit-hearts invest,
# 19221|And hear the paean of these dreamy rhymes,
# 19221|Might I but speak of my departed rhymes,
# 19221|And thou wouldst sing of my empoet-swans.
# 19221|O no! it is not fancy-free, I see:
# 19221|Even Poetry murmurs in a pensive dream,
# 19221|And like a breeze sings the blue mists of night.
# 19221|O no! it is not fancy-free, I see:
# 19221|Even Poetry murmurs in a pensive dream.
# 19221|How soft the zephyrs, how the languid hours,
# 19221|As in the noonday shade their limbs recline!
# 19221|How mellow streams the mossy banks assume!
# 19221|How solemn woods, low-chiselled, sumptuous green,
# 19221|Nod to the cadence of their hoarse refrain,
# 19221|As if the deep-toned muses' melodies
# 19221|With half-averted plaints and half-averted plumes
# 19221|In some majestic temple's quietness
# 19221|Had to the silver twilight slowly come.
# 19221|How solemn woods, low-couched, around thee lie:
# 19221|--Or is it fancy-free, and melody
# 19221|That makes the dull night long in worship held?
# 19221|Methinks I hear the harp's harmonious sound
# 19221|In some dim wood, when the deep shadows fall;
# 19221|And the low wind, like one that listens, makes
# 19221|In the still woods the harmony of all.
# 19221|Or in the moon's pale beam, on some hoar rock,
# 19221|Lonely and spectral, mourns her feeble woe;
# 19221|And as the slow waves roll, and, ebbing, break
# 19221|In music
# [50000 | 12536.67] loss=1.36 avg=1.76

The rhyming in this sam­ple is so good as to be sus­pi­cious. It might also sound famil­iar—be­cause many of these lines are being copied from , which opens:

The cur­few tolls the knell of part­ing day,
The low­ing herd wind slowly o’er the lea
The plough­man home­ward plods his weary way,
And leaves the world to dark­ness and to me.
Now fades the glim­m’ring land­scape on the sight,
And all the air a solemn still­ness holds,
Save where the bee­tle wheels his dron­ing flight,
And drowsy tin­klings lull the dis­tant folds;
Save that from yon­der ivy-­man­tled tow’r
The mop­ing owl does to the moon com­plain
Of such, as wand’ring near her secret bow’r,
Molest her ancient soli­tary reign.

Some spelling dif­fer­ences aside, this intro is almost entirely copied from the 8 copies of Gray’s poem in the cor­pus; this exten­sive copy­ing is not some­thing I spot­ted in the GPT-2-poetry sam­ples I looked at, sug­gest­ing that the scaf­fold­ing of the meta­data did indeed help with learn­ing.

Also inter­est­ing­ly, the copy­ing only goes so far, as imme­di­ately after the final line about the owl, where Gray con­tin­ues:

…Be­neath those rugged elms, that yew-tree’s shade,
Where heaves the turf in many a mould’ring heap,
Each in his nar­row cell for ever laid,
The rude Fore­fa­thers of the ham­let sleep …

GPT-2-poetry-prefix instead con­tin­ues:

…Me­thinks I see her in her bliss­ful dreams:
—Or, fan­cy-­like, in some mirage she lies,
Majes­tic yet majes­tic, and of seems
The image of the uncon­quer­able skies.
Methinks I see her in her bliss­ful dreams:
—Or, fan­cy-­like, in some majes­tic cell,
Where lordly ser­aphs strew their balmy dreams
On the still night, or in their golden shell.
There, in the calm of some Pla­tonic dream,
Sits she, and views the unclouded moon arise
Like a fair lady full of realms divine;
And, all at once, a stony face and bright
Glit­ter­ing in moon­light, like the noon-t­ints of a night. …

That is, it focuses on the female fig­ure of the Moon in a way more ode-­like than ele­giac. These lines also do not seem to be extracted from the rest of Elegy either, as words like “bliss” or “mirage” or “dream” or “ser­aph” or “Pla­tonic” do not appear in it. Some of the phrases like “bliss­ful dreams” do appear in the rest of the cor­pus, but oth­ers like “some mirage” or “mirage she” do not. Nev­er­the­less, the style is con­sis­tent through­out the entire sam­ple and the qual­ity is good, sug­gest­ing that while GPT-2-poetry-prefix has man­aged to mem­o­rize to a lim­ited extent, it is nev­er­the­less fully capa­ble of gen­er­at­ing good orig­i­nal text.

An addi­tional exam­ple of mem­o­riza­tion has been spot­ted; sam­ple #17 in the 1000 uncon­di­tional sam­ples is almost entirely a mem­o­rized copy of Percy Bysshe Shel­ley’s “To a Sky­lark”:

======================================== SAMPLE 17 ========================================
 of our mortal life, and made it known
32373|To be man's also,--as to make us one;
32373|A thing apart, yet of an individual
32373|In frailty and high elemental own.
32373|Hail to thee, blithe Spirit!
32373|Bird thou never wert,
32373|That from heaven, or near it
32373|Pourest thy full heart
32373|In profuse strains of unpremeditated art.
32373|Higher still and higher
32373|From the earth thou springest
32373|Like a cloud of fire;
32373|The blue deep thou wingest,
32373|And singing still dost soar, and soaring ever singest.
32373|In the golden lightning
32373|Of the sunken sun
32373|O'er which clouds are brightening,
32373|Thou dost float and run,
32373|Like an unbodied joy whose race is just begun.
32373|The pale purple even
32373|Melts around thy flight;
32373|Like a star of heaven,
32373|In the broad daylight
32373|Thou art unseen, but yet I hear the music of thy singing.
32373|Keen as are the arrows
32373|Of that silver sphere,
32373|Whose intense lamp narrows
32373|In the white dawn clear,
32373|Until we hardly see, we feel that it is there.
32373|All the earth and air
32373|With thy voice is loud,
32373|As, when night is bare,
32373|From one lonely cloud
32373|The moon rains out her beams, and heaven is overflowed.
32373|What thou art we know not;
32373|What is most like thee?
32373|From rainbow clouds there flow not
32373|Drops so bright to see,
32373|As from thy presence showers a rain of melody.
32373|Like a poet hidden
32373|In the light of thought,
32373|Singing hymns unbidden,
32373|Till the world is wrought
32373|To sympathy with hopes and fears it heeded not;
32373|Like a high-born maiden
32373|In a palace tower,
32373|Soothing her love-laden
32373|Soul in secret hour
32373|With music sweet as love, which overflows her bower;
32373|Like a glow-worm golden,
32373|In the silver dew,
32373|Scattering unbeholden
32373|Its aerial hue
32373|Among the flowers and grass, which screen it from the view;
32373|Like a rose embowered
32373|In its own green leaves,
32373|By warm winds deflowered,
32373|Till the scent it gives
32373|Makes faint with too much sweet these heavy-winged thieves.
32373|Sound of vernal showers
32373|On the twinkling grass,
32373|Rain-awakened flowers,
32373|All that ever was
32373|Joyous and fresh and clear thy music doth surpass.
32373|Teach us, sprite or bird,
32373|What sweet thoughts are thine:
32373|I have never heard
32373|Praise of love or wine
32373|That panted forth a flood of rapture so divine.
32373|Chorus hymeneal,
32373|Or triumphant chant,
32373|Match'd with thine, would be all
32373|But an empty vaunt--
32373|A thing wherein we feel there is some hidden want.
32373|What objects are the fountains
32373|Of thy happy strain?
32373|What fields, or waves, or mountains?
32373|What shapes of sky or plain?
32373|What love of thine own kind? What ignorance of pain?
32373|With thy clear keen joyance
32373|Languor cannot be:
32373|Shadow of annoyance
32373|Never came near thee:
32373|Thou lovest; but ne'er knew love's sad satiety.
32373|Waking or asleep,
32373|Thou of death must deem
32373|Things more true and deep
32373|Than we mortals dream,
32373|Or how could thy notes flow in such a crystal stream?
32373|We look before and after,
32373|And pine for what

The 87 lines begin­ning with “Hail to thee, blithe Spir­it!” are all Shel­ley (with per­haps slight spelling dif­fer­ences), much sur­pass­ing the mem­o­riza­tion for Thomas Gray. Con­sid­er­ing the top-k sam­pling method, it’s amaz­ing that the sam­ple could so exactly fol­low “To A Sky­lark”. It turns out there are ~12 copies of the poem in the PG cor­pus (it’s a pop­u­lar poem), so in ret­ro­spect some degree of mem­o­riza­tion is not sur­pris­ing, but that’s still a lot of mem­o­riza­tion. The 4 lines before­hand don’t appear to be copied from another Shel­ley poem, mak­ing it even more amaz­ing. It’s a pity that that sam­ple did not con­tinue fur­ther because one won­ders whether it could have repeated the entire poem and what it would’ve done when the orig­i­nal poem end­ed.

Unconditional samples

“How the clouds
Seem to me birds, birds in God’s gar­den! I dare not!
The clouds are as a breath, the leaves are flakes of fire,
That clash i’ the wind and lift them­selves from high­er!”


For both GPT-2s, I gen­er­ated 1000 sam­ples as fol­lows:

python src/ --top_k 40 --temperature 0.9 --nsamples 1000 --seed 0 \
 --model_name 2019-03-06-gwern-gpt2-poetry-prefix-projectgutenberg-network-224474
# ======================================== SAMPLE 1 ========================================
# |But I shall tell thee of the glorious days
# 1008|Of that old strife, wherein the truth of it
# 1008|Atoned, though at the most points of the pulp
# 1008|Be passed through: and about the torment, which
# 1008|A Hermit in his youth thoughte, yet made
# 1008|Still worse by his words, where he said, 'Peace!
# 1008|Keep silence here; because in any place
# 1008|I heard of charity the while I dwelt,
# 1008|And of her tears made e'en my lot a sting:
# 1008|And, but for those, in truth, to heaven were found
# 1008|The spirit of him offered by the sea,
# 1008|So that, out struggling, he made no denial.
# 1008|But the proud spirit, soon as he was silent,
# 1008|Embraced me, even in bidding me speak therefore,
# 1008|And with such gentle voice and such great front
# 1008|Came up unto my Saviour, that he made
# 1008|Full many a sign, how that all virtue he
# 1008|Would at his own discretion hear and mark;
# 1008|Then saw he the chief flower of love delight
# 1008|Within the living light, and made such pact,
# 1008|That it would make bliss vouchsafe to him.
# 1008|Paradiso: Canto XXIX
# 1008|The day, that most of us had sojourning
# 1008|'Mong the sweet spirits, from the highest shaft
# 1008|Of their kind master had not looked on them,
# 1008|Into the eyes forth from the bark they came;
# 1008|Two palms it bore of one and of the letters,
# 1008|And the other two with wings outspread.
# 1008|Not in beatitude do I behold them,
# 1008|But in and out of hope they uttered a
# 1008|Heaven-toned word of comfort, so that reached
# 1008|The upper light, which in attention hung
# 1008|With the low shores, that to the nether shores
# 1008|Were as the utmost thread 'to the high point.
# 1008|And that unceasing after-effiguring
# 1008|Of the eternal Sire, upon the which
# 1008|One live soul only drew its breath mysteriously,
# 1008|Up to the point, so pointed to the source
# 1008|Of what it had been, I never it knew,
# 1008|And hence never would have recognized.
# 1008|What verdure of an undivided heart
# 1008|Resolved, I say; and he complained e'en thus:
# 1008|"What from thy soul unto the Good I send
# 1008|I ever pray to, and by grace of that
# 1008|Pointing myself I pray thee to the world,
# 1008|To point thee the great mystery of love,
# 1008|From this, the bottom to the source of all
# 1008|Concerning thee; and not by its green leaves
# 1008|Of science so unsullied was the thought,
# 1008|As a small-handled cup, acquired by men.
# 1008|The mind's eye, taking from the mortal world
# 1008|All that it asks of bar or of the gold,
# 1008|With the same fury burns as it was wont;
# 1008|Now it may be by lantern or by shining,
# 1008|Since both thy and my love has made me its."
# 1008|The Almighty Father in his thunder made
# 1008|Resenting, and all round about Him round
# 1008|Went down his smitten steps, so that the air
# 1008|Impregnate came not from his visitations,
# 1008|Setting a day of darkness on all sides.
# 1008|Therefore mine eyes I lifted to the ground,
# 1008|And I beheld a river by the ice
# 1008|Chained up and flowing back along the ice,
# 1008|And suddenly before my feet it melted;
# 1008|And what it now behoves me to retrace
# 1008|The cause I had of it in heart I felt.
# 1008|As the Sicilian bull, that rightfully
# 1008|His cries first echoed in the mountains,
# 1008|Did so rebellow, with the sound of which
# 1008|It made my very blood to quicken well,
# 1008|The dolorous accents which envenom'd me,
# 1008|Forthwith I hasten'd unto where reply
# 1008|Was made: "O Ro! Brunhild"
# ======================================== SAMPLE 2 ========================================
# |Hear the tale that the funeral chant is telling,
# 2491|For the sorrows of other's children that dwell
# 2491|Like sweet flowers upon the wold?
# 2491|'Tis the tale of a life which is fled and gone,
# 2491|And the star of a hope which shone
# 2491|Bright above it, though dark may it be,
# 2491|For the hopes of a brighter day are fled
# 2491|And the joys of a happier lot?
# 2491|'Tis the tale of a life with the weary and sad,
# 2491|Where sorrows begin and rest.
# 2491|For only a song can the widow's soul glad
# 2491|Who sits musing 'mid shadows drear.
# 2491|And only a music, sad with its sighs,
# 2491|Till sad to the soul as death draws near
# 2491|As life on her fragile bark!
# 2491|I hear their voices faint in my slumbrous sleep,
# 2491|The music of lives that seem less real
# 2491|Than phantoms are dream-bound in duty's mystic keep,
# 2491|With music that seems to be more real
# 2491|Than phantoms are dream-bound in duty's mystic keep
# 2491|For souls that sin may not see!
# 2491|All round about us seems, in every place,
# 2491|As far off as the eyes of kith and kin,
# 2491|The ever-tremulous busy world's harmonious race,
# 2491|And I hear the mighty ocean tides,
# 2491|Feeling their strength, their might, their rhythmic din,
# 2491|Are calling me all into one wide choral face,
# 2491|And I hear the infinite singing of the winds,
# 2491|That seem to make me simply live!...
# 2491|The world seems a world that is full of sound and motion;
# 2491|A world of beauty and of music, where it lies;
# 2491|Yet all that is and has for me seems one more treasure
# 2491|Than all the world dreams leave in the skies.
# 2491|I hear the mighty tides of life,
# 2491|They're crying to me,
# 2491|They rise and sink in a restless strife
# 2491|Of endless song.
# 2491|Yet every stroke of sorrow's sword
# 2491|Comes surely from afar,
# 2491|That is true peace which is hard on board
# 2491|Though oceans be dark and terrors war.
# 2491|I hear the myriad singing words
# 2491|Of ocean's depths,
# 2491|They come like a song of broken birds,
# 2491|The music floats on the air and stirs
# 2491|My life to bear its measure in calms
# 2491|Of perfect peace, and it is good,
# 2491|But all is false peace only.
# 2491|When first I heard the autumn rain
# 2491|Sink down the hollows on the plain,
# 2491|I held it very near,
# 2491|And as I spoke to March again
# 2491|I felt the long, slow throbbing rain
# 2491|Creep from the earth in sudden flight
# 2491|Through all the veins of earth again,
# 2491|And in the sunlit, silent night
# 2491|The world grew far forlorn.
# 2491|And April came with rushing rains,
# 2491|And leaves about the naked lanes.
# 2491|I saw again the August noon
# 2491|Roll round the world in blazing heaps.
# 2491|And in the sunlight and the dark
# 2491|A thousand germs their pageant crush.
# 2491|And from the earth the maples bloom
# 2491|In odors of the breath of bloom
# 2491|And from the meadows and the hills
# 2491|The rosy clouds drop down their spilled spilled spilled spilled
# 2491|And drunken with the rain it kills.
# 2491|And soon above the hills shall crash
# 2491|The thunder of rain-wings,
# 2491|And all the naked trees and shrubs
# 2491|Shall lie, like naked, naked blades.
# 2491|Out on the hills there shall be rain,
# 2491|And the maples down the windy lane
# 2491|Shall bleed, and flowers shall weep again
# 2491|Through the weary hours of rain.
# 2491|They shall lie where the maples lie
# 2491|Deep in their bosoms, cold and numb,
# 2491|Each with its wound on either arm,
# ...

Down­load links again:

Some fun pas­sages I noticed in the first 100 uncon­di­tional sam­ples:

======================================== SAMPLE 2 ========================================
|Hear the tale that the funeral chant is telling,
2491|For the sorrows of other's children that dwell
2491|Like sweet flowers upon the wold?
2491|'Tis the tale of a life which is fled and gone,
2491|And the star of a hope which shone
2491|Bright above it, though dark may it be,
2491|For the hopes of a brighter day are fled
2491|And the joys of a happier lot?
2491|'Tis the tale of a life with the weary and sad,
2491|Where sorrows begin and rest.
2491|For only a song can the widow's soul glad
2491|Who sits musing 'mid shadows drear.
2491|And only a music, sad with its sighs,
2491|Till sad to the soul as death draws near
2491|As life on her fragile bark!
## Sample 3:
37804|The white-petalled white fox
37804|Opens himself to coolness
37804|In the late evening.
37804|But when the last child started
37804|The white fox to his feet flew,
37804|And the old fox was master
37804|Of all the magic heathen.
37804|Till when the faint huntsman
37804|Had snuffed the fragrant water
37804|Over his plump ears and skin,
37804|In the old way he knew not
37804|Till morn had almost shone;
37804|And then the fox came slowly
37804|And left the place unguessed;
37804|The white fox was not master,
37804|Although he had been master,
37804|Although he had been servant
37804|And now he could be master
37804|Of all the magic powers
37804|That keep the place enchanted
37804|In the wide earth and water.
## Sample 9:
36661|And the morn breaks, and, all the day,
36661|Red-clover'd birds with silver bill
36661|Flutter from tree to tree in flower,
36661|A quivering dew, a wind that wafts
36661|To haunts among the ancient woods.
36661|The golden-crested ilex, here
36661|Doth vine her purple cup; the deer,
36661|The wild-goose; and, in troops, the sheep,
36661|The goat, the sylvan-haunted elm,
36661|And the green-faced oft-gadding pine
36661|Blossom with purple.
36661|The lark soars up,
36661|And the hare loud answer make!
36661|Doves, willows, dunes, aslant the lake;
36661|Pair after pike sounds warbling;
36661|The reeds a triumph!
## Sample 14:
37452|I had a vision
37452|Of an old and stubborn old man,
37452|His hair was pale, and thin,
37452|His face was all forlorn,
37452|And the moon was full in the air,
37452|And a spirit passed over his brow,
37452|And its face was all for ever.
37452|And he spoke:
37452|'Have we ever a dream?
37452|Have we ever a vision
37452|Of the ghost's ghost?'
37452|The Master gave the word:
37452|'By the breath I know
37452|The meaning of Death:
37452|Can it be 'hush?
37452|Have we ever a dream?'
37452|The spirit said:
37452|'By the breath I know,
37452|The meaning of Death,
37452|You will see a ghost
37452|Stand by the door
37452|And enter.'
37452|And the spirit said:
37452|'By the breath I know,
37452|The meaning of Death
37452|You may understand:
37452|Can it be 'hush?
37452|Have we ever a dream?'
37452|The Spirit said:
37452|'By the breath I know,
37452|The meaning of Death
37452|You can see a ghost
37452|Stretched toward the door,
37452|And see a spectre
37452|Pass by the chamber door.
## Sample 24:
1333|Then, sweet heart, whisper, sweetheart,
1333|"Thou art sweet, but thy love is vain."
1333|I do love thee, my love,
1333|In a word, in a song,
1333|With the heart and the will,
1333|And the power of my heart;
1333|The power of my whole
1333|Of the poet's soul,
1333|And the heart and the soul!
1333|As the winds take the leaves
1333|As the flowers take the flowers,
1333|As the floods take the dew,
1333|As the salt runs in floods,
1333|As the salt runs in floods,
1333|As the snow in the seas,
1333|As the rain in the logs,
1333|As the wind comes and goes,
1333|As the sleet in the coppice,
1333|As the snow in the coppice,
1333|As the snow in the bogland,
1333|As the hail in the river,
1333|As the snow in the river,
1333|As the snow in the county,
1333|As the snow in the county,
1333|As the snow in the county,
1333|As the rain in the vale.
1333|As the stars take the dew,
1333|As the sparks fly from eye,
1333|As the sparks fly,
1333|So the hand of my heart
1333|As the heart of my art
1333|As the tongue of my lips,
1333|As the heart of my heart
1333|As the flame in the eye.
======================================== SAMPLE 39 ========================================
|And as the summer twilight,
34237|When the golden vinewood
34237|Strikes the silent midnight,
34237|Stands mute beside the brook,
34237|With a ghostly sense of the human heart
34237|Forgotten, yearning, sighing.
34237|I do remember how, long years ago,
34237|At the spring by the vistaed stream,
34237|I stood as 'neath the orchard, in the June,
34237|To the sound of the grass and the dream.
34237|I know the moss where the violets
34237|Quested the dew and the sun;
34237|The air above 'mong the orchards
34237|Murmuring ever of bees;
34237|And the heart that was filled with the music
34237|That came to the listening trees,
34237|While the bluebird's notes, as he piped again,
34237|Awoke the robin's golden throat;
34237|And the sound I heard, long years ago,
34237|Came through the wood and the dells,
34237|Bringing the sound of the violets
34237|And the perfume of dying wells.
34237|And the song I heard in the August dusk,
34237|In the August dusk by the lake,
34237|Was sweeter, from the full-leaved orchard,
34237|Than the sound of a happy brook,
34237|When it came to the school of my childhood,
34237|And to the school of the land,
34237|Oh my home of the woods, where the wild-flower
34237|Loses itself and dies!
34237|They give me back the old-time delight,
34237|The pleasant and the calm,
34237|When still the wind was blowing in the woods,
34237|And the children stood in the warm, glad school,
34237|And smiled as the dear lad asked.
34237|They give me back the pleasant book
34237|That gave my heart its fire,
34237|Those childish words, the constant brook,
34237|Those childish words, the tire;
34237|They made my soul to loiter!--Yes,
34237|They do, they make me blest!--
34237|The rest of the household, and the rest
34237|Of the parents whose hearts were filled with care,
34237|And who were sad in their care!
34237|Their voices!--Yes, and they do--
34237|'T was aye! 'T is aye! 'T is aye!
34237|And the dear friends, so dear to me,
34237|They still will live and die!
34237|I have not a moment now
34237|To forget when the morn is gray--
34237|To be happy, and cherish so
34237|The rose that is on her way.
34237|The evening breezes blow,
34237|And the stars shine out to-day--
34237|But I would not live in to-day,
34237|If I were as happy to stay!
34237|I hope that maybe one day,
34237|When all my work is done,
34237|My darling's coming away,
34237|To meet me in the sun;
34237|I hope that maybe I can see
34237|My Peggy's smile upon me.
34237|The evening wears an old, old gray,
34237|Which softly slants upon the way,
34237|Its shadows on the sunny day,
34237|Its shadows on the sunny day.
34237|O'er life, a sad, unwritten scroll,
34237|The words are like the gentle dove,
34237|That sails upon the nightly soul,
34237|Though none may read or hear reproof.
34237|And drooping o'er life's weary way,
34237|God grant the book may never end,
34237|The gentle words that cheer my way,
34237|The gentle words--they come to blend--
34237|The tender words of comfort and of love,
34237|The kindly words--they come to bring me joy.
34237|I know not if my path shall be
34237|Through the world's wild, woeful wild;
34237|But I know that sometimes, in the night,
34237|The dark will come, with wild delights,
======================================== SAMPLE 64 ========================================
2620|And be glad as the lark
2620|When the skies are clear;
2620|And send forth a breeze of love
2620|As of wings to our bark,
2620|And away with a joyous song
2620|As of streams in our ears,
2620|And away with a joyous tune
2620|As of birds in the spheres,
2620|And away with a joyous tune
2620|As of voices in trees,
2620|As of swans in the summer time
2620|When the grass is green
2620|And the air is keen,
2620|And the leaves are young--
2620|Then away with a song of praise
2620|As of flowers in Maytime
2620|All the sunny days!
2620|O beautiful, gentle, and clear,
2620|Illimitable and strong!
======================================== SAMPLE 72 ========================================
, he who had no need to fly;
24869|For in this moment of dismay
24869|The king who held that evil foe
24869|Threw Indra's son as he drew down
24869|The Lord of Life shaft-headed and bow.
24869|Then Indra, lord of every woe,
24869|The Vánar legions, with a shout,
24869|The Vánar legions met and fought,
24869|And straight they broke the tyrant's yoke,
24869|And hurled him at the giant, broke
24869|The mighty bow the giant broke,
24869|Which Indra, King of all the Blest,
24869|Had thrown by Rávaṇ's(924) mighty breast,
24869|The monstrous coil, the brawny hand,
24869|The monstrous mouth, the jaw, the jaw,
24869|The jaw, the jaw and bleeding jaw,
24869|The ungovernable host, the jaw,
24869|And the great bow which never bends,
24869|The arm, the fist, the knee, the ends,
24869|The body laid with mighty stroke,
24869|And the great bow which never bends.
24869|So, when the giants fought, and fell
24869|With murderous strokes, the giant fell,---
24869|So falls the tree with all his trunks
24869|Terrific in its death, that shoots
24869|Wild volley at the mighty trunk,---
24869|So fell the tree with all its boughs
24869|While all the vipers dug and sowed---
24869|So fell the tree with all its boughs.
24869|But Ráma's heart was sad within:
24869|He wept and mourned his captive's sin,
24869|For he had wrought a ruin yet
24869|O'er Raghu's son in his wrath,---
======================================== SAMPLE 78 ========================================
 on the bosom of
11014|King Deshav, son of Bhishma, sat in the shade of the trees,
11014|Humbu, the great, strong, beautiful, fortunate Brahmin,
11014|A king, a keeper of the law, a guide of the realm,
11014|His name unfolded through all time and space,
11014|A ruler of the realm, a keeper of the realm,
11014|And was worshipped, as was meet, by the Great Spirit of God.
11014|And all the days of his life he kept on striving with God
11014|For the union of faith; and at last all-wise he spoke to
11014|"Lord, I am the Brahmin's lord--and I hold thee thine inmost
11014|As I cast my life away from thee, my Lord, to-day!
11014|Therefore I cast mine body away from thee, my lord."
11014|And that, by constant penance, I might win thy favour
11014|So in the spirit's depths he plunged it into the sea,
11014|But, as the wave closed over it, the wandering wind
11014|Caught up the ship's chattels, and bore it with it to the beach.
11014|And Bhimasena seeing there the empty space behind,
11014|The wandering ship rocked in the dark and glowing heat.
11014|He sat upon the bosom of the Mother of God,
11014|He sat upon the emerald seas, meditating death
11014|Of the great sea.  He sat and pondered in his mind
11014|Upon the mystery of the sea, what gods the daring man
11014|Must have to tell of,--and this mystery,--when, in the morning,
11014|As, in the after days, the Lord of life should pass away,
11014|And leave the body alone to ride the ocean's force,
11014|To die in solitude, unknown, untroubled,--and unto him
11014|His world was opened; and as yet no living creature.
11014|And all the night he sat there, gazing in the east,
11014|Until the morning sunlight faded from the hills
11014|And dawn came, bringing darkness and the darkness awful,
11014|And to his soul came holy light from God, to cleanse
11014|All doubt and all resistance, till, in the morning of life,
11014|The coming of the Lord beheld his face.
## Sample 95:
24869|Canto XXI. Lakshman's Speech.
24869|He ceased: then Raghu's son repressed
24869|The sovereign of the giant kind,
24869|And thus with soothing words unsoft
24869|To Ráma Rávaṇ spake:
24869|"Come, with thy brother Lakshmaṇ, friend,
24869|And Lakshmaṇ, and the dame agree.
24869|Thou in the woods shalt soon be found
24869|And bathed in pleasant waters clean;
24869|Where thou shalt sit, and rest, and save,
24869|Well clad in coats of bark and hide,
24869|What days, what nights, what hours will pass
24869|That thou in holy heaven mayst see
24869|Thy darling with her night-made tressed
24869|Far from the forest. Thence will spring
24869|Sweet smells of pleasantness and light
24869|And bliss from the celestial string.
24869|Thence on the ground shalt thou be borne
24869|O'er the bare earth, O Queen Mosteer,
24869|And on the fresh bright earth where thou
24869|Shalt sit in state with Queen Sítá,
24869|In glorious heaven the nights and days
24869|Thou wilt be rapt by the great bliss
24869|E'en as the Lord of Gods is hearkening.
24869|The nights and days are thine, O best
24869|Of giant lords, and I, the best
24869|Of all who love the Lord of Lords,
24869|Whose might can turn the firmament,
24869|Whose might can sway the leafy bowers
24869|And turn each flower and leaf and bower
24869|To holy joy and blissful flowers.
24869|Ah me, the languorous days are come,
24869|And not a moment shall I see
24869|The happy days of Ráma's Queen
24869|Far from the light that round her glows,
24869|And marked with darkening leaves and boughs.
24869|Ah, whither would her steps be turned,
24869|And where the woodman's art had burned?
24869|Ah, whither would her steps be bent
24869|To turn her toil-worn heart once more,
24869|When all her hours were joy and peace,
24869|And all her hopes were set on store?
24869|Ah, let thy soul be comforted,
24869|Let trembling fancy still excuse
24869|The burden of a weary time
24869|That mars a saintlike life and use.
24869|Ah, if thy love were still the same
24869|That now I watch with toil and pain,
24869|That I could be for aid or flame,
24869|Could not my heart and bitterer gain."
24869|And Lakshmaṇ to the forest came
24869|And told his tale with welcoming.
24869|He saw the tree where he was set
24869|With burning buds and leaves beset.
24869|He saw the tree where he was brought
24869|By Sítá of the glittering thought,
24869|And when the leaves were fallen, he
24869|Spoke of his lord the tallest be.
24869|"O Lakshmaṇ, I the deer will slay
24869|From thicket, cave, and mountain gray,
24869|Ere long shall I this forest seek,
24869|And Lakshmaṇ in the covert seek.
24869|O'er hill and wood the Vánar bands
24869|And watch the beasts of wood and sands."
24869|He spoke: and Lakshmaṇ's love obeyed
24869|Nor did he speak as he was prayed.
# Sample 100:
38475|O Liberty, the patriot's sure defence!
38475|True to the man who fears a tyrant's eye,
38475|Preserve thy rights, and own his glorious cause,
38475|And yield the haughty title to a lie.
38475|No longer now on mean estate depend,
38475|And England owns thy sovereign vital force,
38475|And her best sons succeed to guard her laws,
38475|Or her best sons bestow a deedless course.
38475|Now, from that happy climate freedom's hope had birth,
38475|And made one day a milder country bleed,
38475|To the great cause that gave her aid is given,
38475|And to mankind one sure reward is even,
38475|Whilst I, perhaps, to distant climes must speed.
38475|To the same cause who has the cause to join?
38475|What foes against mankind may rise to arms,
38475|Boldly they fight, in actions of design,
38475|Yet all the same, and every day they charms.
38475|Ah, Washington! who can thy cause design?
38475|What can the nation do, or me, subdue,
38475|But still go on, in humbling folks admire!
38475|That we may praise thy conduct, that we fire,
38475|And for thy conduct many a hero dare,
38475|That we may rise, and cast the tyrants down,
38475|And tyrants fall, and fall the people crown!

Not bad.

These sam­ples rep­re­sent roughly top decile poem sam­ples (~10 out of the first 100), at least by my selec­tion.

Scott Alexan­der & com­menters high­lighted a few more sam­ples:

Thou know'st how Menoetiades the swift
Was dragged, of Hector and the fierce compeers
And Phrygian warriors. So, we will dispatch
Your bodies, then, yourselves to burn the ships
In sacrifice; with torches and with bells
To burn them, and with oxen to replace
Your gallant friends for ever. But I wish
That no man living has so long endured
The onset of his foes, as I have power
To burn or storm; for mighty Hector erst
Was slain, and now returns his safe return
My heart, why come you here alone?
The wild thing of my heart is grown
To be a thing,
Fairy, and wild, and fair, and whole
And they have seen the last light fail;
By day they kneel and pray;
But, still they turn and gaze upon
The face of God to-day.

And God is touched and weeps anew
For the lost souls around;
And sorrow turns their pale and blue,
And comfort is not found.

They have not mourned in the world of men,
But their hearts beat fast and sore,
And their eyes are filled with grief again,
And they cease to shed no tear.

And the old men stand at the bridge in tears,
And the old men stand and groan,
And the gaunt grey keepers by the cross
And the spent men hold the crown.

And their eyes are filled with tears,
And their staves are full of woe.
And no light brings them any cheer,
For the Lord of all is dead
Fair is the Lake, and bright the wood,
With many a flower-full glamour hung:
Fair are the banks; and soft the flood
With golden laughter of our tongue
How the clouds
Seem to me birds, birds in God's garden! I dare not!
The clouds are as a breath, the leaves are flakes of fire,
That clash i' the wind and lift themselves from higher!
In the dark the sun doth gleam,
And in the dark the moon doth seem
But now the evening is begun--
Gone is the sun upon the earth!
The silver moon doth like a cup
Of blood-red wine, and as that cup
Is drained of life, doth quench no drop.
What man will drink such wine?
There is no soul of earth or birth
Which man hath never known of earth.
There is no soul who doth not sit
And sing to it, and cry, "Drink!"
There is no soul whose feet are set
On youth's eternal paradise;
For all is a solemn harmony,
And all is a perpetual chant,
And all the world is a song of God.
There is no soul so wholly free
There comes a murmur low and sweet
As of far-off streams in a dream,
Or a murmur of many birds,
Or chime of little evening bells,
As of wedding-bells in the dells,
Soft, sweet and slow,
As of wedding belles that come and go.
A little green ribbon of lilies
By the door of my dear one's room,
A kiss on her cheek, and she whispers,
"I am the bride of the loveliest flower."
A moment we stand in the garden
Of dreams and things,
Dreaming of fairyland
And the fairy music there,
Sweet bells and dreams, and the fairy music,
The fairy songs of the air.

The top per­centile of poems are prob­a­bly quite good, espe­cially with some light human edit­ing to fix up the more glar­ing issues. To get a decent num­ber of top per­centile poems would require a lot of read­ing, but on the other hand, there is no rea­son why select­ing or rank­ing poem sam­ples could not itself be treated as a super­vised learn­ing task for retrain­ing GPT-2-117M-poetry on, by using selected/non-selected as labels and train­ing to pre­dict the prob­a­bil­ity of a given sam­ple being select­ed, and then such a NN could be used to pri­or­i­tize like­ly-­good GPT-2-poetry poems (or any source of poet­ry) for human review (and, in a form of “active learn­ing”, the results of the man­ual review can be fed back in as addi­tional data to help dis­crim­i­nate between the best and the merely good sam­ples).

GPT-2-poetry-prefix completions

Prompted sam­ples can be done like this:

python src/ --top_k 40 --temperature 0.9 --seed 2000 \
    --model_name 2019-03-06-gwern-gpt2-poetry-prefix-projectgutenberg-network-224474

The down­side of using the stock OA inter­ac­tive prompt is that it returns on the first new­line, so one either deletes new­lines or uses a sin­gle line. Nei­ther is good: a sin­gle line is hardly any con­text, while smash­ing many lines into a sin­gle super-­long-­line is dan­ger­ous because nei­ther GPT-2 has ever seen poems for­mat­ted that way (on­ly, per­haps, some prose that snuck in) and new­lines have impor­tant seman­tic func­tions in poet­ry. So, to avoid either prob­lem, I bypassed the inter­ac­tive prompt entire­ly, and I mod­i­fied the Python script to replace input (for tak­ing 1 line of key­board input) to instead read stan­dard input (import sys; so I could sim­ply pipe in mul­ti­ple lines from files or from the copy­-­paste buffer using xclip -o.

> import sys
<         while True:
<             raw_text = input("Model prompt >>> ")
<             while not raw_text:
<                 print('Prompt should not be empty!')
<                 raw_text = input("Model prompt >>> ")
>         for i in 1:
>             raw_text =
## Example use:
# echo "<|endoftext|>" | python3 src/
#                         --top_p 0.99 --model_name 117M-clean --nsamples 1000

The next issue in prompts is the ques­tion of the meta­data; given that all the train­ing data was prop­erly labeled with ori­gin and learn­ing the meaning/associations was much of the point, it does­n’t make sense to not exploit this con­trol in gen­er­a­tion. If I was using authors, as with my pre­vi­ous char-RNN exper­i­ments, the pre­fix is sim­ply what­ever author one wants com­ple­tions from, but in this case, it’s not quite so sim­ple since we only have book IDs

If an author is already rep­re­sented in the PG cor­pus, hypo­thet­i­cally one could look them up in it and see what IDs their poems were included under and use that, but that is a pain and does­n’t work for ones out­side the cor­pus like Gins­berg. So, one could instead sim­ply ask the model what pre­fix it thinks a prompt should use by feed­ing in the input sev­eral times and see­ing what pre­fix it ‘hal­lu­ci­nates’ in the sam­ples, and then adding that to the input for the real sam­ples. If GPT-2-poetry-prefix con­sis­tently returns a spe­cific pre­fix, then that is what it has learned and is use­ful scaf­fold­ing for the inputs; if it can’t do so con­sis­tent­ly, then the pre­fixes aren’t use­ful for this par­tic­u­lar input and it does­n’t mat­ter.

So, to gen­er­ate sam­ples con­di­tioned on rel­e­vant meta­data, I pipe in the whole input unmod­i­fied sev­eral times, look at the gen­er­ated sam­ples for an ID, and if there is a con­sis­tent ID, then pre­fix it to the input and sam­ple again sev­eral times.

Of course, now that every­thing is trained & I have a good input method, I want to see how GPT-2-poetry-prefix does on the same poems as GPT-2-117M before!


First, “Howl”. Given that the Project Guten­berg cor­pus is entirely old poetry and would­n’t include much in the vein of “Howl”, I did­n’t expect this to be good. The fine­tun­ing would wipe out the knowl­edge of free verse.

Find­ing a good pre­fix was hard, also unsur­pris­ing—not much like it in the PG cor­pus! I ulti­mately had to set­tle for a “1997” pre­fix from a rel­a­tively free-­verse sam­ple for gen­er­at­ing the 3 sam­ples:

## Sample 1:
1997|Mona, gazing, voluptious! exhaling human blood.
1997|Bare, the sun, in a golden mist,
1997|Leans on the meadow; the clouds are gone;
1997|Yet the heaven of green mists around
1997|Cannot but give them a fresher light.
1997|The sun hath a juice that no time can stay--
1997|And a golden night-cap.
1997|How softly the quiet clouds go by:--
1997|One sees a star through the evening sky
1997|Shine in the valley.
1997|There is no sound but the stream that strays--
1997|The very air how soft, and oh!
1997|How the waters of the meadows flow,
1997|In the warm afternoon;
1997|And the gentle wind from the hills above
1997|Is a murmurous tune.
1997|Through the still wood the cicalas sing,
1997|And every blade grows young and wan:
1997|A man is carrying a heavy heart
1997|To save himself from the cold, and then--
1997|And the last of it comes.
1997|He makes his coat, and, as he goes,
1997|His garment of a color dies.
1997|It is the night-wind, blowing over and under,
1997|With its great peasings of thundering wonder and wonder.
1997|It is the night-wind, echoing and shaking,
1997|With its great peasings of wonder and wonder.
1997|The night-wind suddenly blows a rosy bud--
1997|It is the sound of the forest trees breaking out,
1997|And a great rumbling of branches, and hoarse, and heavy and strong,
1997|And it is a shrill voice, and clear;--and the wind that gives answer cometh out.
1997|it is the loudest the voice alone in the cry of the silence of the noises in the darkness and the sky, and the moon:--and the water in the little wind in the trees.
1997|sing of the raving, and the tree with its silence that followeth:
1997|And it too long and a quiet water and they that cry of the
1997|tree-playing sea, and the air and the silence.
1997|flashing of the sky.
## Sample 2:
1997|The thundering guns! the splintering masses! the thunderous mass! the splintering guns! A welcome!
1997|Gitcheers! Welcome! the river that from thee will pass:--the storm of flame burst!
1997|The fire that is thine will be there:
1997|Hear it, my Sun-god, breathe upon me;
1997|Be strong! We ask not one again.
1997|Thou, mother Earth, if this is done,
1997|That thou wouldst be for aye to-morrow,
1997|What is thy name? to what will follow
1997|The sun that to thee sang one breath,
1997|What is thy name? In what will follow
1997|The sun in darkness, breathless, fell?
1997|The sun must rise in a day's breaking:
1997|What is thy name? to what will follow
1997|The sun in the world's last spinning-wheel?
1997|There comes a word of a holy token
1997|From Jove, the cloudless king of rain:
1997|"What canst thou, then, that I should follow
1997|The sun in heaven as on earth again?"
1997|The Lord and the Lord are lords together:--
1997|The sun will rise in a day's faring!
1997|The cloud will darken all earthly matter;
1997|The sun will burst in a day's strong-wing.
1997|The earth will moan through the black night
1997|And utter a little scream behind:
1997|"What is thy name, then, to what wilt thou,
1997|Lulled in the silence by Jove of rain?"
1997|The Sun would sink in his blue heaven
1997|And be a torch toothed in the main;
1997|The earth would be a chamber filled once more
1997|And the voice, and the heavens rise of the sea.
1997|And break a voice of the music of the heavens and the rain.
1997|Erebell of that speaks the wind again, the thunder.
1997|And the day and the flowers of all things that sang, and ocean's breath of the sun,
1997|And thou and the morning, and all that sang thee and thunder, the rain and the rain and the rain.
1997|And thou, and all.
## Sample 3:
1997|The whole cathedral of the church; the human procession, _Lyral._ 4. It had hardly gone
1997|The temple, the Capitol, the nation, was still seated there, and the altar, the
1997|pulpit, the water-mill in which it was used by the Greeks, was a perpetual murmur and in
1997|the old church, or the great, or the small, or rather the less, would have gone on at
1997|The altar of God--the altar, which had not yet been raised to the new, or to the
1997|"Blessed are they, blessed, among those who went down! and now is my will so
1997|And they asked the priests and the virgins of God to give them a prayer, so the
1997|dwellers on the earth, in which they are thinking, "a little space is granted for
1997|ever. If he were his own son he is guilty of a wrong. But he is guilty of the loss of
1997|righteousness! If he were to die before his race, his race and parents have been
1997|And they answered with voices of joy, "Remember righteousness, and remember the
1997|Thescore, the obstinate, bad and bold, at the word of the priest, was crucified.
1997|And the pious fathers and the pious men had gone to tell their sons, and to tell
1997|Then the father replied, "I would ask not the grace of thy God to form the
1997|The son was seized with the thought of death.
1997|The priests, he received the gift with the body, and the priest at the same time threw him
1997|turban and the priest's mantle. This, however, was the ransom for the son received
1997|given, and the father increased family increased his blood. The priest departed; but his body was given, he went away with tears that a widow, he died. The sated the widow gave into the father in a
1997|And it was raised, after his body, for a sowing blood in compassion of him.
1997|inery. The venerable face was brought him into the house in all the funeral and buried.
1997|giveness. But before him. The priest, together with himself.
1997|The shepherd took the widow married is thus, the priest,

While they may be OK on their own and plau­si­ble as uncon­di­tional sam­ples, they are dis­ap­point­ing as con­di­tional com­ple­tions, largely ignor­ing both the vocab­u­lary & style. It would seem that the fine­tun­ing wiped out what­ever it was GPT-2-117M was using to gen­er­ate its amus­ing “Howl” com­ple­tions.


For “Ozy­man­dias”, I fed it in a few times, and it seemed to like numeric IDs start­ing with ‘88’, so I used this as a prompt:

8820| I met a traveller from an antique land
8820| Who said: Two vast and trunkless legs of stone
8820| Stand in the desert... near them, on the sand,
8820| Half sunk, a shattered visage lies, whose frown,
8820| And wrinkled lip, and sneer of cold command,
8820| Tell that its sculptor well those passions read
8820| Which yet survive, stamped on these lifeless things,
8820| The hand that mocked them and the heart that fed;
8820| And on the pedestal these words appear:
8820| 'My name is Ozymandias, king of kings;
8820| Look on my works, ye Mighty, and despair!'
8820| Nothing beside remains. Round the decay
8820| Of that colossal wreck, boundless and bare
8820| The lone and level sands stretch far away.

Yield­ing (3 sam­ples):

## Sample 1:
8820|... the last captive died of his own shall
8820|and the stateliest of towers was the king...
8820|And he entered the city.
8820|He sat ironing in his counting-house,
8820|Flinging his burdens from a hundred kings;
8820|(A queen had one, and her name was Pepban)
8820|And the king was the son of Pepban.
8820|Then he said: 'O son of Hernando!
8820|Not a crown of gold nor iron can he save,
8820|But he shall conquer by his own right hand,
8820|And the palace of the king be free and save,
8820|And the crown of the head be free and save,
8820|And the wealth of the world be free and save,
8820|And the wealth of the world be free and save,
8820|And the king be free and save.'
8820|And the seer took the gift. He received it:
8820|And he wrought a wonder in his brain, and died:
8820|But the seer, like a god, whose history
8820|Is unendowed, and expoundeth unto it,
8820|--So shall one tell the story of his glory....'
8820|And the seer surveyed his old hard case.
8820|But the king was old and rough, he died
8820|Of age and weakness; and the seer replied,
8820|'I will tell them to my tale, and they shall learn
8820|Who sing this song, and live like equal lives
8820|In the kingdom of the king.'
8820|Again the seer said, 'We will go and speak
8820|A gentler story in a palace: it shall keep
8820|These seers in silent wonder still and deep
8820|As sleep within his grave.'
8820|And the seer took the gold:
8820|'I have no doubt that I shall tell it out,
8820|And I knew that for me shall there be no fear,
8820|If I tell a lie or clear.
8820|But tell it forth,--the poet shall be clear,--
8820|For never any hero
## Sample 2:
8820| Alluding to the curious fable of the Aztec king, who, in his
8820| The palace called319iform; the people of the region of Bahr
8820| The palace called31920|--Rua, behold! a new king came to mingle these two
8820|_So that I wept beside the dead,
8820|And buried deep in thought I wept.
8820|I, like a desert in a glade,
8820|Keep wistful vigil all the night--
8820|And on the palace built a pile
8820|Of lamps that, kindled, slowly shed
8820|A ceaseless light upon the dead.
8820|On this I saw the shadows grow;
8820|And, dreaming there, I, too, lay down--
8820|The dead.  I wept.  I wept--upon
8820|A couch and saw the shadows too.
8820|And yet, indeed, I wept.  For such
8820|A cold to-morrow's light will gleam,
8820|Or such a flame toward one like this.
8820|Then shall the spirit's eyes grow dim;
8820|And in thy face she read my heart
8820|I sometimes read a thousand rhymes
8820|Of hopes, and fears, and hopes have part.
8820|A moment as I read thy thoughts,
8820|My heart was moved; I was athirst;
8820|Yet I was well content to be
8820|The slave of some great memory;
8820|Then, in this vision of the night,
8820|My heart grew very sore with grief,
8820|And I fell prone.  Oh, it was sweet,
8820|As I lay trembling near the dead.
8820|And, as the dead-man's face, that slept,
8820|Groping beneath the churchyard yestreen,
8820|I heard the murmur, soft and deep,
8820|Of a sweet spirit, wafted thin,
8820|That made the shadows seem to creep,
8820|And murmur, soft and low and mild--
8820|And watching, through the moonlight, where
## Sample 3:
8820| The ebbing tide bears only the sands
8820|And trees that wave their boughs to the western breeze--
8820|A broken landscape, where faint breezes waft
8820|The lingerers on their way to Indian seas--
8820|A land whose shores are sunset skies,
8820|Where the pale mists of evening slowly sail,
8820|And, as we steal, the rivers murmur strange
8820|To the low-murmured wind at fall of night,
8820|And to the distant rim of the far sky:--
8820|Here, silent as a solitary tomb,
8820|With all that life can purchase and wealth can lend,
8820|With youth's immortal spirit, thou, O Friend!
8820|Art not for me to give thyself to Art?
8820|To thee, who art the latest breath of lands,
8820|Whose fame has swept the world for which men sigh,
8820|I give my heart up with indignant hands,
8820|To thee, who art the only one I seek.
8820|For thee, alone with life and light impearled,
8820|I cast my eyes from out the vast of time
8820|Where nothing dwells but Destiny's brief stage,
8820|Unshrouded, till, in its frail waste of time,
8820|It shall re-risen and grow once more sublime.
8820|And then, within the bounds where all is strife,
8820|And death, and life, and love, are all my own,
8820|I take this heart up with indignant breath,
8820|That seems to harden with the weight of years,
8820|And with the bitter hardihood of tears,
8820|And with the heart that has no beat or fear,
8820|And with the hand that has no strength to bear
8820|Only the world's and with the heart that fears,
8820|And by its sorrows only known to keep
8820|The beautiful and glorious years we keep.
8820|Here are the little walls we meant to build.
8820|Here stands the doorway, opening to the East,
8820|The child-like likeness of a poet's child

Sam­ple #2 is over-in­flu­enced by some prose footnotes/commentary which appar­ently were in PG, but the anal­ogy of Ozy­man­dias to Aztecs is a poten­tially fruit­ful one. And sam­ple #3 here is a par­tic­u­larly good exten­sion.

Essay on Criticism

Not clear what text exactly Scott Alexan­der used from Alexan­der Pope’s Essay, so I quoted the famous begin­ning sec­tion of Part 2. 3 sam­ples strongly indi­cated Pope-­like writ­ing was asso­ci­ated with a pre­fix of ‘385’ (if not nec­es­sar­ily a full pre­fix) so I used 38511 for the fol­low­ing 3 sam­ples:

## Sample 1:
38511|Pride even in numbers; wit's a kind pretence
38511|To something foreign still, but ne'er to sense;
38511|A constant waste of words, the world produces,
38511|A lazy sap, that nothing that's foreign to expels,
38511; that's foreign still unknown to the soul;
38511|A young, like, but in the world in motion,
38511|Obscending not, which smells all our own the worse than our own our own the soul's, and soul;
3851166|Like sense; which is like, but in all our whole.
38511|Which thus far more like, but in all things that's an excellence; and ne'er unchanged by which is folly's the worse, they give the worse maintained by which
38511|If sick of sense;
38511|Wholubil, or snug. ills, we know our own our first in sense the worse maintained between the worse, soon expired.
38511|Is blind and first blown on both being free from sense;
38511|From sense the worse maintained
38511|Wholuteness seems at once more used to sense the worse maintained by which
38511|Wholublime or enjoying sense; and first made to the worse, will's the worse maintained to sense;
38511|For sense; by which smells now discharged, and kept unseason'd from sense;
38511|Whose.  that's soon revived.  and then past the worse maintained, by birth to sense; by sense the worse, with weightyselves;
38511|Mankind by being all else barren; the worse maintained and last by birth to sense;
38511|Would cast, since built in nature lies from sense; for which smells and last, by repugither to sense;
38511|Whole; for our present, and first in life at all else to sense so long since built o'r by life to life, is soon revived by contact with heav' we know our own th
e worse maintained the worse it burns first made equal right;
38511|Is free.
3851166|Or dead: thus far more; who survey.
38511|Or wry's profuse and then dead: but what oft the worse maintained and next to life.
38511|From all; and
## Sample 2:
38511|There lies, that write the very best of all;
38511|For the lov'd fool, for those he courts and chokes,
38511|Is but a thorn inborn martyr in grief and sin,
38511|Who would all bawls and rattlantoms.
38511|Some hazels and isle from a thorn, or a starv'd for breaking hearts abh, to hogs;
38511|Or movel sooner writ, when by the starvels, or fombe.
38511|For men of any faultless wox and bribes.
38511|For wagers who should cut apart, and wak'd to make 'gin rights, for stink; for lamb, or chase; for lamb.
38511|Pounders or cast heel or a rope. For, for lamb, for lamb, for lamb or for lamb; for lamb, for lamb or starve.
38511|For no mean; for lamb, for sheep or for lamb, for lamb, lamb, for lamb, for lamb or for lamb, for lam, for mares.
38511; for lambs, a-heats. Suchley, for mares, for mares, for themselves.
38511. (for lambkins.
38511|mells; lam, lamb, lamb; lamb, lambkins; and other's for wares, lambkins; for struts; for sheep, lamb; or pricks, lamb, lambkins; for wer clothes; for mares: for sheep for lambkins; for goats for lamb.  for goats, lamb; for lambkins; and for moo: for hethers, for wark;
38511, lamb: for babes.
38511 unman; and for lamb.
38511; for lambkins; for mares; and the wox: for wheroat.
38511; for other goods, lamb. (let, frothriars, for spoles; a drum); for goats for lamb.
38511d:) for spoons, for spoons; for goats for whelornes, lamb, for mares: for syr trespass, for goats for mares: for lamb; for mares; for goats for sycam;
38511 theirs, for dog; for mawds.
38511; for whel
## Sample 3:
38511|They talk of constancy and faithless love,
38511|A seraph trembles at the specious glove;
38511|Nor in the rich confin'd relief of state,
38511|Find proud pretence, nor in the disdiscoveries of fate.
38511|For when misfortune makes choice remains the conduct's the prize half known,
38511|Can we secret soul without due, they fear of sense of more known.
38511|Some rise where' rights, they make it pays due.
38511861. Sense and judgment, as equal prize seem meanly, the reward the joy, as much possess the prize paid, as well reckon the prize we do not less dare not less keenly wise.
38511861|We see;
38511861|Of happy lovers ought, as well done, like a friend.
38511861|Know they ought, 'tis ev' is the other joy, as well worth a right;
38511861|The joy, as well might, as well may, as well may all is great.
38511861|Nor need of joys not as well maysters, as well as well may they give; but as little store; but as well as well as well may shewn, as much, as well we know, as well as well can be sure might prove, as well may well as well as well as well as well may view;
38511861|The mind: as well as well as well as well as much the fair as well as well as well as well as well as well as well may,
38511861, as thou woe.
38511861, when all.
38511861|Well done by the fair as well might rise, who merit thus prove by 'tis most of all are one;
38511861ly play the few.
38511861|There's well as well as well as well as well as well as well as well as well as well as well as well as well as not less admire;
38511861|Apostorably bright.
38511861|Well done by sight.
38511861, as well as well as by contact;
38511861|For they; the main aim

Alexan­der described his GPT-2-117M sam­ple from Pope:

It under­stands there should be line breaks, it under­stands the approx­i­mate cor­rect length of a line of iambic pen­tame­ter, it under­stands how to talk like an overe­d­u­cated 18th-cen­tury dandy—but it does­n’t appre­ci­ate rhyme or meter. In ret­ro­spect this isn’t sur­pris­ing; GPT has no idea words sound like any­thing; it would be shocked to learn any­one uses lan­guage as any­thing other than text strings.

GPT-2-poetry-prefix still has “overe­d­u­cated 18th-cen­tury dandy” down pat, but it man­ages to improve on the rhyming aspect: there’s quite a few rhyming lines in sam­ples #2 & #3 (#2 seems to be screwed up by tak­ing a digres­sion into foot­notes defin­ing words and then bad sam­pling get­ting it trapped), like “pre­tence”/“sense”, “soul”/“whole”, “love”/“glove”, “state”/“Fate”, “bright”/“sight”, and a num­ber of almost rhymes like “right”/“great”. One won­ders if it’s learn­ing by brute force and mem­o­riz­ing spe­cific pairs of rhymes (although could there really be that many rhymes of “state”/“Fate” in even 3m lines of old poet­ry?), or if it’s doing some­thing more equiv­a­lent to infer­ring the latent pho­net­ics from the co-oc­cur­rence of byte-­pairs? (That may sound unlike­ly, but word embed­dings do many unlike­ly-­sound­ing things with no more super­vi­sion than co-oc­cur­rence11.)

More con­cern­ing­ly, the sam­ples are ter­ri­ble. Pope’s poetry should be straight­for­ward for GPT-2-poetry-prefix, as it fol­lows stan­dard meters and rhyme and relies on a clas­si­cal vocab­u­lary well-rep­re­sented in the PG cor­pus. Why, then, are they so bad? I sus­pect this may reflect the cor­pus itself doing Pope a dis­ser­vice. Pope’s inclu­sion in the PG cor­pus appears to con­sist of the fol­low­ing (grep­ping for “Alexan­der Pope”):

32190|The Works of Mr. ALEXANDER POPE. London: Printed by W.
32190|The Works of Mr. ALEXANDER POPE. Volume ii. London: Printed
32190|Letters of Mr. ALEXANDER POPE, and Several of his friends.
32190|The Works of Mr. ALEXANDER POPE, in Prose. Vol. ii. London:
32190|The Works of ALEXANDER POPE, ESQ.; vol. i. with explanatory

Check­ing PG entries and look­ing through the 32190 pre­fix, it starts:

32190|INTRODUCTION                                                 xv
32190|The Works of Mr. ALEXANDER POPE. London: Printed by W.
32190|BOWYER for BERNARD LINTOT, between the Temple Gates, 1717.
32190|This volume consists of all the acknowledged poems which Pope had
32190|The Works of Mr. ALEXANDER POPE. Volume ii. London: Printed
32190|by J. WRIGHT, for LAWTON GILLIVER, at Homer's Head in Fleet
32190|Letters of Mr. ALEXANDER POPE, and Several of his friends.
32190|London: Printed by J. WRIGHT for J. KNAPTON in Ludgate
32190|Street, L. GILLIVER in Fleet Street, J. BRINDLEY in New Bond
32190|Street, and R. DODSLEY in Pall-Mall, 1737. 4to and folio.
32190|The Works of Mr. ALEXANDER POPE, in Prose. Vol. ii. London:
32190|Printed for J. and P. KNAPTON, C. BATHURST, and R. DODSLEY,
32190|The Works of ALEXANDER POPE, ESQ.; vol. i. with explanatory
32190|Notes and Additions never before printed. London: Printed
32190|commenced printing his particular section of the octavos when the
32190|Quo desiderio veteres revocamus amores
32190|Atque olim amissas flemus amicitias.
32190|Nutrix mea fidelissima M. Beech, obiit 5 Novem. 1725, aet. 77.
32190|Edwardus Blunt, vir amicissimus obit, Aug. 1726.
32190|Francisc. Atterbury, Roffens Episcopus, vir omni scientia clarus,
32190|The fourth volume contains the Satires, with their Prologue,--the
32190|alterations. --_His Last Will and Testament._--WARBURTON.

This is per­haps not good train­ing mate­r­ial for GPT-2-117M-poetry/prefix and explains the bizarre degen­er­a­tion—it is ‘expect­ing’ sud­den ran­dom irrup­tions of large­ly-ir­rel­e­vant prose such as intro­duc­tions or foot­note-an­no­ta­tions (ren­dered inline by PG’s text for­mat­ting). Other entries in the cor­pus will be more free of schol­arly or prose appa­ra­tus. (In ret­ro­spect, a pre­pro­cess­ing step like drop­ping lines longer than ~60 char­ac­ters might’ve been a good idea.)

8 Famous First Lines

The pre­fix trick does­n’t work on the 8 famous first lines nearly as well as it does with the long excerpts from “Howl” etc; I assume they are sim­ply too short to home in on a rel­e­vant pre­fix. Nev­er­the­less, I tried.

“Ulysses”, Lord Alfred Tennyson

“It lit­tle prof­its that an idle king,” yielded no con­sis­tency in pre­fix­es, so I skipped adding one. 3 sam­ples:

## Sample 1:
2428|And he who wears a crown, may die, but little lives.
2428|It was a pleasant day, when snugly shut,
2428|For many happy moons and many a sun
2428|Had lit the laughing waves with constant scarf,
2428|And heard the pealing waters on the run.
2428|'T was noontide; and with such a sky-lark flight
2428|The fleet white pieces flew in waves of sound;
2428|The silver stars, their fitful little light,
2428|Were beating on the liquid night around.
2428|Down went the cold, and all the dreariest blasts
2428|That ever chanced to blow, came mopping near:
2428|The first and fierce, the youngest, was our statesman here,
2428|Old Man-Autumnus!--An oak-headed oak!--
2428|The oak that in our court has grown so hard!
2428|"_Here am I_," answered he, "when I am dead,
2428|And round these sacred ashes I shall cling;
2428|And all the while my heart, unsatisfied,
2428|Holds but one clue,--and how must I be king?"
2428|How could he hope that a short space of day,
2428|And the warm blush of eve, and a fierce warning felt,
2428|Could prove the son of an old hunter, killed
2428|By the wolf's hounds in the wood! He crouched, and leant
2428|Against our shadowy robes; and the boughs bent low,
2428|Like the snow-flakes that cover the world below,
2428|Bent with a conscious motion. As the breeze
2428|Fluffs away, and the suns shivered and sighed,
2428|A shadowy motion, like a sigh, went by.
2428|It seemed a shadowy grief, and a mysterious
2428|Pestilence, like the voice of one who has sinned.
2428|At length, to his own chamber silently,
2428|We came, and gathering up the golden rim
2428|Of the great golden moon, looked down and wept.
2428|Then in a voice so soft, so
## Sample 2:
36287|Weary of wandering,
36287|Weary of loving,
36287|Can love a tawny cheek,
36287|Or a eye of ruby beaming through?
36287|My heart, why come you here alone?
36287|The wild thing of my heart is grown
36287|To be a thing,
36287|Fairy, and wild, and fair, and whole;
36287|With but the lion-heart to mate,
36287|And yet the heart to feel and know
36287|But in the wildest nightingale;
36287|The tongue you have for praise or blame,
36287|The heart that is more soft and tame
36287|Than the wild flower, this summer day,
36287|Shall never know your own dear way.
36287|Though cold and bleak your cheek I feel,
36287|Though bleak your mouth,
36287|Though cold your hand I would not steal,
36287|Oh, love, a thought beyond all words,
36287|Could I but win that word of birds,
36287|Could I but win that heart of yours!
36287|O sweet and tender is the lute,
36287|And soft the song the scarlet throat,
36287|When song-birds wake its wildwood lyres,
36287|And wild-bees through the glens go by.
36287|To you the wild-flower opens wide,
36287|To you one ripple stirs the tide,
36287|One long sweet longing seems to be
36287|A look of love made yesterday.
36287|To you the wild-flower bows its head,
36287|To you the wave is musical,
36287|Each kiss a kiss--but what of that?
36287|Ah, love, beyond your reach you keep
36287|All joys for you--ah, hear you me!
36287|Above your grave the winds are laid,
36287|Love lulls asleep within your shade;
36287|There is no rest for weary feet,
36287|The years are unfulfilled, I trow,
36287|And 'neath your sad and smiling eyes
36287|The sun tracks every path that lies
36287|Across your
## Sample 3:
1020|Who had too much of common human things,
1020|Being at peace with all he had on earth,
1020|Had something of a high and godly pleasure
1020|To do without him, since he had been
1020|In all the world so utterly,
1020|And that he had not far from his ambition,
1020|Had done no act for any human duty,
1020|Could not have felt his life's inheritance,
1020|If some good friend had lived to teach him thus
1020|His father's story of himself and country,
1020|And to restore his kingly name to men.
1020|O, the days that grow in my lament,
1020|And the nights that grow more still,
1020|When the earth lies awake,
1020|And the heaven it girdles,
1020|And the woods all shake,
1020|And the bright stars glitter
1020|Through the shadowy mist;
1020|When the night comes trailing her star-tinted hair,
1020|And the stars are the gray of our being,
1020|And the night wind is like some old sorrow
1020|Whose sweet touch is as of heaven,
1020|With its shadows and with its stars,
1020|Gathered in one place, is a child's face,
1020|A child's face made of golden sea foam:
1020|And the tears and the prayers that we utter
1020|Will be as those broken tears
1020|That fall upon bosoms, at the hour of their birth,
1020|With a sad and sorrowful melody,
1020|With a last, far-off singing of earth.
1020|Yet the night is a golden sleep,
1020|The day is like gold,
1020|And the stars go out alone
1020|Through the dark leaves and the darkness, alone.
1020|Night above and stars about,
1020|All our wanderings ended,
1020|Time above and darkness about,
1020|And the sun all hidden,--
1020|We are two two; the world is two.
1020|Howsoever the hours clap and clap,
1020|All the earth is happy,

“Sailing to Byzantium”, Yeats

“That is no coun­try for old men.”, no con­sen­sus. 3 sam­ples:

## Sample 1:
28591|I would to God that it all came to pass,
28591|That my old line of trouble and poverty
28591|Was left here desolate of my poor discontent,
28591|That the first cause of all my own heart's moan
28591|Was to be born in this sad world--a lone lone lone way?"
28591|I read it in my sonnet; his deeds were so brave,
28591|So kindly a husband, so cunning a guide;
28591|So tender a heart he went with his work,
28591|That his life never stinted its comforts at home.
28591|I read it in my sonnet--and he was so true,
28591|So loving my child, that it ever proved best,
28591|And though I was forced from his tender command,
28591|I felt I was growing up, when his voice, all tender, said,
28591|"If Love be unfaith, then he is the earth's father."
28591|I read in my sonnet--and he was so sweet,
28591|So eloquent was his look and his tones,
28591|That I smiled when I saw him, and ran forth to meet
28591|His father--and asked him in all things to meet.
28591|But some were too haughty, the most malgracious,
28591|To love their loved Father--the more to annoy;
28591|And some were too cold and abandoned, and when
28591|They saw how much reason has melted and flowed,
28591|They cried, "O Love, give us to love, give us fire,
28591|So we may die, as we will; O perfect, as dead,
28591|With the dead silence!" Then each took his own way,
28591|And, in its nature, an exquisite sorrow lay,
28591|Like love in a garden of fresh green;
28591|They were happy--too happy--by nobody knows!--
28591|The world, too, was happy by Roger and John,
28591|Though some at her heart cannot understand,
28591|Yet the whole world was happy on that first day.
28591|O thou who art so beautiful!
28591|What is the world without thee?
## Sample 2:
1365|O, there's a country, there's a town,
1365|And here's the place that I have stroll'd.
1365|See! there's a town!
1365|And here's the place, and here's the goal;
1365|And here's the flower for me and you--
1365|For you and me.
1365|Here's my little girl;
1365|And here's the spot that I call mine--
1365|"Come over here!"
1365|He's under the lilac-tree
1365|Where I loved so often to dwell;
1365|He's near his own tree-trunk,
1365|But here's his knee--
1365|And he's floating away
1365|To the roses in the churchyard,
1365|To the fields where I was born.
1365|Here's the lily-thorn, and here's
1365|"But where's the rose?"
1365|Here's a little lad;
1365|But a marigold he's brought
1365|Here's a ribbon at most:
1365|"Spring comes in the sun," says he,
1365|"And you dream of flowers gay
1365|On the hills of Arcady."
1365|That's the way he's trod;
1365|He shall be a king, and he
1365|Must have to sing--
1365|Ay, a king--a queen--a grave--
1365|Whose name was love, his heir--
1365|For the royal blood he gave
1365|At the royal marriage feast,
1365|And the crimson wine he drank
1365|At the royal marriage feast,
1365|And the crimson wine he drank
1365|At the royal marriage feast.
1365|They love to wake alive,
1365|To gaze at life and die;
1365|But they do not understand
1365|How many thousand voices call'd
1365|Out of the ages by the dead,
1365|Out of the ages by the living's voice.
1365|How many voices, and what else
1365|Did they call the living dead?
1365|All which of them was gone
## Sample 3:
1020|A time will come for tears and cries,
1020|And then I'll know her by their eyes
1020|I had her, but I cannot tell,
1020|And I am lonely in my cell
1020|Since the time the fair one came to me,
1020|I who loved her since my first year,
1020|Though in my youth she had been dead.
1020|I had grown quiet as the night,
1020|And I had won her tender sight;
1020|And the soft notes of her voice
1020|Would haunt me without joy or fear,
1020|Though the day had come.
1020|This breath in the air took the damp,
1020|And was gone to the damp,
1020|And was gone to the black,
1020|And was gone to the black,
1020|And was gone to the black,
1020|And was flown to the white,
1020|And was gone to the white,
1020|And was flown to the black,
1020|And was flown to the black,
1020|And was flown to the black,
1020|And was gone to the black,
1020|And was carried away in the mould
1020|With its all-written countersign.
1020|I thought the words would mean the fight,
1020|But the fighting face was not my sight,
1020|I thought the men would take a share
1020|Of pain and strife and weariness
1020|And leave me on the dusty street,
1020|Where I would meet that sorry man
1020|And tell him tales and talk to him;
1020|He said, "I'm sure I've done my best,
1020|But I'm not the man to meet the west."
1020|He said, "I'm going on our track,
1020|With my best friend here at the back,
1020|And my master, as I understand,
1020|And the rest of the country side,
1020|Where people go and people ride,
1020|And some end never to decide."
1020|"I can't," I ordered, "I can't!"
1020|He said again, "I say

Sonnet #29, Shakespeare

; no con­sen­sus, 3 sam­ples:

## Sample 1:
28287|With heart and hand he went--
28287|The boy, like any lad of five,
28287|Was never seen to pair;
28287|For he ne'er met with a taste of grog;
28287|And now, howe'er his foes may say,
28287|I wish the boy would do all that he can,
28287|And so will merely go to bed;
28287|For, ere he goes to bed,
28287|He'll do most things, not more.
28287|He's gone to bed as readily;
28287|Then, before breakfast, he politely shuts
28287|The book, unless he's said,
28287|And talks of how the breeches blows,
28287|And how the shadows fall;
28287|And how a lady steals a cheese;
28287|And what a consternation comes to dress
28287|That baby sleeping there!
28287|He knows it is a drowsy child,
28287|And therefore it is very wild:
28287|For he ne'er goes to bed--'tis true--
28287|But he is there, and he is fast asleep.
28287|My dears, he says, if you can look
28287|In your writing, he'll just show you just
28287|How very fond you were of me,--
28287|And for how long I loved you so!
28287|And 'tis a very different thing
28287|He reads me, as he so loves me,
28287|Just to think and write of me.
28287|The moon's a very different thing
28287|I should prefer to read to him;
28287|The dogs do bark, the parrake buzz,
28287|And so do all upon me.
28287|One day there was a certain man
28287|Who said, "I love you better,"
28287|Because he had four teeth, they say,
28287|And still was most inquisitive.
28287|They sent him to a stylish drive,
28287|And, though they griev'd him greatly,
28287|They gave poor vulgar names--I trow--
28287|The names--and what do you think?
## Sample 2:
1852|He sees his wretched wife's fate, and dies.
1852|"And she, perhaps, may die in future time."
1852|In Moore's "Ballad of Trees and stones"
1852|A friend to Boston, now far known,
1852|In Boston town, "Is Alexis," sung
1852|A strain of triumph, and these words she had:
1852|"The city's honour, name, and wealth have I!
1852|"For this, a man of blood, toil, battle, strife,
1852|Mechanics labor, die, like this of life;
1852|The whole world's fame, the wind of it, the wave
1852|Of some great deep, that rolls to heaven or cave;"
1852|In that same song, by all the prophets sung,
1852|It is the song to tell the mighty throng
1852|Of those who for their country's honour take
1852|Life's journey to that sacred hill of Fame,
1852|When with great souls, that have been, and are still,
1852|The first great men, to do great things for ill;
1852|When, from no other world, the world's heart breaks,
1852|That noble man who, though he singly speaks,
1852|Has less of good to do than such a song,
1852|And lives in little more than we of him.
1852|What cause has he, this loss of friends and kindred?
1852|What hope of fortune? Wherefore has he perished?
1852|Why is he living on one man's estate?
1852|The poet sings. Where goes the soldier?
1852|The soldier dies. One man lives, all is one.
1852|Let him by no means lily stand alone,
1852|For one man is not worthy of it--all
1852|Is to be tasted by a thirsty sword,
1852|Which, while he lives, will drink up all he can.
1852|There's good in war, but better does he die.
1852|No, this, no more--that man is not the man.
1852|He may be used to giving freely what
1852|He gave, that he should dare
## Sample 3:
392|So many a noble man he scorns in fight,
392|Away he wanders, and his heart declines
392|To be among the dead, and live in death;
392|And now he meets with fear and trembling fear;
392|His courage fails, his faith is hardened, dead,
392|His courage fails, he cannot hope to appear
392|Forward he flies, now threatening danger dread,
392|He calls his lords, and charges them and flies;
392|In dust and ashes, as in sands that bee,
392|Now drives his foot, or in the grass that lies.
392|The trembling sailors, in their care to keep,
392|Now of their craft, and now of theirs surprise,
392|As if they wanted courage, speed, and sleep,
392|And all their thoughts on their audacious prize.
392|The watchful Franks at last had seen this feat,
392|Their flags down to their knees out-stretched they lay,
392|Their head all bare, their helmet on their feet,
392|Their bodies all of gold, their locks of gray,
392|Their bodies all so fair that death might deal
392|The blow, and break the shield, the hauberk too,
392|They drew their swords, and on their backs struck through.
392|After the first they flew in fierce career,
392|Like devils monsters come from hell and fire;
392|They changed their hue, their tunes, their forms, their cheer,
392|And drew their weapons; what their foes desire,
392|They turn and fly, and both their bodies spare.
392|The hosts, whose courage, arms, and arms, were fled,
392|Yet no dismay it had, when, with a spell,
392|The king's enemies were forced to stay,
392|That night each knight was used to aid them well.
392|But now the day drew on with dreadful ending,
392|The Franks, in fury, spurred them on their way,
392|And, for that great king's fall, had turned their backs,
392|And fled, before that host, to God and day.
392|To God or mistress, had they made request,
392|Save of his presence to those lords of rest,
392|That they would go, ere

“Invictus”, William Ernest Henley

; no con­sen­sus, 3 sam­ples:

## Sample 1:
27139|And the stormy winds that blow
27139|All day long across the world
27139|I shall fly from earth below.
27139|"Oh, I am so weak and bowed!"--
27139|And the sun, in dazzling pride,
27139|Shook, and loosened as he heard
27139|All the turmoil of men's rights.
27139|In the town I should not be
27139|Wasted, wounded, and tormented;
27139|Made my spirit drunk with joy
27139|In the wonder of the sun.
27139|Oh, to see the sun shine through,
27139|And to hear the voice I knew,
27139|And to kneel because of old,
27139|And to speak, though it were gold!
27139|How it works, how it glows,
27139|In the sun, in the moonlight,
27139|And on earth, in the stream, in the odorous air,
27139|For the beauty of the moon.
27139|How it shines, how it glows,
27139|In the softness and the glow,
27139|And on earth, in the heaven, in the dusky air,
27139|Is the beauty that we know.
27139|Little, little, little sun,
27139|On your breast like the dawn of day,
27139|How I know about you always,
27139|With my heart and soul astray.
27139|Oh, to shine forever in your light,
27139|To shine forever in your beams,
27139|To shine forever in your brightness,
27139|To be shining, and to be!
27139|I knew the birds were going
27139|Out in the fair,
27139|Singing and working away,
27139|Going and staying and waiting
27139|Just as it might be.
27139|But the song of the children's singing
27139|Came every day to me
27139|Sweeter, and more than the singing
27139|That I may ever see.
27139|It rang so sweet, so clearly
27139|And far away,
27139|Making the angels, after
27139|Their work, but not a
## Sample 2:
38839|Where are the hours when we shall sleep?
38839|The night grows deeper, ruddy now,
38839|And on the east the morning light;
38839|No longer are the tears that weep,
38839|The noon-tide's drowsy murmuring
38839|That rises up through crimson leaves,
38839|As though the soul had fled away,
38839|To hide forever in the day.
38839|How sweet is the valley of Noodles!
38839|How quiet the little brook-heads there!
38839|What little bells by the cottages stood
38839|Tolled 'neath the heavy hours' flare!
38839|What tinkling bells the brook-folds heard
38839|As they crotoned by the millstone there?
38839|How soft the little brook-faces were
38839|Dimpled beside the cottage-door,
38839|And from the windows and out the trees
38839|The long-lost love-notes peeped once more.
38839|He came: the little brook he loved;
38839|And the brook beside his hearth he loved
38839|Where the crook whispered low, and the grasses
38839|That fringed its little patch of moss
38839|Beneath, the mossy water flowed.
38839|He came, and in God's blessed air
38839|He felt new life in that sweet air,
38839|New life and everlasting youth,
38839|And morning freshness, spring and sun.
38839|Weep not for me; our Father sent
38839|His little ones up to be content;
38839|As He had promised our poor ones
38839|We're glad to be thank'd like them;
38839|And let our taker daily say
38839|We're spared for our old love to-day.
38839|A rosy red rose is the rose
38839|And dear to me as children use;
38839|And though we sing it day by day,
38839|We know that mother's waiting here,
38839|And mother is the waiting-maid.
38839|God's plenty here, and He has sent
## Sample 3:
25953|The night that gives its glory,
25953|And the blue skies of the heavens;
25953|Sidney were worthless in that sight,
25953|All in a narrow space apart,
25953|I only had my life preserved,
25953|Left to myself to serve and guard,
25953|To guard and cherish, rest and guard,
25953|Of me the noblest was the king.
25953|Thus has my soul its homeward way,
25953|Thus has my soul its strength obtained,
25953|That what is dark may there appear,
25953|But evil in the light it feared,
25953|And in the light now burns my brain.
25953|With such a life, a wondrous life,
25953|Sounding through many realms and seas,
25953|The soul to millions makes at ease,
25953|And when it leaves the pleasant place,
25953|Binding in union is of grace.
25953|He who the body's life confers,
25953|From aught that is not, never slights,
25953|And when his body's life is spent
25953|The spirit cannot be content.
25953|I asked him what way he went,
25953|How many leagues he had not seen.
25953|The foolish wish I dared to own,
25953|To give the secret now I dare;
25953|I hid my foolish thought away,
25953|And lest the truth should be revealed,
25953|I thus began to test my will,
25953|And thus the answer was expressed:
25953|"I wish, O soul, that thou wouldst keep
25953|At present, in thyself at rest:
25953|But thou from sense hast naught to fear,
25953|And I my truth would not deny.
25953|And now by strength to hold thee fast,
25953|I do thee scorn, and I will give
25953|Henceforth my soul, and seek thy love
25953|As of the race of gods above."
25953|Then quickly I the thought requir'd,
25953|And thereupon this answer grew:
25953|"O soul, what wonder hath our Lord,

“Pioneers! O Pioneers!”, Walt Whitman

; no con­sen­sus, 3 sam­ples:

## Sample 1:
31314|Do you think this summer would be just one?
31314|Did not all the gods in their thunder-storm,
31314|Say, would make the trees, temples, rocks, and tops, . .
31314|And now would they think these trees didn't know,
31314|But they couldn't be built, I think they do,
31314|And would like to live on in a greener place,
31314|And the gods have made it to be a good history;
31314|And we haven't the time, you see, to say.
31314|The woods, the fields, are green,
31314|And the flowers everywhere.
31314|The sun and the grass,
31314|And the birds and the fishing-trees,
31314|Are quiet everywhere.
31314|And there is the sun,
31314|And the flowers everywhere.
31314|The sun and the flowers,
31314|And the flowers everywhere.
31314|The sun and the flowers,
31314|And one after one,
31314|Are quiet all around.
31314|But there is an end.
31314|There is another place.
31314|The end of the road
31314|To which to go.
31314|The end of the road
31314|To which to go.
31314|Ah, why do you go
31314|Into the garden there into the forest,
31314|And into the wood, and into the village,
31314|And into the field?
31314|Why does the rain
31314|Make me uneasy?
31314|Do you see,
31314|Do you see?
31314|Listen to me,
31314|I am in a hurry,
31314|And away.
31314|There is a strange thing:
31314|I have that alone.
31314|It is not at all like I married:
31314|There is a new man,
31314|Who has just been married.
31314|I have a new hat,
31314|That's in my hat.
31314|I wish I could find out another,
31314|But then there is one . . .
31314|Oh, why do you
## Sample 2:
19|Let us sing this song of mine:
19|Where I am, there I'm,
19|Tell the mighty, mighty sounding,
19|Ocean's awful son of old,
19|In the islands of the blessed,
19|In the groves of Arcadian
19|In his cradle, cold and cold.
19|We, the Fairies, we the children,
19|We the islanders, the bold!
19|We are all that has been fashioned
19|In the wondrous dreams of old,--
19|We, the revelers, the giants,
19|We the children, and the bold!
19|O the wondrous song of battle!
19|O the spoils of men of might!
19|O the spoils of conquest, conquest,
19|Where the many are not quite.
19|By the walls of ancient stories,
19|By the marble-mantled wall;
19|By the chains of dread OENEUT,
19|And the marvels of the fall.
19|By the ramparts of the giants,
19|By the caverns of the deep,
19|By the graves of men immortal,
19|By the caverns of the deep,--
19|By the temples of the Morning,
19|By the temples of the Night,
19|Where the warriors and the giants
19|Met in vision met in fight,
19|And the dying captive maidens,
19|Sat in still and stately light.
19|Child of Earth, too fair for sorrow,
19|Mother of the Light of Life,
19|Fairest daughter of bright radiance,
19|We have sung thee a new song!
19|By the ramparts of the giants,
19|By the temples of the night!
19|By the peaks of Alleghany,
19|Where the eagle cleaves the sky,
19|Trojan prince of fire and glory,
19|Comes in might the ivory-headed:
19|By the palaceer Hippolyta,
19|And the brilliant dame who leads him
19|Where the women weep and smite them
19|On the great steed Aristagrus,
19|Sorrowing, binds his loins of golden
## Sample 3:
1165|Bring us back our olden treasures.
1165|In the small white house with the crumbs for food and fire,
1165|How you cursed the stars and drummers!  I hear you drumming.
1165|Are you getting sleepy-hearted?
1165|Are you writing toiling while night was ebbing?
1165|Do you yet look in the face so white?
1165|How your furrows, how you pined away!
1165|Up the ladder--who would fain be strong?
1165|Who would hunger after freedom long?
1165|Are you being hungry, child, and tired?
1165|Mother, mother, come blow me a song.
1165|Out of doors a man came and sat beside me:
1165|He was black, he was white, but I could not see him;
1165|He was robed in a sackbut of very little gold,
1165|And the words that he said were, "Hotel-fever!" . . .
1165|He said:
1165|He has seen Barra's vision.  He is not old;
1165|He is not fit for the desert.  He can find
1165|Sight and freedom, voice and heart and mind,
1165|And he has not learnt to forget . . . yet he
1165|Has a vision of Paradise.
1165|But, alas! the thing is over,
1165|And there is no chance can take me.  That is why
1165|I looked, he said.
1165|The place is filled with flowers,
1165|With curiosities and secret pain;
1165|One has a face like those of flowers,
1165|One has an accent like a bell.
1165|The small sad music of my days
1165|Moves on.  The grassy fields and lawns
1165|Are not more silent than the stones;
1165|But one face moves beneath the stars. . . .
1165|All this is very beautiful, perhaps;
1165|The hills and woods, the fields and meadows,
1165|The clouds and clouds and all the sky,
1165|The sea's sky and the hills' sky.

“The Love Song of J. Alfred Prufrock”, T.S. Eliot

“Let us go then, you and I,”; no con­sen­sus, 3 sam­ples:

## Sample 1:
22142|When the summer comes again,
22142|When the birds on the sunny land
22142|Make the winter to come again,
22142|I may say that the happy hour
22142|When the harvest-time comes again,
22142|When the heart aches for the land of my love
22142|And the day is my heart's desire,
22142|When the harvest home is come
22142|And the days are my heart's desire."
22142|"Oh! what if we both would wander over the sea,
22142|Afar from the home on the lowland, and stray
22142|O'er the hillside and over the dale, as we stray
22142|O'er the hillside and over the moor, through the wood,
22142|By the light of the moon on the hillside and stray
22142|Till the golden mist o'er the landscape is gray.
22142|And often for me the olden log-house is seen,
22142|The cabin'd log and the swinging door,
22142|The house where I lay till the break of day
22142|Till the sun shone out and the shadows drew away,
22142|While the shadows still wandered o'er valley and hill,
22142|Till the heart stood still, till the hush of the hill
22142|Came o'er the meadow and wandered awhile,
22142|In the sweet early gloaming with autumn awhile,
22142|To lie in the light of the long autumn days,
22142|We two.
22142|The gray-haired woman that leaneth on my side
22142|And holdeth me fast in her arms,
22142|Hath bid me lie down 'neath the old oak tree
22142|That so thickly embosoms me.
22142|The greening spring came with its silent voice
22142|When the autumn leaves hang their green,
22142|And the winds from the woods whisper'd a strange
22142|remembrance of many a long vanished year,
22142|Till the gray shoon dropp'd 'neath the gray shoon's shade,
22142|'Mid the gray shoon's shade;
22142|But where is my home
## Sample 2:
30488|Over the plains and slopes, where the wild roses blow,
30488|And the low valleys and the brown hills meet the sea,
30488|And the winds hurry by with a cry, "We want you! we want you!"
30488|And we stand on the roadway, and walk in the rain,
30488|And weep for the dead that have gone from us--we are waiting,
30488|Waiting the word that the wind shatters over the hill.
30488|We have come back from the hills, from the plains where our
30488|blood was shed,
30488|From the fields where the grass was the ground and the
30488|We have come back from the valleys, we have come back
30488|from the hills!
30488|We have come back from the plains, from the plains where
30488|the grass grew green;
30488|And now that the voice of our winds shatters the
30488|world between.
30488|We have come back from the plains, have we come back
30488|to the graves,
30488|And the wind shatters the breath from the hurrying
30488|ship o'er the waves.
30488|We have come back from the plains, have we come back
30488|to the dead years,
30488|And the winds cry aloud, "Go up, and seek
30488|the faces of the gods!"
30488|We have come back from the plains, have we come back
30488|to the years,
30488|And the winds call out, "Go up, and seek the
30488|wilder world appears!"
30488|We have come back from the plains, we have come back
30488|from the days,
30488|And the swift leaves quiver, the bright leaves dance on
30488|our happy way;
30488|We have come back from the plains, we have come back
30488|to the dead years,--
30488|For the voices of men, or the gods' high
30488|heavenly joys!--
30488|We have come back from the plains, and
30488|from the long ago,
30488|Where in the cold and the gloom, on the
## Sample 3:
27494|To meet old friend Angelico's,
27494|And kiss the little one's beard.
27494|We shall have tea in the library;
27494|We shall have tea in town;
27494|By my good friend, Michael dissentund,
27494|I wish all you gentlemen!
27494|The sun has a face as fair as ever,
27494|As fair as ever was seen;
27494|As fair as ever was seen.
27494|There's a smile on his beautiful lips, my boy,
27494|As bright as the eyes of a lark;
27494|As bright as the sun at day-break gray, my boy,
27494|As bright as the eyes of a lark.
27494|There's a smile on his lips, my boy, my boy,
27494|As soft as the kisses he yields;
27494|As bright as the sun at noon-tide's end, my boy,
27494|As bright as the smile of a lark!
27494|Oh, there's something quieter than sleep, my boy,
27494|That twinkles in every eye.
27494|There's a smile on his lips, my boy, my boy,
27494|That shines like a lamp on the earth;
27494|As soft as the sun at noonday gray, my boy,
27494|As bright as the smile of a lark.
27494|A soft, pure love, that endures but ill, my boy,
27494|A soft Persuader that calms not nor charms,
27494|A king of the world-flowers and balm of the sea, my boy,
27494|A bright-hearted lover that seeks and finds, my boy,
27494|A rainbow of Heaven without a stain, my boy,
27494|A rainbow of Heaven without a stain.
27494|A sweet, pure love, that abides in a magical land,
27494|A kingdom-king, and a world-workeen band;
27494|A queen of the wildwood, a maid that is only one name,
27494|The bright-faced one calls her by name.
27494|A sweet, pure love, that all other names name, my boy,

Hamlet, William Shakespeare

“To be, or not to be: that is the ques­tion:”; some con­sis­ten­cy, so pre­fix “1006”; 3 sam­ples:

## Sample 1:
1006|Shall wretched mortals, for a certainty
1006|Of cowardice, from forth their secret places
1006|Take flight, and turn in view the point askance,
1006|From whence they can, on any object bent,
1006|So that the mirror may not abstain
1006|To 'GOD,' as in the sky grows colourd there
1006|The sunbeam.  If I had, I should prove
1006|The fountain's well-accustom'd form there.
1006|I should not be a word, that is not parting
1006|Betwixt the sun and moon, nor they, for sowing
1006|In opposite direction, doth import
1006|Their night and their moonshine to that shadowy land.
1006|Therefore into dark caves are we brought,
1006|Where underneath is alabaster, save
1006|Of dusky idol, in which shining moors
1006|'Cross-bows do us sometimes view.  Rumm, there,
1006|That consecrates the world, are seen below.
1006|How many stairs, as 't were, to o'erpass the threshold
1006|Of my triumph, when I did before thee,
1006|Enter'd, and were below, though light were not
1006|But in those hours to which this chant is not.
1006|For thereon is my mind alone intent,
1006|And this there be the singing souls, that wait
1006|Their Master's will.  If but e'en now thou choose
1006|To listen to this be melody sweet,
1006|Here may thy heart find rest.  If this thou hear,
1006|Thenceforth what argument there smote thy love;
1006|That, well perceiv'd of me in its ascent,
1006|Thou mayst at last the good from that have learn'd
1006|Of me made perfect, and of now prove o'ercome.
1006|That next I saw, which, hearing, thou shalt take
1006|For guide, arose from out my last embrace,
1006|As a good mother should, that her embrace
1006|Around each form Adams in her son,
1006|The same who
## Sample 2:
1006|Shall they not fear?--let one fear have his throat?
1006|No: for I see, and can perceive it not,
1006|That they, who were before me, were lamenting
1006|For the nature of the world, and in its light
1006|Have pity on their prison-gries, and therefore
1006|Spirits of low estate do not accuse.
1006|As oft along the still-perduing Nile
1006|Along the Indian isle the night-owl thronged,
1006|And with her breath had led them to the rivers,
1006|Thus each along the shore, in company
1006|With the fair consort of a lady, moved
1006|To supper with a working-place of eggs,
1006|Whereon all love was painted; and "O Lord!"
1006|The singer, as he finished singing, struck
1006|On the sea-shore at left and right below,
1006|Stay'd his steps, and thus in accents clear did:
1006|"Behold a wonder! behold a thing
1006|Which, whoso of mortal thing may tell,
1006|Hastes before him, dark and terrible,
1006|Of an eternal weal; and this accursed
1006|Foul thing, which may the doubting furnish try'd
1006|To feebor, I have set before him all,
1006|Who must to nothing turn.  But list: resound
1006|With me the patience, and the pity move."
1006|He then: "Now, sister!  think upon thyself;
1006|For how may I inform thee, if thou hearest,
1006|How soon and how slow time and means he takes?
1006|What world or language may at last he ask,
1006|Who to the world may'st give thee all things fair,
1006|Thou know'st, who hold'st it?"  Compassing that,
1006|And beck'ning the approach of his approach,
1006|I answer'd: "By new stars, and by George, spirit
1006|Of sign all cardinal virtues I'm assured,
1006|That wrong thy sight; but
## Sample 3:
1006|What became of thatch of yours
1006|That you sally forth so suddenly?'
1006|I did not know him; but he felt
1006|As freshening blood the air beneath: and tow'rds me
1006|He moved, and I towards him moved, and fond
1006|As father, and in words, and with the sweet
1006|And with the sweet comply'd; as he had will'd,
1006|I wept, and to his bosom lifting me
1006|Upward, and all the while distill'd my blood.
1006|Proceeding from the river, and I thence
1006|Shelter'd and still, I lifted up mine eyes
1006|Toward the sun upon the matin light,
1006|And looking, ever taking it in view,
1006|Again beheld it tow'rds the other bank,
1006|From those eternal Poubs where first it was.
1006|Unto the chasm precipitous, I found
1006|A narrow path and new to me expos'd,
1006|And to the left hand and the right mine eyes
1006|Directed, and between, on the sharp rocks
1006|Directed, and between, on each side me,
1006|And on the ground, were dark and cavern'd depths.
1006|Then to the right it spake, in semblance such
1006|As sometimes Indian women have beheld
1006|Saw in their dreams the latter realms of Hell:
1006|"Why partest from me with looks enchanted,
1006|"That farthest from thine eyes withdraw thee not?"
1006|I said it somewhat, and turn'd to the right,
1006|And when the chasm was gone, and I had reach'd
1006|The bottom, where the stairs were not beheld,
1006|Backward my eye alone its course directed,
1006|Directed to another care and deep,
1006|That in its bark it led me; and the sage
1006|To my recalling them returnd I not.
1006|"Of all that I have shown thee, since that gate
1006|We enter'd first, whose is the self-"

Romeo & Juliet, William Shakespeare

“Romeo, Romeo! Where­fore art thou Romeo?”; some con­sis­ten­cy, with 1006 pop­ping up again as a pre­fix (Shake­speare per­haps is mem­o­rable enough for GPT-2-poetry-prefix); 3 sam­ples:

## Sample 1:
1006|If once thou see, the sport is vanity.
1006|But fix thine eyes beneath: the shameless'st thou.
1006|Ah! yet no better sight hath ever brought us
1006|To the destruction of Eve, from the rape
1006|Of the fair world, where wickedness lay hid.
1006|Ah! now before the good Augustus returns
1006|In daily deeds, let our pray'rs to the Furies
1006|Be changed; for that he longs counterfeited
1006|The world's adornment, which in books for us
1006|His saintly he still wears, who for us did
1006|Lays his beloved, us'd have thereby
1006|Been so enamoured of this roundelay."
1006|As it was spilt by him who severs the dry land
1006|And, each eve earlier than the setting sun,
1006|Wines underneath the rose that Tarquin leaves;
1006|So ravished I the more the fair, the more
1006|The great disdain, the pity, the long-sought,
1006|Toward the ribald nature of this law,
1006|Which the new world much sooner would be war
1006|With my sweet-will, the which the will hath oft
1006|Been usefully perverted.  I beheld
1006|Four mighty shades, and they besought for speech,
1006|But spake but slowly; whereto I re-turned,
1006|They showed me the grievous shame that clustered
1006|In their resplendence, who had all confounded
1006|With Adam, that Adam's fair abode
1006|Long time abode in peace.  "Remember," thus
1006|The teacher cried, "Adam, I the three."
1006|"And is this the true mettled nymph, that shine
1006|Thy twins now in th' ascent, as thou dost now
1006|Below?"  I said: and he, "My son, who know'st
1006|The art of song, and who but hear, may be
1006|Lest thou divide the loaves from five to six."
1006|Thereat the
## Sample 2:
1006|If truly the mad people of Raymond dread thee,
1006|Well hast thou 'scaped yourselves, they are already
1006|Each other's torment; and the wretches, mortals!
1006|Are to the shades of thy polluted ways."
1006|Whence she again replying, after us?"
1006|Purgator to the Red relates this; and Dante,
1006|"That spirit of the just hath spoken and said,
1006|'Purge now the mountain of the unquench'd, and make
1006|The sea of fire purge otherwhere he sees,
1006|Where Hood and other vain adventures 've led;
1006|Let the o'er-shadowing bridge ill-beseem'd
1006|Yoke up the mighty demons while they walk
1006|In different paths, and divers guides disentangle
1006|The tangled ways, so that no stork may turn them back;"
1006|And the good Master to me: "See thou findest
1006|E'en thus, how in the little one the race
1006|Is to be cumber'd with the broken blossom.
1006|That from this blow across theims of brightness
1006|Has not so virtue worried thee, it seems,
1006|That one with good intent in passing 'bove it
1006|Thou seest the harmonies of the mind,
1006|And therefore dost not comprehend them.
1006|That the disturbance which the sound invades
1006|Interpreteth not, maketh thy fancy
1006|Take in good hieroglyphs of everichoon;
1006|Thou seest that not only he who bids it
1006|Be, but that other who behind the hill
1006|Comes from step's shadow, comes in solitude;
1006|So that the evil one, who, in disdain
1006|Of creature freedom, 'bove the other waits,
1006|The same, who hoping in complacency
1006|To aid thee in thy loftier thoughts, like them
1006|I see now aids, which thee, thus overcome,
1006|With light returneth to thy mind from thee."
1006|Ill shapes that course which
## Sample 3:
1006|If thou no longer liv'st in the old age,
1006|To stop the running of the vein thou well;
1006|Thou by thy deeds hast purged away my youth,
1006|Strengthen'd by me in good deeds and old,
1006|And banish'd with my beauty from my song.
1006|O how much better were it I had Ruthven,
1006|With Beatrice and high excellence
1006|Of soul! to have companion'd such, who yields
1006|To her for intermingling of the world!
1006|But I see thee already clasp'd in thine,
1006|Not in thy only Heaven, but in the life
1006|To which thou crav'st it, and dost here renew it.
1006|This good companion, speak from whence it is
1006|That I should fear not, when the hour arrive
1006|That to your sighs a second time shall bring,
1006|Then with her grace, which that affection gave
1006|And your fair looks, which with her will I scan,
1006|Till my changed nature prove itself as yours."
1006|Whence her reply the thoughtful Sage imposed;
1006|"And I am He, whom thou didst long for, dead
1006|Cord de'il me not," he answer'd, "yet for thee
1006| hamm'st me.  Heart of Potentate was I,
1006|And life was in the willow.  That the words
1006|Which now I hear, shall marvel thee, wert thou
1006|A dearth in love.  Ah, how I long, long to
1006|For it am pain and grief!  that knowest well
1006|My soul's fair metes on earth!"  His sovran speech
1006|Bespake the meek thanks of the high Lord.
1006|I do not over 8 oceans turn," he then
1006|Began, "if new device can be of late
1006|Received in thee, tell us, who thou wast, and by
1006|The heels of th' mountain, times such words to me."
1006|"That I frizzano elsewhere is

“Jabberwocky”, Lewis Carroll

Upon request, I gen­er­ated 100 sam­ples of Lewis Car­rol­l’s “”. Exam­in­ing pre­lim­i­nary sam­ples, the clos­est pre­fix was #24650, cor­re­spond­ing to The Jin­gle Book, Wells 1899, an anthol­ogy of humor­ous chil­dren’s verse (which makes sense). “Jab­ber­wocky” itself does not appear in the PG cor­pus but the “Jab­ber­wock” is men­tioned in one of the poems in Wells 1899, the acros­tic poem “An Alpha­bet Zoo”, so, close enough.

Some sam­ples:

24560|He found a foxy in the brake,
24560|A cunning fox of scarlet dye,
24560|And from that foxy followed make
24560|The scrawny fox in glee.
24560|He followed with his dam and horn
24560|To where the river-water runs,
24560|And as his living current on
24560|The river-water likes him up
24560|A mighty rocky heifer heaves,
24560|And in a single field, or twain,
24560|Shows like the yellow corn;
24560|And when the wind doth blow, so too
24560|Low in his bottom lies his head,
24560|And in the grass leaps up again,
24560|In fearful freedom unbetrayed.
24560|And while the sword-strokes gushed and ran
24560|A-gathering like a hurricane,
24560|The wildest war-cries, loud and shrill,
24560|Came surging through the hills.
24560|At times a wild yell rose in the air,
24560|From steep to steep, from moor to moor,
24560|A scream of horror and despair,
24560|Of death approaching neathernock,
24560|Of dying embers in the pore.
24560|"Thy shield, my son! thy sabre bright!
24560|My hands! thy lips! thy lips of flame!
24560|O hasten! I have done the same."
24560|It made the blood tingle and squall
24560|To hear that wounded warrior's cry;
24560|It made the life blood quickly fall
24560|On the proud swine that fought and died---
24560|"Say on! say on! thou wounded swine,
24560|Say on! thou hewer of the sanguine!
24560|Say on! thou hewer of the sanguine!
24560|Say on! thou hewer of the sanguine!
24560|Say on! say on! that pierced youth
24560|Who crossed the heath in days of yore,
24560|And when he crossed it, "Say on!"
24560|It made the blood tingle and squall
24560|To hear that wounded warrior's cry;
24560|It made the life blood quickly quail,
24560|To hear that death did soon draw nigh.
24560|It made the blood tingle and squall,
24560|To hear that death did soon draw nigh.
24560|"Say on! say on! thou wounded swine,
24560|Say on! thou hewer of the sanguine!
24560|Thou art not doomed to fetch a pail,
24560|'Tis not the ebbing blood so drain."
24560|And then it made the blood tingle,
24560|To hear that agony within:
24560|"Thou liest"
24560|"It is morn!" said the clover-bush,
24560|As the sun came up the sky;
24560|"It is morn!" said the cattle-cock,
24560|As it drifted by.
24560|Then loud cried the blackbird, and forth
24560|On the moor the clamorer fell;
24560|"The rusty ruts!" he cried, "is morn!"
24560|And he opened his little cage,
24560|And went to his morning cell.
24560|Two ravening birds sat at his feet,
24560|One flew away and one fell,
24560|The blackbird with a shining face,
24560|Sang these grand welcomes to his race.
24560|"Who is the Blackbird?" they said,
24560|"Singing high, sing low, sing low,
24560|Sings the song for the giver and light?"
24560|"I am the Blackbird---a joy befall
24560|To the young in the woods so fair.
24560|I am the wild-duck, which roams at will
24560|Up and down in the gulphyry rill.
24560|I, the giver of rivers fair.
24560|I, the joy-devoted mole,
24560|I, the joy-devoted mole.
24560|I, the humming-bird, which sings so sweet
24560|I, the giver of rivers fair.
24560|I, the bird-cail'd eggs, with a silver sound,
24560|I, the joy-devoted mole.
24560|I, the giver of goodly things,
24560|I, the swift-wing'd eagle, I,
24560|I, the joy-devoted mole.
24560|From the sunny, sunny south,
24560|From the sunny south,
24560|The swarm departed,
24560|But woe to every wicked wight
24560|That ever them befel!
24560|"O shame to every wicked wight
24560|That ever them beguile!"


In May 2019, Ope­nAI released the nex­t-largest model, which increases the para­me­ter count from 117 mil­lion to 335 mil­lion, an increase of almost 3×. The GPT-2-345M model has increased layer depth & more atten­tion heads but appar­ently sim­i­lar win­dow size; as such, while it may not be much more able to main­tain coherency across long sam­ples, its coherency & qual­ity should be supe­rior to GPT-2-117M within each win­dow, as it can absorb more knowl­edge into its para­me­ters & the increased depth may allow for more ‘think­ing’ at each step.

The reg­u­lar text sam­ples from the GPT-2-345M model struck me as some­what sub­tly but notice­ably high­er-qual­ity than GPT-2-117M, so while I was hop­ing some­one would super­sede GPT-2 entirely by releas­ing a more advanced model (like a large-s­cale Trans­former XL or Uni­ver­sal Trans­former, or even newer mod­els like the which mar­ries bidi­rec­tional & uni­di­rec­tional Trans­form­er­s), I decided to train GPT-2-345M on the PG cor­pus to com­pare it with GPT-2-117M.


This proved more dif­fi­cult than GPT-2-117M. The GPT-2-117M model was already large, at 480MB for the whole, so mak­ing it 3× larger bloats it to 1.4GB on disk; and the VRAM use on a GPU is even worse: with GPT-2-117M, a train­ing mini­batch of n = 2 could barely fit on a 1080ti’s 11GB, but at GPT-2-345M, n < 1! The main cul­prit seems to be the self­-at­ten­tion lay­ers, as reg­u­lar self­-at­ten­tion scales more than lin­ear­ly, so GPU VRAM gets eaten up fast, and appar­ently 16GB might not have been enough for GPT-2-345M either. While I have enough sys­tem RAM to train GPT-2-345M with­out any tricks, my Thread­rip­per CPU is still ~14× slower than a 1080ti, and if one guesses that GPT-2-345M takes 3× longer to train than GPT-2-117M, and GPT-2-117M takes 1–2 days, and CPU is 14× slow­er, then that’s <84 days for the poetry fine­tun­ing, which would not be fun.

To solve this, nshep­perd extended his GPT-2 train­ing code­base to employ a tech­nique Ope­nAI helped intro­duce (and pre­sum­ably used in train­ing GPT-2, although the GPT-2 paper is silent on the detail­s): “gra­di­ent check­point­ing”. Gra­di­ent check­point­ing is a space-­time trade­off which throws away some of the inter­me­di­ate states of a NN, poten­tially greatly reduc­ing total VRAM use, but at the cost of some slow­down when those inter­me­di­ate states need to be recom­puted for doing the back­prop­a­ga­tion; the slow­down, for­tu­nate­ly, turns out to be fairly mod­est.

The down­side of gra­di­ent check­point­ing is that for GPT-2-345M, it is still not mem­o­ry-­ef­fi­cient enough to train it just like GPT-2-117M—the self­-at­ten­tion lay­ers check­point nicely (as the remarks12 apro­pos of need­ing extremely wide Trans­former win­dows to accom­plish ), but it’s not enough, due to the giant word/BPE-embedding, which blows out RAM usage. (Although it’s pos­si­ble nshep­perd did­n’t imple­ment gra­di­ent check­point­ing quite right for GPT-2, as the Ope­nAI papers don’t men­tion any dif­fi­cul­ties related to the embed­ding or using gra­di­ent check­point­ing.) His ini­tial solu­tion was to sim­ply dis­able train­ing of the embed­ding and train only the Trans­former lay­ers, rea­son­ing that the generic Eng­lish embed­ding prob­a­bly would­n’t need to be trained that much as the Trans­former lay­ers are where the real work is done; much lat­er, it occurred to us that the Adam SGD opti­mizer was part of the mem­ory prob­lem, as, being an adap­tive momen­tum-based SGD opti­miz­er, it must store a mean/variance for every para­me­ter to adjust its updates per-­pa­ra­me­ter, which greatly increases mem­ory use (and which gra­di­ent check­point­ing does noth­ing about); when we switched to sim­ple SGD, that freed up enough RAM to re-en­able the embed­ding. This is impor­tant in part because the learn­ing rate for Adam & SGD dif­fers by orders of mag­ni­tude: a LR <0.01 seemed good for me for SGD, but Adam wanted a LR more like 0.00001. With n = 1 mini­batch­es, the train­ing loss is extremely noisy and it is dif­fi­cult to see the impact of any hyper­pa­ra­me­ter changes for the usual hand-­tun­ing, so nshep­perd also imple­mented a sim­ple ‘val­i­da­tion loss’ func­tion, which was help­ful toward the end.

So, the upshot seems to be that GPT-2-117M can be trained end-­to-end with Adam on a com­mod­ity GPU in 11GB VRAM; and GPT-2-345M must be trained with gra­di­ent check­point­ing, and one must choose between either fancy SGD opti­miz­ers or full end-­to-end train­ing includ­ing the embed­ding; and 744M (re­leased 2019-08-20) can’t be trained at all. Toward the end, I switched from Adam+­Trans­former-only to SGD+all, and this seemed to drop my GPT-2-345M-poetry val­i­da­tion loss by ~0.01 to a final 1.915 (which is not noth­ing, so per­haps the embed­ding did need some adjust­ing for a more poetic vocab­u­lary).

In total, I trained GPT-2-345M-poetry for 815,326 steps (mini­batch n = 1), with an Adam LR ~ 0.00001 and SGD LR ~ 0.001, over ~7 days (2019-05-04–2019-05-13) on 1 Nvidia 1080ti; the nec­es­sary train­ing time, with the ben­e­fit of hind­sight, was prob­a­bly closer to 3 wall­clock days. GPT-2-345M-poetry con­verged to a final loss of 1.915, an improve­ment of ~0.1 over GPT-2-117M’s ~2 loss (so, in some objec­tive sense, which is indi­rectly related to gen­er­ated poetry qual­i­ty, one could say that GPT-2-345M is 5% bet­ter than GPT-2-117M). I had expected some­what more quan­ti­ta­tive­ly, so I won­der if more aggres­sive train­ing meth­ods like + would have worked if they were imple­mented in this code­base & I had the patience to wait a week or two for mul­ti­ple cycles? In any case:


XKCD #1263, “Reas­sur­ing” (cf “Reas­sur­ing-­Para­ble-­Gen­er­a­tor”)

Training samples

[285999 | 5982.87] loss=2.07 avg=2.29
Saving checkpoint/run1/model-286000
Generating samples...
======== SAMPLE 1 ========
1170|Thy soul, thy very soul is burning!
1170|I look, the old-time woman I love
1170|Comes in for a kiss.  The room returns
1170|A ghost of its faded glow the same,
1170|But the old woman is gone as before;
1170|The moon has come down with a veil,
1170|And the old woman sits by my side
1170|Like a new moon rising from my grave!
1170|One day, I saw a bird full ripe:
1170|And a little flower in my garden
1170|Was the signboard's sign of peace to be.
1170|Like unto one, it told of the spirit
1170|Who dwells with all things and can see them not.
1170|The birds of all colors sang freely,
1170|And the flowers of every hue and scent
1170|Ran on like a dance up heaven's stair,
1170|With songs and flutterings of the blue and red,
1170|And sunbeams, glittering the branches overhead,
1170|And the wild sunbeam streaming upon them.
1170|And my soul was filled with a sweet desire,
1170|A need that was strange and divinely dear;
1170|And the birds sang so that the day was gone;
1170|And the flowers came full when night was near,
1170|With full hearts of light and fragrance blown,
1170|And the flowers and the birds would sing again
1170|Toward the evening that would bring them near.
1170|And the sunbeam was white as the sea,
1170|The flower was white as the evening skies,
1170|The star on the rainbow was gay and bright,
1170|And the bird of fire was a flutter of wings,
1170|And the flower and the bird would sing once again--
1170|Till I knew my soul was full of its sweetest dreams.
1171|I see how the leaves of the trees fall
1171|In the sunlight of morning;
1171|And in many a wood, with a thousand sounds
1171|And a thousand eyes, a thousand suns to behold,--
1171|I see how the leaves of the trees fall.
1171|How they fall in summer on the slopes
1171|And the meadows and the streams and streams!
1171|How they fall in the winter, soft and slow,
1171|How they fall in spring!
1171|Yet this fall must ever be beautiful;
1171|I think of the leaves, I hear them speak,
1171|The stars in heaven see them pass and pass,
1171|The rains are silent with them in their fall.
1171|The leaves of the trees are lightening and falling;
1171|The leaves of the trees are falling;
1171|And the leaves of the trees are all of their leaves
1171|Fallen on the fields,
1171|And they fall, and they fall on the fields,
1171|And I see how the leaves of the trees fall.
1171|The stars of the heavens look above,
1171|And the leaves of the trees are falling and falling,
1171|And bright winds sing,
1171|As in gladness, softly, soft and low,
1171|Loud and low;
1171|And in every star--what a wonder is it!
1171|The leaves of the trees are falling on the plain,
1171|And light-begotten clouds are lifting up the leaves,
1171|That fall like drops of rain.
1171|They fall on the flowers; and a thousand roses,
1171|And a hundred white lilies,
1171|And a thousand roses, and a hundred blossoms,
1171|And a thousand white lilies.
1171|They fall on the flowers, and droop and decline,
1171|And sleep in the sun.
1171|The leaves fall on the grass with an eddy they weave,
1171|And a thousand golden lilies
1171|Dream of the night of the leaves that was swept away.
1171|Their dreams are done.
1171|The leaves fall on the fields. They sleep in the sun,
1171|And the flowers shall be green,
1171|For the leaves of the trees are falling in the sun,
1171|With their rich lights of blue.
1171|I go to the door to-night,
1171|And look in the

[286000 | 6003.19] loss=1.03 avg=2.28
[347999 | 52462.43] loss=1.98 avg=2.06
Saving checkpoint/run1/model-348000
Generating samples...
======== SAMPLE 1 ========
 closed, or wearily
24869|Lest some foul fiend should be o'erthrown,
24869|Our hosts of fiends in wild rage we
24869|Began with furious rage to fight.
24869|We met them at the ramparts high,
24869|In numbers few and in strength not few:
24869|Whence we, our life like water spent,
24869|Abandoned the walls and basely fled,
24869|And the great flood swept like a raging sea,
24869|That o'er the deep ocean's bound
24869|Comes rushing on a torrent sea.
24869|And now the fiend's fury we met,
24869|And, like an eaglet chased by an eagle,
24869|We met, O King, with our lives in stake,
24869|As if our lives were forfeit by the foe:
24869|For in the raging tide that swept
24869|Our spirits from the wall and low,
24869|We lost and were scatheless once more.
24869|No foe may stand the fury of the king
24869|Who comes so fierce and bold to fight.
24869|And ere night's dark clouds have passed away,
24869|Our fleet will fly, and, safe return,
24869|We shall be in our country won.
24869|So will he take his own, O King,
24869|As some strong bull that fights in sport.
24869|The fiercest foe we ever saw
24869|Has strength to fight, has courage long,
24869|Is he whose flesh is fiercest, bold,
24869|Or who has lost his princely sway."
24869|They heard with trembling hearts and fear
24869|The angry speech of Rávaṇ sad:
24869|In trembling hearts and fearful fear
24869|The brother of the King of Thieves.
24869|His anger at her words they felt,
24869|And many tears they hurriedly shed:
24869|The king who ne'er to others shed
24869|His rancour of unwholesome truth.
24869|Like some vast crocodile or snake,
24869|Like the vast crocodile or snakes,
24869|He who his pride of power would show,
24869|His brother whom he scorned and belied,(856)
24869|Saw the sad news with many a sigh
24869|To Ráma by that voice revealed,
24869|And thus he spoke, most pitiful:
24869|"If thou wilt fight with Ráma, then
24869|The giant King, thy vassal, may
24869|This city all take securely in:
24869|But thou with faithful Ráma fight.
24869|'Twere well that you the city bring,
24869|Our lives in danger should ye die.
24869|To-day and to the morrow, he,
24869|Great king of giants, may he meet,
24869|The giant king I know who stole
24869|From thee thy darling Lakshmaṇ, when
24869|Our fathers on the day of vengeance
24869|Saw thee in this city slain.
24869|To-day my son is born to thee,
24869|To-morrow to the King may he
24869|Thy Ráma and thy Vritra slay,
24869|And thy own heart-sick brother, too.
24869|To slay my brother thou hast vowed
24869|This day and day in turn to dwell.
24869|Why, Ráma, is the world thus full
24869|Of fierce wrath, full of deafening noise?
24869|Thy brother's life and thee's, I ween,
24869|Lie evermore in my eyes.
24869|For all a life like this can yield
24869|Is precious little: thou must play
24869|With thy own brother, and a prey
24869|To thy most furious foe consent,
24869|If Lakshmaṇ and thy brother die.
24869|Now I would fain the King of Thieves
24869|A thousand worlds in strife destroy,
24869|And all the race of fiends who dwell
24869|Deep in the wood, a million moons.
24869|I would not seek retribution here
24869|For thine own hand upon the dead:

[348000 | 52483.09] loss=2.75 avg=2.06
[531999 | 9443.46] loss=2.09 avg=1.97
Generating samples...
======== SAMPLE 1 ========
 better his life and wealth to him
2136|I will set the gold in store."
2136|He spake no word, but bowed his gray head low
2136|And left him swiftly, and he came
2136|Through the great doors his sons to see.
2136|And there within the court his sons brought
2136|The golden treasure, all bound in gold,
2136|The precious gems and the wise gems round
2136|They held, and bore them back and brought
2136|To King Olaf, and from them cast
2136|As gifts before the king his gold.
2136|And as he heard the sons bring in the gifts
2136|He spake unto King Olaf,
2136|"My mother, look ye here and see
2136|What hap lies layed before me here."
2136|On the ground the King his mother spake
2136|From the dark wall, where her breast showed white
2136|With misgiving and dread:
2136|"Nay, but this man's heart that did breed
2136|Fell full of bitter grief as mine shall fall,
2136|And grief is for the day that comes,
2136|And sorrow is for heart to know."
2136|"Nay, O King, to-day is good and fine
2136|Withal as glad and long to be;
2136|But to-morrow may bring mickle pain
2136|Withal for grief," the king replied.
2136|"And shall be."  "That will I say."
2136|Then rose and said, "My friend, I stand
2136|Before the king in glory now:
2136|"The gold now lies at my royal hand,
2136|The gems of my folk's great king there be:
2136|And from my mother, the dark woman, bear,
2136|The dark woman, the great lord's daughter fair
2136|And mother of our good king and lord,
2136|This golden thing:
2136|This ruby and this chrysolite,
2136|Gleaming bright with golden light."
2136|And Olaf spake, for quickly he spake
2136|Sooth and plain, in full or hidden wise,
2136|And as the king in council him eyed
2136|Stood to speak, saying, "Be it so, now,
2136|Here is for mourning and for mourning sake
2136|A costly gift for our father's sake."
2136|"Goodly gift as I ween," the king rejoin'd,
2136|"And I will take it all, I ween,
2136|Than for grief or for sorrow some wise need."
2136|And so spake the good knight Grettir, and said,
2136|"My son, let the gift-bag now bring too,
2136|With silver wrought in gold and satin wrought
2136|And gold, and then our lord and master true
2136|May go in as wont:
2136|"And we will bring it," the king replied, "with speed."
2136|And so he took the golden gift and laid
2136|The ruby and the chrysolite by him,
2136|Which the cold hand that laid it must give
2136|And let him lay it next before him
2136|Behind the arm to bring it to the king.
2136|And straight the two princes went to the king,
2136|But went in silence, gazing at the golden ring
2136|Unto the lord of gems, whose shining face
2136|With a low voice them said, "Why dost thou stand
2136|And look all gleaming in the kingly door
2136|Where kings and heroes pass?
2136|"I knew not that thy heart so glowed with hate
2136|As to look on me, though I stood there high:
2136|Now know I nought," the warriors said, "dear king,
2136|Thy great heart's wrath shall not thee displease.
2136|"Not thou thine anger or thy great displeasure:
2136|For now at last with death thy father dies,
2136|And in thee as in his sons doth the will

[532000 | 9459.46] loss=1.31 avg=1.97
[534999 | 1390.41] loss=2.18 avg=1.89
Saving checkpoint/run1/model-535000
Generating samples...
======== SAMPLE 1 ========
1728|the ground: the other two with their hands each on his
1728|sleeve laid flat against his face, and strove to cover
1728|himself with the cloak. Then each was moved by pity, and
1728|clothed him in the cloak. And the hero Alcinoüs rose
1728|straightway at his father, and gave him his spear, and spake
1728|in his turn to Alcinoüs, saying:
1728|'Alcinoüs, no longer endure to sit, as though a
1728|mere man, and let the great Menelaus come in; but come in
1728|and see the son of Nestor, the Achaean hero; and ask him
1728|all about thy deeds and thy father's house, whether to have
1728|stayed, or to have returned to thine home. Say, is he still
1728|living, or hath he now gone down to the house of Hades
1728|gods to abide by them?'
1728|Then the noble son of Nestor answered him, saying: 'Father,
1728|I was not yet the son of Menelaus, when a god set me
1728|here of mortal man; yet he will not lay an hand on me,
1728|unless I too go down to the house of Hades, through fear
1728|of Zeus, lest I repay the wrong that He hath done. My noble
1728|son has gone with thee to the halls of godlike
1728|Eurymachus, and hath vowed thee unto death with a spear,
1728|on account of thy evil counsels; and in a fit of anger
1728|he promised to slay thee by the hands, on pain of death.'
1728|Then Antinous, the son of Nestor, answered him and said:
1728|'Yea now, my son, I will go thy house and seek thy
1728|father, that he may tell thee all aright.'
1728|Thus he spake, and with him went forth the noble
1728|son of Nestor. Then Alcinoüs and the other Achaeans
1728|greeted the king, and each one offered gifts. But he did
1728|not speak to mortal man about his ransoming of the
1728|gods: as for the sons of Nestor, he made them wait a long
1728|time at the ships, for he was fain for gifts in form of
1728|merry or gold, and a goodly store of raiment, and himself
1728|a well-beloved son. In like manner he minded that the
1728|children of men should offer gifts for his sake, and
1728|ponder his counsel; and the other captains and counsellors
1728|would be listening in their halls.
1728|Now the noble son of Nestor had gone down to be dead,
1728|for his house was near, and the people thronged that way about
1728|him, and at the first dawn they found him, sitting
1728|at the entrance of his hall, where he had left his golden
1728|and he looked great tears upon their faces. Then his spirit
1728|came not again to his own body with him.
1728|So long as he dwelt beneath the waves of the dark sea,
1728|some god had nursed him in the dark, but as soon as the
1728|winds swept him from the sea to the land, he fled to the
1728|tomb of Zeus. There a great swan lay in wait for him, the
1728|lordly Erystus, grandson of old Oceanus, and was first
1728|to woo him, being the first who came to woo a maiden,
1728|for surely there was none else in the wide deep to
1728|prevent him. So the lordliest son of Nestor was minded to
1728|offer him a goodly fair gift of goodly gold and bronze,
1728|and he did so, and the gift was the best, the fairest of
1728|all. And the king Alcinous and the other chieftains and
1728|favored of goddesses then brought him to the house of
1728|Zeus, who is godlike and lord of generous souls. And
1728|the blessed gods showed great joy of the deeds they
1728|had done in the sight of all. As for him he lay lying
[659999 | 5522.96] loss=2.36 avg=1.90
Saving checkpoint/run1/model-660000
Generating samples...
======== SAMPLE 1 ========
In vain: there is one true knight that we
1728|shall meet, if we seek it so, and there will be
1728|no need for anything."
1728|So he spake, and stirred the blood of Thrace, where
1728|the thick-sown reek of the battle had made a
1728|great pond, and the blood-drinkers were all weary
1728|of battle for their wine-youth, which they had got in
1728|their hands. And now they set fire to their braziers
1728|and went to bed, and their eyes still looked toward
1728|Ithaca, for they thought to see the haven called
1728|Lacedaemon, and the good news of Agamemnon.
1728|Now when they were come to their place, they feasted
1728|on plenty, and by great signs the Trojans led
1728|themselves about the gates, and when they had put
1728|everything of weight in, they took their stand
1728|straight on them, smote on their helmets with their
1728|hoofs, and the loud clamour of the fight was made
1728|heard and known; so there were great confusion and
1728|revenge in their breast, and the stout men
1728|thrust down their armour from them and dashed
1728|them upon the ground. Thus they fought in the
1728|thick of the battle. But the valiant Dardanian
1728|strong in fight found his strong men among the horses
1728|of the foremost, and caught them up in one. And he
1728|sitting by those heroes, put to their mouth a
1728|salt, and he sprinkled them with the drops of
1728|grapes, and spake and hailed them:
1728|'Friends, it is meet, O friends, that we should show ourselves
1728|good men, who know not warlike service by a name
1728|otherwise than this, a mind of mortal men: for war
1728|shows weakness in the hearts of men. Let us be men,
1728|and fight with heart and hand. As for this word of
1728|sorrow that ye bear after so long sorrow, know
1728|none the less that it is the will of the gods:
1728|we have seen great evils in our father's house, and
1728|this very day the people have perished for a
1728|little, for there was no man in them save but three
1728|old brothers, men of renown in the halls, yet they
1728|wrenched apart themselves and went their ways as wolves and
1728|foxes. And the daughter of Poppaea bare them sons,
1728|And the lord of the Credans, a man of evil mind,
1728|came at the fourth year to Mantinea: one came in the
1728|twentieth year, and begat seven sons, and he was
1728|the lord of the city of Mantinea, where he made an
1728|immediately for himself an offering from his own substance.
1728|Therefore we will not fight with these people, but let
1728|them take their way across the sea from the land, and
1728|the old man's tale he answered with bitter mouth:
1728|'"If this will not win ye to the passage of the
1728|sons, then bid your men stand guard at once upon the
1728|shield-wall, lest that too I take the sons of my
1728|father and sons-in-law in the house and see the
1728|evil deed done in all my flesh. As for me, I do not
1728|believe that I am of any worth whatsoever, for ever as now
1728|I am lying under the earth with bare shoulders,
1728|having lost even my manhood and the strength of my
1728|hands, and ever through the days of my life I am without a
1728|strength, and ever in the counsels of my heart I was
1728|Then wise Telemachus answered him, saying: 'Stranger,
1728|not till all these things have come to a pass and
1728|the people of the Achaeans make great feast and
1728|have a banquet, then wilt thou boast to thee of thy own
1728|courage and thine own strength in going and bringing
1728|them by strong armament over the sea,
[656999 | 2795.58] loss=2.49 avg=1.85
Generating samples...
======== SAMPLE 1 ========
 hope the night shall cease before the day.
27441|As the last year lies at last,
27441|And the last year's sun
27441|Ascends the hill-tops once seen in;
27441|So th' last year dies, and shall not die;
27441|For the day is spring's and spring's heir.
27441|When the last year sleeps,
27441|When the last year's day
27441|Darkens the hills,
27441|When thy light is no more, but only thy shade,
27441|Then will love make the hills and valleys glad.
27441|When the last year lies at peace
27441|And the last year's sun,
27441|Shines, as it before, upon thine altars alt,
27441|Then to thee, O God, th' offering will be offered.
27441|To the music of the spring-time we come in,
27441|We love to trace old Ocean's flow
27441|'Gainst the banks of every shallow lea,
27441|Or watch the rolling green of every hillock green
27441|Like gold in the dawning;
27441|We love to hear the sea run screaming and laughing--
27441|We love to hear the deep grow wilder and wilder--
27441|We love to hear the rolling of the sea-waves
27441|Crashing through all the coasts of the world.
27441|We love to watch the birds and the rising sun,
27441|Like a great stream through an isle of gold;
27441|We love to see the birds in their lovely green,
27441|And a sunbeam in their shining;
27441|We love to hear the sun beating, beating--
27441|It's a joy to be young when you're old!
27441|The spring-time's in its rosy prime.
27441|All the birds are flying;
27441|We're happy, and we love to be young, in summer too!
27441|'Twill come to us still, as old as we,
27441|To sing our story ere we sleep,
27441|And wake, and find the old world bright in the morning!
27441|_The_ WORLD, we know, is passing away,
27441|With all its pleasures and its woes:
27441|But one thing, sure, must remain:
27441|Our story, on the morrow, we'll tell.
27441|On the morning of the morrow I, the sea-king, lay in a lonely deep,
27441|And heard a strange old song;
27441|A sound like thunder came from the heavens, and it seemed the old sea
27441|The night has long been cold and dark and dead,
27441|Yet a ghost-like light gleams from the eyes of the stars,
27441|And a strange old song I heard.
27441|The starry heavens grow dim;
27441|The waves are foaming white and red;
27441|But I, in a lonely, weary way,
27441|Hear the great waves roar and roar,
27441|And the old sea-song still sounds on
27441|In the lonely hours of the night.
27441|Sweet-voiced as the misty-colored birds,
27441|The wind on my beach has been singing;
27441|Ah! who is this that sings of the sea?
27441|What are the songs of the sea
27441|That I have heard afar on the shore?
27441|'Tis a ghostly thing the winds are singing:
27441|'Tis a strange old song; O who is this that sings?
27441|It is many a summer o'er the waters
27441|After the sun has sunk,
27441|The sea-bird, high in heaven,
27441|Has never been seen by mortal eye;
27441|But he has been, and it has been well
27441|The waters passed, so he would stay.
27441|With the leaves of the forest all glisten,
27441|And the star of the sky above
27441|Looks lustily with his golden bars,
27441|And the trees are bending low;
27441|He brings the night a blessing, and he makes it clear,
27441|He lulls the weary stars to slumber,
27441|And he lulls the waters under till dawn may keep
27441|The dawning of the day.
Generating samples...
======== SAMPLE 1 ========
 let you know
9578|What God will never do to you in trouble.
9578|If it's through some brave, stalwart soldier-man
9578|You strive to follow, out-fighting him will spring
9578|Like the break in a wall the water makes
9578|Just when it's nearly over!
9578|A soldier's a soldier for life, and must
9578|Rise up on either hand, firm and growing,
9578|To the work he's called to do, unshaken!
9578|Your country calls you to it; rise up and do
9578|Whatever it asks of you.
9578|It may not be great fighting; may be home drubbing;
9578|May be a life of disgrace and shame;
9578|But never, never, last of any man who strives
9578|To do God's will for country or God's will for home.
9578|Your heart is in it; up and do your work!
9578|God bless you, soldier!
9578|And when the shadows lengthen
9578|And it's very dark inside,
9578|And you hear the droning of traffic in the street,
9578|And the wet fog of the night is about you,
9578|Remember that you are heroes and that your work
9578|Is never done well enough;
9578|That your battles of yore are not lost though you grope
9578|Wildly at your work, nor yet finished nor done;
9578|God bless you, soldier!
9578|Never let your fame be counted
9578|A victory or a defeat!
9578|God loves all great and mighty men,
9578|Not great or mighty in the least,
9578|And He knows that among the sons of men
9578|There are some who cannot reach Him!
9578|"He hath called his seer."
9578|Now that I've tried, God rest my soul!
9578|My feeble prayer failed of a dose.
9578|You'll find a lot of soldier-people, I'm told,
9578|Who'd like to be really great, and they'll try.
9578|I had a vision of a vision once
9578|Of two white boys, one who wore a hood,
9578|And one who, more like a black-hooded ghost,
9578|Held up in his hand the stick of wood!
9578|A soldier boy from the house before me
9578|Stepped forward in his work-shirt white,
9578|And, holding up my old rifle and my old butt-stock,
9578|He turned the wrench of the safety down!
9578|"You'll never," said the seer, as I stood at attention,
9578|"See such a rifle anyhow!"
9578|And that was a vision false and dim,
9578|An apparition of the past,
9578|Of a gray boy from the village of my childhood,
9578|And of my boyhood at home and here,
9578|Who had turned the wrench of the safety to decide
9578|Whether from that dark future or no,
9578|The rifle I would choose at the other's risk.
9578|Not much longer I had written to Colonel Taylor
9578|In the hope of getting prompt reply;
9578|But I found in Colonel Taylor's file a page or two
9578|Where I could search; and there myself said he,
9578|"If a black boy from the village of my childhood
9578|Were to go down into the world to-day,
9578|Would you send up your heart to him? And then
9578|Imagine how the earth would hear and bow,
9578|And smile on you and help you up, and set you free,
9578|And then send up its pitying bosom to you!"
9578|The sun sinks mournful down on the dying of day,
9578|And the leaves are turning into night;
9578|But let not your heart fail
9578|To mourn forever the boy,
9578|Who died before his birthday,
9578|For he dreamed of a great future, too,
9578|And dreamed that the world was his to share.
9578|He dreamed that his step was on the way
9578|To that bright and glorious thing,
9578|Where all who are gifted, gifted, shall go,
9578|To the far, far shore,

Calculating validation loss...
100%|███████████████| 40/40 [00:17<00:00, 2.34it/s]
[658000 | 3737.42] validation loss = 1.93

Random Samples

Test­ing GPT-2-345M-poetry, a slightly higher tem­per­a­ture felt war­rant­ed, so to gen­er­ate 5000 ran­dom poetry sam­ples:

python src/ --top_k 40 --temperature 0.95 --nsamples 50000 \
    --batch_size 10 --model_name 345M-poetry

I also gen­er­ated 500 con­di­tional sam­ples for Yeat­s’s “The Sec­ond Com­ing”.

Read­ing through train­ing & ran­dom sam­ples, they feel notice­ably more coher­ent; it feels eas­ier to extract mean­ing­ful sub­sec­tions which form rea­son­able poems. (In par­tic­u­lar, the pas­tiches of clas­si­cal epics or Dante have got­ten remark­ably good.)

Some fur­ther sam­ples:

======================================== SAMPLE 530 ========================================
16265|In their little room with the door ajar
16265|And the candle hanging on the wall ajar,
16265|I have come across the word "Rise"
16265|With a face as grave and flat as you please.
16265|The one thing I remember of "Rise"
16265|Is the way it makes you feel--so bad, so bad.
16265|And I've come across many words to-night
16265|That are so like "Rise"--so like--so vague, so vague.
16265|"Elegance," and "Artistic Vigour,"
16265|But "Rise" is far above the rest,
16265|And I cannot hear--or see--the word,
16265|I will just stop here (I'll stop if I can).
16265|If you don't know what "Rise" means, try.
16265|"Rise" says--but don't ask.
16265|Well, I believe that I do--
16265|And if you don't know what "Rise" means
16265|You won't either!
16265|It's when the sun makes his round
16265|On clouds a-shine;
16265|And the little stars come out
16265|From under the silver dawn
16265|To shine about me so.
16265|When the sun puts out his light,
16265|And closes his earth-door,
16265|And with shadows blackens the blue
16265|Of heaven's blue;
16265|When all the stars I own
16265|Have put their clubs in bed,
16265|And I lie just like a log
16265|In the deep, still earth;
16265|When I don't hear my mother
16265|Any more in the hall;
16265|When the cat is out of the kitchen,
16265|And my brothers and I
16265|Are up to something unsaid
16265|That we've long been keeping--
16265|I have been like to catch a cold
16265|For weeks beyond reach.
16265|And I know that this is so--
16265|And the fever that wrings
16265|My bones is a cunning one
16265|That will not go well
16265|With the morning.
16265|And when I feel all warm and well
16265|And young again,
16265|I have a thought--and I say it--
16265|Of the night that's come.
16265|'Tisn't much: a summer's night--
======================================== SAMPLE 570 ========================================
18007|With a face of a smile, and a mouth that was redder than wine,
18007|And a hand, and a mouth that was whiter than snow,
18007|And a voice that was softer than dew of the morn!
18007|And I leaned my face between his, and I held him--I
18007|Holded him, and I kissed him, and--ah, there's a stain!
18007|I held him, to prove him faithful, while I dreamed
18007|Of a face of a smile, of a mouth that was redder than wine,
18007|Of a hand, of a mouth that was whiter than snow,
18007|Of a face, of a mouth that was blacker than black.
18007|But all in vain; and the moon, the moon, the moon
18007|Came a-drifting, like a careless sister, and stole
18007|Away from the village, and left them aghast,
18007|When they found him at last in the house of his wife--
18007|Sick with love, with a heart that was blacker than black.
18007|One moment as he wept, and then, like a priest,
18007|He stood weeping, with a pallor on his face,
18007|He stood with his head bowed, and prayed, and was silent;
18007|Then he turned to his wife and said, "Wife," and I
18007|Shuddered--and kissed her, and said a poor thing, and bad;
18007|And--there's a stain!
18007|And then came the moon that came so seldom this year,
18007|And vanished so swiftly that I forgot to say "Good-night."
18007|And she answered, "Kiss me, my sweet; it is time."
18007|And she rose and went to the door; and then--I cried,
18007|And--there's a stain!
18007|Then I thought that I had forgotten all,
18007|All that I had felt and done and said
18007|For I had little to lose, and I had not much to say;
18007|But I knew what did come after I turned in the door,
18007|And that is, that the stains will never go away.
18007|What matters it whether the moon come again,
18007|Or the sun rise, or the little birds sing;
18007|That the little things that I have known and done
18007|Will hurt me still, and stain my lips with black.
18007|The moon is low, and from the windows, low,
18007|The little stars wheel in their canteens,
18007|Ripening for a night when the world is gray,
18007|In the west, far, long ago;
18007|Like flowers, they come, the stars of midnight, bright,
18007|Like flowers, in a cloud of gold and snow.
18007|The stars of midnight hang like ripples of wine;
18007|The flowers have bloomed and vanished away,
18007|Too early to be precious to us all.
18007|But the things I love the best, they have not died,
18007|And shine through the twilight, gold and red,
18007|Too late for our love, too soon for our tears;
18007|So dark and drear for their tender glow,
18007|But yet they are bright, for they will not die.
18007|In the land of shadows I will seek
18007|The star of night that shone above her urn;
18007|And she will laugh in the darkness again,
18007|When I have been gone with a heavy heart,
18007|And she will smile in her lonely home on the shore,
18007|When I shall come no more.
18007|Out of the West, out of the night,
18007|A star has come.
18007|He has rolled in glory before
18007|My soul and I.
18007|I think I have felt him before,
18007|But I never was human.
18007|Now the stars are turning into night,
18007|And the clouds are brightening.
18007|For I never was half so fair
18007|As he now is;
18007|And I never shall see him, only,
18007|Till our bodies meet, heart to heart,
18007|In the bosks of love.
18007|When all is
======================================== SAMPLE 610 ========================================
29700|Hark! from those shadowy depths thy voice
29700|Mournfully echoes, "AUTH".
29700|That was the night when from her fane
29700|Mangu Damoneo, her priest,
29700|Sang by moonlight in the glade
29700|Of her garden; but, alas,
29700|Her lovely form was gone.
29700|The night of sudden twilight,
29700|Of sudden funeral, laid
29700|The flowers of her beauty by,
29700|And left her lying dead.
29700|The mourners, with no pity
29700|For that fair maid, who had died
29700|So young and sweetly, they
29700|Drew from the crowded bier.
29700|Now the long twilight mourns
29700|In dark convulsions slow,
29700|The sun, whose brightness, when it climbs
29700|To meditate its last,
29700|Turns the great globe of heaven
29700|To a vast blackness; and the moon,
29700|In the blue distance lost,
29700|Waves her orb all palely pale
29700|O'er earth and ocean's bed,
29700|While, at her last sad funeral,
29700|The waters are still.
29700|"Where, O, where," the poet cries,--
29700|"Where is she, whom so fair,
29700|So pure a form has laid,
29700|Whose heart was true, and tender,
29700|And soft as summer air!"
29700|Yet there is sorrow in the words;
29700|For in the middle space,
29700|The grave is silent, but the sea
29700|Is louder still than I.
29700|The poets write how fair
29700|She lay in moulded tomb,
29700|A shape of beauty, soft of feature,--
29700|Yet still she's dead!
29700|I look to distant lands,
29700|Yet still I see her face,--
29700|How fair she lies in tomb!
29700|And all that Nature left,
29700|Which never came to pass,
29700|Is on her cold dead breast
29700|That cold dead face seems to look,
29700|As if his own it were.
29700|From sea to sea he sailed,
29700|And then upon the shore,
29700|The man-child found his mother's breast,
29700|The mother's bosom bare.
29700|On many a foreign shore,
29700|When sorrowed he passed along,
29700|The deep-embattled ocean cried,
29700|"A bitter heart, I've had!"
29700|The storms and the tempests of his youth
29700|He turned upon himself alone;
29700|And though the storm and the tempest had
29700|Their rage, he had passed with the wind,
29700|With God,--his only sign;
29700|And, ere the cloud of the tempest blew,
29700|His soul was with the world at play.
29700|He looked to the stars, and the stars smiled,
29700|And the moon in the heaven looked;
29700|And, as he looked, he beheld her light,
29700|And all the heaven smiled with him.
29700|When winds and tempests fly,
29700|When floods and fires fail,
29700|As their wake doth meadow and fen,
29700|'Tis the man-child's heart that craves.
29700|And I--I shall be bound,
29700|With the hoary-headed, strong, old,
29700|To earth, and the graves of the dead,
29700|Whose feet are mowed down, as they lie;
29700|And I shall rest my weary head,
29700|In the silence of Eternity,
29700|In the peaceful arms of God.

Here is a ‘failed’ exam­ple, where GPT-2-345M-poetry imi­tates the schol­arly appa­ra­tus that unfor­tu­nately con­t­a­m­i­nates the PG poetry cor­pus; it is quite plau­si­ble-­sound­ing, even includ­ing plau­si­ble-look­ing Lat­in:

======================================== SAMPLE 620 ========================================
38566|and of the two main branches of the lyric poets, he is
38566|of the first. This latter is more like a novel, the former
38566|a mere exercise in poetic technique. It is true that some of the
38566|longer poems are only in two or three of the seven acts,
38566|(as 'Cytherea and Amphithoüs', 'Odysseus and the Daughters of Pose)
38566|The metre of the whole of the tragedies is at most
38566|two or three distinct feet, two or three acts, or perhaps
38566|no more than one line of one act. But there is always no
38566|clear, distinct point of passage, and the metre may change
38566|after the action has begun, and in doing this the danger of
38566|distinction. As regards Lucan, we might as well look to one of
38566|the last days of the Republic, the 'Ante-ratu' of The Deed
38566|(cl. 27)--'Ad locum triplice trieter hoc sibi sibi dicas, etc.
38566|Præestus huic erat precatus in sæpe Lucania per Bettini
38566|Ergo sæpe diadema'--quoting an asp of the same name as the
38566|antagonist,--'Ad mihi quærent horribilem fratres' being a better
38566|means than the one named 'Hæcam.' And thence we can infer from this
38566|that the 'Ante-ratu' of the other tragic poems was a regular and
38566|accidental epigram of Lucan himself,--the kind of thing which
38566|Sextus in his Annales, 65 says of the poets of A.D.
38566|Mihi sæcula, quærent Hæc morbis,
38566|Nec coelo mutum modo vel fratris
38566|Rara ferus.
38566|If it is right to treat of the tragic poets of a later period, it
38566|should not surprise us that Lucan is so honoured in our day. Already
38566|GREEK, and still more after that, in the days of A.D. 60 and A.D.
======================================== SAMPLE 670 ========================================
36773|But not with the first men of Greece;
36773|Not so much the Greek itself
36773|As what this Greek gave and took from him
36773|His power to change what the past had been.
36773|The past he made more perfect than it was;
36773|Not for the beauty of the things,
36773|Not for the new things it is, and strange;
36773|But for an inner sense and power,
36773|The force of a spirit to penetrate
36773|Truly into Nature; the power of Nature's mind.
36773|But these are, in spite of words, a vain,
36773|A futile spectacle!
36773|For what were all
36773|But scenes of dream
36773|And scenes of glass,
36773|Till that which best is is at last all?
36773|The image of the true?
36773|Not this, as of old; not this, as of old
36773|When, leaning on his pike the Spartan king
36773|Spoke to his multitude like one man alone.
36773|Nor is it so with me; since naught can prove
36773|My nature's image; all I feel is known
36773|And knows what it is: the image of man;
36773|The body; the sense; the mind; the will; the blood;
36773|And so goes ever on to the dark heart of things.
36773|But this image, as the great natural bulk
36773|Of all the world, and the universal frame
36773|And home of all things, is a wordless sound,
36773|A little, but a strong one, heard and seen
36773|In the unspeakable calm of the supreme;
36773|That, like one man's word of it, "A little more."
36773|The soul shall know
36773|How, like the world of things,
36773|And like the thoughts,
36773|Like these our thoughts, our thoughts are many,
36773|Like the world of things,
36773|And like the sounds,
36773|Sounds of life heard and unseen,
36773|Like the world of sounds,
36773|Sounds of thought and thought, they are the same!
36773|Though what we think is like all things, like we,
36773|Though we be like one thing, not like all.
36773|This is the meaning of life.
36773|For each is like to him, like all;
36773|Though all like him are like to him, not all.
36773|Not like, but like how, and with these two
36773|The earth is like most in the sense of being strong,
36773|The rocks and the sky, sea and shore,
36773|And air and water, and all things like to blood.
36773|_With many a rose for the dead_
36773|_Is laid the flower of mine._
36773|_The flowers, the rose, the sea, the air, the water, the land._
36773|'For we are the pictures of our lives, and our ways
36773|Make up a universe.
36773|'We live life out in the open,
36773|And leave the rest
36773|To fancy._
36773|_The mind-for-life of the sea._
36773|'The mind-for-life of the sea
36773|Takes place for us, for us
36773|The little seas, the little woods,
36773|The little leaves, and the little birds,
36773|The little fishes, the little things that live
36773|The round world over,
36773|And love us with its love of the things we see;
36773|And the mind-for-life of the sea
36773|Is our outward life,
36773|And we would leave it undone, undone,
36773|Right, by ourselves,
36773|For any one to do it for us,
36773|Without any help,
36773|For any one to bear such pain,
36773|That we could never be content,
36773|For any one to stay,
36773|As we have been,
36773|And go on being, and on being.'
======================================== SAMPLE 740 ========================================
19385|"While his auld foot beats thae ances,
19385|Ah! daur ye bee at the auld bar?
19385|Ah! daur ye bee at the auld bar?,
19385|An' if ye have ten min'us ances,
19385|Daur ye be at the auld bar?
19385|There's a cauld, dingy bar,
19385|An' ae blythe young lady
19385|Was a' in his auld house,
19385|He had a fondness for her,
19385|She was his auld ha'."
19385|"I am a king o' the Irish clover,
19385|An' nae sic clogs o' gowlds I ken;
19385|But, whisht! I'm wae to hear men bleedin'
19385|At my reign o' the Irish clover,
19385|An' to see women's looves like wee diz,
19385|When the land that I rule is the land o' clover!
19385|I hae a wife an' a daughter fu' o' age,
19385|She lives down in the clover ha'--
19385|The land that I rule is the land o' clover."
19385|Oh, I was sae blest to leave the play,
19385|That, like a ghaist, rusts awa!
19385|On a windy hill, I heard the snowdrift sweep,
19385|But, oh! how gleyerfu' it was to die!
19385|I set me down upon the heather, where
19385|I heard the wild bee as it swarm;
19385|While the linties all were on me blithe and bonny,
19385|In a bonnie ring I lay.
19385|I slept the sleep that fyfu' the night end,
19385|I thought that I was left alone,
19385|But in a swoon I see the light is glint,
19385|I saw the light was west.
19385|My gowld was yellow, my heart was saft,
19385|My hopes o' long ago were fled--
19385|But lo! ilk bird's sweet singing on,
19385|I heard it far and near.
19385|I think upon my past, the while I'm casting
19385|Ae thought o' Heaven above me, yet
19385|I'm ever glad to be gane!--
19385|The land that I rule is the land o' clover!
19385|'Tis a saft breeze that gies me a kiss--
19385|'Tis a saft, fresh caftinge on my love,
19385|That waves upon my face,
19385|As gladness I do kiss, and he kiss me,
19385|Though he be sae slow.
19385|I canna get used to a kiss from your lips,
19385|But I'm wae to think that I'm blest,
19385|Whar the winds are blowing, and the rain's in bar;
19385|That they may be a-kissin' and a-kissin' at the same.
19385|The gowan-stalks grew high in the heather,
19385|The kye lay snug in their beds;
19385|And ne'er in the days o' our childhood
19385|Were we wint o' such joy;
19385|For, as life and its pleasures might come annee,
19385|So we fain was we to lie where the heather lay.
19385|Then he leaned o'er the heugh, and he laughed a hearty laugh,
19385|And the kye looked aghast, and the deil look'd asker anither,
19385|For their heads were sune to change.
19385|But the gowan-stalks they turn'd in the air,
19385|And the kye wad sieve awa',
19385|For, like a saut bird, O, saut were the tears
19385|That we shed as we fain had we seen, a wild bee on,
19385|But what could be a bee on!
19385|And sair, sae we sieve awa!
19385|And sair, sae we sieve awa!
19385|And we fain would lie where the heather lies, close

Gan­bare, GPT-2-chan:

======================================== SAMPLE 800 ========================================
27669|We never say "Thank you"
27669|(Except when it's very late!)
27669|But he's the kind that would go
27669|And do his very best,
27669|If you'd give him a hand
27669|If he only were mine.
27669|_And sometimes when we sit around the fire
27669|We talk of a thing or two:
27669|Of the boy that was mine when I was young--
27669|(No matter!)--we talk of the boy we used to know,
27669|When he played in the wood,
27669|When he went a bowing round,
27669|In the golden olden time;
27669|He was just another boy._
27669|It's a fine old wood and I've tried to walk
27669|Through the boughs above a few leaves,
27669|And I've looked where there's little green places,
27669|To the shade of a great blue sky.
27669|There's the sea, there's the meadows, the white road and yellow sheep,
27669|And there's the old, old, old road where I met him last._
27669|'Twas a wood that he liked; it seemed just green and fair;
27669|And he liked the boughs above the great blue sky.
27669|'Twas a field we used to pass, now he is far away;
27669|And he loves the meadows, the white road and yellow sheep.
27669|And he loves the field when he is dead, but he'd rather be
27669|With his little boy there in the woodland far away.
27669|When he's gone
27669|He's a rich young boy sitting in his mother's lap
27669|And that is why he'll not forget
27669|To kiss and to hug me when he's gone._
27669|_Oh! would you'd come back, this would-be friend,
27669|And we'd talk of a time when it hadn't been too hot
27669|Since we all set out; and how we'd walk
27669|Along the road again, where he'd left us.
27669|And when we'd reached him in the valley he'd say,
27669|With a smile, "Good-by, my dear, good-by!"
27669|And look at me with his dimpled smile;
27669|And I think we'd talk and joke about such things.
27669|And maybe we'd play at our old home game,
27669|If you'd come back
27669|And help to share in our old home joy.
27669|And when you are back
27669|It's just that we two can walk,
27669|Along the road again:
27669|We can watch the sheep and the grasses grow,
27669|And tell each other stories and listen to the wind
27669|Crying to the woods and calling in the trees
27669|Through the blue day, "Old dear, old dear!"_
27669|_As we were riding down that lane of blue,
27669|O'er the brown hill's crest;
27669|And there in the valley between our feet
27669|There stood a house._
27669|'Twas a tiny, simple-minded place,
27669|With a fence, and railings, and railings more,
27669|That made us feel so small.
27669|It could only be called 'Half Price' Town.
27669|There were houses up and down,
27669|In the distance we could scarcely see,
27669|To the horizon's end.
27669|_We sat, in the shade of a timber tree;
27669|Our eyes half closed, and our faces bent;
27669|And with a heavy heart we thought of home,
27669|And of the woman who'd died asleep,
27669|In the house we had grown to cherish.
27669|And the voice was quiet again;
27669|In the silence of the night,
27669|In the light of the moon and stars,
27669|And the night-wind we could feel
27669|The pulses of the sleeping and dying,
27669|In the quiet, moonless house.

This is a pecu­liar one; it starts as a satir­i­cal poem but I can’t make out what it is try­ing to switch to part­way:

======================================== SAMPLE 950 ========================================
1279|The King of Wales is a very clever fellow;
1279|He's sly as a cat, and a thief as well,
1279|He's like to set round a lamb a score,
1279|To take and turn round, and a-mimicking her tone.
1279|But to crown my Lord of Leven's min' fun,
1279|He's just like a dunce in w'at he can do;
1279|He's all in a twirl if he's going to be tail,
1279|For the man that heaps things at Lord Brougham's feet.
1279|Lord of the Whinnybone and the Hooplebooroo!
1279|You're a wit all as fine as Sir Thomas Suckly,
1279|And I'll venture to say, in my noddle,
1279|That when all the rest are out to piss,
1279|Then your Lord of Leven is just the man to go.
1279|Now, the best and the brightest of our ancestors,
1279|Who bore the race formerly known as Bow-wow-wow,
1279|Were all of one gender, and of one sex,
1279|And yet they were men of many other breeds,
1279|And more and more there was variation in their breed.
1279|For example, of the male, who was black and of white,
1279|There was also a fair breed of females all over the country;
1279|And yet 'tis a fact that the best men of the day
1279|Were all of one sex and of one species,
1279|And yet they were men of many hundreds thro many centuries.
1279|So there never was anything like it--there never was,
1279|In any age or clime, a breed like to the first:
1279|All were of one sex and of one sex combination--
1279|So, you see, they were men of many hundreds,
1279|That were all of one sex, and of many thousands,
1279|Whose species combination was myriad times more numerous than yours.
1279|In every locality from Cornwall to the Cape,
1279|There were divers sorts of males, and divers sorts of females,
1279|In every locality and every rank and class;
1279|For example, of the first five generations,
1279|There were divers sorts of weds, and divers sorts of brides;
1279|In every locality, and every rank and class,
1279|There were divers sorts of housings, and divers sorts of shoes.
1279|And therefore in every locality and rank and class,
1279|There was variation, combination, and change of occupation:
1279|There were divers sorts of women, and divers sorts of men,
1279|In every county and every town and town-house too.
1279|The first five generations were all of one sex;
1279|And this was probably why before the time of the fifth,
1279|There were divers sorts of brides, and divers sorts of men;
1279|For example, in the first five generations,
1279|The wife was the opposite of the husband--most probably;
1279|And the husband was usually a bachelor, or a man,
1279|With another wife, and the same sex as his wife,
1279|In every county and every town and town-house too.
1279|In every county and every town and town-house too,
1279|There were divers variations of dress and apparel;
1279|Some were dressed in crape, and crape-cloths and bonnets,
1279|Some in kirtles, and some in hose and hose-at-heel;
1279|Some in high-heels, and some low-heels, and some hose-nigh-loose,
1279|Some in high-heels, and some in hose-nigh-loose;
1279|And some wore noddles trimmed with blue or white,
1279|And some with knobs, and some with rings, and some with rings;
1279|And therefore there was plenty of noddles for all ranks and classes.
1279|The last five generations (which I shall say was one
1279|generation before the last), were all somewhat further off
1279|than the last, and thus left me to interpret
1279|The names of the divers divers divers generations.
1279|A woman in red and white was as happy as a girl;

This one I think must be a mix of and the (but if a wiz­ard offers you rain­bow-­col­or­ful draughts of rum strained through his magic red beard, I sug­gest declin­ing in the inter­ests of hygien­e):

======================================== SAMPLE 1060 ========================================
5185|"To the fish-lake of Pohyola,
5185|To the lake of honeyed Hiisi;
5185|Weep Iwi, weep upon Lempo,
5185|On the blue rock of Hiisi;
5185|Weep the god whose tears enrich;
5185|Weep the hero who subdued
5185|By his iron-handed arms
5185|This enfolding water-brook,
5185|By his magic hurl the rock
5185|To the upper deeps of Hiisi!"
5185|On the floor of clay they throw it,
5185|On the stone on which they write it,
5185|Spake these words in magic measures:
5185|"Lo-ye-Yo-Ru-Raka, son of Hiawatha,
5185|Take this magic measure,
5185|Words of ancient prowess!
5185|Put your senses into it,
5185|Take these mental forms into it,
5185|Turn to toe the enchanted drink
5185|Of the ancient, good, and wise Wabun!"
5185|As they twirl the mental measure,
5185|As they pour the mental formulae,
5185|Words of magic potency
5185|From the wizard's magic red beard
5185|Stream into the magic waters,
5185|Rainbow-colorful draughts of rum
5185|From the six-pending arts of brewing.
5185|Thus at midnight, hour of darkness,
5185|From the six-pending arts of brewing,
5185|Munchausen' of the spirit SAMURAI,
5185|Bitter his bitter sorrowful experiences,
5185|Tears flow to see the brewing measures,
5185|Casting him dark as night to HIAWASH,
5185|To the dancing-bower of Winansi,
5185|To the palace of the Song-element,
5185|To the home of ancient Wabun.
5185|There he sees the wizard fire-breather,
5185|There he eats the fruit of fire-fruit,
5185|Warm the water in the blue-smoke,
5185|Honey of the fire-flies, Sahwa;
5185|Nevermore to rise from Sahri
5185|To the glittering skies of Hiawatha,
5185|To the shining islands in the ocean,
5185|To his home in Winansi's honey-lands.
5185|True he sings no more of former loves,
5185|Sings no more of former partnerships;
5185|Beauty and truth the singer forsakes,
5185|Joys and dreams of CHEWBEL tree-top hollow,
5185|Thus again he sings to little honey-paws,
5185|Sings but one sweet thing, one only,
5185|Sings of the honey-pastoral flowers,
5185|Sings not of the meadows golden-rod,
5185|Nor the soft, corn-fields resting in slumbers.
5185|Young again the wizard sings of combing
5185|Sunny downs and forests of awaking,
5185|Of the golden balls in linden-groves,
5185|In the nooks of meadow-lands adorned
5185|With the purple blossoms of the ash,
5185|And the golden globes of chrysanthemum.
5185|As he sings, he flays and bleaches piarmate
5185|Peppers, onions, garlic, and ears of corn,
5185|For a meal to make his night's banquet.
5185|Straightway Hiawatha asks the artist,
5185|Places in his basket heaps of grain,
5185|Hangs himself to earth in alley lowly,
5185|Lays his hands and knees upon the rafters,
5185|Drops his baskets of grain into annihilation,
5185|Sings in low, third parts, low alto-dingule.
5185|Finally he hastens to the stable,
5185|Finds the black-frost of the winter gathering,
5185|In the center of high rafters framing;
5185|In the center a famine-gathered mass,
5185|In the center a man and his banditti,
5185|In the rafters great store of grain reaped from snow-sledge.
5185|Homeward goes the artist, homeward.
======================================== SAMPLE 1420 ========================================
1381|The loon of the moon!
1381|In the night, to me,
1381|The air was full of stars;
1381|The wild bird piped upon the branch,
1381|And the blackbird, tuned his song
1381|To the sweetest note
1381|That ever a lark knew!
1381|And I heard it ring, above the sea,
1381|In the land of the dreaming sea,
1381|In the land of the dreaming stars!
1381|A star is born when the summer moon
1381|Is full in the sky:  she wets her tears
1381|To kiss the star and sigh to hear
1381|One sigh from a thousand stars.
1381|A star is born when a nightingale
1381|Sings in a blue-bell tree; and she
1381|With such a song and such a tune
1381|Fills all the fragrant heart of the wood
1381|With such a fragrance as is there,
1381|And she sings in the heart of it.
1381|A star is born when on the wildest seas
1381|An Indian maiden dances alone,
1381|Pale as a ghost in the moon's eclipse,
1381|And the stars and the wind sing from the foam;
1381|And as the moon is dimmed in the heaving deeps,
1381|The heart of the maiden lies high and free,
1381|Shaking her curls in the beating breeze,
1381|And her eyes are bright with ecstasy.
1381|A star is born when the wind-flower sings,
1381|Or the hawthorn bathes in dew;
1381|And the nightingale sings like a bird in the sky,
1381|And all the stars and the nightingale ring,
1381|And the heart of the wind-flower sings.
1381|A star is born when the snow-white swan
1381|Drops her white soul to the river:
1381|And the winds and the water sing with glee,
1381|And the white swan lifts her face to the sun,
1381|And the stars sing out in the glory of day:
1381|And the heart of the swan is happy in heaven,
1381|And the heart of the river sings in the sun!

An alt-his­tory where Ger­many won WWI:

======================================== SAMPLE 1550 ========================================
937|And, as you will, I'll be bound to you,
937|And always be and never be forgot.
937|We were a little band of heroes then,
937|With our flags all waving by our side,
937|And our hearts all set on some deeds of fame,
937|And our eyes all full of good-will light,
937|When the Kaiser came along to help them.
937|He was a mighty thing of might,
937|And when with us our flag all flowed,
937|When all men called each other comrade,
937|When the earth was made for happiness,
937|When freedom's dawn was dawning on all lands,
937|Our lives were all to some great cause done
937|We might not be wholly silent there,
937|For our hearts all yearned to join with 'em,
937|And we could not all go quietly by
937|But stood to fight the battle hard to the end;
937|And so, long as we had hearts to move,
937|The Kaiser, to keep back our strong men,
937|Was always there to tell us that we were called
937|To fight for liberty---and all were we:
937|And every man must be a Kaiser
937|When our glorious flag we salute.
937|And now that there's a Kaiser in town,
937|The hearts we had at first to thank
937|For the end of the old war we went on --
937|To keep back the strong men to stand by us;
937|We are so proud to hold in scorn
937|The Kaiser's warning words back.
937|So when you hear our cheering chorus,
937|Look down in the faces of your foes.
937|And when you're called to your father's home,
937|Oh, just smile---and let him go by:
937|It was the Kaiser's own call
937|That your country was called to give.

Tao Te Ching

The (TTC) is a famously enig­matic text, writ­ten in a dif­fi­cult style in an more dif­fi­cult lan­guage, and because of the chal­lenge, has attracted many high­ly-­var­ied trans­la­tions.

Hiræth Weltschmerz com­piled ~270 trans­la­tions of the first verse of the TTC in one text (108Kb/19k word­s), and com­piled per-chap­ter trans­la­tions of the rest of the TTC as well (4.8M/870k word­s). Using TTC snip­pets to prompt GPT-2-345M did­n’t pro­duce good results, so he asked me to train a GPT-2-345M on the TTC to see what it did.

I con­verted the first cor­pus to Unix text for­mat, replaced var­i­ous escaped char­ac­ter enti­ties with their Uni­code equiv­a­lents, and replaced dou­ble new­lines with sin­gle lines, and trained the orig­i­nal GPT-2-345M (I was unsure if using one of the poetry mod­els would help) with the usual set­tings for ~6 GPU-hours, at which point it had reached a loss of ~1.8 and I began to worry about over­fit­ting & stopped.

I gen­er­ated the usual ~1k uncon­di­tional sam­ples:

Some train­ing sam­ples:

[12999 | 10468.36] loss=2.03 avg=1.96
Saving checkpoint/taotehching/model-13000
Generating samples...
======== SAMPLE 1 ========
 can find a way to rid yourself of these issues.
Forced to see them (the spiritual essence), there we receive freedom.
Free from spirituality, we receive nothing but desire.
These are not separate things. They are one in common. Together they are the Mystery.
The Mystery of the World is all illusion.
Everything we see is but a small part of the Mystery.
The true mystery is beyond understanding.
The mere awareness of the world is a mere shell.
By being aware we become a turd.
Those who try to hide the true inner essence will become trapped in the shell.
Those who hide their inner essences are free to become free.
These two pairs of differentiations arise from the same source: nothingness and everythingness.
Their differences are mysterious, but still they are infinitely subtle.
Beyond spirituality, they are the portal to all wonder.
The words we use to express the nature of these differences are not the same as the words we use to express the nature of other differences.
The same name may be applied to different things.
As a mere passing fable, it is the boundary of mysteries.
The real is infinite, but it has limits. The empty of the empty is the gate to all spirituality.
The name that we use to explain it, is not the name of the real.
It is the Mystery of all things; as "the ground of the profundities of the heavens and the abode of the ten thousand names.
The profound subtleties are discovered from the deepest minutiae.
The Uncomprehending is the gate to all mystery.
The unsearchable is the door of all knowledge.

The Tao that can be spoken is not the eternal Tao. The name that can be named is not the eternal name.
The nameless is the beginning of heaven and earth; it is the mother of all non-being.
The named is the mother of all things. From the named it is known what is above and what is under.
Words are ways of naming; words are names for things.
There is nothing beyond the limits of the mind and thoughts; there is something beyond the limits of the mind.
This vision that can be voiced in words is not eternal vision; this that can be voiced in words is not eternal words.
The indescribable is the essence of heaven and earth; it is the Mother of all things.
Exclamatory to the absolute is the threshold of all comprehension.
Descriptive to the absolute is the gateway to all differences.
Non-being is the limit of what is known. Exclamatory to the absolute is the threshold of all transcendence.
Non-being is the origin of heaven and earth. Exclamatory to the absolute is the gate to all diversity.
The name of the universe is not the true name of the nameless.
The Nameless is the gate to all mysteries.
Having a body we see, The body is the door of all wonder.
Hence, The unmanifest is the gateway to every wonder.
Names are ways of naming; Names are limits of names.
Unmanifest is the gate to all things.
The unmanifest is the mother of all things.
Nameless it is the mystery of the universe.
Existence is the domain of all that is.
Obscure is the domain of all that is known.
"It" is the beginning of heaven and earth.
The manifest is the Mother of all things.
This unity, Is the beginning of the universe.
Non-being is the Mother of all things. Non-being is the Mother of all things.
It is the mystery of the universe.
All things are (A Real) if they are perceived as a number, or unity, The mystery is the key to all things.
Names that can be names are not real. Names that cannot be names are real. Nothingness and all its aspects can be perceived as the subtlety of the mind.
The Namelessness or totality is the essence of all things.
Names that can be names are not real. Names that can not be names are real.
Nothingness is the Mother of all Things.
Nothingness is the mystery of all things.
Non-being is the Mother of all things. Non-being is the secret of the universe.
The Teaching which we can name is not the Mystery we can observe. The name that we can give is not the name we can give; but the Name that we can call is the Name that we can see.
The way that we can see is not the Way of the Past; it is the Way of the Past that is manifest, But without lasting, though enduring, we shall see the Unseen.
This we call the Mind, The Mother of the Past and of all Past: It is the gate

Calculating validation loss...
100%|███████████████████████| 40/40 [00:16<00:00, 2.45it/s]
[13000 | 10503.37] validation loss = 1.81
[13000 | 10504.22] loss=2.17 avg=1.96
[15099 | 12666.34] loss=1.95 avg=1.78
Generating samples...
======== SAMPLE 1 ========

The maiden-maiden is the Mother of the Serpent,
The mother of all things, is called Mother,
Who has been named and is called mother, has no share.
He who has no name conceives
He who has an idea conceives
These two, existing without separation,
Have their root in the same source; In the same sort of source,
The One and the Being of the same and both differing in name,
Reaching out from the same source, Dissolves both.
Through these two there arises within all things a measure.
From these two measure and measure, They are called the World-Dao.
So, just as Nameless, let us look upon the mysterious depths of existence;
As Manifest, the sight of its manifestations.
To rid ourselves of desire, let us view its nameless outer form;
As we perceive its unmanifest core, we see its pure Inner Light.
This is the mystery hidden in the entire mystery.
As for the world, Nameless we see its beginning.
As Manifest we see its progression.
That which we are yet not filled with, Is filled with fire.
The fire of all-pervading Truth, Wherece from without can spring, From which flow all those things that are known.
Truly alone, viewing the Unconstructed, I see a mystery deep within the Unconstructed.
That which is Unconstructed is but the beginning of great things;
That which is Named is but the gateway to great things.
It is by working that the Heavenly Way may become manifest,
That through an effort that We may see its full extent; That is, We seek to see the hidden parts, We may see its mysteries,
that is, We must see the Unconstructed.
In that which is unobstructed, the dawn of all things is brought.
In that which is filled with things, the darkness of all things.
These two emerge and separate, Yet together, All things are called darkness and light.
They are both of One Origin, Which is the Gate to all Profoundness.

The Nameless is the door to Mystery,
The Manifest is to a Reasoned Way the gate to all hidden Mystery.

It is from mysteries which we shall not know, That we may approach to the Eternal, In order that we may obtain the Path of Life.

It is not always the case that, While nothing is experienced, The Essential Principle itself is hidden.
This state of Nothingness is the gateway to the Manifestation of all that is Eternal.
It is the origin of that which is lasting, Therefore it is the Emptiness which is the Gateway to the Finality.
That which is without beginning, It is the gate to all the subtleties of the Eternal;
That which has an origin, Being a process, Is the source of all the subtleties of the Eternal.
If you would know the Divine mystery, Become a Deserter. Life is a spring from a Deeper Mystery.

The unvarying Manifest is the Source of the Unvarying;
That which has a name is the mode of manifestation;
That which is unnamed is the mode of existence.

Unity is source of Manifest, Non-manifest the mother of the manifold.
Thus to watch, to wonder; to have desires and desires; All things, Manifest and non-manifest, To return whence to seek its way.

Tribulation is its mother.
The mystery of the Tao is the Body of the Tao.
The Essence of Tao is Tao, Tao the Cosmic Universe;
All non-being Is its mother.
Non-being is its mother Because it has the Secret Principle hidden,
Though without Secret Principle It is the Mother of all.
That Which Is Deeper Goes Before Deeper,
And That Which Is Secret Goes Before Secret;
They Are Called the One and the Mother of All Things.
The Name Is from Mystery,The Creature from Form,The Origins of all things
To perceive the Secret, Become a deserter
Desire Returns from the Secret: It is its Mother The Secret of all existence.
These words:
"Empty Words"
To see them Is to see the Secret,
To understand The Secret Is to understand The Origin
Its Dynam is Manifestness (Of all)
That is Its Mother (Of all)
That which is empty
Remain a Mystery
Beyond Mystery
Beyond Proserely
With all mystery in Its Depth
Has the Serpent walked
To understand Its Secret
Has become a Serpent
As for that which has a name It is the beginning of All Things
The One without a name has originated All things.

What arose is called the Essence of All Things.
These two that were separate
With opposite names But Ever Dissolving Is the

[15100 | 12682.90] loss=1.69 avg=1.78

I then began train­ing it on the full TTC cor­pus, which was split into per-chap­ter files. Remem­ber­ing the prob­lems with run-on poet­ry, I added the <|endoftext|> mark­ers to the end of each file. (It would be bet­ter to add that to the end of each trans­la­tion, but HW did­n’t include any delim­iters for each trans­la­tion, and doing so man­u­ally would be too much work.)

This required sub­stan­tially more train­ing time.

Some train­ing sam­ples:

[88999 | 6150.00] loss=2.33 avg=2.44
Saving checkpoint/taotehching/model-89000
Generating samples...
======== SAMPLE 1 ========
 Tao Tao's Way is universal. It covers all lands. Therefore those who realize it can accomplish all under Heaven. Therefore he who maintains the Tao, can be believed to know the Universe.

There is an ocean and a river, and all beneath the heavens aspire to be carried by them. The people of the Tao country follow these rivers and streams, as they may be carried. The people of the man of the Tao country follow the rivers and streams, as they may be guided; and the people of the man of the Tao country follow the rivers and streams, as they may be measured. All under heaven strive to be carried by the Tao Way, as they might be carried by Heaven. The great man, when he knows the Tao Way, can achieve all under Heaven. When a man who has completed the Tao is believed to know the Tao, this is shown by his ability to govern.

All Tao is infinite; to people like it, it is the very mother of all things. That which flows with the Tao is universal; that which opposes it is not. The Great King possesses the Tao Way, so that he can hold everything under heaven. To know Tao, one must be a man of Tao. To know the Great King is the same as to know Heaven. The Great King who understands Tao can hold everything under heaven. A man of Tao must know everything under Heaven, just like he knows Heaven.

True and Eternal Tao Tao can take care of all things that exist, without needing a reason. The person who is able to hear Tao is capable of capturing everything under heaven. Having enough listening ability, the one who can take care of Tao Is able to know all things in the universe. Having enough being able to listen to Tao, The one able to know everything under heaven is able to govern all under heaven. Thus he who is able to control Tao is the one who will be able to control the universe. Whoever knows Tao can grasp everything under heaven. The person of Tao has no limitations, therefore he is able to grasp everything under heaven.

By Tao I can all things be fulfilled, Without needing to get into details. All things call to the Tao Way Without needing to tell the other Tao people to be fit. I know the Tao Way And all things flow with me. If you're also able to think, You can capture the whole Tao Way. Knowing all things That come to you Can help you to know all things that come to others. When one can know Tao You can also master Tao. When the Master knows Tao He can master the Tao Way. And if you are able to master the Tao You can also take all things into your own hands. You don't have to control Tao or all things.

Under Tao, Everything is fulfilled. Tao never requires a reason for doing something. All things--- even beasts and plants--- seek out Tao. But because of Tao's inner being, All things have this one thing to do with it. Knowing the Tao gives you the inner clarity of Tao. But to have the inner clarity you also have to know that you are a human being. So one who is a Taoist learns that he is a human being. The Master of Tao learned that he was a human being. The Taoist who understands the Tao is not burdened by it; He has the inner clarity he needs.

The Way of Tao: All things call to it, Through but one source. Everything that desires the Tao Must either reject or destroy itself. The Great Thinker has Tao, And can guide all under Heaven. To have the Tao Knowledge is to know the mysterious Source. Knowing the mystery reveals the subtle power of Tao. The Supreme Man can conquer all under Heaven. If he knows Tao, he can be truly the Master of All. To know Tao is to understand the mysterious nature of Heaven. If you know Tao, you also can serve Heaven. The Great Thinker has Tao to serve Heaven, And the Taoist who has Tao to serve Heaven is equal to all things.

All things around a "Way of Tao" "says to its people, "everything"s the natural way,"' and everything that should reject it's it alone. Knowing everything about Tao, people accept Tao, and everything under heaven has Tao in it. To know Tao is to know all things. To know Tao is to know that Tao itself is all things, and the Master of Tao can know all things under heaven. Therefore, he who understands Tao can serve Tao.

Tao was born of the Tao Way It's there for you But not through words, and it's everywhere All things call to it, But not through its Tao Words will not accomplish the feat. You cannot hold on to Tao's place without grasping the primeval essence of this force of nature. Know Tao, or your Tao will know the universe It's how you grasp the primeval essence that matters Don't hold Tao hostage and force it to shape

Calculating validation loss...
100%|██████████████████████| 40/40 [00:16<00:00, 2.36it/s]
[89000 | 6186.11] validation loss = 2.44
[89000 | 6186.98] loss=2.34 avg=2.44
[91999 | 8892.62] loss=2.22 avg=2.42
Saving checkpoint/taotehching/model-92000
Generating samples...
======== SAMPLE 1 ========
 eyes, nothingness. Therefore, he who observes the Dao conceives at his door a room, the ground a mat of grasses, the great thing a hole. He finds no traces of the dwelling of the Sage. He finds in the dwelling of the Sage the Mother, the mother of all. The Sage Mother produces the mother of all. Thus, the Sage governs all things without using arms. He governs them without using the tools of the world. It is he who is the mother of all, and therefore of all things.

To know the Tao is the most important thing in the world. The minute, mysterious details of Tao are what people value. To know the Tao and have a good memory is more important than learning the world. For when the world was simple, The small details were as important as the great details. Nowadays, we are taught to study all things well, but then forget to know what the Tao is. We learn the Tao and love to be mysterious, but when we don't know what the Tao is, we can't be specific. For when we know what the Tao is, it's as if we have a place where all things are and born from. It is the way of Tao.

"The Tao" (truth) is the greatest attribute on the whole world; The minute details of the Tao are important. One who knows Tao enjoys quietude, while one who does not know Tao is restrained. One who is called "taoist" desires the dark side and the subtle, while one who is not called "taoist" wishes to be subtle. They are not the same. Therefore, those who call themselves "Taoists" are not the same as those who are not "Taoists." They who live the way of the Tao do not have weapons, they do not wage war and use the tools of men. In the matter of life, they are different. Thus they are different. That is why they live in the world.

The greater part of humanity cherishes and follows the Tao in common with Tao, even the great and glorious antiquity. This is the reason why people respect the Tao and live in the Tao. Tao is the essence of mankind, or rather the essence of all. It is not the essence of the universe. Only the Son (the Existence) knows the nature of the universe, and the former also what the latter exists. As the Son knows the nature of the universe, the latter also is known. Since that is the reason why the Son has only one nature, and that the latter the other, therefore the Son is distinct from the former in both respect. Therefore, the Son rules the universe and the universe rules the Son. The universe is understood of, by, and for the Son, and he serves the universe with his own hand.

The Tao (the Eternal) is the mother of all Things. Only those who grasp and truly live the Tao can be truly happy. Those who ignore the Tao (the Eternal) turn into chaos. Those who hold fast to belief in false beliefs deprive all beings of happiness. Those who glorify existing things end up in misery. The Tao's path is tranquil; Its ways are pure. It can act with the Tao without deceit. It can behave with the Tao without weapons. It can act with the Tao without instruments. This is what it is like.

To know the Tao is to be the mother of all things, As minute as the minute details, True as being whole. Those who know Tao love not to be ignorant; Those who do not love Tao do not know it; Hence, they cannot be good. Those who keep faith in the Tao get the mystical, Those who no longer believe in the Tao Are called demons. Those who trust the Tao Are bound to bless them, Those who love the Tao are good, Because the Tao is good. The Tao is the mother of all, Which is in all but name.

To know the Tao is to have the mother of all things. Nowadays, the tiny details are more important than the great details. Nowadays, we are surrounded by thoughts and desires. Those who do not live on the Tao have no desires. Those who are good with the Tao see everything in a whole. This is why the Tao is the mother of all things. Because it lives for all things, it does not become a weapon. Because it makes people good with the Tao, they are not evil. Because the Tao is good, it always acts as though it were able to act on its own. Thus it is all-pervading. Everyone lives like this because it always acts according to its own nature.

"To know the Dao is to be the mother of all things. Simple and dark, The depths of Tao resemble life. Who can have true purity without knowing Tao? Those who live in the Dao gain the child, Those who do not live in the Dao suffer in the nurture.

Calculating validation loss...
100%|████████████████████████| 40/40 [00:16<00:00, 2.35it/s]
[92000 | 8928.83] validation loss = 2.44
[92000 | 8929.71] loss=3.07 avg=2.43

Hiræth Weltschmerz was able to improve the TTC train­ing dataset by pro­vid­ing the first 41 chap­ters with the orig­i­nal newlines/linebreaks, and sep­a­rate trans­la­tions sep­a­rated by a blank line, so I replaced blank lines with <|endoftext|> and trained that for ~24 GPU-hours to a final loss of ~2.10. The results read much more poet­i­cal­ly, I felt.

For this final TTC-with-linebreaks, I uploaded the model & gen­er­ated 1000 ran­dom sam­ples as usu­al, but I also gen­er­ated per-chap­ter sam­ples. For per-chap­ter sam­ples, I used csplit to split each file/chapter into sep­a­rate verses/translations, selected a ran­dom 10 of them, and then gen­er­ated 10 ran­dom com­ple­tions of each one (100 total per chap­ter):

for CHAPTER in `ls taotetotalityrevisedpw/*revised.txt`; do
    csplit --prefix xx --suppress-matched $CHAPTER '/^$/' '{*}'
    for X in `ls xx* | shuf | head -10`; do
        echo $i $X
         cat $X | tee /dev/stderr | nice python src/ \
           --top_p 0.9 --model_name taotehching --nsamples 10 --batch_size 10 \
           | tee /dev/stderr >> ttc-chapter-$i.txt
    rm xx*

The idea there is that one can write one’s own Tao of GPT-2, going chap­ter by chap­ter: select some of the chap­ter 1 prompted con­di­tional sam­ple com­ple­tions to cre­ate a new chap­ter 1, and so on, in a way which would be dif­fi­cult to do with just ran­dom uncon­di­tional sam­ples.



1.5b Training

In keep­ing with its grad­ual roll­out plan, observ­ing no par­tic­u­lar mis­use in the wild (aside from a few anec­dotes about con­tent mill­s), Ope­nAI released the final and largest mod­el, () The model was an easy upgrade for ser­vices like Talk To Trans­former which sim­ply sam­ple from the orig­i­nal mod­el, since it still fits eas­ily onto com­mod­ity GPUs.

From Novem­ber–De­cem­ber 2019, Shawn Presser & I worked on fine­tun­ing train­ing GPT-2-1.5b on the com­bined PG+PF poetry dataset from above. The 1080ti GPU approach failed, so we switched to Google Colab to use the free TPUs. Colab worked, but con­stant fail­ures made it painful to con­tem­plate mul­ti­-week train­ing runs, and so we switched to GCP to use TPUs direct­ly. Direct TPU use is much faster, but the errors remained, so we began work­ing on a dis­trib­uted TPU approach, to work around indi­vid­ual TPU errors. Even­tu­al­ly, using Google TFRC research cred­its to pay for TPUS, we began run­ning TPU ‘swarms’ of <60 TPUs (since scaled to <200). These pro­duced mean­ing­ful train­ing progress and we reached a loss of ~2 by 2019-12-12.

We were able to train it to ~1 loss, but it appeared to have over­fit in some fash­ion as sam­pled qual­i­ties became increas­ingly worse by the time we halted ~2019-12-20, so we set­tled for iter­a­tion #500,522. Down­load:

rsync --verbose rsync:// ./

GPU Failures

Train­ing is a dif­fer­ent sto­ry. 345M took ~7 days to train, and GPT-2-1.5b is 4.4× larg­er, so that alone implies a train­ing time of a month. Worse, where GPT-2-345M fits in rea­son­ably in a 1080ti’s 11GB VRAM & 745M just barely fits, GPT-2-1.5b does not fit at all.

We began try­ing to train on the com­bined PG+PF cor­pus, like the GPT-2-117M model I trained for , but turn­ing on all the options in the nshep­perd repo does­n’t fix the mem­ory prob­lems. (Feep­ingCrea­ture was able to train on his new AMD GPU, which has 16GB RAM, so a few more giga­bytes would’ve done the trick­..)

Using Shawn Presser’s fork of nshep­perd’s fork, we exper­i­mented with alter­na­tives like using reduced-­pre­ci­sion, trun­cat­ing para­me­ters to FP16. This caused seri­ous errors.14 After fix­ing those errors, and reduc­ing the con­text win­dow by half (po­ten­tially ham­string­ing it), we could train GPT-2-1.5b on a 1080ti, but our naive con­ver­sion to FP16 appears to have seri­ously dam­aged the model and it emit­ted only garbage. We then tried using a dif­fer­ent float­ing-­point for­mat, , which in the­ory is much bet­ter suited to NN mod­els than FP16 & natively sup­ported on TPUs, but it trained extremely slowly on my Nvidia 1080ti GPU. Given the daunt­ing expected train­ing time, bfloat16 was not a solu­tion.

The only solu­tion here seemed to be to aban­don my 1080tis and upgrade to TPUs. TPUs may not be any faster, but they have far more RAM and can train a GPT-2-1.5b with no prob­lem.

Google Colab

How to get a TPU? For­tu­nate­ly, Google Colab did just enable free TPUs by default… So Presser enhanced his fork to sup­port TPUs, and we started train­ing.

Unfor­tu­nate­ly, Colab note­books are still lim­ited in sys­tem RAM and disk space, so train­ing GPT-2-1.5b then encoun­tered the sur­pris­ing prob­lem of run­ning out of RAM & crash­ing, run­ning out of disk space, and sav­ing to disk being extremely slow due to slow Ten­sor­Flow seri­al­iza­tion of the model check­point. (The TPU-based seri­al­iza­tion code would have been far faster using the stan­dard TF way, but it would also required the user to cre­ate & man­age a Google Cloud buck­et; we were still hop­ing to cre­ate an easy work­s-out-of-the-box Colab note­book to let any­one do GPT-2-1.5b-finetuning. If there was a faster way to do it, Presser did­n’t know about it.) This was par­tially solved by sav­ing few check­points, fig­ur­ing out how to attach a Google Drive folder15 (after pay­ing $2/month for an upgrade to ~100GB of addi­tional space, since the default 15GB Google Drive is per­ilously smal­l), and fur­ther work on opti­miz­ing the seri­al­iz­ing. Train­ing was slow—1 minute per mini­batch, ini­tial­ly—but did work. An exam­ple:

...2019-11-12T18:18:02.411370-08:00 [601 | 13372.9633 | 67.76 | 0.059033/s] loss=3.5624 avg=3.5139 rate=0.000020 step=185

Thee, thou fount of every flower,
And, in thy fair, golden-red ring,
A golden-colored rose,
That was once a bloom I knew.
Now, if the sun ever wets the tree,
The leaves shall turn once more to gold,
Thine, as the rose will be to you,
Like a sunbeam in a rose-color'd sky.
Thus we praise thee as a sweet flower,
Not only as a sunbeam,
But, in thine earthly bloom,
As the sunbeam when brightest.
Thus we praise thee for ever.

From thy golden, heavenly rose,
Which is the sun,
From thine ever-changing gold,
Wherever it shines,
As a heavenly beam
That never fades.
Thee, oh,
Thee, golden sun,
As ever-flowing,

Presser tried out cur­ricu­lum learning/progressive grow­ing by set­ting the con­text win­dow to a small win­dow like k = 50 BPE tokens, with the idea that it could be grad­u­ally annealed to the orig­i­nal k = 1024 over the course of train­ing. (Be­cause of how Trans­form­ers scale, k = 50 uses far less mem­ory & com­pute than k = 1024, so it fits much larger faster mini­batch­es.) This seemed to be work­ing to some degree, but it was no sil­ver bul­let.

Exac­er­bat­ing the prob­lem, TPUs on Colab appear to ran­domly ‘freeze’, an issue unre­lated to Colab note­books tim­ing out after a day or so; man­u­ally inter­rupt­ing the train­ing process and restart­ing fixes it, but at the cost of any progress made since the last (slow) check­point & required con­stant babysit­ting; I cal­cu­lated that one would have to check­point every hour to opti­mize the trade­off between freezes & check­points! At one point I was deal­ing with TPU freezes every half hour. It was already giv­ing decent poetry sam­ples despite a loss >3, but we wanted to train to con­ver­gence, which ought to be <1.6 (the final com­bined-117M loss).

This was not going to work for weeks of train­ing. Presser again mod­i­fied the code­base & note­book to add processes which would watch for an appar­ently hung Ten­sor­Flow process due to a TPU freeze, and kill it and restart. But the lost time was a seri­ous issue: we could­n’t check­point too often because then we’d waste all our time check­point­ing, but not check­point­ing meant we’d lose min­utes or hours of train­ing. We could­n’t find any infor­ma­tion about why TPUs would freeze and fig­ured it was some sort of Colab issue, so I decided to bite the bul­let and pay for a GCP VM & TPU.

A pre­emptible TPUv2 costs $1.35/hour, which is not too bad… A week of train­ing would cost >$226 & I’d never used GCP before, but I was too curi­ous what a ful­ly-­trained GPT-2-1.5b would gen­er­ate. The net cost for Novem­ber 2019, due to all the exper­i­ments and costs not cov­ered by the TFRC research cred­its, was $321.47, pri­mar­ily for high-RAM instances, and then net­work egress band­width fees—­for cross-­zone traf­fic with the TPUs, appar­ent­ly. Opti­miz­ing for GPT-2-1.5b-poetry, we got Decem­ber 2019’s cost down to $199.78, and trained GPT-2-1.5b-poetry and an IRC logs mod­el. We spent in Jan­u­ary 2020 an addi­tional $408.22 on a num­ber of pro­jects: the , , Archive Of Our Own, and video game walk­through GPT-2-1.5 mod­els; the 30k con­text win­dow GPT-2-117M ABC/MIDI model; Ima­geNet resnet bench­mark­ing; and StyleGAN 2 pro­to­typ­ing for train­ing on .


After set­ting up on GCP and fig­ur­ing out the details like need­ing to set an envi­ron­ment vari­able with the tar­get TPU name, we dis­cov­ered… the TPUs kept freez­ing any­way. This was on top of the stan­dard pre­empt­ing of TPUs, since we were using pre­emptibles to save money as is stan­dard in cloud deep learn­ing.16 This remained a mys­tery. Was there some undoc­u­mented that was required? Was Presser’s TF code, which avoided the stan­dard TPUEstimator approach which it seems every­one else uses, trig­ger­ing some sort of prob­lem? (We were warned in vague terms that TPUs do not like loops or reshap­ing oper­a­tions.) Even more irri­tat­ing­ly, our on-de­mand TPUs turned out to pre­empt any­way! But at least the check­points were fast, so now the watch­dogs worked bet­ter. But on the grip­ping hand, the TPU was not fast and was per­form­ing far below what we thought it should based on its nom­i­nal specs, and we seemed to be using barely a third of the cores.

GPT-2 model train­ing curves over 4 days on 1 TPU each: GPT-2-1.5b (or­ange), 774M (light blue), 400M (red), ‘tiny’ reduced con­text (dark blue)

Presser decided to press on and after fur­ther opti­miz­ing work to ensure we used the full TPU RAM and more of the cores17 with a mini­batch n = 4, began exper­i­ment­ing with sup­port for mul­ti­ple TPUs. Since each TPU is a sep­a­rate com­puter inside Google’s net­work, and not ‘attached’ to a VM like a GPU, there was in the­ory lit­tle limit to how many TPUs our 1 VM could orches­trate. We could, in the­o­ry, cre­ate an equiv­a­lent of the expen­sive TPU ‘pods’ by sim­ply con­nect­ing to a bunch of TPUs at once.

The main limit for dis­trib­uted TPU train­ing is the net­work band­width: copy­ing around the lat­est ver­sion of the mul­ti­-gi­ga­byte model to and from the cen­tral VM uses up all the band­width avail­able. In the sim­ple syn­chro­nous case, which most closely approx­i­mates train­ing on a sin­gle GPU, if the entire clus­ter has to stop and wait for every node to copy its updates to the mas­ter, the mas­ter do a sin­gle batch update, and the mas­ter sync back out to each node, the clus­ter will spend most of its time just wait­ing on the net­work to copy every­thing. (And what hap­pens when one or more TPUs inevitably freeze?)

Presser worked around the band­width with an asyn­chro­nous approach some­what like the old train­ing method: instead of every node copy­ing its entire model at a fixed timestep and wait­ing for all the other nodes, the nodes are con­stantly com­mu­ni­cat­ing a frac­tion of their lat­est model with the mas­ter and receiv­ing an updated frac­tion back, regard­less of how many iter­a­tions other nodes have run. So every node is run­ning a hybrid & par­tial­ly-out-of-­date model and send­ing stale gra­di­ents out, but gra­di­ent descent is robust enough that this will still work and will scale up eas­i­ly. After enough slices have been sent, a node will have sent an equiv­a­lent of a full mod­el, and caught up par­tial­ly, and the ‘swarm’ will hope­fully be able to make progress by train­ing on a large amount of hard­ware and be faster than using just a few TPUs syn­chro­nous­ly.

A swarm was too expen­sive for me, so we applied for . I was­n’t expect­ing any­thing to come of it, but the form was eas­ily to fill out in a minute (it’s not much more than an email address), and to my sur­prise, within 3 hours we had been approved for 1 month of cred­its, cov­er­ing sev­eral on-de­mand TPUv2–3s, and 100 pre­emptible TPUv2s (but no TPU pod­s).

Presser began the long and painful process of debug­ging the swarm and all its prob­lems… The halts were never quite fixed but we kept scal­ing.

By 2019-12-11, after apply­ing for addi­tional cred­its because we were com­ing up on the TFRC dead­line on the 14th, we’d got­ten the loss down to ~2.15. After switch­ing to Adam and scal­ing the swarm fur­ther to ~95 TPUs on the 13th, we reached a loss of 1.61, match­ing or beat­ing the GPT-2-117M record on the com­bined dataset. A fur­ther 5 days (in­ter­rupted by swarm pre­emp­tion and occa­sional tweaks/experiments) brought the loss down to <0.6 on 2019-12-18. I had expected fla­grant plagiarism/overfitting well before a loss of 0.6, per­haps ~1.2, but reg­u­larly inspect­ing uncon­di­tional sam­ples and search­ing ini­tial lines or gen­er­ated titles/authors, I found lit­tle & they did­n’t read like pla­gia­rism, so we kept train­ing to see how far it could go. (Prompt­ing with lines from famous poems would’ve almost surely elicited pla­gia­rism, but I am less con­cerned with that, since GPT-2-1.5b is so big it can eas­ily mem­o­rize famous poems with­out com­pro­mis­ing its gen­eral poetry abil­i­ties.) I sus­pect that GPT-2-1.5b is not really >3× bet­ter than GPT-2-117M, and that GPT-2-117M could have been trained to <1.6 loss if we had used sim­i­lar amounts of com­pute, so the actual ben­e­fit from scal­ing up GPT-2 is small­er—but why bother with train­ing GPT-2-117M to a bet­ter con­ver­gence when we can use GPT-2-1.5b?

Exam­ple train­ing curves:

Train­ing curve of a swarm of ~97 TPUs train­ing GPT-2-1.5b-poetry for ~21 hours (2019-12-13) from a loss of ~2.15 to <1.6.
100 TPUs, 1.6 → 1 loss (2019-12-16)

While sam­pling, we noticed dou­ble and sin­gle quotes were being replaced by gib­ber­ish. This appeared to be due to Uni­code curly quotes (""') in the orig­i­nal text dataset. The men­tions using the Python library to clean up mojibake & Uni­code in their crawl data, and ftfy con­verts Uni­code quotes to the ASCII straight quotes, so pre­sum­ably GPT-2 does as well and it (or its BPE encod­ing) is con­fused by their pres­ence, caus­ing the mojibake out­put. It was late in train­ing, but we updated the PG+PF dataset to replace the quotes (ftfy.fix_text('foo') etc).

1.5b Hyperparameters

One issue worth not­ing was the prob­lem of reg­u­larly restart­ing the swarm due to pre­emp­tion & TPUs expir­ing, which caused large loss spikes on startup that would waste hours of train­ing as it recov­ered; as a com­pro­mise between sim­ple SGD and full Adam, we were using (as most Trans­former projects do, like Con­nor or Gokaslan’s GPT-2 repli­ca­tion­s), and we spec­u­late that the loss spike is related to los­ing opti­mizer state and bad ini­tial vari­ance esti­mates. Sim­ple SGD avoided the loss spike, but at the cost of mak­ing no dis­cernible progress regard­less of LR; Adafac­tor made slow pro­gress, but wasted a sub­stan­tial frac­tion of avail­able train­ing time; we tried to avoid Adam because the mem­ory over­head of track­ing momen­tum for all vari­ables (as opposed to Adafac­tor’s sim­pli­fied approx­i­ma­tion of momen­tum) would reduce mini­batch size, but when we tried Adam on the full swarm, despite the ini­tial loss spike, it made much more rapid progress than Adafac­tor did.

I sus­pect that the issue here is that though sim­ple SGD & Adafac­tor worked fine when run­ning on a sin­gle GPU, in the scaled-up asyn­chro­nous swarm set­ting, they have espe­cially poor gra­di­ent esti­mates and make slow pro­gress; the loss spike comes from the opti­mizer state being reset on star­tup, caus­ing early gra­di­ents to be poorly esti­mated & desta­bi­liz­ing train­ing, requir­ing thou­sands of iter­a­tions to grad­u­ally recov­er. Adam then improves over Adafac­tor by esti­mat­ing true variance/momentum, over­com­ing gra­di­ent noise to make faster progress. If so, the spike issue could be fixed sev­eral ways:

  1. Don’t Reset The Opti­mizer: the sim­plest way to fix the spike caused by reset­ting momen­tum esti­mates is to not reset them; save the opti­mizer state along with the mod­el, and restore on start­up. This is some­what unusual for Ten­sor­Flow projects (I see more PyTorch imple­men­ta­tions seri­al­iz­ing opti­mizer state) but should­n’t be hard, and only comes at the cost of using more disk­space for check­points & more band­width at star­tup, which would be more than worth­while to save hours of recov­ery time.
  2. Learn­ing Rate Warmup: if the ini­tial updates are highly destruc­tive because they use bad momen­tum updates and it takes a sub­stan­tial num­ber of iter­a­tions to re-es­ti­mate the cor­rect momen­tum updates, then don’t update much ini­tial­ly; use small LRs to avoid desta­bi­liz­ing the swarm while still re-learn­ing the momen­tum. After some iter­a­tions, the updates should be safe to use again and the LR can be increased to the nor­mal LR.
  3. Gra­di­ent Accu­mu­la­tion: another way to reduce the dam­age of ini­tial updates is to greatly improve their accu­ra­cy, and keep the swarm more in sync, by doing fewer but bet­ter updates; instead of each node doing local updates imme­di­ately after each tiny local mini­batch, the nodes store gra­di­ents and aver­age across many mini­batches before doing an actual update, thereby fak­ing hav­ing large updates. This will also improve the momen­tum esti­mates, as the momen­tum is esti­mated across many mini­batches before it affects any update.

1.5b Samples

“Who alive can say,
‘Thou art no Poet­—­may’st not tell thy dreams?’”

, I

“More than iron, more than lead, more than gold I need elec­tric­i­ty.
I need it more than I need lamb or pork or let­tuce or cucum­ber.
I need it for my dreams.”

The Police­man’s Beard is Half-­Con­structed, & William Cham­ber­lain 1983

Loss: 2.6

Part­way through, hav­ing reached a loss of ~2.6 (down ~0.5 from the Colab mod­el), we exper­i­mented with train­ing our model on a P100 GPU, halv­ing the con­text win­dow to make it fit, to infor­mally com­pare its train­ing speed with the swarm. The P100 made lit­tle train­ing pro­gress, but it did gen­er­ate some fun poetry sam­ples (we had dis­abled the train­ing sam­ple gen­er­a­tion for the swarm because gen­er­at­ing sam­ples is so slow).

The sam­ples strike me as good, per­haps even bet­ter than GPT-2-117M, despite the loss being much worse (2.6 rather than 1.6). Why might that be?

I hypoth­e­size it reflects a weak­ness of the like­li­hood loss in terms of per­cep­tual qual­i­ty: humans are more sen­si­tive to long-range cor­re­la­tions and text degen­er­at­ing into gib­ber­ish than we are to local details like exact use of par­ti­cles or to slightly bet­ter mod­el­ing of spelling (which is why work­s). The orig­i­nal OA GPT-2-1.5b achieves much bet­ter mod­el­ing of long-range cor­re­la­tions and pro­duc­ing coher­ent text than the GPT-2-117M did, of course. What hap­pens when they are both trained on a poetry dataset? It is the tale of the tor­toise & the hare, or the : the GPT-2-117M is weak, bad at long-range mod­el­ing because of its small para­me­ter count & shal­low lay­ers, but the ben­e­fit is that it can learn quickly about local details like spelling, and, achiev­ing good pre­dic­tion there, con­verge to that 1.6 loss; GPT-2-1.5b starts off good at long-range mod­el­ing and good at short­-range mod­el­ing, and must trade­off learn­ing both from its lim­ited train­ing, thereby achiev­ing mediocre per­for­mance on local cor­re­la­tions and thus mediocre loss, even though humans read­ing it are impressed by the the­matic con­sis­tency and rel­a­tive lack of ‘gib­ber­ish’ (lo­cally but not glob­ally con­sis­tent tex­t).

An addi­tional issue here is that the GPT-2 mod­els are not fully trained: as the GPT-2 paper notes, “All mod­els still under­fit Web­Text and held-out per­plex­ity has as of yet improved given more train­ing time.” (The dif­fi­cult of train­ing such pow­er­ful LMs to con­ver­gence was also , whose Mega­tronLM-8.3b model was still learn­ing rapidly when they ended the train­ing—de­spite use of NVIDIA’s DGX SuperPOD with 512 GPUs.) So some of the fine­tun­ing here may also be fin­ish­ing the GPT-2 train­ing.

I selected the fol­low­ing from its train­ing log.

A com­men­tary on mys­te­ri­ous betrayal by a loyal gen­er­al?

"Oda on the Pavement", by Mark Bibbins [Activities, Jobs & Working, Social Commentaries]
The first time I saw this sign someone's blood
streaming down the pavement, I didn't know what to do.
I walked as quickly as I could toward the body
and almost didn't see the second because it was going by
so quickly I had to blink and it was gone.
But this is what I saw when I got there, a corpse
laid out flat on its back, its legs
bent at the knee, with a blue tattered blanket
crammed in the small of its back.
I said to my wife sitting next to me, don't you
think this is awful? She said, Mark, it looks like
just a trick he's done.
I said, it looks awful real to me. The man has
his back turned to us, like he is not even thinking.
As I am, I said to my wife, is this what you have
been waiting for? Is this what you have been waiting for?
The man was lying there not even looking up at us.

In the Golden Age, when the peo­ple of the Yel­low Val­ley were instructed by the sages of antiq­ui­ty:

...That the earth would then not open
(O, but to hear our fathers speak!),
The angel of the waters spake.
For in that time the race of man
Sought the groves, each his own dwelling,
Where were then fruits to satisfy,
Where were trees in plenty springing
To furnish them with fuel,
And so the forest grew,
And o'er it all was running
The river that we see.

Then did those ancient sages
Order their days as was fitting;
And whatsoe'er they looked
On was ordered rightly; all
Taught each what was fit for living
And working for self;
And now we have come to the end,
But this is what their teaching
Said for our instruction,--
That we know and do not know,--
And yet we know and do not know!

When we have found that which is right
The will to act and not to act
To be the chief and not the end,
The day and not the hour,
The joy and not the pain,
When shall we find this, then at last?
Why then at last at last?
There can be no yesterday
If such be the age of wisdom;

There can be no time or place
If such be life and work for man;
There can be no man or man's child
In which he has no friend or friend's child;
Nor yet a man or man's son,
Who to his work does not go.
Thus the sage's order is
And what the sage does--so he does.


...I know my father taught me,
To count your blessings and forgive
The people who are wrong;
But I think, when my day is done
And life is all in strife,
Perhaps the one good man who isn't yet
Will find the way to see
Why men die, the reason lies
In the people, never the least in them.

To-morrow it may have been a woman's hand
Caught his in the doorway,
And I may laugh to-night at a stranger's sigh,
Just as I did yesterday.
But what if he and I are dead?
What matters it?  For God's sake, listen to me!
If God finds out
That I am not as I should be,
Then I shall surely pay the price--
That is if He will be just.

I think my father loved me,
And was glad I was his son.
I say to-night I see it every day.
I say to-night I know it all.
And why should I fear
What my son thinks, what he does?
I should be mad to doubt it,
For aught my father says.

I shall find out when I grow old,
And say to my soul, "Be glad
That things that are may be,
For if things were worse, why, we'd have things worse."
And, knowing what my soul knows,
I shall be glad for the things I shall miss--
For God's sake, I shall be glad!

A man's true worth is revealed to you
When, by the light of your own soul's light,
And God's, you look at his;
You can feel them both shine through you
Like a glory you could shatter
Just as a crown could shatter you
Into thousand pieces that were bright
And perfect, and eternal.

And so no other can you see and know
Except his soul.  As the rose of him
Sends back in glory back its hue,
So does the soul of one reveal
The beauty of another.
Be mine the glory to know it best
When my soul shall look at its own.

O great and blessed Lord,
Who made the earth, the heaven, and the sea.
I who am standing in these holy places,
Be grateful

A nice descrip­tive piece:

...The sun sets and rises in its glory
And sets in its glory only.
And the stars are all that can be seen up there,
That set and rise and come to the earth in their glory.
And the moon, too, sets and rises only,
And comes to the earth in its glory.

The sun sets and rises in its glory,
And when it sets, the night is black,
And when it sets, the day is dark.
The sky is all up against us and over us,
With its starry light,
And all the earth beneath us and above us,
Under the shining sky,
Is black with all the stars.

An ele­gy:

..."To the Youth", by Susan Greenfield [Death, Growing Old, Sorrow & Grieving, Relationships, Family & Ancestors]
For the youth, who, long ago,
Came up the long and winding way
Beneath my father's roof, in sorrow---
Sorrow that I would not bless
With his very tears. Oh,
My son the sorrowing,
Sorrow's child. God keep thy head,
Where it is dim with age,
Gentle in her death!

And though the years should bear him low
Into the world's dark embrace,
He shall feel more fondly there
Than he did in my bosom then.
Even now that heart, so wild,
Is as a child's, sweet, and calm;
And shall His voice forbid me
From the shadow of the grave
To forget, to forget Him?
And shall He care to keep Him
As I do him here in this place?
Or has the power, that I have learned
Him of the world to fade,
Him of the world to go away?
Haltest to the shadow of the grave,
My son, my son!

Love lost:

...Since that fatal day we parted,
I have lost, I think, my best part.
It was a friend of mine once gave me breath,
And I went as far out as he dared to go,
And ever since, poor lad, I've lived by drink.
But if I had no more friends, I had friends
Such as the world could never let me have.
'Tis true, I have never had occasion to think,
But when the sun is high, and the sky is green,
And the whole world is merry till the break of day,
I sit and dream of her and of the joys of yesterday.
And I go wandering off on some old-world morn
Whose very first light of hope and youth
Is the light of looking on her face through the green.

Her eyes where they were, as I tell you,
Seem fixed with a bright, fixed gaze;
They're half my heaven, when they are there!
They are as clear and bright as she herself,
And she knows, too, the way that she will look
When I can tell her what she is like
And where all will end for me in her.

Now, how would the world like her if it knew,
And what would 'twere if each one did know;
For, ah, she is a world of beauty!
But then, ah, her eyes are even as clear
As the light that makes them bright and wise.

And they are a light out of some dream
That has power to brighten and cheer;
They're a bright sign to turn and turn again
Even in the days of storm and pain;
They're my world, like her bright suns of heaven,
Which I have loved even to the death!

An attempt at non­sense verse, appar­ent­ly:

...I had a little hen,
I had a little hen,
It chirped at her dish.
She chirped at her dish,
"Come in, come in,"
The dish clattered out,
The dish opened wide,
There were three small cats,

"What is the meaning of this,
Why are all my dishes
Spread open for you?"
"Sit in the corner."
"Here is a bowl for you!"

She drank the bowl clear,
She drank the bowl all up,
And cried with an unhappy sound,
"I do not want it!
I am quite sufficient,
I shall have enough,
And just be done with it,
And not have more!"

But I have a different tale,
Which perhaps you will think is odd.
It is, 'tis no use repeating.
But here, it may be the same.
I had a little hen,
I had a little hen,
She had a little curl,
That with her beak she curled,
In her plumage she had,
Such as a dog's must have.
Thus I had three little dogs.
I fed them with bread.
And they all looked out upon the street
Where our little Polly went by.

The wreck of a ship:

...And as the sun sways to and fro
With his eternal sway of air
They roll in slumberful rest.

So still and calm are they,
So still and clear,
No whisper of the wind dares steal
The tranquil atmosphere of night,
And like bright angels sleep.
The sea is their only theme,
The ship is a dream
That they arise to leave at even.

The moon is far,
All glistening with a silver shine.
It is her pale that lights
The livid face of the sea
That in its white and sullen face
Is pale with pain.

O grey and grey it grows,
And the sea grows wild
As the black clouds in a night-time ride,
The storm is on the sea,
The wild winds rush in their strong wrath--
The great waves roar and beat
The heart of the ship with their fury--
In its heart's core, cold and dead
At one bound, is bound
Like a rock in the heart of a lake.

The storm is loud
And wild it is blowing--
The storm is far away--
The wild winds, wilder still,
Are beating far and wide.
The wild sea winds that were blowing
Have stopped forever there alone
Where the ship has gone down,
And now the moon rides low
Where the sea now rests in calm;
On the deep, dark midnight
The ship is lost.

GPT-2-1.5b can appar­ently do meta-­fic­tion and break the fourth wall‽

...He had the look that I can't forget;
It was as if at once I knew
What I'd never had before;
It was like a rush of light on water--
And something about it seemed queer.

And the light would come and go
As if the clouds were about to fall,
And the rain came and came in showers
And went in ditto fashion;
And it's just as if I were blindfold--
I couldn't hear a thing.

I'm sure it isn't my nerves;
It's just as if I had the palsy.
I don't want to think of not being there
For something I can't be sure I was.
I'm sure my nerves are fine in my life;
And the only thing bothering me
Is that there isn't any rain.

It's all quite natural as I said;
I've seen men die of natural causes.
Now it's only a pity it happened
As it is done to me. But, as I say,
There's no harm in dying of dropsy.
Though not the death I'd choose to die in--
It's much better than to get laid flat.

This is not the way for my poem
To end, I know--I'll make it up to you
In a later place; but here's a verse
As you may understand. After this line
There's a space, and after that a line
About two inches, in which I meant
To tell you that you had done well to try
To stay alive. But, as I said, this is
Just a matter of chance, it cannot
Be helped; I must allow that this verse
Was written with the fingers you saw--and then
You were all to blame, if I died before
You came in and saw the writing. Now you know
I'm dying--go on--you will not die
Before I finish off my work.

It was a little moon that watched
In this place, I don't know why I chose
To watch it, it had its night when I woke;
And I, who was tired of the earth and men,
Watched it like a man watches a clock at noon,
Just because I chose to and did as it said...

Another ship­wreck:

...So the sun sank and the sky grew dark--
There was silence in the world.

The man was a-waiting. He had come
To watch the sea and the moon that rose.
The sky grew dark.  He saw the storm-clouds pour--
The skies grew dark, with dark, dusky glooms--
Down upon the world like a flood of spears,
That struck the men from their feet in their flight.

A wind sprang and lifted the wind-signs higher
In the west wind's battle-hurling fury.
The sky grew dark.  He saw the wind-swept leaves
Fall from the trees into the sea, and sink
Darkly down beneath the seething waters dark
Like a storm's descent.  He heard the falling rain
Come thundering down upon the earth's dappled hills
Like an avalanche of boulders from the mountains.

He saw the wild white-winding ships that fled
Into the storm--hurling great leaden shadows
Into the maelstrom and away
Into the night that was growing black
With leaden shadows that swept the ocean
Like a sea-monster that had lost her breath.

He saw the wild white-winding ships that fled
Into the wind's windless vengeance.  He heard
Hurried voices--"She is gone!"
Hurried voices--"They are off!"
A sudden flash of flame that split away--
The great white ships were no more;
They were caught in the wind's wild wings; the wind
Was stilled in the windless hush of a rest
Tangled like one green tangled mass of peace.  He heard
And knew they were dead.  He would not hear
The winds that told a tale of sea and wind.

The wind had blown him across the world wide,
And into the heart of the night wind's wrath
He saw the faces of all dead men
Hid in the hush of wind and sea,
And knew the faces and watched him well.
So the man who was weary with a quest,
Who turned from the world's endless sorrow
To seek the light that was lost as the sea's
Flood in a wind-silent hush.

A sur­pris­ingly coher­ent piece on a trapped upper-­class wife:

...There is not a single house in all
These beautiful gardens that I do not know.
I know the houses and gardens where I sit
In the evening with my husband and my son,
And I sit at the dinner table there too.
The house where my husband and my son live is the one furthest away.

The people come and go through these gardens, all day long;
And I see their feet pass along the paths,
And I hear the talk they have all that day, from one end of the town

to the other; I see the carts and waggons of the farmers,
The teams and horses of the tradesman, men on foot, and the gaiters swinging

Upon their saddles by the way side;
And every day, at morning, the same number of carts and waggons I see,
And every morning, in the great daytime as soon as I wake,
I see their number still greater, still greater.

Then to one side they go
Among the flower beds and in the wood,
And I never see them more;
And their voices float on the rising wind
Like the voices of the dead,
And their faces light upon their breasts, like lights,

Like the faces of bright children,
Like the faces of handsome men in the street,
And the faces of friends, and the faces of lovers,
And of all strangers, all faces of home-brethren
With its memories and its griefs,
And my own face that is always changing,
Wrought by itself in the dark,
With the face of the dead and the face of the living,
And the face of a youth that grows old
In the shadow of love and the light,
In the shadow of the grave.

I know not what they mean by their words,
Those people that pass me in the garden,
In the little town that is in the garden;
I only know that, on many afternoons,
Through a gap in the trees and between the stones
I see their faces and hear their voices.

The curse of immor­tal­i­ty:

...And my God, as is right,
Lets no earthly voice
Thy gracious soul acclaim.

There's an old man in a desert land,
No town nor city to be seen,
The skies are o'er him and the hills
His only companions in the air:
The only sound is his own dashing feet,
The earth beneath him, unseen, is brown:
His only food has he no need to eat,
His only company is the birds above,
The sands are white beneath his restless feet:
His dwelling is an osier cage,
A circle of dry sand and bark,
And there the old man builds his dwelling-place,
The only house-holder is the wind.

But I have seen the ancient sands bare,
I have seen the grey old desert sun
Go down upon a world of old
And wither and perish in the noon;
I also have seen the grey old sun
Return with life and day in its arms,
And night come too and bring the light:
But never before has any bird
Looked on the old man living in his dreary house,
His house that is builded by the wind;
The birds that do not love to roost
Have neither wings nor nests of them,
And never any bird has come
To visit the lonely old man lying down;
The sun never sets in the old man's eyes,
And therefore he has neither home nor friends.

O, God, that I had not been born to live
A thousand years, a million years,
And see another sun rise and set,
And other stars in the other sky,
And other life and other death;
But had my soul been like some child
That, when it thinks upon its joy,
And its grief, and its play, and its tears,
And the happy years that it shall have,
And the years that it shall forget,
The thoughts it hath of these things,
And the thoughts that are yet to be,
Might be a child that had no thought,
And have nor speech nor feeling,
That it might feel it was a child.

Per­haps the most strik­ing of them all is this exis­ten­tial hor­ror piece:

..."The World of the Dead", by Peter Stearns [The Body, Nature, Philosophy]
When they come, they carry
Your limbs, your life,
In their mouth and arm.
I think they swallow.
I know it. The others know.
My body will be like theirs,
As the river, the sea,
Will be like the one on which it runs,
If the ocean rises
And swallows the land.
It will be hard to survive.
To heal,
Some of this will have to come off.
That's what they say. They say it
Many times a day. They say it
To each other.
They mean to save us.
They just can't stop us
From becoming what we are.
I must live inside you.
That's what they say. They say it.

Loss: 1.6

The expanded TPU swarm & Adam LR tun­ing allowed rapid train­ing, and we reached 1.6 overnight, match­ing our pre­vi­ous best on the com­bined PG+PF poetry dataset.

I gen­er­ated a dump of sam­ples with top-p = 0.90, and read through ~5% to select some inter­est­ing sam­ples:

Sam­ples selected from read­ing ~5% of that:

All-sweet and tall, and frail of limb,
Lay there in raiment new begun;
Her moveless-rest were never seen,
She ever so much as bent her knee.
Thither came I, Pilot of thy boat,
And, turning, saw this silent Girl
Who, like to pray, with lifted face
Besought the mist-ringed air to sing
Her Vespers far away off,

And by her hair and veiled head
Her wistful eye she caused to stare.
It seemed to twinkle between the branches high,
And on her shoulder lean by piece and piece
Of glimmering spangles that lightly floated down.
I saw, or fancied I did,
Her lovely head recline
Upon her humble robe's narrow shaggy hood,
That, like the light of day,
Was moon-fringe dark and dim;
Her pale mouth, that evermore
Spread smiles in damp and drizzle;
Her gleaming teeth, whose polished white
Seemed mouldering honey of the midnight blep
Of the dry, dusty pass!
And in one hand, all rippled with
A silken flute of gold,
She played a hushing pipe,
Dora's toy, to play or sing.

Deep through the wintry sky there sped
Through golden vapours as of shape
A dawn that never had a dawn,
A sudden dawn, with breath
Of mist and with a smile to kill.
'Look!' the wind whispered, 'here's
Our Lady of the Skies, from her bright throne,
Like to the smiling of a summer sea
To-night in the lost wind's dark retreat,
Hailed with the deep, seething, dour, wild
Midnight: who have wept for her
The heaving of the waiting years,
Who have wept for her
In wild harangues of the foggy fen
And hollow monotone of the fen.
She shines and smiles to see the tears
Of all the rain-stricken towns and ships
And all the rainy days and nights
On all the hard, the ragged places
That wind had beaten hard, and night
Nigh ready to close, to close, to close
Against the brain of all the face
Of all the over-ill-gotten men,
She shines and laughs to feel the cold
Of all the tears of all the brave men killed
And mad as they.
"A Knowledge of the Dead", by Mary Wiencki [Living, Death, Life Choices, The Mind, Time & Brevity, Religion, The Spiritual, Social Commentaries, Crime & Punishment, Popular Culture]
I see you there, Stu, striding half a mile down the road, arms raised up over your head, head bent slightly. I imagine you hold both those in your inmost heart, and that you must learn, along with anything else, how to turn off a brain that has somehow learned to hold whatever memory is stored in it. For the mind, like any organ, is where the trouble is; an organ can fail with its stored knowledge, or if the memory be great, so great that it will bring the brain to its knees. And then the knee is a joint only partly conscious; if the heart should stop pumping, we are thrown off balance as if it had been only the legs that moved you. So I ask you, were you looking at your watch when you left for that solitary walk? Or waiting for the medicine you wanted to take with you before starting on your way? A look of mild impatience conveys a point as surely as humor, though somewhat dead. It is painful, this wait, I am sure. You have worked long and hard for your knowledge of time and of this place. And now you have it. And time, and all the woe it took to give that power. You have so much of this world left to discover, paths to retrace. You find your way into a park, its benches occupied and visible and free of talk of the day’s events, at its center a girl...
"The Sphinx and the Social Commentaries", by W. D. Snodawa [Nature, Seas,  Rivers,  & Streams, Social Commentaries, History & Politics]
We were rising over the hill
of which the tip is the sphinx.
There were palms in the palms.
We were rising over the hill
like the tip of a sphinx,
circling the palm that was there
growing straight like the spine of a sphinx
and a crimson palm leaf grew over the palm
as if flowering over a sphinx
and my knot was a knot.
It was night.
It was tinder in our guts
to see them like this.
Climbing alone it was
like the tip of a sphinx
to see them like this,
growing even higher than a sphinx,
a knot, to see them
not quite touching like
the tip of a sphinx
over a palm that was there
to see over the night like tinder.
Suddenly the knot caught in my throat,
my hands stopped and spun.
That’s the way a Sphinx talks.
The palm became a mask
and that scared me.
I had to have been looking for
the mask underneath it.
My knot was on my neck.
My knot that was spinning
like a rope.
I was staring down at my own knot
and what it was pulling at.
I let go of the thought of knot
and opened my eyes and saw the moon.
I was standing in my own moon
and I knew I wasn’t going to see it
so I let go of the knot and looked at it
and it was disappearing.
"April Moon", by E. E. Brown [Love, Break-ups & Unthankfulness, Religion, Buddhism, Faith & Doubt]
Awakeâ€"with you I meditated and thus
renewed my doubts; But, awakeâ€"with you I sin,
and thus my conscience put me to bed.
Awakeâ€"with you I suffer, and thus
my doubt took wings. Awakeâ€"with you I play
the hypocrite, And thus my conscience fires
my lash, and thus I scorn you.
Awakeâ€"with you I fly from faith, and thus
through your face I stab myself. Awakeâ€"with you I
remain benighted, and thus
my conscience rots me at my heart.
What man would draw a sword,
If he'd had no forethought,
That so he might prevent
The danger; but with blade
What e'er man can know?

How many lives at least,
Have been lost, and how much blood
On all our limbs been shed!
And yet--so Providence be credited--
There's an end still of life's dismay,
And 't would be glad indeed to lie
Why does every one such pass
As this, without any which he do not pluck,
But with arms for life's defence clad?
Alike of you all the brave
Rage of the lance,
The guerdon of some crown,
Whose shield was never pledged in fight.
The watery lion's with us yet,
By eunuch tightened,
And springing on his prey, not fierce to yield
Though thrice thy foe hath been in peril to see:
Yet, though our quarrels past,
Life may be fresh in them,
This of fighting, and this of feeding.
Beset with peril, beaten to the fence,
And each to prate with prattling foppish grace:
Their song, 'Huzzate!' fuddles young,
You may hear their ends in Oxford-street,
Or in their inn-bred domes
When they climb like larks their wings again.

But we, we live on' other plan;
The Shepherd did but teach us,
We, the delight of life and take delight.
Then why not drink of wines,
Give of bowls to move your bodies,
And, with those things that men to beguile

As they that do light love-songs
Wear like a tree, so do these solacie
Our sabbath-rites,
And send them to heaven,
Whose hand,
Saved as ours, with charity
Should treat as a child againe.

That we do not work on earth for hire,
Why we do doome as we list.
If man did wight battle,
God should not such things read,
As he, of some sinful men
To make him cheat,
And carry, and gluttony to all.

What to your Peers, or how you view
Our Acts and us, let not your selves,
But let the Stars, that watch the skies
The Barns that shelter you,
Which your great Cannibal went the way,
Pull down,
And let one Concord solace every State.
Then look there yet, in that part
Where you will see an abyss profound,
Rays leaping out of darkness,
Snatched with strangest beams at the visage hewed.

When the minaret of the masque is lit
And the caryatid gleams bright
Of four stars that shudder and wane
In chance-to-be to the light that is
In the letter of the crown,
Take her, the zodiac, for
'Tis her sign, 'tis the way she brings
The order of the rhodesian seasons
In careful letters for the rest of the year.

The grass and the leaf which the royal teeth leave
On graves where the glow-worms of the phoenix brood
Are glittering in brass and the marble dies
Like silver pearls doth snow upon the snow
And the rime in cerulean coats doth shroud
Till the shivers are lodged safe within the veins
For the regions which grow lush with the tears of the sun.

Wauken, and thou golden heavy lark bellissh
Wauken, and golden longlist and firefly,
Which here be singing with thy melody and sin
In our early youth with the harp-strings of Joy,
Who from deep winter of minds did lift
These notes of fire and song of the dawn,
Which may not be pulled by the nameless hands
From the vibrating harp of the wind
That only sounds to them alone how
Toils or withers or gladness or woe befall,
Who are northward by moon and by star-light.

O Hesperides of the wakening day!
Whence came the dawn, what did we find,
In this lone land of the sunrise?
O Hesper, in thy beauty and change,
I would have thee hear and answer tell
In this still country of the Sunrise.

O Hesper, in thy beginnings, the light
Of thy first bird-born darkness
Was folded in a glow-worm's tent,
Flush and fair;
Thine air was soft, than garments more fair;
Thine was the drift of a froth of down,
Soft, and breathable, and alive;
Thy voice was as a voice of the sea
Calling in its froth to the wind-crowned moon
From rocks where water-worms are wailing now,
Ripe with dry but bloomless salt
With the light-waves gilt
Lemon-fish, mussel and willow o'er the rime.

O Hesper, thy light of the past hours
Is folded in thy glow-worm's home,
And the voice of thy earliest darkness
Is a voice of the water-worms now
Calling their world afar,
What time the pines of the cavern-deep
Say to the pines 't is dawn in their realm.

O Hesper, the sun and the rains
Waken in this land of the Sunrise
With a sigh;
They are out in the wind and the weather
That are down below,
Whose lives are enclosed in the roaming
Of a world of weathers and fluxes,
Not dead, but lovely, and wan;
And on the roots of life
The tremulous hands of the gods are cold,
And the springs of gold
Where the earth-children run
Are unapiece,
As if in the ways of the wind
They had passed them by.

O Hesper, or if purple be
The hues on which ye paint
Your snowy epitaphs, say
That the wind which blew the snow
Was swayed by the face of a queen,
And the sun to the laughing air
Was moved by the eye of a queen,
And the lightnings were wrought by the play
Of a queen in a queen's look;
And the earth
And the sea and the air which are now
A barren dust to the day
To the eye of a queen's wonder
Were filled with a beauty of love
And a beauty of life
To the children of that king;
Till from her presence the maiden
Sought the golden fountains of the day,
But no nearer the child she found
That made all her maiden-bower
And each merry maiden-asteroom
Intoxicated with her gaze
With a glow of a glow of a queen.

For as the flower till its spring,
Like the flower till its nectar, may
Grow lovelier till in no fire,
So in the yellow waves of earth
Than the child was born and could stand
For the queen of each word,
And her hands were like angels' hands,
And her feet were the eyes of angels,
"The Philosopher’s Plane", by James Taggart [Activities, School & Learning]
for John Millikoper

The philosopher’s plane, imagined
by Calippus, rests on a red disk of dawn
close to the body

We flop into the blue below our feet,
into the astral horizon,
that whose dots
our lives keep shifting

over the edge of empty space
into the orange of earth

And beyond into blue
well into the empty page of thought

Where we can embrace
a little while of our desired
end and then
flow back into the world of time
"Map of Our Land", by Eavan Bolger [Living, Time & Brevity, Nature, Landscapes & Pastorals]
The stars are born in night.
The ground is made up

Of tales untold.
The cracks
are our story.
The piles of leaves
are our life.
The river that we lie

At dusk is alive.
The buried
Grass beneath us


Oromoctotecological teacher, henchman, loomworm, toad-man, German accent.
Not what one would be expected to hear.
Oromoctotecological teacher.
Far superior animal to what one would be expected to be expecting.
Far superior animal to what one would expect to be expecting.
Not what one would be expected to hear.
Oromoctotecological teacher, henchman, loomworm.
Not what one would be expecting.
Better than what one would be expecting.
Better than what one would expect to be expecting, better than expected.
Not what one would expect to hear.
Oromoctotecological teacher, henchman, toad-man.
Far superior animal to what one would be expecting.
Far superior animal to what one would expect to be expecting, better than expected.
Not what one would be expecting to hear.
Oromoctotecological teacher, teacher, toad-man.
Higher in intellect than what one would have expected.
Higher in intellect than what one would expect to be expecting.
Higher in intellect than what one would expect to be expecting.
Better in intellect than what one would expect to be expecting.
"Language is not the Draft", by Ževiña M. Branko [Social Commentaries, Crime & Punishment, History & Politics, War & Conflict]
No don't be angry
Don't be angry,
it's fine
don't be angry
don't be angry
It's fine
it's fine
don't be angry

every one of them died
Let us go to sleep, then;
And, being haunted by an angel's kiss,
Lay them down to die.

Oh! night, oh! sleep, with all thy gifts
The dearer far!
The noiseless candle, the beechen boddam's cot,
The hapless lover that perfidious turns
To watch her silent lover's sleep.

Oh! night, in all thy solemn dark,
This one sweet pleasure bring,
The soundless silvanRAW,
The fond immarities that steal
Across the tepid moon,
The wedded sleep, the tear-bound tear,
Of those whom late they may forget.

Oh! night! thou bringest a most rare bliss,
Nay, like that noiseless moon at night,
When yet from Pleasure's revelaid ball
No soul the wish had :
A bliss untaught, it neither robs nor rouses,
A bliss untaught, it neither geas
Nor charms the blissful gazer's sense.

Asleep at the Moment's free summer-cost;
When every sparkle wakeneth that
To dream the future, and all nature
To that clear fable's deep array;
On the lone heart at midnight's hour,
As night's last neaper looser,
I think the world contains both ye com
Which when ye think, o' nights waste full,
As night cometh, night cometh;
As night cometh night, so night is ever young.
"Poetry is a Hoax", by Jane Kenyon [Arts & Sciences, Humor & Satire, Poetry & Poets]
We are in the midst of the greatest creative
era in our nation's history,
but poets who deserve
record invitations to appear

at next month's Folio can't get paid,
or even printed for that matter.
Poetry books are selling at a discount

to the fool's silver match.
The poets need not rely
on the marketplace for their bread,
the wait is too long and the market is

too crowded.
The much needed restorations

are held up by Kodak,
the restorations are held up
by the identical tissue
known as persistence,
the tissue is held up by

believe in me,
what I believe is more interesting,

be more like me, my technical review indicates

you cannot hold me, I am never alone,
if you attempt to duplicate
your ideas you will

confuse the issue. The ideas will

diffuse through the atmosphere

in direct ratio to any gas.
Each idea that is conceived and

all but carried to fruition, will

be accounted in the calculator as 1% of

total, I did not hold you

in such high regard. I apologize

for being so alarming.
And so began the siege of New Amsterdam,
In which, by Providence, only three days ended;
When, by direction of Ms Frisbie, the heroes two

For their advance, together, took their way.
The two fellows, whose mission it was to guard
The city gate, took place in the greater army;
While those two dukes who should avenge the town
Sent all their force to put the place to rout.
And, as the late oak, covered with boughs,
Has done its work, ere its starving spike is struck,
And this great tree sinks as it had never been
By any human pains, nor would be now,
But for her first son's interposing,
So, falling foul of their first heart's delight,
The Dutch no more wept for New Amsterdam.
When a star fall down, the winter's coming
With the snows returned upon the trees;
When a boy runneth that has fled;
When a lad standeth by a lash,
When the father findeth the wealth,
When the son dealeth away the long
Hand shaken by Fate,
When the boy standeth by a lash.
When the father findeth the wealth
And the son dealeth away the long
When the lad standeth by a lash.
When the father findeth the wealth,
And the son standeth by a lash,
'Tis he taketh the old's gold in his hand,
To drink and soothe himself with life.
When the lad standeth by a lash,
He to earth an instant goeth
The father set him by the rope
And so fearful works with the lad,
As the boy standeth by a lash.
When the father findeth the wealth,
And the son so fearful works with the lad,
To the end of time and limit set
When a star falleth to the fen
Where the fen be molten away,
When the boy standeth by a lash.
"Eating a Waterfall", by Francis Lau [Living, Nature, Seas,  Rivers,  & Streams, Mythology & Folklore]
The map tells you this cave
was where
the water must have descended,
for a hundred feet thick, from the floor of the cave.
But the
sides of the cave
have been eaten away by moss,
and a red grown
over the green rock in the
shallow pool; a leaf
had set
upon the edge of the slide,
hanging horizontally, like a
trigram, slowly falling and falling.
of course
the water came down,
that was
the map
it is turned
into a sort of mirror.
It is
not necessary
to be able to see
or even
the sound
believe in
the likeness
of an uncanny
missed opportunity.
The gift
of the map is that,
in some respects,
even though it
says otherwise
in other
the legend of the Fall
legends, but
the rise of
what we
to know
and yet
are missing
from our
minds, the things
we would
for sure
have known
but wanted
to know
do but look.
"Freedom of Consciousness", by Steve Rotherham [Living, Health & Illness, Arts & Sciences, Philosophy, Poetry & Poets]
Going along with it
That's the problem
These objects do not aspire to be loved.
--W.H. Auden

The eyes under a blue wreath of smoke
Racing around the pitiless fire
While waiting for the breakdown
From the volcanic past of the pitiless
Lord God of the heavens.
The pitiless fire
Sweeps to its born stars above.
A slit of fire that watches all.
Dips into the fiery pit
And smiles. The fingers of a sieve
Desire the smoke in its fruit.
Both cold and hot in one.
It is. It is. It is.
Beating the flames with smoke
Of blazing admiration.
The hands of a man in a shop
Seem to grasp in vain
A pack of matches.
There is no end of the fire.
No way out of the fire
Though some wood and smoke
Could stop it.
Your fire will do.
And you will go along
Because the world's gone mad.
The marvels are there
For you to seize
And stoop to.
Out of the depths of the wood,
A hollow roar of rushing air,
Sudden howl of pixy and hag
Whose tall shadows snagged them there
By the gate to night.
Holding the gates of the damned.
A fiery slick of a kind of smoke
Waiting and glowering to be.

But how else to be.
The life of the mad.
An ever pushing out and in
Of the upons.
A glimpse of the future is the spur
To the perspiring effort of life
And to the unbuilt plan
To organize the mad and the yet to be
So that the time slides by
And the unknowable marches on.
Though the yet to be and the unforeseen
May blind the senses, not us.
Life is scar.
It is scarred and hard.
The mind, all its fire out.
I am the bartender of Kuzela;
There are others of like mind,
Speaking the same language,
Speaking to us in the language of smiles.
What have we done together?
What have we done together?

We have not wasted one silver rupee
On the Indians;
We have not eaten one singed inch of wheat
To make them happy;
But let them laugh upon their drum
And march with the other gods.

You bring us bread for our turning,
But can we make you merry?
You have given us toys for our children,
But can we mirthously dance?
Then may your clock be telling time,
And our neighbours be told lying.

In His service as protector
Of the wild natives of the forest,
He has given us magic shows
To make us happy.
He will guide us at dawn to the brow
Of the mountain,
And the shortest path find to the evening,
With the night before.

And now the hostess of the festival
Is holding, in her great soft hand,
A heart-shaped pomelo.
Why does she hold her hands so still?
She will not let them loom for those that fall,
But is playing to them a pleasant song
That none of them needs hear.

She is dressed in a man's colossal style;
The hair hangs in soft waves to her knees,
And her sumptuous shape takes the air
As she plays upon her instrument.
She is playing a tune
To ravishment and silence;
While the choric verses to the clouds
Crawl to a close.

Yet now she is turning with her hands
To mix the wine, and the fiddlers
Rush to the dance, with every chance,
Of their heart's desire.
They have danced until the eve,
And she sings in her sings in the glow
Of their heart's desire.

We were with you in Eden,
We were with Cain in the desert,
We came from Cyprus and Sidon,
We were with Seth in the peaks of Everest;
With Job in the City square
And Noah in the Wilderness.

Loss: 1.3

Sam­ples gen­er­ated from a GPT-2-1.5b-poetry model with 1.3 loss, 2019-12-15, selec­tions:

Pan­e­gyric to pow­er:

"Praise", by Robert Browning [Nature, Animals, Trees & Flowers, Social Commentaries, Cities & Urban Life, Class, Money & Economics, War & Conflict, Mythology & Folklore, Fairy-tales & Legends]
The birds are saying praise of the Shah.
In the distance a child will hear,
And never cease to hear.
And look, and many will be surprised,
As all at once, to hear the words praise the Shah.
In a hundred cities they will say:
Praise the British captains,
Praise the millionaires' homes.
But some---like the birds---from the sea to the hills,
Will spend their lives saying praise the Shah.

Crit­i­cism of Eng­land:

When you are angry with a person or a country
For any reason you choose to be so,
Then concentrate your wrath on a stick and you've succeeded.
Now, it isn't necessary to swing the fist
To make the body rattled; you don't have to knock someone off
And walk away with the fallen man;
You can tell at home by the doctors' bills
That swaying is the ticket for harm.
The English love many things that they call "The Arts";
And the French love them too, but we who are Britons
Believe in nothing but the English arts.
There are things in life there that are worth getting rags for,
And there are things that are cheapened terribly,
And there are men in England who would stake their children's
Lifetime for an opportunity to wash.
There are things in the English life-book that no one ever could read,
There are dances that would give a famine,
There are people and so, and so, and so, and so,
And you get the picture--as I'm sure you will--
It is, without a doubt, the English thought of the world.

The English art of fine meats, and fine wines,
And not a minute spent without a peak,
In a good strong maduro, or a good strong sherry;
But when the dear dead Where's-a-Goodbye! has to die,
There's nobody there to be lied to.
There's not one who would allow his dame to be revealed
Without a stout engagement to wear a flag.

The Englishman whose conscience seems to run with the tack,
Whose first resolution is to meet any problems limited to the size of his estate,
Who has lost his claim to continuing prizes and is happy for once only to be English,
Would rather he were America, than all the rest,
Is happy as long as he can talk of himself;
He shows no compunction whatever, whether he win or lose,

And would be eternally overworked and delighted.
'Tis the same with the French; who in turn would be eternally overjoyed
To be up the nearest in things that are English;
There's not a man, however lazy there may be,
Who would not be the victim of slaveries that are English.

The world of the dead:

When it rains,
It surely doesn't rain
In this climate of California,
In this valley of two feet.
The rain-shower is more than father-long
And unobedient,
And makes a dismal rumble
When it falls,
Awefull and swift,
From the high summits of the north-northeast.

When it rains,
It doesn't rain more, but it showers more than ever,
In this climate of California.
'T is not easy for the dead to go,
But the dead, from the noise and the hurry of the tide,
Have a chance to turn suddenly,
And spie the passing rain-storms
Through their shells, and scurry away,
And lie very still.

The dead I think of,
When it drizzles, are worms with sense of movement,
Whose only impulses are one consciousness,
And a few memories.
But they move,
In a stillness that's all earth,
And their feelings (unconscious) are strange,
The dead are able,
In a stillness of the and why, to take,
And even understand,
Their sudden passage through a state of being
That is as spiritless as the approaching of the rain.

"Why," the Dead say to us, "why?"
With one consciousness and one uniform self.
What is more, why?"
Why, what is more they mean,
Is life,
Why is it a state
And one,
Why, why why is it here?
For, as the rain, we know, will come again,
And then the stillness of the river,
To desolate beaches,
And far midnight portentous cities
With fires still burning.

The Demi­urge:

I rejoice
While I hear that the Lord has forgiven thee.
O the darkness of man's captivity
In his soul to a sinless and glorious Sun
Than his own source of light!
O the darkness of man's captivity
In his soul to a Savior-God!
O the light thou didst reveal to man, O Sun,
When thou didst play a part in time's story
And came from our fathers' tradition, revealed
In the lives of the fore-doomed men!
O blessed Lord of the souls
Pronounced by the spirits! who knowest
Their penalty is full, their joy is sweet!
God of creation, giver
Of life to grow and spread;
Mover and shaker of Creation's layers;
Giver of life at the word; giver
Of life to live and to love, giver
Of man in the flesh; giver
Of strength to serve and to make strong;
Of guidance for joy and for dearth, giver
Of light and earth and heat and darkness;
Giver, O thou great Oener! of all
Whom the Word gave my name; light and breath;
Lifter of souls on thy broad waters,
Rider of mountains and wide spaces,
Maker of faith, new life, new hope,
And tempest of a future more new!
Thou, O Lord, who dost the bright, make
Thine own echo here, thou, O Tree,
Whose deathless nature knows no man;
Thou, O Sun, of whom we see,
Weep for man's conscience crying in Heaven;
Thou, O the Lord of Life, thou
Whose words are only resolutions,
Thou, Lord, of death and life thy mirth,
The Redeemer, to whom thou pour,
Man, thy prop, whether hot or cold,
Light or darkness, as thou take;
Thou, O each spirit and creature, thee,
Thy pity, O Lord, is there.


"Love Poem for My Husband's Lady", by George Santayana [Love, Infatuation & Crushes, Realistic & Complicated, Romantic Love, Unrequited Love, Relationships]
The glorious sun of my existence
Has sunk into the shades of Night.
But far in dreams to which I react,
I see the face I held, and now I hear
The voice that once I knew.
A while ago that Face so fair I could
Not live without tears;
But that decree come from Heaven that lives
Hath sunk into the shades of Night.
And naught to me is dear as now I
Can say there is in Heaven no sun nor moon,
Nor star nor air.
My heart is in my thoughts and my days are
Dark with that which in them is not.
I have seen visions and heard dreams that
Would ever only be as long as
I remember that I know.
Yet there are graves of manslaining things
Among the dreams of a young man.
And thou hast seen visions as drowsy
As any stump, and as untrue.
But we are on different lines with equal nepotism:
Thou art a number, and I a thought.
And my desires are now of narrower compass.
And many things that have been said are
Empty of meaning than the wave.
Wherefore, it may be, we are at peace,
Since all that touch us are yet only
An occult behind the surface.
A love-tried woman hath more substance
Than some thin air that flies between
Theurge and maiden.

...Like the red rose the white thorn, and the rose
Caught from the branches of a mighty tree,
In whose swift light a maiden may climb up
From earth to heaven: so doth thy sacred flesh
Still hold the charms of youth.

War. War never changes:

"Recuerán, the Mercenary", by Maggie Dietz [Activities, Jobs & Working, Relationships, Family & Ancestors, Social Commentaries, History & Politics, War & Conflict]
What does it matter where it happened or who it was?
The bullet traveled what did it matter when it fired
Where, far from where it all happened.
Do we matter, the years, who left us and where,
Remember the who and why,

Left, left to us, who are to be freed who and why?
I ask in my yet young memory what I know about you,
I ask in this yet young and still dark memory.
I ask you, have you ever told you're sorry,
Have you ever, down through the years, ever said who it was
To do this, to be a mercenary?

What does it matter in the who or what
To be forgiven, forgotten, forgiven who and why?
Your mercenary smile.
Your smile mercenary, mercenary now, does it matter where
It happened, when, or who?
In a tavern on a lane, behind a dark smoke,
A year ago, the answers were,

The secret gar­den:

No length of ways and no shore,
No looking in error through,
No darkling bough that hides
The white hush of its peace,
No chasm in the fountain-fall
That gives no hint,
No things that cannot be,
The corn, the worm, the sheath, the breath,
Of the things of the garden,
Are for ever concealed from sight
In their hidden places.

The power that whirls the earth of things
To soundless music, swinging like the blades,
And waits a boundless invitation
To its feast by the vale!
The obolus at last
Gave into our keeping
The things of the garden.

And it is forbidden
In our seeing
To put the garden under lock,
While yesternight
Black fall and storm
Mutter the red hearts of men!

But I know the secret, hidden gate
Where since it all fell to earth,
What lies and is hidden down there
That no man sees
That may not yet be known!

And it is forbidden
And dim and awful
To follow men to that dark place
To see what the garden withheld
When it shut.

But I know the secret.
And I knew, while I lay here,
That I know how.
And my cunning is as a star
In the garden's light.

And my thoughts are garden flowers
The secret's reflected flowers.
It is forbidden.
I feel the garden's darkness,
The hand that made flesh on me.
But I know what I know.

There are things that are for sea and sky,
For grovelling trees and trembling walls,
For the languor of the vale and stream,
And many for earth; for fields and streets
And river's arches and the sea
I hear a voice from the house so low
The night is trembling as it holds me
Hearing the house cry, "We need you not!"
And the lights, one by one, begin to flare
And tear their velvet red and purple
Throwing here and there and running everywhere
To reach some half-rhyming note of the house,
While under seas the pulses of terror beat.

I don’t really get this one but the rep­e­ti­tion and inver­sions make it inter­est­ing to read:

"Tell the Riders", by Bruce Smith [Living, Marriage & Companionship, Sorrow & Grieving, Relationships, Family & Ancestors, Home Life, Men & Women, Arts & Sciences, Language & Linguistics]
You say, "Sue, tell the riders."
They say, "Tell the riders."
Why not, say, "tell the horses,"
or maybe, "tell the music,"
Or the other one, "Tell the Fans"?
But you say, "Sue," no one
needs to know. Or the Other One,
Or the Fans, or, better still, "Tell the Fans."
You say, "Sue," they are sure to hear.
Ask the Others, or better still, "Tell the Fans."
It's in their nature to keep secrets,
even though you would. Why not tell the Fans
and get out of the way?
When I was a child, I never needed to ask
which finger was up or down.
Now I am a man, I never need to ask,
which door do I go through?
On our walk, they laugh, they think
something's coming today, a possibility
or threat, but what? I tell them, "The Fans."
They say, "Tell the Fans." I say, "Never
have I, in every case."
In fact, my mother often
forgot to say "do, dears."
She thought I didn't hear as many
as I did. "Tell the Fans" they cry,
or better yet, "Tell the Riders."

Love lost:

"Note in Bear Memoriam", by Vivian De la Noy
Who will remove my stitches
who will undo this confession,
who set these lines of words
drawn in wood? what voice will sing
their melody, irrepress the sting,
let you speak your name, my name,
what has become of it, left there,
left inside me, voiceless, mute?

Urban vs rural life:

"In These Cool Cities", by Ted Koochet [Living, The Mind, Activities, Travels & Journeys, Nature, Winter]
In these cool cities, they forget the trees' autumnal glow.
In those other places, the trees are burning.
Everywhere is winter here.
Frost and snow.
But in my streets it's still autumn.
Last night, an hour or two before the snow,
I glanced out my window and --- nothing.
Nothing but the jumble of bodies
and words like sea gulls yapping in the dark.
And those morning papers' faces of every morning.
The snow was quieter and paler.
No fall --- but a blue cloth.
Pelted like little pellets and faded.
And last, the crows' dim hang-hips.
I hear the crows now.
And somewhere below, another peal.

Social media satire:

"Conversation with a Friend on Romans", by J. Frederic Jagger [Relationships, Arts & Sciences, Humor & Satire, Social Commentaries, History & Politics]
We're fond of quoting people, but it's an illegal act

To put our words in someone else's mouth.
For instance, if I say to you, "I like Hitler,"
You would be liable, of course, and-legal.
One may be bitten by ants-by-dogs- in the District

But it is entertaining, and sometimes amusing.


We all can sing of love,
We all can sing of life.

What shall make each of us noble
And shine out in the eyes of all men,
That in triumph and in grief we know
We are all of us children of the sun?
What man has not, on the lonely moonlit heights,
Seen the glory of the lances' fires,
And heard the singing of the beaten arrows?

What can the earth and ocean and sky
That music not disclose?
What voice of human choirness e'er came nigh
That was not poured into that music,--
The Voice of the All About Us?


"The Mask Maker", by James Haimes [Religion, God & the Divine, Arts & Sciences, Social Commentaries, Mythology & Folklore, Heroes & Patriotism]
When we've grown weary of the hero, there is a certain
triumph to be found in the figure of the mask.
There, the underworld of fantasy and the ideals that carry it, is
God dressing as a minstrel. To us he is immaculate,
regardless of his skin, be it shabby or silvered, or if o on
behalf of the fair or beefed, or just an old suit, a bald head,
headdress not unparalleled in its beauty but pitiable in its
When that other menagerie called the human face is viewed, we
find it to be pitiful indeed.
Still it comes,
after all these years.
"On a Bridal Shower", by Alice Notley [Living, Life Choices, The Body, Nature]
I am thinking now of bridal showers, of the feeling of getting them--- I mean the giving of pleasure, of giving yourself to other people.
They come to mind: how many sweeter feelings there are: Nectar vaults, lockable vaults.
The most brilliant of eyes---that of my friend, now dead---was such a doll. Light and easy, of voice easy as a moonbeam.
And she wasn't thinking of birth, but wealth, the dream. Wealth, the sparkling.
It's the eyes alone that gleam, the loose and luminescent body. And the mouth. And how deep and wide and light-wide and luminous mouth.
So many of us gave ourselves to others. We didn't know we did.
"Intangible Things", by J.R.R. t cm = 120"", from "Meditations", by Alfred, Lord!
It seems we have been given possession of the senses, to see, hear, taste, touch, smell,
it is given to us. And so we have them. But we can't keep them.
We lose them. They fade. It's like losing a baby. And it's like losing a lover.
It's like losing a race. We race down the straight in a battle suit. We are breaking away.
We are at the front. The one in the front is the one who falls away.
But it's like losing a race. It's like the gully. It's like the catch. It's like the hole. It's like the ball. It's like all the possibilities.
But it's like a hole. It's hard to get into. It's hard to get out of.
We have the means to prolong it. We have the senses to maintain it.
It's ours. But what's ours is just as it's always been. It's like a dot that appears once in a lifetime.
It has been. But it has now become something like ours. It's still temporary. It doesn't last. It's just like us.

An ele­gy?

O my little sister, do not be frightened by these fairy rings,
by the bright white flowers that grow on the trees and the bushes,
by the voices of girls and boys, and the old old mume
with the silver stars, and by the tall pale idols
that shine through the candles:
do not be frightened.

O my sister, if you are frightened, I am afraid for you,
I shall never be able to tell you, for my eyes are
seared with blood; my head is broken, my fingers are frozen,
I am coming to you, I am coming to you;
the stars and the candles are moving in their spheres,
and the sea and the trees, and the idols and the girls
and the graves, and everything that you see.

O my little sister, listen to me,
I shall say what you have said before,
I shall fold my white hands and pray to you,
and you shall not speak; I shall give you
several loaves,
and you shall take one of each and one of the other
with you;
and the stars and the wind shall be changed
to gold.

O my little sister, do not cry,
and I shall tell you a story,
one of many, of the eyes that were not born
for one's heart and the hands that were not given;
you shall be glad,
for if your heart is not
your hands have been good.

O my little sister, listen to me,
I shall say what you have heard before;
I shall be calm and cheerful,
for I have heard the same things,
and I know what I hear.
I shall change the stars to gold,
O my little sister,
and the candles to golden balls,
and you shall take one of each and one of the other
for you and me,
and the dead shall live.

One more I noticed while gen­er­at­ing from a ~1-loss model on 2019-12-16, although I did not read through for selec­tions (see also ~0.7-loss model sam­ples, 2019-12-18):

"The Dead Dead Trees", by William E. Stafford [Living, Death, Sorrow & Grieving, Time & Brevity, Nature, Trees & Flowers, Arts & Sciences, Poetry & Poets]
for William E. Stafford

The fire out of nature comes, and all things die.
The old trees walking around the forest,
Old homes grown up so high they watch the sky;
Old friends grown old in the lights of the city
Grow fatter and slow, and cast a long frown.
But all the trees of sadness are sad, old friends,
You gather in your trees the great aim that can/ Save by crying. The fire out of nature comes, and all things die.
O bright Biblical Valentines of our rock marriage
Twice Sweet Roses, once a widowed Love would bear,
O give us the first vintage that points toward faith
And the second would add a marriage bond,
O give us a cup with the words of mine spouse,
O give us the wine of the West enough to brew
Two potions to govern whatever we deem
Treacher-of-heart and/or mind.

O give us two basic cakes: one
Higher than the other.
O give us two lovely days.
One a bright noon, the other at night.
Two better than two diluted drinks;
One a day without a bud that caressed the ground,
One side of which is born black, the other white.

One side deep in knowledge, the other one wet and shiny
And light as a great bird's eye.
One an eye that twinkles when you flash it,
One a kiss that devours.
O we rock over just enough to find the sun.
To make each other better.
Just enough to make us wiser.
"November", by Lee Herrick
All of what we think we know
of seasons slips away,
and beautiful things take their course---
a cascade of moss, a snow
granular fall. In these rare
nights gorgeous and bluer
than winter, there is no cold,
and no fierce wind. In these
few bright days the already
red sunset burns on the hills---
until the nighttime is blind
without the sun.
"Ticks", by Ben Lerner [Living, Death, Health & Illness, Relationships, Family & Ancestors, Philosophy, Social Commentaries]
Ticks are tiny bloodhounds with a bulldog
mouth and a life span of about five seconds.
They are small, pink, mosquitoes with a tan
to their wings. On your face, they leave
a pouty frowny face. Tick-tock, tick-tock.
They are everywhere, like crickets in a park
quietly singing. Tick-tock, tick-tock.
I get nervous around them. They tell me
they are hunting wildebeest. Tick-tock,
I get nervous around them. Their very presence
makes me nervous around them.
They look like feral dogs and
they hunt like wolves. Tick-tock, tick-tock.
Tick-tick, tick-tock.
I'm serious about this. Tick-tock.
"My Mother's Snare", by Ben Lerner [Living, Death, Relationships, Family & Ancestors]
My mother's snare
sounds like rain
picking up the strings of the rain,
then the trees, picking up the strings
of the trees.
It's the last refrain of a small song
my mother wants to finish, but cannot.
"The Lady's Speech", by Sir Walter Ralegh [Relationships, Family & Ancestors, Social Commentaries, History & Politics, War & Conflict]
A child was sitting in the sun
When another child said to him:
"Sun-child, tell me why
This great house is such a pit'
In which all people lie.
Tell me, why does no one die
Here in this pit?"
"Death is not here
Except in the sun
And only happens when the sun shines."
"Then who dies then?" asked the first child.
"Everyone dies then."
"My Brother the Bomb", by Mark Rudman [Living, Death, The Body, Time & Brevity, Religion, Faith & Doubt, God & the Divine]
For Joe Miller

In heaven, we worship every fruit,
From grape to peach to plum;
We go to earth and find it full
Of thorny thorns and braches,
Wrestling with itself to get away---
To explode on us.
This is our way of saying hello.
This is how we express gratitude:
By giving, by making things happen.
In hell, we worship every bomb
That people would drop on each other---
On Hiroshima or Dresden,
Or any other night where everyone
Has been too sleepy to turn on the light.
This is our way of saying goodbye.
This is how we make sure no one dies.
O world of radiant sunshine,
How far above you seem
The clouds o'er the sea,
Above the rooftops of New York
How high and distant they fly.
Your beauty seems almost painful--
For all the rain and mist.

O world of golden skies,
How near you seem to be
To souls that wander, lost and free,
Through fields of corn and wheat.
Though all below seems dark and drear,
Each height and hill is bright and fair.

O world of sparkling dews,
How near you seem to be
To women whose lips are wet
And cheeks that blusher are
Than mine or thine or even hers.
We smile because we're happy
And strangely jealous of each other.


Sub­jec­tive­ly, the out­put shows a lot of poetry knowl­edge, much bet­ter than the char-RNN sam­ples. There’s rhyming, themes are con­tin­ued for shock­ingly long pas­sages com­pared to char-RNN, and there are many pas­sages I feel could inspire a poet or even be cleaned up a lit­tle to be pass­able poems on their own. Adding the meta­data did help—GPT-2-poetry is worse than GPT-2-poetry-prefix. Some of the ones I liked most are (first lines) ‘We never say “Thank you”’, ‘Thy soul, thy very soul is burn­ing!’, ‘“It is morn!” said the clover-bush’, ‘And they have seen the last light fail’, ‘There comes a mur­mur low and sweet’, and prob­a­bly the best is ‘The sun is gone, and the night is late’.

Is GPT-2-poetry-prefix bet­ter than GPT-2-117M at poetry com­ple­tions (since GPT-2-117M will prob­a­bly hardly ever gen­er­ate poetry with­out a promp­t)? Prob­a­bly, with excep­tions. “Howl” is far worse, but that is for good rea­son related to the old­ness of the PG cor­pus; if any­one could assem­ble an equally large cor­pus of more recent poet­ry, I’d expect GPT-2-117M fine­tun­ing to pro­duce bet­ter com­ple­tions. The Pope sam­ples from GPT-2-poetry-prefix are clearly bet­ter (be­fore diverg­ing into pros­e). I would argue that the Shel­ley sam­ples are some­what bet­ter. And the 8 famous line com­ple­tions are over­all of much higher poetic qual­ity (sev­eral of the GPT-2-117M com­ple­tions are just prose, unsur­pris­ing­ly).

So, if one is look­ing for poetry com­ple­tions in an old-­fash­ioned vein, it deliv­ers, but at the cost of flex­i­bil­ity like more pros­e-­like (and hence con­tem­po­rary) poems. This is an expected and fix­able prob­lem, and over­all, I con­sider GPT-2-poetry-prefix to be suc­cess­ful as a poem gen­er­a­tor & bet­ter than my pre­vi­ous char-RNNs.


Nor is this near the limit for Trans­former-based poetry gen­er­a­tion, as there are many pos­si­ble improve­ments which could be made, all of which I’d expect to deliver sub­stan­tial gains:

  • Make It Big­ger:

    • big­ger NN mod­els: our ini­tial results used the pub­licly-re­leased GPT-2-117M, which deliv­ers infe­rior results on all tasks com­pared to the unre­leased GPT-2-1.5b: the sam­ples gen­er­ated by Ope­nAI & asso­ciates from GPT-2-1.5b are much bet­ter than GPT-2-117M sam­ples, indi­cat­ing that sim­ply scal­ing up con­tin­ues to deliver gains. Our GPT-2-1.5b poems turned out sub­stan­tially bet­ter.

      Nor did the var­i­ous GPT-2 model sizes appear to reach any nat­ural limit with GPT-2-1.5b, indi­cat­ing that the Trans­former NNs can be increased much fur­ther before hit­ting zero mar­ginal gains. (This is con­sis­tent with other large-s­cale NN research, par­tic­u­larly on CNNs where even bil­lions of images can be use­fully trained upon.) Ope­nAI’s Greg Brock­man has said (Feb­ru­ary 2019) that Ope­nAI intends to keep scal­ing GPT-2-1.5b up with aspi­ra­tions of train­ing 10–1000× larger mod­els (although it’s unclear what these fac­tors refer to: data cor­pus size, model para­me­ters, win­dow width, layer depth, or all of the above). Given that GPT-2-117M deliv­ers such qual­i­ta­tively infe­rior results to the ~10× big­ger GPT-2-1.5b, and if GPT-2-1.5b is equiv­a­lently qual­i­ta­tively infe­rior to a hypo­thet­i­cal 10× big­ger ‘GPT-2-huge’ and a 1000× big­ger still ‘GPT-2-enormous’ is pos­si­ble, the qual­ity leap from GPT-2-117M poetry to a hypo­thet­i­cal ‘GPT-2-enormous’ would be stag­ger­ing.

      These pro­jec­tions for GPT-3 have since been borne out—GPT-3 (pub­lished 2020-05-28) has 175b para­me­ters (166× more), and GPT-3’s untrained ran­dom poems are as good or bet­ter (!) than our GPT-2-1.5b poems.

    • bet­ter NN mod­els (which will prob­a­bly need to be big­ger): the most painful limit is the small con­text win­dow, which has a num­ber of pos­si­ble solu­tions like recur­ren­cy, mem­o­ry, effi­cient atten­tion vari­ants, or var­i­ous approx­i­ma­tions. other options include more atten­tion heads or more lay­ers or exter­nal mem­ory func­tions or on-the-fly adap­ta­tion; there are many pos­si­bil­i­ties here. (The pre­fix can be seen as an extremely crude kind of recur­rency or mem­o­ry, and helped a lot; how much more so a real mem­o­ry?)

    • more & bet­ter data: quan­ti­ty-­wise, the PG cor­pus is barely a tenth of a giga­byte and exhibits many enor­mous omis­sion­s—all of mod­ern poet­ry, for exam­ple, not to men­tion most for­eign poet­ry, or non-Eng­lish poetry as a whole (why not a mul­ti­-lin­gual GPT-2 if suf­fi­ciently large? neural machine trans­la­tion approaches improve the more lan­guages they have access to, why not reg­u­lar lan­guage gen­er­a­tion?). There are many places addi­tional poetry could be obtained from, such as Wik­iSource, Poetry Foun­da­tion, Lib­gen, or the Inter­net in gen­eral (per­haps write a poet­ry-de­tec­tor Trans­former to search through a dump like Com­mon Crawl for poet­ry?). Qual­i­ty-­wise, the PG cor­pus is good but still has a num­ber of flaws: a lot of prose, just enough non-Eng­lish poetry to screw things up (espe­cially Lat­in), mostly pre-1923 poet­ry, & min­i­mal meta­data (ide­al­ly, poems would be indi­vid­ual units rather than book-length streams, and meta­data like author would be avail­able to use in pre­fix­es).

      GPT-3 expands the dataset greatly to more of Com­mon Crawl plus 57 bil­lion words from 2 delib­er­ate­ly-­vague­ly-de­scribed Inter­net book cor­pus­es, but play­ing with it, I feel that GPT-3 is still weak on areas like sci­ence—­for exam­ple, I observe GPT-3 to eas­ily write machine learn­ing paper abstracts, but it does not do quite so well when I try to extend it to the rest of papers, and it does­n’t spon­ta­neously quote from within papers the way it quotes abstracts. Does this reflect the fact that many papers exist only as PDFs, and only a rel­a­tively small frac­tion of all sci­en­tific papers have clean read­able HTML ver­sions (while they usu­ally all have read­able HTML abstract­s)? If so that may weaken the GPT rea­son­ing & com­mon sense abil­i­ties con­sid­er­able; after all, while it does not usu­ally come up in reg­u­lar writ­ing that giraffes have two eyes instead of three eyes, prob­ing def­i­n­i­tions & study­ing excep­tions & manip­u­lat­ing causal arrows in unusual ways are all the bread & but­ter of sci­en­tific writ­ing, and could implic­itly teach that bet­ter than reg­u­lar writ­ing.

  • Gen­er­ate Smarter

    • using a bet­ter sam­pling strat­egy than top-k, like (but curi­ous­ly, not —beam search gives sub­stan­tial improve­ments on what the nucleus sam­pling authors call “closed” text gen­er­a­tion tasks like trans­la­tion, but while beams search helps char-RNN a lit­tle, it dam­ages results badly the wider the beam, and gives par­tic­u­larly bad results on GPT-2; Kyle Kast­ner says that beam search can work in con­texts with heavy con­straints, like being con­strained to gen­er­ate rhyming lines or explicit rep­e­ti­tion penalties)

      Nucleus sam­pling has been imple­mented in nshep­perd’s Ten­sor­flow & Hug­ging Face’s PyTorch GPT-2 sam­pling code.
    • use tree search meth­ods: any deep, thor­ough, search inevitably becomes a tree; tree searches are use­ful for enabling kinds of ‘back­track­ing’ and ‘revi­sion’ or ‘chang­ing its mind’ about mul­ti­ple pos­si­ble vari­ants of a poem, as opposed to the usual sam­pling approaches which tend to com­mit to each word and force all-or-noth­ing choic­es. (My pro­posal for back­prop reward opti­miza­tion would have sim­i­lar advan­tages, as each iter­a­tion step allows ‘think­ing’ about how to improve a given input, approx­i­mat­ing a search implic­it­ly—even if not explic­itly like a MCTS or -like approach.) The chal­lenge here is to fig­ure out a tree search which avoids the rep­e­ti­tion trap.

  • Train Bet­ter, by fix­ing the loss (eg 18), or switch­ing to the RL set­ting to directly max­i­mize gen­er­a­tion qual­i­ty:

    • richer losses: the stan­dard GPT uni­di­rec­tional pre­dic­tion loss is not the only pos­si­ble (dif­fer­en­tiable) loss; it is not even, strictly speak­ing, the best—­mod­els like / using more sophis­ti­cated losses like bidi­rec­tional loss­es, which force the model to pre­dict a word miss­ing from any­where in the string (as opposed to only miss­ing from the end), typ­i­cally out­per­form GPT-2 on lan­guage tasks. A model like uses a denois­ing objec­tive where a 15%-­long chunk is replaced by a miss­ing token & T5 must pre­dict all the miss­ing text based on con­text; these sorts of objec­tive losses allow learn­ing much more from a given dataset. (In­deed, such mod­els typ­i­cally out­per­form GPT-2 on every­thing but lan­guage gen­er­a­tion. Odd­ly, they typ­i­cally do quite badly at that, which is a major rea­son every­one uses GPT-2 for gen­er­at­ing new texts, and BERT etc for every­thing else like gen­er­at­ing embed­dings or clas­si­fi­ca­tion.)

    • adding global end-­to-end loss­es, which enable train­ing to opti­mize non-d­if­fer­en­tiable prop­er­ties rather than easy (but par­tially irrel­e­vant ones like pre­dic­tive losses such as cross-en­tropy in pre­dic­tion of the next word). For exam­ple, rules defin­ing accept­able meter or rhyme use or penal­iz­ing total rep­e­ti­tion—these can­not be done via the nor­mal train­ing because no indi­vid­ual dis­crete word is respon­si­ble and para­me­ters can­not be smoothly adjusted to decrease/increase a global prop­erty like ‘rhymes’ which is the result of all words con­sid­ered together as a whole. (This sort of RL loss has been employed in other nat­ural lan­guage tasks like machine trans­la­tion, where met­rics like pre­dic­tive loss do not map onto the desired goal of seman­ti­cal­ly-­cor­rect trans­la­tion, and word-by-­word gen­er­a­tion of trans­la­tions yields sim­i­lar issues as here, but there are met­rics like BLEU or ROUGE or gram­mar check­ers which pro­vide a crude mea­sure of global qual­i­ty. RL approaches have .)

    • using sub­jec­tive qual­i­ty-based losses, like :

      instead of train­ing a NN to pre­dict indi­vid­ual nex­t-char­ac­ters as accu­rately as pos­si­ble or imi­tate a text cor­pus as well as pos­si­ble, we really just want them to pre­dict good nex­t-char­ac­ters to write text as well as pos­si­ble—which is not the same thing at all, any more than accu­rately pre­dict­ing a human Go play­er’s next move on aver­age is the same thing as play­ing Go super­hu­manly well.

      This encour­ages more global coheren­cy, more the­matic pro­gres­sions, use of rare words when appro­pri­ate, sur­pris­ing sub­ver­sions or twists which work well when tried but don’t appear in the orig­i­nal cor­pus, learn­ing esthet­ics, and so on. If it works and the new GPT-2-poetry is able to suc­cess­fully pro­duce new poems which con­sis­tently get the top score from the critic and no fur­ther improve­ment is hap­pen­ing, then you sim­ply read a bunch of its new poems, pick which one in each pair you like, retrain the critic on the expanded dataset to detect the remain­ing flaws in the ones you dis­liked, and then keep train­ing GPT-2-poetry to avoid gen­er­at­ing the ones you dis­liked & gen­er­ate more poems like the ones you liked. Repeat with many cycles, and it should gen­er­ate excel­lent poems while avoid­ing all the flaws of crude like­li­hood train­ing and even cruder top-k sam­pling which hob­ble GPT-2-poetry right now. Even bet­ter, you could cre­ate a web­site to crowd­source the rank­ings to keep it train­ing 24/7 and improv­ing indef­i­nite­ly.

    • using “expert iter­a­tion” archi­tec­tures like Alp­haZero to do much more sophis­ti­cated search over pos­si­ble poems, cre­at­ing an iter­a­tive boot­strap

    • adding cre­ativ­ity losses along the lines of , Elgam­mal et al 2017, where updat­ing GANs encour­age diver­sity

      • one could attempt to invent new styles of poetry by tak­ing inspi­ra­tion from evo­lu­tion­ary meth­ods, such as the “Pop­u­la­tion-Based Train­ing” vari­ant employed in Deep­Mind’s AlphaS­tar League which cre­ated diver­sity by delib­er­ately scram­bling the ‘rules’ for each lin­eage of agents. The “AlphaS­tar League” used a pop­u­la­tion of mul­ti­ple NNs, each forced to spe­cial­ize in using a par­tic­u­lar unit or rewarded for achiev­ing par­tic­u­lar goals like defeat­ing a spe­cific NN (rather than win­ning in gen­er­al). The AlphaS­tar League was cred­ited for forc­ing the over­all AlphaS­tar pop­u­la­tion to explore strate­gies reliant on par­tic­u­lar kinds of units and fig­ur­ing out coun­ter-s­trate­gies to suc­cess­ful ones. Some­thing sim­i­lar could be done with poetry rules: train many dif­fer­ent agents, each given a spe­cific rhyme scene or meter or vocab­u­lary for their reward func­tion, and in pref­er­ence-learn­ing approach­es, the best poems can be pro­vided to human crit­ics for rat­ing & improv­ing the NN crit­ic. Poten­tially excit­ing new com­bos could emerge as pro­duc­ing the best poems as rated by the humans.

Given that GPT-2-117M is far from the state of the art as of Feb­ru­ary 2019, and hard­ware & gen­er­a­tive NN research is advanc­ing rapid­ly, it will be excit­ing to see what sort of poetry can be gen­er­ated given another 4 years!


Archive of Our Own (Ao3) GPT-2-1.5b

Aaron Gokaslan scraped the large fan­fic­tion web­site (Ao3) and cre­ated a text dump (12GB raw; 190,931 sto­ries; 2.06b words; no meta­data).19 We trained a GPT-2-1.5b on it, under the the­ory that it might be use­ful as a basis for tex­t-game-­like appli­ca­tions such as , under the idea that since AI Dun­geon 2 is essen­tially col­lab­o­ra­tive sto­ry-telling, start­ing with a sto­ry-based model ought to give bet­ter results. Whether that is true remains to be seen, but we gen­er­ated Ao3 text sam­ples and the dataset & model can be down­load­ed:

rsync rsync:// ./ # 11GB
rsync rsync:// ./ # 2.8GB

Video Game Walkthrough GPT-2-1.5b

Twit­ter user me_irl pro­vided a 50MB scrape of video game walk­throughs, which he’d used pre­vi­ously with GPT-2-345M and requested we do fine­tun­ing on that as well: video game walk­through text sam­ples (). me_irl has sug­gested that they could be used as hypo­thet­i­cal game designs for com­pe­ti­tions or art pur­pos­es. The com­bined dataset/model can be down­load­ed:

rsync rsync:// ./


In Decem­ber 2019, Shawn Presser trained a GPT-2-117M model for a few mil­lion steps on the /r/DoTA2 sub­red­dit, as part of the Sub­red­dit Sim­u­la­tor train­ing pro­ject. (The final GPT-2-1.5b & dataset for Sub­red­dit Sim­u­la­tor has not been released at the pro­jec­t’s request.)

While the final model check­point appears to have been lost (oop­s), step #562,971 from 2019-12-18 has been upload­ed:

  1. rsync -v rsync://
  2. Mega (433MB)

Bradley-Terry Preference Learning

intro­duced a deep rein­force­ment learn­ing archi­tec­ture for learn­ing “I know it when I see it” sub­jec­tive­ly-de­fined reward func­tions from human feed­back: a human makes com­par­isons of actions/datapoints/episodes to select the ‘bet­ter’ one, a NN is trained to pre­dict the bet­ter one based on these com­par­isons, and another NN is RL-­trained based on the pre­dicted com­par­isons inter­preted as a reward. Since the human is unable to write down a con­ven­tional reward func­tion in soft­ware, the pre­dic­tor NN (anal­o­gous to a Dis­crim­i­na­tor in a GAN or a ‘critic’ in actor-­critic RL) learns the reward func­tion by exam­ple, and then the RL agent NN (anal­o­gous to a Gen­er­a­tor in a GAN) learns by tri­al-and-er­ror what sequences will opti­mize this com­plex reward func­tion, and the human feed­back pro­vides addi­tional guid­ance on new parts of the prob­lem as the pair of NNs boot­strap into bet­ter per­for­mance. This is demon­strated on video game or robot­ic-style sim­u­la­tions, but appears equally applic­a­ble to other sequence prob­lems where reward func­tions are impos­si­ble to write and exist­ing losses like max­i­mum like­li­hood are imper­fect for gen­er­a­tion (such as music or poetry com­po­si­tion).

As orig­i­nally framed, the pre­dic­tor merely does com­par­isons, receiv­ing & pro­vid­ing binary feed­back. This is jus­ti­fied as being implic­itly equiv­a­lent to a stan­dard pair-comparison/competition mod­el, the (akin to the famous ELO), where each dat­a­point has a latent vari­able on a com­mon car­di­nal scale (often, like a , scaled to for con­ve­nience), pro­duc­ing a total order which effi­ciently extracts all pos­si­ble infor­ma­tion from the com­par­isons.

I sug­gest that this is not nec­es­sar­ily the case, as exam­ples from GANs indi­cate that such a pref­er­ence-learn­ing archi­tec­ture may be learn­ing some­thing odder (such as mem­o­riz­ing com­par­ison­s), and that the archi­tec­ture could be improved by remov­ing the implic­it­ness of the B-T rank­ing, com­put­ing the B-T rank­ings directly (which can be done even with non-over­lap­ping com­par­isons by using a Bayesian model with pri­ors and using covari­ates such as the pre­dic­tor’s own esti­mates), thereby pro­vid­ing absolute qual­ity scores for cor­rect­ness of com­par­isons, more effi­cient regres­sion, RL rewards, and mean­ing­ful inter­pretable scores for down­stream uses.

Moved to the GPT-2 pref­er­ence learn­ing page.

Efficient Attention

One of the most frus­trat­ing lim­i­ta­tions of is the con­text win­dow: 2048 BPEs runs out fast when you start prompt pro­gram­ming some­thing hard, and hacks like have nasty & sub­tle side-­ef­fects. How do we get future Trans­form­ers with rea­son­able con­text win­dows and/or mem­o­ry, which we can use for research papers, books, struc­tured text, images, video, genomics, and so on, where we need to han­dle sequences with lengths in the mil­lions? (Such improve­ments would per­mit not just doing things GPT-3 strug­gles to do, like write coher­ent nov­els, but many bet­ter archi­tec­tures, like mul­ti­modal Trans­form­ers which can learn jointly from images & text, access­ing image-based datasets like PDFs, and learn­ing far more accu­rate human-­like rep­re­sen­ta­tions & tacit knowl­edge with less data & smaller mod­els, pro­vid­ing large mod­els use­ful for almost all con­ceiv­able tasks—e­spe­cially robot­ic­s.)

Below I com­pile & cat­e­go­rize research on break­ing the dense atten­tion qua­dratic bot­tle­neck (overviews: Lil­ian Weng, Madi­son May; review: ; bench­mark suite: ):

“Table 1: Sum­mary of Effi­cient Trans­former Mod­els pre­sented in chrono­log­i­cal order of their first pub­lic dis­clo­sure” (Tay et al 2020)

  1. A Trans­former is a con­sid­er­ably dif­fer­ent archi­tec­ture than an RNN, and is not that easy to explain, as it uses mul­ti­ple con­vo­lu­tions to imple­ment “atten­tion”, allow­ing flex­i­ble inter­nal con­trol flow, over a large but finite input win­dow, with­out any recur­rency or hid­den state or LSTM units nec­es­sary. For increas­ing­ly-tech­ni­cal expla­na­tions, see:

  2. 774M requires changes to nshep­perd’s check­point­ing, specif­i­cal­ly, remov­ing the layer == 10 restric­tion in, and let­ting the check­point­ing code check­point as much as pos­si­ble, which enables train­ing mini­batches n≤10 on my 2×1080tis. Diff:

    diff --git a/src/ b/src/
    index 4e942d8..71092bc 100644
    --- a/src/
    +++ b/src/
    @@ -124,10 +124,10 @@ def block(x, scope, *, past, hparams):
         with tf.variable_scope(scope):
             nx = x.shape[-1].value
             a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
    -        x = x + a
    +        x = x1 = x + a
             m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
             x = x + m
    -        return x, present
    +        return x, present, x1
     def past_shape(*, hparams, batch_size=None, sequence=None):
         return [batch_size, hparams.n_layer, 2, hparams.n_head, sequence, hparams.n_embd // hparams.n_head]
    @@ -161,9 +161,9 @@ def model(hparams, X, past=None, scope='model', reuse=tf.AUTO_REUSE):
             pasts = tf.unstack(past, axis=1) if past is not None else [None] * hparams.n_layer
             assert len(pasts) == hparams.n_layer
             for layer, past in enumerate(pasts):
    -            h, present = block(h, 'h%d' % layer, past=past, hparams=hparams)
    -            if layer == 10:
    -                tf.add_to_collection('checkpoints', h)
    +            h, present, x1 = block(h, 'h%d' % layer, past=past, hparams=hparams)
    +            if layer < 48:
    +                tf.add_to_collection('checkpoints', x1)
             results['present'] = tf.stack(presents, axis=1)
             h = norm(h, 'ln_f')
  3. It would require either high­-end GPUs with ≥16GB VRAM, or TPU instances (which were used to train it). GPT-2-1.5b can’t be trained on my 1080tis with either the nshep­perd code­base or Shawn Presser’s fork, although Presser has a Google Colab note­book using TPUs which can train it.↩︎

  4. Other exam­ples of fine­tun­ing are Face­book Mes­sen­ger logs, nshep­perd’s unpub­lished Linux ker­nel C source code & IRC-log train­ing4, and story prompts. And, while it does­n’t use GPT-2-117M, too good to not men­tion is “Stack Roboflow: This Ques­tion Does Not Exist”.↩︎

  5. GPT-2 com­ple­tions of 26 prompts: “Ozy­mandius”/“One Art”/“The Road Not Taken”/“Where the Side­walk Ends”/“Because I could not stop for Death”/“Infer­no, Canto I”/“In Flan­ders Field”/“O Cap­tain! My Cap­tain!”/“Howl”/“The Tyger”/“Out­sight”/“Zuang Zhou Dreams of Being a But­ter­fly”/“Son­net”/“Oh, the Places You’ll Go!”/“The Hol­low Men”/“The Sum­mer Day”/“A Just-Fin­ish­ing Can­dle”/“A Psalm of Life”/“Still I Rise!”/“The Sec­ond Com­ing”/“Do not go gen­tle into that good night”/“Kubla Khan”/“Edge”/“The Raven”/“There Will Come Soft Rains”/“The Lorax”.↩︎

  6. For exam­ple, 2 peo­ple fine­tuned GPT-2-117M on an IRC chan­nel’s logs, get­ting losses of 1.95 & 2.3; why was the lat­ter’s loss 18% worse com­pared to the for­mer when they were using the same IRC chan­nel, GPT-2-117M pre­trained mod­el, train­ing code­base, & had both appar­ently con­verged? Because. while the IRC chan­nel was the same, they used dif­fer­ent IRC clients which had dif­fer­ent IRC log for­mat­ting con­ven­tions—the for­mer’s logs had the full time­stamp pre­fixed to each line, and the lat­ter did­n’t. Said time­stamps made up ~20 char­ac­ters of ~110 char­ac­ter lines, or, ~18% of each line! So the mod­els were per­form­ing iden­ti­cally on the con­tent that mat­tered, and the much lower loss was sim­ply because of near-per­fect pre­dic­tion of the high­ly-repet­i­tive & pre­dictable time­stamps on every line. (In­deed, given the lim­ited win­dow of GPT-2-117M, arguably the model with the worse loss would be bet­ter in terms of gen­er­at­ing fun coher­ent sam­ples.)↩︎

  7. I have 2 GPUs but nshep­perd’s code does not (yet) sup­port multi-GPU train­ing eas­i­ly. Some sup­port using Horovod for multi-GPU has been added but I can­not vouch for it.↩︎

  8. I dis­cov­ered this while being puz­zled why --batchsize 32 did not lead to instant out­-of-mem­ory errors for train­ing; sim­i­lar­ly, if you make the mis­take of sam­pling with the option --top 40, what you are actu­ally doing is sam­pling with the default --top_k 0. Oops.↩︎

  9. It is pos­si­ble that the addi­tional train­ing was help­ing, because the remain­ing tiny changes in the loss might trans­late to large per­ceived qual­ity improve­ments—while the loss did­n’t change, the sam­ples from later on did strike me as bet­ter. This was some­thing I thought I noticed with char-RNN as well, that the loss became a bad guide to qual­ity when the NN had mostly con­verged. On the other hand, with larger GPT-2s, like GPT-2-1.5b, the rela­tion­ship between loss and per­ceived qual­ity seems even more opaque, with qual­ity some­times wors­en­ing even as the train­ing loss decreases rapid­ly.↩︎

  10. One might worry that by tak­ing up space in the mod­el’s lim­ited con­text ‘win­dow’ of inputs, because the Trans­former has no hid­den state or ‘mem­ory’, such inline meta­data would be a bad thing as it will push real words out of the con­text win­dow, thereby degrad­ing qual­ity and mak­ing it even more inco­her­ent & ram­bling.

    But on the other hand, if it does learn to asso­ciate spe­cific IDs with genres/topics, then rep­e­ti­tion of the inline meta­data serves as a ‘mnemonic’ for global infor­ma­tion which is avail­able to all sub­se­quent iter­a­tions of the mod­el, serv­ing as a crude mem­ory itself.

    For exam­ple, if Home­ric pas­tiche has ID #16452, then as long as the final iter­a­tion of the model over­laps for just the ID with the first iter­a­tion of model dur­ing sam­pling and both see “16452”, all mod­els will be able to con­sis­tently agree on gen­er­at­ing Home­ric pas­tiche rather than some other pas­tiche because they all see the same ID some­where in their con­text win­dow & that guides their gen­er­a­tion.↩︎

  11. starspawn0 has col­lated some of the results:

    • All U.S. pres­i­dents and Russ­ian lead­ers in tem­po­ral order, where the order was not spec­i­fied in the doc­u­ments used; also, all ten­nis cham­pi­ons in inter­na­tional com­pe­ti­tions over the years. So, tem­po­ral order can be extract­ed.
    • The lon­gi­tude and lat­i­tude of cities in the U.S. and Europe, along with their rel­a­tive dis­tances.
    • The rel­a­tive size of many kinds of objects, like cars, ele­phants, humans, hous­es, and so on—which object is larger than which oth­ers.
    • The exact sizes of many objects in meters, with rea­son­ably small error. For exam­ple, it might say the dimen­sions of a wind­shield are about 1.4 meters by 1 meter.
    • Which kinds of ani­mals are dan­ger­ous, which are not; and which kinds of objects (e.g.“fire”) are dan­ger­ous, and which are not (e.g.“water”).
    • Which ani­mals are smarter than which other ones; which ani­mals are faster, which are slow­er; which ani­mals are heavy, which are light; which ani­mals live in water, which do not.
    • Which cities cause arousal (e.g.“fun”, “excit­ing”), and which do not; which are expen­sive, which are not; which are dan­ger­ous, which are not; which are reli­gious, which are not; which are large, which are not; which are hot, which are not; which are wealthy, which are not; which have a rec­og­nized intel­lec­tual cul­ture, which do not.
    • Which kinds of clothes are appro­pri­ate for dif­fer­ent age groups; which kinds cause emo­tional arousal; which kinds are expen­sive; which kinds are appro­pri­ate for dif­fer­ent sex­es; which kinds you expect to find in dif­fer­ent loca­tions; which kinds are asso­ci­ated with wealth; which are not appro­pri­ate for hot weath­er, which are; same for cold weath­er.
    • Qual­i­ties of mytho­log­i­cal crea­tures—­like the ones for ani­mals.
    • Qual­i­ties of pro­fes­sions and pro­fes­sion­als: age, arousal, dan­ger, gen­der, intel­li­gence, loca­tion, valence, wealth.
    • Qual­i­ties of sports and sports­men / wom­en: arousal, dan­ger, gen­der, intel­li­gence, loca­tion, speed, wealth.
    • Qual­i­ties of states: cost, intel­li­gence, polit­i­cal, reli­gios­i­ty, size, tem­per­a­ture, wealth.
    • Qual­i­ties of types of weather and weather phe­nom­ena (e.g.“tor­nado”): dan­ger, tem­per­a­ture, wet­ness.
    • Phys­i­cal prop­er­ties of objects, such as rigid­ness and strength. Prob­a­bly also includes trans­paren­cy, soft­ness, hard­ness, round, prick­ly, angu­lar, and so on.
    • Rela­tions like whole-and-­part, and rel­a­tive loca­tions of a part within an object: e.g. hand is con­nected to arm, arm is con­nected to shoul­der, shoul­der is con­nected to neck, neck is con­nected to head.
    • Prop­er­ties of coun­tries and cities: geolo­ca­tion, GDP, GNI per-­cap­i­ta, CO2 emis­sions per-­cap­i­ta, fer­til­ity rate, amount of inter­net use, call­ing code, mil­i­tary expen­di­ture, life expectan­cy, energy use, pop­u­la­tion, places imported from, how long they’ve had a national anthem, kinds of sports, GDP growth, crime rate, and so on.
    • Binary attrib­utes of coun­tries and cities: con­ti­nent, time zones, con­tained-by (which regions con­tain which coun­tries; which coun­tries con­tain which cities; which bor­oughs are con­tained in which cities; and per­haps even rela­tions between the bor­ough­s—which bor­der which oth­ers, how they are shaped, and how large they are), lan­guage, high or low crime?, mil­i­tary con­flicts, ath­letes, medals won, orga­ni­za­tions found­ed, schools found­ed, com­pa­nies found­ed, weath­er, type of gov­ern­ment, offi­cials, and many more.
    • It’s even pos­si­ble to pre­dict the qual­i­ties of objects not in the train­ing cor­pus, using some­thing called the

    [Some ref­er­ences:]

  12. Child et al 2019:

    We also intro­duce (a) a vari­a­tion on archi­tec­ture and ini­tial­iza­tion to train deeper net­works, (b) the recom­pu­ta­tion of atten­tion matri­ces to save mem­o­ry, and (c) fast atten­tion ker­nels for train­ing. We call net­works with these changes “Sparse Trans­form­ers”, and show they can model sequences tens of thou­sands of timesteps long using hun­dreds of lay­ers. We use the same archi­tec­ture to model images, audio, and text from raw bytes, set­ting a new state of the art for den­sity mod­el­ing of Enwik8, CIFAR-10, and Ima­geNet-64. We gen­er­ate uncon­di­tional sam­ples that demon­strate global coher­ence and great diver­si­ty, and show it is pos­si­ble in prin­ci­ple to use self­-at­ten­tion to model sequences of length one mil­lion or more.

    5.4. Sav­ing mem­ory by recom­put­ing atten­tion weights

    Gra­di­ent check­point­ing has been shown to be effec­tive in reduc­ing the mem­ory require­ments of train­ing deep neural net­works (), (Gruslys et al., 2016). It is worth not­ing, how­ev­er, that this tech­nique is par­tic­u­larly effec­tive for self­-at­ten­tion lay­ers when long sequences are processed, as mem­ory usage is high for these lay­ers rel­a­tive to the cost of com­put­ing them.Us­ing recom­pu­ta­tion alone, we are able to train dense atten­tion net­works with hun­dreds of lay­ers on sequence lengths of 16,384, which would be infea­si­ble on mod­ern hard­ware oth­er­wise. In our exper­i­ments, we recom­pute the atten­tion and feed-­for­ward blocks dur­ing the back­wards pass.

    …For each sequence length, we attempted to train the largest model which could entirely fit into 16GB V100 accel­er­a­tors with­out model par­al­lelism. Over­all, we found that increas­ing the sequence length by a fac­tor of 4 requires a reduc­tion in model capac­ity of approx­i­mately 4 × √4 = 8. Thus we found we could use fac­tor­ized self­-at­ten­tion on sequences over 1 mil­lion timesteps long, albeit with extremely few para­me­ters (3 mil­lion).

  13. Because it is so large, I am pro­vid­ing an alter­nate rsync mir­ror: rsync --verbose rsync:// ./↩︎

  14. The code turns out to mul­ti­ply by a large num­ber as a way of set­ting a default ‘highly unlikely’ value for each pos­si­ble BPE, but in FP16, it can’t be rep­re­sented and over­flows, and so the out­put amus­ingly just becomes the BPE 0, which is the char­ac­ter ‘!’, so it kept print­ing out ‘!!!’. Indeed.↩︎

  15. The Colab envi­ron­ment has spe­cial sup­port for mount­ing Google Dri­ve, so a mag­i­cal incan­ta­tion like this will mount a Google Drive folder as a nor­mal (slow) direc­to­ry:

    from google.colab import drive
    drive.mount('/content/drive', force_remount=True)
    #!rm -f ~/drive
    #!ln -s /content/drive/My\ Drive ~/drive
    !mkdir -p /drive
    #umount /drive
    !mount --bind /content/drive/My\ Drive /drive
  16. We noticed that, in fact, our pre­emptible TPUs would always pre­empt pre­cisely at mid­night. I spec­u­lated that, as dis­cussed in the Google SRE hand­book where they cover how the Chubby ser­vice is delib­er­ately taken down at ran­dom to live down to its uptime promis­es, pre­emptible TPUs were being delib­er­ately taken down to stop users from treat­ing them like on-de­mand TPUs, and this sim­ply was­n’t doc­u­ment­ed. Dis­cussing our prob­lems with TFRC, this appar­ently was cor­rect.↩︎

  17. Presser is con­vinced that TPU power is greatly over­rated and most TPU projects have made poor use of the avail­able pow­er, as they get far less speedups than one would expect over GPUs. Odd­ly, there does­n’t seem to have been much work on using mul­ti­ple TPUs out­side of a TPU pod con­fig­u­ra­tion, and TFRC did not seem to know of an equiv­a­lent to the ‘swarm’.↩︎

  18. also demon­strates prob­lems with like­li­hood decod­ing strate­gies in GPT-2 for story gen­er­a­tion.↩︎

  19. Alter­na­tive Ao3 fan­fic­tion dumps, among oth­ers, are avail­able on the Inter­net Archive.↩︎