GPT-2 Neural Network Poetry

Demonstration tutorial of retraining OpenAI's GPT-2-small (a text-generating Transformer neural network) on a large public domain Project Gutenberg poetry corpus to generate high-quality English verse.
topics: statistics, NN, fiction, shell, GPT, tutorial
created: 3 March 2019; modified: 29 Oct 2019; status: finished; confidence: likely; importance: 8

In February 2019, following up on my 2015–2016 text-generation experiments with char-RNNs, I experiment with the cutting-edge Transformer NN architecture for language modeling & text generation. Using OpenAI’s GPT-2-small (117M) model pre-trained on a large Internet corpus and nshepperd’s finetuning code, I retrain GPT-2-small on a large (117MB) Project Gutenberg poetry corpus. I demonstrate how to train 2 variants: “GPT-2-poetry”, trained on the poems as a continuous stream of text, and “GPT-2-poetry-prefix”, with each line prefixed with the metadata of the PG book it came from. In May 2019, I trained the next-largest GPT-2, 345M, similarly, for a further quality boost in generated poems. In October 2019, I retrained 117M on a Project Gutenberg corpus with improved formatting, and combined it with a contemporary poem dataset based on ’s .

With just a few GPU-days on 1080ti GPUs, GPT-2-small finetuning can produce high-quality poetry which is more thematically consistent than my char-RNN poems, capable of modeling subtle features like rhyming, and sometimes even a pleasure to read. I list the many possible ways to improve poem generation and further approach human-level poems.

OpenAI announced in February 2019 in their creation of “GPT-2-large”, a Transformer1 neural network 10x larger trained (like a char-RNN with a predictive loss) by unsupervised learning on 40GB of high-quality text curated by Redditors. GPT-2-large led to large improvements over GPT-1’s natural language generation, is close to or , and demonstrated high performance on untrained NLP tasks (see the paper for more details: , Radford et al 2019). By large improvements, one means that the best samples like the ones included in the OA announcement have started to reach an uncanny valley of text, capable of telling entire semi-coherent stories which can almost fool a sloppy reader—certainly, the verisimilitude is better than any char-RNN output I’ve seen. (A dump of many more samples . There is also an interactive word-by-word .) The full GPT-2-large model was not released, but a much smaller one a tenth the size, GPT-2-124M was released in February 2019, which I call “GPT-2-small” to avoid confusion.

GPT-2-small was used in most initial experiments with GPT-2-based text generation. OA’s next largest models, GPT-2-355M & were released in May & August 2019, and the final, largest, GPT-2-1.5b model was released in (too late to be used in most of these experiments); 355M–774M turn out to just barely be trainable on commodity GPUs.2 Also worth noting is the release of Gokaslan & Cohen 2019’s , which produces good samples if perhaps not quite as good as the OpenAI GPT-2-1.5b, but which was not trainable at the time on desktop GPUs3 although it does still at least run (allowing for sampling/prompting if not training). OpenAI notes that “Since February, we’ve spoken with more than five groups who have replicated GPT-2”, and some have or gone further than GPT-2-1.5b: replications/extensions include Gokaslan & Cohen 2019, , Hugging Face (a NLP startup), , Nvidia’s , Google’s , and MS’s .

So, all of the following experiments use OpenAI’s 2 smallest models.

GPT-2-small: Generating Poetry

“‘I don’t speak’, Bijaz said. ‘I operate a machine called language. It creaks and groans, but is mine own.’”


Naturally, people immediately used GPT-2-small for all sorts of things, and I applied it myself to generate surreal anime plot summaries & dialogue for .4 Even more naturally, just as with char-RNNs, GPT-2-small works well for poetry:

The quality of the results is limited by sometimes only having access to smaller models and difficulty in running larger models at all; that can’t be fixed (yet). But quality is also reduced by GPT-2-small being trained on all kinds of text, not just poetry, which means sampling may quickly diverge into prose (as seems to happen particularly easily if given only a single opening line, which presumably makes it hard for it to infer that it’s supposed to generate poetry rather than much more common prose), and it may not have learned poetry as well as it could have, as poetry presumably made up a minute fraction of its corpus (Redditors not being particularly fond of as unpopular a genre these days as poetry). Finetuning or retraining the released GPT-2-small model on a large poetry corpus would solve the latter two problems.

The poetry samples above did not exploit finetuning because OpenAI did not provide any code to do so and . Fortunate, , which I could use for adding more interesting samples to my TWDNE and for retraining on poetry corpuses to compare with my previous char-RNN poetry attempts back in 2015–2016 (see the top of this page). An with support for training on GCP TPUs has been created by Connor Leahy (), who .

Training GPT-2-small To Generate Poetry

Data: The Project Gutenberg Poetry Corpus

“My heart, why come you here alone?
The wild thing of my heart is grown
To be a thing,
Fairy, and wild, and fair, and whole”


For the poetry corpus, Allison Parrish’s public domain (“approximately three million lines of poetry extracted from hundreds of books from Project Gutenberg”) will serve admirably. A few other possibilities surface in , like , but nothing particularly compelling.

As far as the text formatting goes, GPT-2-small is flexible, you can dump in pretty much any text into a text file to use as the corpus, but some text formats are better than others. You want something which is as regular as possible (in both syntax & semantics), but also one which is as close to the kind of text you want generated, but also which wastes as few symbols as possible. Regularity makes learning easier, and you don’t want to have to massage the output too much, but on the other hand, GPT-2-small has a narrow ‘window’ and no memory whatsoever, so if each line is padded out with a lot of formatting or even just whitespace, one would expect that to considerably damage output coherence—as most of the fixed ‘window’ is wasted on meaningless repetitive whitespace, while other changes like replacing newlines with the poetic convention of ’ / ’ are worse than nothing (since newline is 1 character vs 3 and maximally dense). Minimizing formatting also makes the cross-entropy loss easier to interpret or compare across datasets/runs: if there is a lot of formatting which is easy to predict or little formatting, the loss can look misleadingly good (or bad) as it easily predicts the formatting but struggles on more meaningful content.5

The PG corpus has a strange format: each line is a separate JSON object, consisting of one line of poetry and a numeric ID for the work it’s from. Fortunately, the file as a whole is in order (if the lines were out of order, training on them would destroy the long-range language modeling which is the Transformer’s raison d’être!), so to turn it into a clean text file for training on, we can simply query it with jq and strip out the remaining formatting. This provides a pretty good format over all: the newlines are meaningful, no symbols are wasted on leading or trailing whitespace, and it looks like what we want. It is imperfect in that metadata/formatting we would like to be there, such as author or poem title, is not there, and things we would prefer not to be there, like the prose prefaces of books or annotations, are, but hard to see how to fix those easily. Another flaw I learned about only afterwards is that the PG corpus has been censored ad usumm711 Delphini “egregiously offensive content” such as “racist/sexist/ableist” words like “Pakistan” or “homogenous” (while, of course, permitting words like “shit” or “fuck”). This corpus is unsuited for any serious academic work, but it should be fine for playing around with generating poems.

Setting up the GPT-2-small training environment & obtaining the poetry corpus:

There is an additional step before beginning training. GPT-2-small works with text in a “byte-pair encoding”, which is somewhere in between a character embedding & a word embedding. The point of this BPE encoding is that it is somewhat more efficient than raw characters, because it can chunk more common sub-words or phrases & this gets more complete words or phrases into the Transformer’s fixed ‘window’ of n symbols, but BPE still assigns symbols to individual letters, and thus arbitrary outputs can be generated, unlike word-level NNs which are more compact but trade this off by having a restricted vocabulary of m words seen in the training corpus and must treat everything else as the unknown token <UNK> (especially bad for rare words like proper names or variants of words like pluralization or tenses). The training code will encode the text corpus at startup if necessary, but for 117MB of text this is so slow that it is worth the extra work to run the encoding process in advance & store the results before training on it:

Training GPT-2-poetry

Then training proper can begin; my Nvidia 1080ti6 can fit a minibatch size of 2 (GPT-2-small is still a large model), and I’d rather not see too much output so I reduce the frequency of checkpointing & random text generation:

WARNING: the Python library “fire” used in the OA GPT-2 code is very treacherous— if you typo a command-line option! Double or triple-check any new options you set against the available arguments defined by train_main in, and keep this gotcha in mind if setting an option doesn’t appear to be doing anything. (I discovered this while being puzzled why --batchsize 32 did not lead to instant out-of-memory errors for training; similarly, if you make the mistake of sampling with the option --top 40, what you are actually doing is sampling with the default --top_k 0. Oops.) While nshepperd has removed use of “fire” in favor of saner CLI options, watch out for this if you are using the original OA code or other derivatives.

Some hyperparameters could use tweaking:

  1. runtime, temperature:

    ‘Temperature’ (0–∞) is used in sampling: the top-k most likely words are generated, and then selected randomly from; at 0, the most likely word is always chosen, while 1 means each is selected according to its likelihood, and it degenerates to a uniform 1 in k probability with higher values. In other words, the higher the temperature, the more chaotic or unlikely the generated sequences will be.

    In the original nshepperd code release, the default temperature setting for the samples during training, 1.0, is not the usual 0.7 everyone uses for GPT-2 prose sampling—although it turns out for poetry we don’t want it at 0.7 as that forces too many repeated lines & 0.9–1 turns out to be much better, so use temperature in that range when generating samples. (Higher still may be better but I have not experimented with >1.)

    If you are sampling after 15 May 2019, it may be a better idea to use a new sampling strategy, (which essentially sets a different k at each step to avoid sampling extremely unlikely words and greatly reduces the repetition problem), which can be enabled like --top_p 0.9.

  2. train time, learning rate (LR):

    A key NN hyperparameter as always.

    In nshepperd’s code, the Adam SGD learning rate is left at its TensorFlow default of 0.001, which works initially, but appears to be much too high for this purpose (perhaps because the minibatch is so tiny on 1 GPU). After training overnight, the loss was not decreasing below 2.5, so I decayed it manually to 0.0001 & resumed training (editing line 136 of to read tf.train.AdamOptimizer(learning_rate=0.001*0.10)), eventually decaying it again (to 0.001*0.0001) to get it down to a loss of ~1.95. (nshepperd has since added a --learning_rate option so manual editing of the source is no longer necessary.)

GPT-2-poetry Samples

After training an hour or two, a sample

Overnight samples during training:

# ...[115999 | 2375.96] loss=1.10 avg=2.31
# ======== SAMPLE 1 ========
# The sun is gone, and the night is late,
# The lights are few, and the chimneys close,
# And yet the moon hangs lonely
# Above the town, where the tides divide,
# On the shores of the silent tide.
# The tide is still, and the tide is at rest,
# And the sands are many, and many a breast
# And breast of the tide is weary,
# And far away the tide's last foam
# Stands silent under the waves;
# But the restless tide creeps slowly,
# And the tide is strong, and the tide is deep,
# And the sea, like a white-faced phantom,
# Floats motionless over the deep.
# The tide is past, and the tide is at rest,
# And the sands are many, and far apart,
# And the sands are many, and far apart,
# And the sea, like a white-faced phantom,
# Floats motionless over the sea.
# 'Tis evening, and he comes to me,
# With scent of flowers, from the village lane;
# And the wind is blowing faint and low,
# And the dew, like tears, on the hill-tops pale,
# Is drifting faint and cold from the eastern sky,
# As he floats adown the western wave;
# And the trees that bend their branches sigh,
# And the winds that waft a sigh,
# Are fading dimly away,
# As the tide goes wandering,
# Tossing waves of sand.
# And the cloud that floats and swells
# Has passed, like the wave on the bay,
# And the moon is slowly sinking low,
# As he sinks adown the western bay.
# There are waters blue and bright
# On the breast of the silent sea;
# And the foam is like a bright
# And lingering pearl of the sea.
# Oh, I would my heart might still
# Ever be where the waves are free.
# Where the foam-flakes fall and swell,
# Falling into the silent sea;
# Where the foam-flakes fall and swell,
# Falling into the silent sea;
# Where the foam-flakes fall and swell
# Falling into the silent sea.
# Oh, I would my heart might still
# Ever be where the waves are free.
# Far away from the sea and the tide,
# In a quiet little bay,
# Where the foam-flakes fall and swell,
# And the rocks are telling of a day
# When I would my heart could stay;
# Where the foam-flakes fall and swell
# Falling into the silent sea;
# Where the foam-flakes fall and swell
# Falling into the silent sea.
# Oh, I would my heart might still
# Ever be where the waves are free.
# With a love that never will,
# With a passion never still,
# With a memory never still
# For a love that never will,
# With a sorrow never still
# For a love that never will,
# With a passion never still,
# With a passion never still,
# With a passionate love that never will,
# With a sorrow never still
# For a love that never will.
# Oh, I would my heart might still
# Ever be where the waves are free.
# With a love that never will,
# With a passion never still,
# With a sorrow never still,
# With a passion never will,
# With a passion never will,
# With a sorrow never still,
# With a love that never will.
# With a love that never will,
# With a passion never will,
# With a sorrow never still,
# With a love that never will,
# With a sorrow never will,
# With a sorrow never still,
# With a love that never will,
# With a grief that never will,
# With a sorrow never will,
# With a love that never will.
# Oh, I would my heart might still
# Ever be where the waves are free.
# With a love that never will,
# With a sorrow never still,
# With a sorrow never still,
# With a sorrow never will,
# With a love that never will,
# With a sorrow never will,
# With a sorrow never will.
# With a love that never will,
# With a sorrow never will,
# With a sorrow never still,
# With a sorrow never will,
# With a sorrow never will,
# With a love that never will.
# With a love that never will,
# With a sorrow never will,
# With a sorrow never still,
# With a sorrow never will,
# With a sorrow never will,
# With a sorrow never will,
# With a sorrow never will,
# With a sorrow never will,
# With a love that never will,
# With a sorrow never will.
# With a love that never will,
# With a sorrow never will,
# With a sorrow never still,
# With a sorrow never will,
# With a sorrow never will,
# With a grief that never
# [116000 | 2385.67] loss=3.33 avg=2.32
# ...
# [133999 | 5919.61] loss=2.89 avg=2.21
# ======== SAMPLE 1 ========
# "When the night is gone and the sun goes out,
# "There goes my bride and my lover:
# "Can she never come to her wedding-ring
# "With the light of the seven candles?
# "Oh! to bed with the bridegroom and bride,
# "Or to lie with my lover!
# "He shall not come to my bed at night
# "For a little while longer.
# "She shall not come to my chamber
# "For a little longer."
# The night is spent, the sun has gone;
# The bridal of the bride hath done;
# The bridal of the bridegroom comes
# To wed the bridegroom's attour.
# I come to see the wedding-feast.
# He cries to me--"Wilt seek the bride,
# "Wilt seek the bridal?"
# "Oh! I have sought her mother's bower
# "And never found her! oh! my flower!
# "Would we should love as brides should do!
# "I shall not find her!"
# The bridegroom at the bridegroom's door
# Gave his bride a ring and a prayer.
# "Now, bridegroom, sing a bridegroom loud!
# "I shall not find her!"
# The bridegroom at the bridegroom's door
# Gave his bride a ring and a prayer.
# What means the bridegroom or the bride?
# The bridegroom's bridegroom waits to ride.
# And looks with wonder at the bride;
# "And does she dream?"
# "Oh! I have dreamed!
# "She dreams of my youth,
# "As one that hears--it cannot be -
# "The story of a marriage vow!"
# And a voice answers:
# "The story of a marriage vow!"
# And the words reach the bridegroom's door
# As the bridegroom at the bridegroom's door
# Kisses the ring with a bridegroom's kiss on his cheek.
# They are wed! They are wedded!
# Each is in his bridegroom's bower;
# Each hath his bride in his bosom now!
# And each hath his bride in his heart.
# She is wedded!
# With each is the bridegroom's bride!
# And love is a bridegroom's bride!
# She is wedded!
# With each is the bridegroom's bride!
# "They are wedded!
# They are wedded!
# They are wedded!"
# A man from a fair tower, where the birds of the air
# Fairer and fainter and fairer the flowers and the trees,
# When the sweet light fades from the gardens of day he found
# Came he in quest of a maiden, whose form was wildly wild,
# So she came at the summons. He came, and stood by her side;
# And he gazed on the dream of her marvellous face, and smiled;
# And he said:  "I came by the river's side, when the day was still
# And I think of my bridegroom--he--whom ye all have a will!
# She is wedded!
# And he loves me not, my bride!
# And he looks at her eyes with a love that seems to him divine,
# Then he pines, and thinks of her eyes with an inward passionate shine
# They are wed! They are wedded!
# By the river's side they are, bridegroom, led!"
# He hath summoned her maiden with the will, and she answers
# That he needs her for his bridegroom still.
# A man from a fair tower, where the birds of the air
# Fairer and fainter and fairer the flowers and the trees,
# What is the meaning of that--no--is it, Lord, where ye bear
# All the wonders of the world in this fair maiden's hair?
# What doth the meaning of that--no--nor what doth she know,
# She is wedded!
# What doth the meaning of that--no--she hath chosen so?
# She is wedded!
# She is wedded!
# She is wedded!
# She is wedded!
# She is wedded!
# She is wedded!
# She is wedded!
# And the will is good,
# In the light of days,
# In the heat and stress,
# It is wise and wise
# To be wedded!
# She is wedded!
# And the love is good,
# In the light of days,
# It is hard to live
# And bear the wreath!
# And he girdeth her to his bosom, and thinks of her
# As she stands, all white, in the open air,
# With the full moon facing her face; and her breath
# Cries and whispers and wails and cries
# ...[134000 | 5929.46] loss=2.12 avg=2.21
# ...[135998 | 6934.07] loss=1.96 avg=2.16
# [135999 | 6934.58] loss=3.06 avg=2.17
# ======== SAMPLE 1 ========
# re
# Their love and beauty are as one in dream,
# A visible sign of the things that were.
# But I have seen these things by all men's eyes,
# Felt them as kindred of man's earthly life,
# And with the instinct of the unseen spirit
# Been caught by likeness of the thing he is.
# My love is but the wind, and I have blown
# From earth to where I am, and I have seen
# The things that no man dreamed of; yet at last
# I know by my soul's sense of a sense of things
# That are not, and may be, but the things that were;
# And yet I know these things are not, but are
# As earth and heaven, if earth and heaven and hell
# Are but the same things that it seems. Yea, then
# I am the wind. God knows the ways of men,
# He knows the insensate secrets of delight,
# And they are mysteries, if there be any to be seen."
# But with that word the wind in wonder strode.
# He heard the rustle of the leaves, and saw
# The shadows move about him, and he leaned
# Against the doorway like a god, and knew
# The inner meanings of the leaves and streams.
# There where the trees lie down at their root-holes,
# There where the wind smells of the blossoming boughs,
# He saw the grass, and felt the green blades come,
# As if it were the buds and boughs upon air,
# And heard the green birds sing. He saw the fields,
# The trees, the rivers, and the flowers within,
# The birds, the grasses, and the living things,
# And the strange river on the shore that rolls
# Through all its quiet marge into the sky.
# There let him live till time should come, and then
# Let love be like the heaven, and we be one
# To love, and not be one, being all in all.
# And if he had not done me the good work
# Had it been well not I. The things that he said
# Should never be fulfilled by simple sense;
# For all must have a meaning in themselves.
# But he that works out of his mind is one
# With whom the things that are and are not are,
# And makes them meet and good. 'T were a good thing
# For him to work and win for me, and so
# If he were not I would have it all.'
# But he that lives and not lives in the world
# Was not more worthy of the hand of Fate,
# And knows life's meaning, and would seek for it
# Through failure, and in death's despite. For him,
# Who hath been stricken with me through the brain,
# Forget to tell me how his brother, he
# Whom he had saved and murdered--so let it be
# By some great memory left.
# But at last,
# As I said this, he saw me, and he said
# To one, whose face was grey with tears in me,
# "What is it? let me tell you who I am.
# Do you see the things that you have seen before?
# What is it?"
# "They are more wise
# Than wise men think of wisdom and good will,"
# Replied the other. "What I deem is good.
# The gods are good to mortals as they are,
# And they know well whereby we are born: but they
# Who have loved God and died to him the most
# Of all the gods are fallen into ill things:
# For God we know is good, and hath not been,
# And therefore must be, so it be, with men
# Who love, and love because we loved them not.
# Alas, I do not think that God alone
# Hath power over the earth to let the gods
# Face to face with the world. I hate at times
# The gods that made them: the gods that knew
# Their names are our own gods, and would not know
# One other reason, for I have the power,
# And all the gods are fallen into ill things."
# Then she said to me, "What may have been
# To have known, before I came into this land
# To find you in some other place and knew you,
# And know you, seeing so many and strange,
# And knowing such a godlike way to go
# Among the gods and suffer such long-sought.
# I can take my crown of gold and wear a garland,
# Take some crown for my sake, and the happy crown
# And let it be for all the years long held
# That I have known, and felt so like a god
# Some few suns live. My heart is all in all
# To live again, my life upon earth dead."
# So I said to the god that loved me well
# And longed to have him come back into my prayers,
# [136000 | 6944.39] loss=3.15 avg=2.18
# ...

The loss here is the usual cross-entropy we often see in architectures like a char-RNN. Typically, the best text generation results come when the model has trained down to a cross-entropy of <1, and 2–4 tend to be incoherent gibberish; in this case, GPT-2-small’s original poetry modeling capability is not too shabby (as demonstrated by the various prompted samples), and it shows decent poetry samples starting ~3.5. Given how large & powerful GPT-2-small is, even with this much poetry to work with, overfitting remains a concern—memorizing poetry is not amusing, we want creative extrapolation or mashups.

For this model & dataset, I trained for 519,407 steps to a final loss of ~2 in 72 GPU-hours; almost all of the learning was achieved in the first ~16 GPU-hours, and training it additional days did not do any apparent good in terms of the loss itself. (It is possible that the additional training was helping, because the remaining tiny changes in the loss might translate to large perceived quality improvements—while the loss didn’t change, the samples from later on did strike me as better. This was something I thought I noticed witch char-RNN as well, that the loss became a bad guide to quality when the NN had mostly converged.) This suggests that GPT-2-poetry was underfitting the poetry corpus & would benefit from an even larger model size.


Before sampling from any new finetuned version of GPT-2-small, remember to copy encoder.json/hparams.json/vocab.bpe from the 117MB model directory into the new model’s directory. I find higher temperature settings work better for poetry (perhaps because poetry is inherently more repetitive than prose), and top-k appears to work fine at OA’s top-40. So unconditional sampling can be done like this to generate 2 samples:

python src/ --top_k 40 --temperature 0.9 --nsamples 2 --seed 0 \
    --model_name 2019-03-06-gwern-gpt2-poetry-projectgutenberg-network519407
# ======================================== SAMPLE 1 ========================================
# --
# And it must be decided.
# It must be decided,
# And it must be decided.
# It must be decided,
# And it must be considered.
# It will be decided,
# Though the hill be steep,
# And the dale and forest
# Hold the land of sheep.
# And it must be decided,
# There's a jolt above,
# And its paths are narrow,
# And its paths are long.
# Yes, it is decided,
# And it is completely.
# All the hills are covered
# With grey snowdrifts,
# Shaded with a shimmer of misty veils,
# And the hills have a shimmer of hills between,
# And the valleys are covered with misty veils,
# And there lie a vast, grey land, like a queen,
# And they are not, in truth, but many and many streams,
# O'er the purple-grey sea whose waves are white
# As the limbs of a child of ten.  And there
# The river stands, like a garden-fair
# In the valleys of the north, the valleys of the west,
# Blue and green in the summer, and runneth softly forth
# To the blue far upland beyond the sea;
# And over the high white upland far away
# Floats a white and tender water, and wearily
# Through the trees the rosiest water-lilies play
# In the sun, and rise and fall--the purple and red
# Of the streams.  The waters are hidden in their bed
# By the stone o'er the darkling hills.  The waters run
# Like a ringlet under the stone.  The water flows
# Through the rocks like a river, and the stream
# Is a ribbon of gold spun by the sun.  It gleams
# Like a gold sunbeam shining through the gleam
# Of a sudden silver, and silently falls
# On the pool, and is lost in the darkling deeps--
# Sink, sink in the shadows, ere it flee
# Into the darkling depths.  And the waters sleep
# In the light of the moon and the silver of dawn,
# And silently float past the mountains of heaven.
# As we gazed the city fades into the clouds
# Of the sky, and we are above the roofs.
# And suddenly as the moon, flurrying,
# Dazzles the sea with her swan-throated song,
# And there is a faint far singing of birds,
# And a sound from the land, as of swarming seas,
# The grey sea, and the land that hideth rest,
# And the sky that hides the lovely green of God.
# So we are caught, like the moving sea,
# That calleth unto its sleeping
# Soft and still, like the moon that calleth
# In the twilight depths vast and hoary--
# Till we see the City changing toward the dark,
# And its changing towers in the distance darken.
# In the city is a calm and quiet street,
# Full of sunlight, and a smell of rain,
# That falls from unseen towers like soft white feet
# On sleeping city's rue and misty pane.
# There is peace, and a vague peace over death,
# And a far-off singing in the city's breath.
# And all fair cities must go to dust,
# And every body be one tomb--
# And all white houses dwindle and grow dull,
# And the city's breath is a dull death-blow.
# But this place is a place of peace and trust,
# And it is but a little street,
# Whose idle heads and sunken faces
# Are bright with light that makes them bright.
# Then it is not alone fair Town that lies,
# With open pillared streets beneath a sun,
# And many a weary world and dusty town,
# And a sunflowers and a great tide onward run
# In the blue of the heavens that are not gray,
# But only blue and pale, like tender wings
# Sailing with wide-spread, languid, luminous eyes.
# This place is the very heart of it,
# Whose quiet hours with its peace throng
# The silent nights and the perpetual sea.
# The City slept with her silent towers,
# A stream that ran in an idle stream,
# And a mist hung at the windows of the tower.
# And it was a street--a sunlit dream,
# A dream of a world that lay
# Open in the summer morning,
# And in its heart a joy all gay.
# For its sunshines and palaces were there,
# Till a wind came softly here.
# And it was a new, new city,
# A city that arose in the early morning;
# That opened its gates on June morning,
# With a sunset and a moonrise sweet.
# The city was a cathedral;
# And out of the sound of the bells and t
# ======================================== SAMPLE 2 ========================================
#  of the world
# The best, that, when once dead, is found again.
# And what is this?  Where can we find a place,
# Save in the solitude, where he may be
# The friend of all beneath the sun, and be
# An unseen presence, if the traveller's eye
# Can follow where he cannot:  there he stands
# Dark in majestic pomp, like those whom owls
# Could once have told down with a lion's maw.
# His form is like his fathers, and the crown
# Of all his race:  the very colours are
# As his to-day, which we must see and bear;
# The only parent is the creature's he.
# His face, where we have marked it, is but veiled
# In twilight, when we see, and he appears
# Himself in all his nature--where, if man
# Can recollect, he saw it in the frame:
# 'Tis clay wherever found--and so is called,
# When nature gives him back her clay.  It means
# That clay was form'd; but clay is form'd elsewhere;
# He needs must feel through all this frame, and, lo,
# The horse he rears, is human in his mind.
# So too, his nature is a thing apart
# From the great Nature, which has made him thus
# A likeness of himself:  and he beholds
# The creatures that he knows, and not intends
# To visit them, and only in their hearts
# Deserts them; and if they come indeed,
# And if the sea doth bring them, then the man
# Is still a child of theirs.  He can recall
# His mother's features and the father's look.
# And often he has said that he foresaw
# The sea, the winds, that he may all at will
# Be sea.  In short, the man is all he sees.
# He fears the sea may hurt him.
# Lashed to the helm,
# The ship was in the sea, and, on its moor
# And the sails furled, in silence sat the maid
# Motionless, like a star; no sound was heard
# Save of the distant ocean's fitful hum;
# The sounds of tempest came to him, his ears
# Mercurially listless, and his heart
# Disturbed like a distempered sea; he stood,
# And gazed from heaven in an unblest thought;
# He had not heard his mother's voice; he gazed;
# The mother's look was of a loftier mood;
# He had not heard his own; he had not heard
# What ever was, where his own heart has been;
# He had not understood the very thought
# Of his own heart, where life could find no shore.
# The sea beats on:  the vessel's bell strikes six:
# Dive down, O death! to earth, to heaven! to heaven!
# And it is sweet thus to be two souls alone:
# Dive down for home, and to the air renounce
# The galling bonds of everlasting life
# In some lone bark, that, dying, to the last
# Are still as death without her:  so to him,
# The mother's voice, still sweeter, spoke of home;
# And as the young man fell upon her breast,
# The mother's oracle, the words of death,
# Even as he spoke, a living death arose:
# He feels his heart rise, and ascend the sky.
# The wreck shall surely reach the sea; he dies,
# A mortal change, as earth, in which it was;
# And God, though dead, had still a dying man.
# But when they parted, he can never die.
# There are thousands, yes, there are thousands who,
# Without a mother, could not die unheard
# Of by a hand unseen:  yet some are sad,
# Lonely and wretched here, without a mate;
# Or if the grave touch, the great hearts' light
# Have no soft touch, even of a brother's grief
# Scarce suffered, they shall each a new life yield;
# And one, once more on earth, to heaven, or God,
# Shall meet his father's face, or bless his grave.
# Not vainly on these mocking thoughts he breathes;
# They sink to nothing when he sinks to rise:
# The tears of fatherly compassion reach
# The mother's eyelids, her, but not her eyes.
# And now a voice was heard by the wild bird,
# With words of comfort from the infant boy.
# Oh, had it stayed the angel's birth, and then
# Those tresses streaming, would have felt the strain
# For the bright star, and for a glorious man.
# It is a noble deed:  and, through the world,
# Doth woman triumph, though she suffer loss
# And poverty and pain, and,

Not bad.

Cleaning Project Gutenberg & Contemporary Poetry

“In the dark the sun doth gleam,
And in the dark the moon doth seem
But now the evening is begun—
Gone is the sun upon the earth!
The silver moon doth like a cup
Of blood-red wine, and as that cup
Is drained of life, doth quench no drop.
What man will drink such wine?”


noted the issues with the Project Gutenberg corpus and, as book-level transitions are solved, suggested a heuristic for reconstructing the blank lines denoting (presumably) stanzas: use the line numbers in the original JSON and look for gaps in the transitions. Since stanzas are still connected, <|endoftext|> is used for the book-level transitions, and a blank line is used for the stanza-level, preserving as much of the semantics as possible.

Presser’s Python script:

This is converted to the NPZ and trained as usual. I retrained the previous non-prefix 117M PG poetry model for ~30k steps (>16h?) down to a loss of ~1.73. (I used 117M instead of 345M for compatibility with my concurrent experiment in preference-learning training.)

The results are quite nice, and competitive even with 345M. Some selected samples:

The gods are they who came to earth
And set the seas ablaze with gold.
There is a breeze upon the sea,
A sea of summer in its folds,
A salt, enchanted breeze that mocks
The scents of life, from far away
Comes slumbrous, sad, and quaint, and quaint.
The mother of the gods, that day,
With mortal feet and sweet voice speaks,
And smiles, and speaks to men:  "My Sweet,
I shall not weary of thy pain."

...Let me drink of the wine of pain
And think upon the agonies of hope,
And of the blessed Giver of all good things;
For, man for man, mine is the deepest love
That sorrow takes upon the humblest soul;
But who hath learned how sorrow turns to gall
The places where my feet have trod before.

...And 'stead of light, o'er earth, o'er rocky mountains,
A slowly falling star,
Its pointed pointed splendor far uplifting,
Heaven's flowery path bore down;
Each cranny of the air a gracious feeling,
It waved divinely round,
It called us hence, "Come what wouldst thou here?"--
Sweet mountain, that I love,
With that bright tint of heaven above,
'Twould make me still to see
One like to thee,
As fades the light that seeks the wandering eye.

...The skies are smiling sweetly on,
And summer's fairest hours are gone.
Oh, blessed Mercy! how the blest
Taste life itself can truly taste.
Thy morn of days, with all its past,
May on life's tempest paint the last.

...When you come to die,
Every nerve and bone
Soon lulled in sleep,
Secure and free,
Sleep will seize on you.
When you come to die,
Every nerve and bone
Soon lulled in sleep,
Sleep will seize on you.
When you come to die,
Every nerve and bone
Soon lulled in sleep,
We'll still be free,
And you'll never escape from our woe!

...I would be all that I can do
And this to carry with me
Along with me, O brother,
And bid my lagging days relent
For every worthy deed done,
And glorious though the world be,
They never will repent me,
But in God's name endureth ever,
Whose blessed hope my soul abides
For refuge through the awful doors of death.

...We are old men, who pass
On the sands with gaze
Out of the narrow world of fashion;
We are old men, who stay
On a river's flow
And a common day
Where the life of youth is waiting,
And a longing grows
For the world of youth and beauty
Where the old man goes.

...When I am dead, my dearest,
Sing no sad songs for me;
Plant thou no roses at my head,
That by that token may grow cold.
My dirge shall be a muffled noise,
My trentals stiff with dread,
For he who once his faith hath won
Will never know it read.

...O beautiful, golden-bosomed ships!
O sunburned ships on the sea; O ship which breams
Above the waves and beams; O songs of love
Sent from the wide West, that shall sing us songs
In our hearts afar, as a summer star.

While training that, I recalled that my other major beef with the PG corpus was its absence of more contemporary poetry. I didn’t really feel like trying to scrape the Poetry Foundation (PF) website myself, but I gave Google Dataset Search another try and to my surprise, discovered . Aside from being large, it comes with interesting metadata: the title and author, but also a somewhat eccentric set of ‘tags’ describing the poem. They would be nice to add inline, allowing some degree of controllability (like my use of author or book ID prefixes, or CTRL’s use of subreddits).

The Kaggle Poetry Foundation scrape has numerous issues I had to fix before its format was acceptable for GPT-2:

  • I replaced prefixed whitespace, trimming leading/trailing whitespace in all fields
  • replace 3+ spaces with newlines
  • deleted all 2+ spaces
  • dropped poems with <100 characters (generally a scrape error)
  • remove Unicode junk
  • serialize it as title+author+tags (if any) / poem / <|endoftext|> (ie the inline metadata trick, allowing for potentially better learning and a small degree of control in conditional generation)
  • the final formatted corpus

Once the cleaned PG was done, I then swapped out the PG for PF dataset and began finetuning. (I could train on the combined dataset, but the PF dataset is only 20MB and at ⅙ the size of PG, training on that would take a long time to pick up on PF’s details.) Surprisingly, the PF dataset trained down to ~0.60 loss after ~10k steps, as compared to PG’s ~1.73, a decrease much greater than I had expected from providing some metadata, suggesting that modern poetry, being so prose-like, is much easier for GPT-2 to model—which doesn’t strike me as a compliment.

Some selected samples from the PF-finetuning:

"The New York Movie", by William Collins

for Billie Holiday

Billie Holiday is dead. I am unhappy,
standing there like a gray stone,
standing there in my sweater,
thinking I can’t kill you: I would like to be a big, red dog
with snow. I wish you a good night
with your face, your cheek and your legs, your legs.
I want to throw away this falling Sunday
with your face, my ugly body, my ugly body.
Nothing to give my body to,
nothing to give my heart to.

..."To a History Store", by Yehuda Amichai [Living, Death, Time & Brevity, Nature, Fall]

How hard it is for the living to die
holding only memory, and then to study the minutes
how like a trumpet of the future it sounds.

Gentlest of birds, patient as a body, cautious,
clear as the tongues of previous vervain birds.

Perhaps you are close to it, beautiful world.

..."Zagajas", by Robertizing [Religion, Christianity, Faith & Doubt, God & the Divine]

Our Lord who bore a child upon his knee.
We'll ask no more.
As sandpaper for what it wears.

Autumn, a new day.
Remember November, a new day.
Remember November, a new day.
Think of the silver
smiling into the barracud

Already at the edge of day.
Think of the hardson of earth
crumbling into something new

Into every tingling ring
to the full edge. Think of what

New, bright, revolutionary clouds

scattered out of the blue:
the clouds that dissolved them,
soundless, self-edesfilled.
Think of the cycles in and around of this

crescent myriads of ants starting
to collect beneath each other,

their seeds suddenly

burning each to the other, each

moving and flashing.

..."The Bean Eaters", by Rudyard Kipling [Relationships, Home Life, Pets, Nature, Animals, Landscapes & Pastorals, Winter]

The fairies were wonderful.
They trod the snow, chasing
the catkins to the north.

Frosty violin-skins were flying
and they began to sing,
leaving an echo of singing.
Then, as the she-torches rang,
a second spring
flowed up from the fur brush.
It was the strangest sight
all through the wintry night.

It was the woods, falling in long grass,
and I was thinking of you, Little Brother,

in the sweet marsh,
that I might recognize, Little Brother,

as I think of you.

..."In Golden Gate Park", by James Jenny Xie [Living, Coming of Age, Time & Brevity, Activities, Jobs & Working, Philosophy]

In Golden Gate Park’s the day is breaking, only
the timeless moments of the night sketch the sky’s
high promenade of flying goldenness now
and never a late, dissolving splinter of black glass.
But in Golden Gate Park’s the morning breaks. The sidewalks
bask to me like cars at a funeral or the stars
like blind lights waiting on cars long since gone.
There, to the streaming windowpane, the little birds
scarve to get ready to swoop, and the sky’s yellow
and gold. It is the end of hunger that slays the bird.

..."To Theodore", by Kenneth Slessor

Death may forgive, but love is better.

He that loves the rose
Whose pale cheek glows
With one hand swift and close,
Whose fingers move
The gold hair of the rose,
Gone to pass.
Where his lips draw breath
The bitter thong
Sigh as if Death had
No part with them,
He hears the song,
Hears the shout,
Saying me,
As I must.
Love is better, they say,
Than the loss they know;
Dreaming is worse, they say,
Love must hate so.
As his torch I carry the air;
He shakes my wings;
He speaks no word;
Saying me,
As I reach,
As he calls me,
Call him, O dear,
Call him, oh dear.
Love has been my constant care.

The contemporary PF samples properly mimic all the metadata and formatting, and are good, for what they are. (If you doubt this, read through a random selection of PF poems.) The ones I liked seemed like they benefited greatly from the PG pretraining. There are still a lot of flaws in the unclean PF data: run-on lines are particularly irritating, and appear to be flaws in the Kaggle scrape dataset rather than the original PF website. I have brought up the problems on Kaggle, but I doubt they’ll be fixed soon.

With PF done, I combined it with PG and trained on the combination dataset for another ~20,000 steps, yielding a final loss of 1.61. The combined model appears able to do both datasets well (the weighted average of a dataset with a loss of 0.6 and another dataset 6 times larger and a loss of 1.7 would be ~1.55, close to the model’s ~1.6), and the samples don’t appear to differ much, so I don’t excerpt any. But the combined model should make a great starting point for RL preference-learning training.

Random samples & model downloads:

Training GPT-2-poetry-prefix

The first version of the PG data for GPT-2-poetry just runs all the lines together, erasing the metadata about what book each line comes from. A good model should nevertheless gradually learn about the transitions between poems & whole books, but that is hard and there may not be enough transitions in the data to learn effectively.

Much like the char-RNN experiments on this page, there is no reason one can’t inject that metadata in a structured way to see if the model can learn to exploit the metadata; even if it cannot, the added metadata shouldn’t hurt that much because it is so regular & repetitive. Inserting the metadata also allows for some degree of control in conditional generation; one should be able to put in the book ID for, say, Homer’s Iliad as a prompt and get out a long block of consistent Homeric pastiche.7

Ideally, there would be unique IDs for every author, poem, and book and these would appear at the beginning of every poem and the end of the poem would be delimited with the <|endoftext|> symbol that OA’s GPT-2 models were trained with, but unfortunately only the book ID is available in this particular dataset. (Project Gutenberg ebooks do not include any metadata or formatting which would cleanly split each discrete poem from each other.) Like before with authors, the book ID metadata can be formatted as a prefix on every line with a delimiter like the pipe character.

Rather than start over with GPT-2-small again, GPT-2-poetry can just be further finetuned on this new prefixed version of the PG corpus to produce what I call “GPT-2-poetry-prefix”:

The loss of GPT-2-poetry-prefix will be much lower than GPT-2-poetry because the prefix is so predictable, but it will hopefully learn interesting things beyond that.

In other samples, the generated IDs switch in the first two lines, and while that’s not much to judge with, GPT-2-poetry-prefix seems to ignore keywords from the first line when the IDs change, and doesn’t repeat them in the rest of the sample or attempt to rhyme off them, which is further evidence it is successfully associating & learning to mode-switch.

Like GPT-2-poetry, GPT-2-poetry-prefix converged quickly to a final loss of ~1.6 after 224,474 steps taking 31 GPU-hours, not improving much after the first ~8 GPU-hours despite decreasing the learning rate. (Diminishing returns appear to set in quickly for finetuning GPT-2-small even if one has a relatively large new corpus.)

GPT-2-poetry-prefix Samples

Training samples

# ...[33999 | 4308.76] loss=2.45 avg=1.86
# ======== SAMPLE 1 ========
# 11|But I have had a day that was gone when I saw you in a dream,
# 1322|A year ago, a year ago.... I'm going back
# 1322|Oh, the years are many and the years begin!
# 1322|You and I have traveled far and wide,
# 1322|You and I have wandered far and wide
# 1322|Through the land I love so well,
# 1322|Where the beautiful land of long ago
# 1322|Died and ebbed and glowed,
# 1322|In the little green land of long ago,
# 1322|In the little green land of long ago,
# 1322|In the little green land of long ago.
# 1322|You and I have traveled far and wide
# 1322|Through the land of long ago;
# 1322|You and we have traveled far and wide
# 1322|O'er the land we love so well,
# 1322|In the little green land of long ago,
# 1322|In the little green land of long ago.
# 1322|You and I have traveled far and wide
# 1322|Through the land of long ago;
# 1322|And the years have slowly swept us on
# 1322|Till the dust has left the sod,
# 1322|And we all must rue the little day
# 1322|When we all are gone from God....
# 1322|And we all must rue the little day
# 1322|When we all are gone from God....
# 1322|I know a little garden very old,
# 1322|Built in a nook near the western sea,
# 1322|With doors and windows closely locked and barred
# 1322|And steps that made no sound but the world outside,
# 1322|And a wall of white-walled flowers all in a row
# 1322|Of violet bloom that just began to blow,
# 1322|And a dial that looked like a golden globe,
# 1322|Cool, square, and green, with the leaves and the grass,
# 1322|And a dial that looked like a needle of bronze;
# 1322|And I wonder if these petals are lost to-day,
# 1322|These petals will be forgotten to-morrow.
# 1322|"A rose would be a rose,"
# 1322|I wonder much at it;
# 1322|It does not matter what or who,
# 1322|So very, very much it matters:
# 1322|It is so nice to think
# 1322|The world is such a place for petals,
# 1322|And so much like a pink.
# 1322|"A purple fox-hawk lived down by the sea
# 1322|And danced and slept in the white-walled flowers,
# 1322|And I thought about a tree,
# 1322|A rose that dropped as an apple falls
# 1322|Into the water, and fell in showers."
# 1322|And I wondered what people said,
# 1322|And wherefore took the trouble,
# 1322|And wherefore came my little red bird
# 1322|To look at me and wave its wings
# 1322|For joy of self and for its red,
# 1322|And why I cannot tell;
# 1322|For it is so nice to think
# 1322|The world is such a place for petals.
# 1322|And so I sat and played,
# 1322|And the old clock set the chimney-top
# 1322|To ring time to itself and stop
# 1322|The time with a slow, frost-ridden drone,
# 1322|And a rustle as of bees.
# 1322|The house was old, and had grown lean
# 1322|As if it were but a leaf,
# 1322|But I found there was only one room
# 1322|And I slept on my stem at the end of the porch
# 1322|Like a child asleep.
# 1322|The door was old, but very young,
# 1322|And very, very young;
# 1322|And a weary smile on its satin lips
# 1322|Hung like a crown upon the night,
# 1322|And the roses crept into the room
# 1322|By the little red door in the wall,
# 1322|And I woke, and felt as a king is,
# 1322|And thought of all the trouble and pain
# 1322|Of a little life and a little love
# 1322|When only a boy is born.
# 1322|For my heart was old, and restless with the years,
# 1322|And my face with no trace of the things I had
# 1322|And no remembrance when I was twenty-five,
# 1322|And
# [34000 | 4318.77] loss=2.15 avg=1.86
# ...
# [47998 | 11504.75] loss=1.91 avg=1.84
# [47999 | 11505.25] loss=2.37 avg=1.85
# ======== SAMPLE 1 ========
# |So when the sun has touched the hill
# 30599|And made the earth a little heave
# 30599|Of gold-dust on the firmament.
# 30599|The sea is a molten pearl
# 30599|Silvering the yellow moon.
# 30599|Gold is the poppies of purple plush,
# 30599|Which make a star within a pool
# 30599|Of purple, blue, and green and blue.
# 30599|The sea is a molten pearl,
# 30599|Whose petals burn as light
# 30599|In the rose-yarn of the sea.
# 30599|The gold star beckons, sparkles, shines.
# 30599|The water gleams and glows.
# 30599|The poppies turn to a golden rim,
# 30599|And a thousand petals, one by one,
# 30599|Come dancing through the purple waves
# 30599|Of purple.
# 30599|My soul goes up to the sun.
# 30599|The sun, all gold, is gone.
# 30599|The sun is gone, the sun
# 30599|Is not more gold than my soul.
# 30599|What is the rainbow, my soul?
# 30599|The rain is falling in the tree
# 30599|In dewdrops falling, that are wet
# 30599|With dewdrops falling.
# 30599|I have heard music in the woods
# 30599|Under a great sky.
# 30599|Their notes, on a hundred harps,
# 30599|Dance by a liquid, falling star.
# 30599|The song of the whole universe
# 30599|Rings like a loon's tune.
# 30599|The rain is falling, my soul,
# 30599|In the tree-tops, the rain is falling,
# 30599|And the rain is not more musical.
# 30599|The trees, like great globed fruit in a garden of heaven,
# 30599|Are as full of little shining blossoms
# 30599|As the face of a child of ten minutes.
# 30599|The wind is playing a soft tune
# 30599|Like the silver notes of a bell.
# 30599|The grass is a dance for a child,
# 30599|And the sun is going down.
# 30599|The rain is playing a soft tune
# 30599|Like the golden notes of a bell.
# 30599|The rain is playing asleep in the meadows
# 30599|Like a sea of dreams,
# 30599|And the wind is playing a soft tune.
# 30599|The leaves are wet with the tears:
# 30599|There is only the wind that comes.
# 30599|The leaves are wet with the tears.
# 30599|The trees have bent to the wind,
# 30599|Like heavy blossoms, and nothing stays.
# 30599|The earth is flooded with tears
# 30599|Like small white stars in the sky.
# 30599|The wind's song is marching by
# 30599|Like a song in a wind from a string.
# 30599|It comes from nothing, it comes not from the years;
# 30599|An atom of dust and a tide,
# 30599|The ceaseless rain falls heavily over the world,
# 30599|The summer flowers are red
# 30599|And one like a gold cup, in a crimson cup,
# 30599|Flows out from nothing, and goes
# 30599|Without speech or motion nor sound.
# 30599|The yellow flowers all slip,
# 30599|All their leaves are wet,
# 30599|And their crimson petals are wet.
# 30599|The rain has driven the sun outside,
# 30599|The wind has driven the rain outside,
# 30599|The moon is going out and in,
# 30599|With the stars on the roof like snow.
# 30599|The rain has driven the rain outside,
# 30599|The moon is driving the clouds in the air
# 30599|Like white, wool flakes in a snow.
# 30599|The trees are wet with the tears,
# 30599|They weep in their night-black tatters,
# 30599|They moan like mortals who lost their way
# 30599|Because they were glad of the rain.
# 30599|The rain is falling, my soul,
# 30599|It has torn away the flowers,
# 30599|They are wet with the tears of the sun.
# 30599|In the black ooze and the snow,
# 30599|The cold winds, shuddering, are blowing,
# 30599|The leaves are blown like blown corn,
# 30599|There is only the wind that comes.
# 30599|Black and long!  The night is
# [48000 | 11515.52] loss=2.14 avg=1.85
# ...
# [166998 | 5752.38] loss=2.17 avg=1.52
# [166999 | 5752.88] loss=2.10 avg=1.53
# ======== SAMPLE 1 ========
# 26|His golden radiance is no more."
# 16452|"I shall not think of men in Argos more
# 16452|Than they are now, who many a bloody deed
# 16452|Wrought on the Greeks, nor yet of Ilium's king
# 16452|In arts and arts like these can speak the rest.
# 16452|But they--their kings--the Trojans and their sons
# 16452|Have fallen. Their deaths, the Grecians and their friends
# 16452|Have fallen in battle, from whom little hope
# 16452|To escape the battle, but the steadfast hearts
# 16452|Of heroes and of Trojans have become
# 16452|Inglorious still. The immemorial strife
# 16452|Shall rise for ever in a glorious day,
# 16452|When wars are waged between us and the Greeks.
# 16452|The battle shall be theirs, the mirth, the song;
# 16452|The mirth which all the Olympian people share,
# 16452|Shall bless the younger warriors with a joy
# 16452|So great, so glorious, and a greater fame,
# 16452|That all the Greeks shall learn, that in the van
# 16452|Ye stand yourselves, and they will praise your deeds.
# 16452|But I beseech you, if indeed by mine
# 16452|Unknown dishonour you be wrested hence,
# 16452|That with your lusts, illustrious and august,
# 16452|All others ye may vanquish. Now, my friend,
# 16452|Behold this prize to crown your father's pride.
# 16452|He said, and shaking both his palms, assent
# 16452|That I should also wish it. Thou art brave;
# 16452|Thou know'st how Menoetiades the swift
# 16452|Was dragged, of Hector and the fierce compeers
# 16452|And Phrygian warriors. So, we will dispatch
# 16452|Your bodies, then, yourselves to burn the ships
# 16452|In sacrifice; with torches and with bells
# 16452|To burn them, and with oxen to replace
# 16452|Your gallant friends for ever. But I wish
# 16452|That no man living has so long endured
# 16452|The onset of his foes, as I have power
# 16452|To burn or storm; for mighty Hector erst
# 16452|Was slain, and now returns his safe return
# 16452|To the Thesprotians, and each other's wives
# 16452|And tender children, and all other babes
# 16452|Assemble round me now, for ye have more
# 16452|To suffer than they know. Go then--the rest
# 16452|Will bear you safely; if ye dare to use
# 16452|The Grecians with your ships, leave not the flames
# 16452|Or fire, or slay yourselves and leave the rest,
# 16452|Ye may, at least, escape, but Hector's hands
# 16452|Will set you free; the vultures on the hills
# 16452|Held forth, and I will pierce those hollow shades
# 16452|Adown the trench, and ye shall find them there
# 16452|Scaped unaware. If, then, the Gods above
# 16452|Should bring thee to the fane of Peleus' son,
# 16452|In his despite, thou wouldst be left alone,
# 16452|Beside the yoke unwearied, there to stand,
# 16452|And from the Trojans scatter all thy dead.
# 16452|So spake the son of Atreus, and retired.
# 16452|Then, leaving Hector's body in the tent
# 16452|Of Periphas, King Agamemnon seized,
# 16452|And in his royal hand his spear adorned
# 16452|With silver, and with gold, studded thick all o'er
# 16452|With glittering tin, the weapon of Mars thus toil'd.
# 16452|Brave as he is, and of unwearied arm,
# 16452|So shall be thy revenge, if thou shalt make
# 16452|Fierce contest for the Danaï. But fly
# 16452|With shout and clamour fill'd throughout the host.
# 16452|So shall ye perish, slain by Hector's hands,
# 16452|Each living soul, and, flying into fire,
# 16452|Sink under Troy's wide gulfs. Be this the work
# 16452|Of your brave sires, and all the parents' hearts
# 16452|Shall prosper, one and all. But ye in haste
# 16452|(For ye are valiant) shall
# [167000 | 5762.79] loss=1.32 avg=1.53
# [167001 | 5763.29] loss=1.60 avg=1.53

One training sample is worth remarking on:

# [49999 | 12526.44] loss=1.83 avg=1.77
# ======== SAMPLE 1 ========
# ._
# 19221|The curfew tolls the knell of parting day,
# 19221|The lowing herd winds slowly o'er the lea,
# 19221|The ploughman homeward plods his weary way,
# 19221|And leaves the world to darkness and to me.
# 19221|Now fades the glimmering landscape on the sight,
# 19221|And all the air a solemn stillness holds,
# 19221|Save where the beetle wheels his droning flight,
# 19221|And drowsy tinklings lull the distant folds.
# 19221|Save that from yonder ivy-mantled tower
# 19221|The moping owl does to the moon complain
# 19221|Of such as, wandering near her secret bower,
# 19221|Molest her ancient solitary reign.
# 19221|Methinks I see her in her blissful dreams:
# 19221|--Or, fancy-like, in some mirage she lies,
# 19221|Majestic yet majestic, and of seems
# 19221|The image of the unconquerable skies.
# 19221|Methinks I see her in her blissful dreams:
# 19221|--Or, fancy-like, in some majestic cell,
# 19221|Where lordly seraphs strew their balmy dreams
# 19221|On the still night, or in their golden shell.
# 19221|There, in the calm of some Platonic dream,
# 19221|Sits she, and views the unclouded moon arise
# 19221|Like a fair lady full of realms divine;
# 19221|And, all at once, a stony face and bright
# 19221|Glittering in moonlight, like the noon-tints of a night.
# 19221|Methinks I see her in her blissful dreams:
# 19221|--Or, fancy-like, among the moonlight pale,
# 19221|A heavenly lark twitters with every gleam
# 19221|Of the rapt rapt beauty that she craves to hail.
# 19221|Or in the moonlight gleams she seems to see
# 19221|A virgin glory, and a power divine,
# 19221|And every motion of the looking-glass
# 19221|Is full of love and divine gentleness.
# 19221|Methinks I hear her in the twilight dim:
# 19221|--Or is it fancy-free, and musical,
# 19221|And full of music? Or some holy hymn
# 19221|Over some water-lily's long-lost lute?
# 19221|O no! it is not fancy-free, I see:
# 19221|Nay! fain would I thy spirit-hearts invest,
# 19221|And hear the paean of these dreamy rhymes,
# 19221|Might I but speak of my departed rhymes,
# 19221|And thou wouldst sing of my empoet-swans.
# 19221|O no! it is not fancy-free, I see:
# 19221|Even Poetry murmurs in a pensive dream,
# 19221|And like a breeze sings the blue mists of night.
# 19221|O no! it is not fancy-free, I see:
# 19221|Even Poetry murmurs in a pensive dream.
# 19221|How soft the zephyrs, how the languid hours,
# 19221|As in the noonday shade their limbs recline!
# 19221|How mellow streams the mossy banks assume!
# 19221|How solemn woods, low-chiselled, sumptuous green,
# 19221|Nod to the cadence of their hoarse refrain,
# 19221|As if the deep-toned muses' melodies
# 19221|With half-averted plaints and half-averted plumes
# 19221|In some majestic temple's quietness
# 19221|Had to the silver twilight slowly come.
# 19221|How solemn woods, low-couched, around thee lie:
# 19221|--Or is it fancy-free, and melody
# 19221|That makes the dull night long in worship held?
# 19221|Methinks I hear the harp's harmonious sound
# 19221|In some dim wood, when the deep shadows fall;
# 19221|And the low wind, like one that listens, makes
# 19221|In the still woods the harmony of all.
# 19221|Or in the moon's pale beam, on some hoar rock,
# 19221|Lonely and spectral, mourns her feeble woe;
# 19221|And as the slow waves roll, and, ebbing, break
# 19221|In music
# [50000 | 12536.67] loss=1.36 avg=1.76

The rhyming in this sample is so good as to be suspicious. It might also sound familiar—because many of these lines are being copied from , which opens:

The curfew tolls the knell of parting day,
The lowing herd wind slowly o’er the lea
The ploughman homeward plods his weary way,
And leaves the world to darkness and to me.
Now fades the glimm’ring landscape on the sight,
And all the air a solemn stillness holds,
Save where the beetle wheels his droning flight,
And drowsy tinklings lull the distant folds;
Save that from yonder ivy-mantled tow’r
The moping owl does to the moon complain
Of such, as wand’ring near her secret bow’r,
Molest her ancient solitary reign.

Some spelling differences aside, this intro is almost entirely copied from the 8 copies of Gray’s poem in the corpus; this extensive copying is not something I spotted in the GPT-2-poetry samples I looked at, suggesting that the scaffolding of the metadata did indeed help with learning.

Also interestingly, the copying only goes so far, as immediately after the final line about the owl, where Gray continues:

…Beneath those rugged elms, that yew-tree’s shade,
Where heaves the turf in many a mould’ring heap,
Each in his narrow cell for ever laid,
The rude Forefathers of the hamlet sleep …

GPT-2-poetry-prefix instead continues:

…Methinks I see her in her blissful dreams:
—Or, fancy-like, in some mirage she lies,
Majestic yet majestic, and of seems
The image of the unconquerable skies.
Methinks I see her in her blissful dreams:
—Or, fancy-like, in some majestic cell,
Where lordly seraphs strew their balmy dreams
On the still night, or in their golden shell.
There, in the calm of some Platonic dream,
Sits she, and views the unclouded moon arise
Like a fair lady full of realms divine;
And, all at once, a stony face and bright
Glittering in moonlight, like the noon-tints of a night. …

That is, it focuses on the female figure of the Moon in a way more ode-like than elegiac. These lines also do not seem to be extracted from the rest of Elegy either, as words like “bliss” or “mirage” or “dream” or “seraph” or “Platonic” do not appear in it. Some of the phrases like “blissful dreams” do appear in the rest of the corpus, but others like “some mirage” or “mirage she” do not. Nevertheless, the style is consistent throughout the entire sample and the quality is good, suggesting that while GPT-2-poetry-prefix has managed to memorize to a limited extent, it is nevertheless fully capable of generating good original text.

An additional example of memorization has been spotted; sample #17 in the 1000 unconditional samples is almost entirely a memorized copy of Percy Bysshe Shelley’s :

======================================== SAMPLE 17 ========================================
 of our mortal life, and made it known
32373|To be man's also,--as to make us one;
32373|A thing apart, yet of an individual
32373|In frailty and high elemental own.
32373|Hail to thee, blithe Spirit!
32373|Bird thou never wert,
32373|That from heaven, or near it
32373|Pourest thy full heart
32373|In profuse strains of unpremeditated art.
32373|Higher still and higher
32373|From the earth thou springest
32373|Like a cloud of fire;
32373|The blue deep thou wingest,
32373|And singing still dost soar, and soaring ever singest.
32373|In the golden lightning
32373|Of the sunken sun
32373|O'er which clouds are brightening,
32373|Thou dost float and run,
32373|Like an unbodied joy whose race is just begun.
32373|The pale purple even
32373|Melts around thy flight;
32373|Like a star of heaven,
32373|In the broad daylight
32373|Thou art unseen, but yet I hear the music of thy singing.
32373|Keen as are the arrows
32373|Of that silver sphere,
32373|Whose intense lamp narrows
32373|In the white dawn clear,
32373|Until we hardly see, we feel that it is there.
32373|All the earth and air
32373|With thy voice is loud,
32373|As, when night is bare,
32373|From one lonely cloud
32373|The moon rains out her beams, and heaven is overflowed.
32373|What thou art we know not;
32373|What is most like thee?
32373|From rainbow clouds there flow not
32373|Drops so bright to see,
32373|As from thy presence showers a rain of melody.
32373|Like a poet hidden
32373|In the light of thought,
32373|Singing hymns unbidden,
32373|Till the world is wrought
32373|To sympathy with hopes and fears it heeded not;
32373|Like a high-born maiden
32373|In a palace tower,
32373|Soothing her love-laden
32373|Soul in secret hour
32373|With music sweet as love, which overflows her bower;
32373|Like a glow-worm golden,
32373|In the silver dew,
32373|Scattering unbeholden
32373|Its aerial hue
32373|Among the flowers and grass, which screen it from the view;
32373|Like a rose embowered
32373|In its own green leaves,
32373|By warm winds deflowered,
32373|Till the scent it gives
32373|Makes faint with too much sweet these heavy-winged thieves.
32373|Sound of vernal showers
32373|On the twinkling grass,
32373|Rain-awakened flowers,
32373|All that ever was
32373|Joyous and fresh and clear thy music doth surpass.
32373|Teach us, sprite or bird,
32373|What sweet thoughts are thine:
32373|I have never heard
32373|Praise of love or wine
32373|That panted forth a flood of rapture so divine.
32373|Chorus hymeneal,
32373|Or triumphant chant,
32373|Match'd with thine, would be all
32373|But an empty vaunt--
32373|A thing wherein we feel there is some hidden want.
32373|What objects are the fountains
32373|Of thy happy strain?
32373|What fields, or waves, or mountains?
32373|What shapes of sky or plain?
32373|What love of thine own kind? What ignorance of pain?
32373|With thy clear keen joyance
32373|Languor cannot be:
32373|Shadow of annoyance
32373|Never came near thee:
32373|Thou lovest; but ne'er knew love's sad satiety.
32373|Waking or asleep,
32373|Thou of death must deem
32373|Things more true and deep
32373|Than we mortals dream,
32373|Or how could thy notes flow in such a crystal stream?
32373|We look before and after,
32373|And pine for what

The 87 lines beginning with “Hail to thee, blithe Spirit!” are all Shelley (with perhaps slight spelling differences), much surpassing the memorization for Thomas Gray. Considering the top-k sampling method, it’s amazing that the sample could so exactly follow “To A Skylark”. It turns out there are ~12 copies of the poem in the PG corpus (it’s a popular poem), so in retrospect some degree of memorization is not surprising, but that’s still a lot of memorization. The 4 lines beforehand don’t appear to be copied from another Shelley poem, making it even more amazing. It’s a pity that that sample did not continue further because one wonders whether it could have repeated the entire poem and what it would’ve done when the original poem ended.

Unconditional samples

“How the clouds
Seem to me birds, birds in God’s garden! I dare not!
The clouds are as a breath, the leaves are flakes of fire,
That clash i’ the wind and lift themselves from higher!”


For both GPT-2s, I generated 1000 samples as follows:

python src/ --top_k 40 --temperature 0.9 --nsamples 1000 --seed 0 \
 --model_name 2019-03-06-gwern-gpt2-poetry-prefix-projectgutenberg-network-224474
# ======================================== SAMPLE 1 ========================================
# |But I shall tell thee of the glorious days
# 1008|Of that old strife, wherein the truth of it
# 1008|Atoned, though at the most points of the pulp
# 1008|Be passed through: and about the torment, which
# 1008|A Hermit in his youth thoughte, yet made
# 1008|Still worse by his words, where he said, 'Peace!
# 1008|Keep silence here; because in any place
# 1008|I heard of charity the while I dwelt,
# 1008|And of her tears made e'en my lot a sting:
# 1008|And, but for those, in truth, to heaven were found
# 1008|The spirit of him offered by the sea,
# 1008|So that, out struggling, he made no denial.
# 1008|But the proud spirit, soon as he was silent,
# 1008|Embraced me, even in bidding me speak therefore,
# 1008|And with such gentle voice and such great front
# 1008|Came up unto my Saviour, that he made
# 1008|Full many a sign, how that all virtue he
# 1008|Would at his own discretion hear and mark;
# 1008|Then saw he the chief flower of love delight
# 1008|Within the living light, and made such pact,
# 1008|That it would make bliss vouchsafe to him.
# 1008|Paradiso: Canto XXIX
# 1008|The day, that most of us had sojourning
# 1008|'Mong the sweet spirits, from the highest shaft
# 1008|Of their kind master had not looked on them,
# 1008|Into the eyes forth from the bark they came;
# 1008|Two palms it bore of one and of the letters,
# 1008|And the other two with wings outspread.
# 1008|Not in beatitude do I behold them,
# 1008|But in and out of hope they uttered a
# 1008|Heaven-toned word of comfort, so that reached
# 1008|The upper light, which in attention hung
# 1008|With the low shores, that to the nether shores
# 1008|Were as the utmost thread 'to the high point.
# 1008|And that unceasing after-effiguring
# 1008|Of the eternal Sire, upon the which
# 1008|One live soul only drew its breath mysteriously,
# 1008|Up to the point, so pointed to the source
# 1008|Of what it had been, I never it knew,
# 1008|And hence never would have recognized.
# 1008|What verdure of an undivided heart
# 1008|Resolved, I say; and he complained e'en thus:
# 1008|"What from thy soul unto the Good I send
# 1008|I ever pray to, and by grace of that
# 1008|Pointing myself I pray thee to the world,
# 1008|To point thee the great mystery of love,
# 1008|From this, the bottom to the source of all
# 1008|Concerning thee; and not by its green leaves
# 1008|Of science so unsullied was the thought,
# 1008|As a small-handled cup, acquired by men.
# 1008|The mind's eye, taking from the mortal world
# 1008|All that it asks of bar or of the gold,
# 1008|With the same fury burns as it was wont;
# 1008|Now it may be by lantern or by shining,
# 1008|Since both thy and my love has made me its."
# 1008|The Almighty Father in his thunder made
# 1008|Resenting, and all round about Him round
# 1008|Went down his smitten steps, so that the air
# 1008|Impregnate came not from his visitations,
# 1008|Setting a day of darkness on all sides.
# 1008|Therefore mine eyes I lifted to the ground,
# 1008|And I beheld a river by the ice
# 1008|Chained up and flowing back along the ice,
# 1008|And suddenly before my feet it melted;
# 1008|And what it now behoves me to retrace
# 1008|The cause I had of it in heart I felt.
# 1008|As the Sicilian bull, that rightfully
# 1008|His cries first echoed in the mountains,
# 1008|Did so rebellow, with the sound of which
# 1008|It made my very blood to quicken well,
# 1008|The dolorous accents which envenom'd me,
# 1008|Forthwith I hasten'd unto where reply
# 1008|Was made: "O Ro! Brunhild"
# ======================================== SAMPLE 2 ========================================
# |Hear the tale that the funeral chant is telling,
# 2491|For the sorrows of other's children that dwell
# 2491|Like sweet flowers upon the wold?
# 2491|'Tis the tale of a life which is fled and gone,
# 2491|And the star of a hope which shone
# 2491|Bright above it, though dark may it be,
# 2491|For the hopes of a brighter day are fled
# 2491|And the joys of a happier lot?
# 2491|'Tis the tale of a life with the weary and sad,
# 2491|Where sorrows begin and rest.
# 2491|For only a song can the widow's soul glad
# 2491|Who sits musing 'mid shadows drear.
# 2491|And only a music, sad with its sighs,
# 2491|Till sad to the soul as death draws near
# 2491|As life on her fragile bark!
# 2491|I hear their voices faint in my slumbrous sleep,
# 2491|The music of lives that seem less real
# 2491|Than phantoms are dream-bound in duty's mystic keep,
# 2491|With music that seems to be more real
# 2491|Than phantoms are dream-bound in duty's mystic keep
# 2491|For souls that sin may not see!
# 2491|All round about us seems, in every place,
# 2491|As far off as the eyes of kith and kin,
# 2491|The ever-tremulous busy world's harmonious race,
# 2491|And I hear the mighty ocean tides,
# 2491|Feeling their strength, their might, their rhythmic din,
# 2491|Are calling me all into one wide choral face,
# 2491|And I hear the infinite singing of the winds,
# 2491|That seem to make me simply live!...
# 2491|The world seems a world that is full of sound and motion;
# 2491|A world of beauty and of music, where it lies;
# 2491|Yet all that is and has for me seems one more treasure
# 2491|Than all the world dreams leave in the skies.
# 2491|I hear the mighty tides of life,
# 2491|They're crying to me,
# 2491|They rise and sink in a restless strife
# 2491|Of endless song.
# 2491|Yet every stroke of sorrow's sword
# 2491|Comes surely from afar,
# 2491|That is true peace which is hard on board
# 2491|Though oceans be dark and terrors war.
# 2491|I hear the myriad singing words
# 2491|Of ocean's depths,
# 2491|They come like a song of broken birds,
# 2491|The music floats on the air and stirs
# 2491|My life to bear its measure in calms
# 2491|Of perfect peace, and it is good,
# 2491|But all is false peace only.
# 2491|When first I heard the autumn rain
# 2491|Sink down the hollows on the plain,
# 2491|I held it very near,
# 2491|And as I spoke to March again
# 2491|I felt the long, slow throbbing rain
# 2491|Creep from the earth in sudden flight
# 2491|Through all the veins of earth again,
# 2491|And in the sunlit, silent night
# 2491|The world grew far forlorn.
# 2491|And April came with rushing rains,
# 2491|And leaves about the naked lanes.
# 2491|I saw again the August noon
# 2491|Roll round the world in blazing heaps.
# 2491|And in the sunlight and the dark
# 2491|A thousand germs their pageant crush.
# 2491|And from the earth the maples bloom
# 2491|In odors of the breath of bloom
# 2491|And from the meadows and the hills
# 2491|The rosy clouds drop down their spilled spilled spilled spilled
# 2491|And drunken with the rain it kills.
# 2491|And soon above the hills shall crash
# 2491|The thunder of rain-wings,
# 2491|And all the naked trees and shrubs
# 2491|Shall lie, like naked, naked blades.
# 2491|Out on the hills there shall be rain,
# 2491|And the maples down the windy lane
# 2491|Shall bleed, and flowers shall weep again
# 2491|Through the weary hours of rain.
# 2491|They shall lie where the maples lie
# 2491|Deep in their bosoms, cold and numb,
# 2491|Each with its wound on either arm,
# ...

Download links again:

Some fun passages I noticed in the first 100 unconditional samples:

======================================== SAMPLE 2 ========================================
|Hear the tale that the funeral chant is telling,
2491|For the sorrows of other's children that dwell
2491|Like sweet flowers upon the wold?
2491|'Tis the tale of a life which is fled and gone,
2491|And the star of a hope which shone
2491|Bright above it, though dark may it be,
2491|For the hopes of a brighter day are fled
2491|And the joys of a happier lot?
2491|'Tis the tale of a life with the weary and sad,
2491|Where sorrows begin and rest.
2491|For only a song can the widow's soul glad
2491|Who sits musing 'mid shadows drear.
2491|And only a music, sad with its sighs,
2491|Till sad to the soul as death draws near
2491|As life on her fragile bark!
## Sample 3:
37804|The white-petalled white fox
37804|Opens himself to coolness
37804|In the late evening.
37804|But when the last child started
37804|The white fox to his feet flew,
37804|And the old fox was master
37804|Of all the magic heathen.
37804|Till when the faint huntsman
37804|Had snuffed the fragrant water
37804|Over his plump ears and skin,
37804|In the old way he knew not
37804|Till morn had almost shone;
37804|And then the fox came slowly
37804|And left the place unguessed;
37804|The white fox was not master,
37804|Although he had been master,
37804|Although he had been servant
37804|And now he could be master
37804|Of all the magic powers
37804|That keep the place enchanted
37804|In the wide earth and water.
## Sample 9:
36661|And the morn breaks, and, all the day,
36661|Red-clover'd birds with silver bill
36661|Flutter from tree to tree in flower,
36661|A quivering dew, a wind that wafts
36661|To haunts among the ancient woods.
36661|The golden-crested ilex, here
36661|Doth vine her purple cup; the deer,
36661|The wild-goose; and, in troops, the sheep,
36661|The goat, the sylvan-haunted elm,
36661|And the green-faced oft-gadding pine
36661|Blossom with purple.
36661|The lark soars up,
36661|And the hare loud answer make!
36661|Doves, willows, dunes, aslant the lake;
36661|Pair after pike sounds warbling;
36661|The reeds a triumph!
## Sample 14:
37452|I had a vision
37452|Of an old and stubborn old man,
37452|His hair was pale, and thin,
37452|His face was all forlorn,
37452|And the moon was full in the air,
37452|And a spirit passed over his brow,
37452|And its face was all for ever.
37452|And he spoke:
37452|'Have we ever a dream?
37452|Have we ever a vision
37452|Of the ghost's ghost?'
37452|The Master gave the word:
37452|'By the breath I know
37452|The meaning of Death:
37452|Can it be 'hush?
37452|Have we ever a dream?'
37452|The spirit said:
37452|'By the breath I know,
37452|The meaning of Death,
37452|You will see a ghost
37452|Stand by the door
37452|And enter.'
37452|And the spirit said:
37452|'By the breath I know,
37452|The meaning of Death
37452|You may understand:
37452|Can it be 'hush?
37452|Have we ever a dream?'
37452|The Spirit said:
37452|'By the breath I know,
37452|The meaning of Death
37452|You can see a ghost
37452|Stretched toward the door,
37452|And see a spectre
37452|Pass by the chamber door.
## Sample 24:
1333|Then, sweet heart, whisper, sweetheart,
1333|"Thou art sweet, but thy love is vain."
1333|I do love thee, my love,
1333|In a word, in a song,
1333|With the heart and the will,
1333|And the power of my heart;
1333|The power of my whole
1333|Of the poet's soul,
1333|And the heart and the soul!
1333|As the winds take the leaves
1333|As the flowers take the flowers,
1333|As the floods take the dew,
1333|As the salt runs in floods,
1333|As the salt runs in floods,
1333|As the snow in the seas,
1333|As the rain in the logs,
1333|As the wind comes and goes,
1333|As the sleet in the coppice,
1333|As the snow in the coppice,
1333|As the snow in the bogland,
1333|As the hail in the river,
1333|As the snow in the river,
1333|As the snow in the county,
1333|As the snow in the county,
1333|As the snow in the county,
1333|As the rain in the vale.
1333|As the stars take the dew,
1333|As the sparks fly from eye,
1333|As the sparks fly,
1333|So the hand of my heart
1333|As the heart of my art
1333|As the tongue of my lips,
1333|As the heart of my heart
1333|As the flame in the eye.
======================================== SAMPLE 39 ========================================
|And as the summer twilight,
34237|When the golden vinewood
34237|Strikes the silent midnight,
34237|Stands mute beside the brook,
34237|With a ghostly sense of the human heart
34237|Forgotten, yearning, sighing.
34237|I do remember how, long years ago,
34237|At the spring by the vistaed stream,
34237|I stood as 'neath the orchard, in the June,
34237|To the sound of the grass and the dream.
34237|I know the moss where the violets
34237|Quested the dew and the sun;
34237|The air above 'mong the orchards
34237|Murmuring ever of bees;
34237|And the heart that was filled with the music
34237|That came to the listening trees,
34237|While the bluebird's notes, as he piped again,
34237|Awoke the robin's golden throat;
34237|And the sound I heard, long years ago,
34237|Came through the wood and the dells,
34237|Bringing the sound of the violets
34237|And the perfume of dying wells.
34237|And the song I heard in the August dusk,
34237|In the August dusk by the lake,
34237|Was sweeter, from the full-leaved orchard,
34237|Than the sound of a happy brook,
34237|When it came to the school of my childhood,
34237|And to the school of the land,
34237|Oh my home of the woods, where the wild-flower
34237|Loses itself and dies!
34237|They give me back the old-time delight,
34237|The pleasant and the calm,
34237|When still the wind was blowing in the woods,
34237|And the children stood in the warm, glad school,
34237|And smiled as the dear lad asked.
34237|They give me back the pleasant book
34237|That gave my heart its fire,
34237|Those childish words, the constant brook,
34237|Those childish words, the tire;
34237|They made my soul to loiter!--Yes,
34237|They do, they make me blest!--
34237|The rest of the household, and the rest
34237|Of the parents whose hearts were filled with care,
34237|And who were sad in their care!
34237|Their voices!--Yes, and they do--
34237|'T was aye! 'T is aye! 'T is aye!
34237|And the dear friends, so dear to me,
34237|They still will live and die!
34237|I have not a moment now
34237|To forget when the morn is gray--
34237|To be happy, and cherish so
34237|The rose that is on her way.
34237|The evening breezes blow,
34237|And the stars shine out to-day--
34237|But I would not live in to-day,
34237|If I were as happy to stay!
34237|I hope that maybe one day,
34237|When all my work is done,
34237|My darling's coming away,
34237|To meet me in the sun;
34237|I hope that maybe I can see
34237|My Peggy's smile upon me.
34237|The evening wears an old, old gray,
34237|Which softly slants upon the way,
34237|Its shadows on the sunny day,
34237|Its shadows on the sunny day.
34237|O'er life, a sad, unwritten scroll,
34237|The words are like the gentle dove,
34237|That sails upon the nightly soul,
34237|Though none may read or hear reproof.
34237|And drooping o'er life's weary way,
34237|God grant the book may never end,
34237|The gentle words that cheer my way,
34237|The gentle words--they come to blend--
34237|The tender words of comfort and of love,
34237|The kindly words--they come to bring me joy.
34237|I know not if my path shall be
34237|Through the world's wild, woeful wild;
34237|But I know that sometimes, in the night,
34237|The dark will come, with wild delights,
======================================== SAMPLE 64 ========================================
2620|And be glad as the lark
2620|When the skies are clear;
2620|And send forth a breeze of love
2620|As of wings to our bark,
2620|And away with a joyous song
2620|As of streams in our ears,
2620|And away with a joyous tune
2620|As of birds in the spheres,
2620|And away with a joyous tune
2620|As of voices in trees,
2620|As of swans in the summer time
2620|When the grass is green
2620|And the air is keen,
2620|And the leaves are young--
2620|Then away with a song of praise
2620|As of flowers in Maytime
2620|All the sunny days!
2620|O beautiful, gentle, and clear,
2620|Illimitable and strong!
======================================== SAMPLE 72 ========================================
, he who had no need to fly;
24869|For in this moment of dismay
24869|The king who held that evil foe
24869|Threw Indra's son as he drew down
24869|The Lord of Life shaft-headed and bow.
24869|Then Indra, lord of every woe,
24869|The Vánar legions, with a shout,
24869|The Vánar legions met and fought,
24869|And straight they broke the tyrant's yoke,
24869|And hurled him at the giant, broke
24869|The mighty bow the giant broke,
24869|Which Indra, King of all the Blest,
24869|Had thrown by Rávaṇ's(924) mighty breast,
24869|The monstrous coil, the brawny hand,
24869|The monstrous mouth, the jaw, the jaw,
24869|The jaw, the jaw and bleeding jaw,
24869|The ungovernable host, the jaw,
24869|And the great bow which never bends,
24869|The arm, the fist, the knee, the ends,
24869|The body laid with mighty stroke,
24869|And the great bow which never bends.
24869|So, when the giants fought, and fell
24869|With murderous strokes, the giant fell,---
24869|So falls the tree with all his trunks
24869|Terrific in its death, that shoots
24869|Wild volley at the mighty trunk,---
24869|So fell the tree with all its boughs
24869|While all the vipers dug and sowed---
24869|So fell the tree with all its boughs.
24869|But Ráma's heart was sad within:
24869|He wept and mourned his captive's sin,
24869|For he had wrought a ruin yet
24869|O'er Raghu's son in his wrath,---
======================================== SAMPLE 78 ========================================
 on the bosom of
11014|King Deshav, son of Bhishma, sat in the shade of the trees,
11014|Humbu, the great, strong, beautiful, fortunate Brahmin,
11014|A king, a keeper of the law, a guide of the realm,
11014|His name unfolded through all time and space,
11014|A ruler of the realm, a keeper of the realm,
11014|And was worshipped, as was meet, by the Great Spirit of God.
11014|And all the days of his life he kept on striving with God
11014|For the union of faith; and at last all-wise he spoke to
11014|"Lord, I am the Brahmin's lord--and I hold thee thine inmost
11014|As I cast my life away from thee, my Lord, to-day!
11014|Therefore I cast mine body away from thee, my lord."
11014|And that, by constant penance, I might win thy favour
11014|So in the spirit's depths he plunged it into the sea,
11014|But, as the wave closed over it, the wandering wind
11014|Caught up the ship's chattels, and bore it with it to the beach.
11014|And Bhimasena seeing there the empty space behind,
11014|The wandering ship rocked in the dark and glowing heat.
11014|He sat upon the bosom of the Mother of God,
11014|He sat upon the emerald seas, meditating death
11014|Of the great sea.  He sat and pondered in his mind
11014|Upon the mystery of the sea, what gods the daring man
11014|Must have to tell of,--and this mystery,--when, in the morning,
11014|As, in the after days, the Lord of life should pass away,
11014|And leave the body alone to ride the ocean's force,
11014|To die in solitude, unknown, untroubled,--and unto him
11014|His world was opened; and as yet no living creature.
11014|And all the night he sat there, gazing in the east,
11014|Until the morning sunlight faded from the hills
11014|And dawn came, bringing darkness and the darkness awful,
11014|And to his soul came holy light from God, to cleanse
11014|All doubt and all resistance, till, in the morning of life,
11014|The coming of the Lord beheld his face.
## Sample 95:
24869|Canto XXI. Lakshman's Speech.
24869|He ceased: then Raghu's son repressed
24869|The sovereign of the giant kind,
24869|And thus with soothing words unsoft
24869|To Ráma Rávaṇ spake:
24869|"Come, with thy brother Lakshmaṇ, friend,
24869|And Lakshmaṇ, and the dame agree.
24869|Thou in the woods shalt soon be found
24869|And bathed in pleasant waters clean;
24869|Where thou shalt sit, and rest, and save,
24869|Well clad in coats of bark and hide,
24869|What days, what nights, what hours will pass
24869|That thou in holy heaven mayst see
24869|Thy darling with her night-made tressed
24869|Far from the forest. Thence will spring
24869|Sweet smells of pleasantness and light
24869|And bliss from the celestial string.
24869|Thence on the ground shalt thou be borne
24869|O'er the bare earth, O Queen Mosteer,
24869|And on the fresh bright earth where thou
24869|Shalt sit in state with Queen Sítá,
24869|In glorious heaven the nights and days
24869|Thou wilt be rapt by the great bliss
24869|E'en as the Lord of Gods is hearkening.
24869|The nights and days are thine, O best
24869|Of giant lords, and I, the best
24869|Of all who love the Lord of Lords,
24869|Whose might can turn the firmament,
24869|Whose might can sway the leafy bowers
24869|And turn each flower and leaf and bower
24869|To holy joy and blissful flowers.
24869|Ah me, the languorous days are come,
24869|And not a moment shall I see
24869|The happy days of Ráma's Queen
24869|Far from the light that round her glows,
24869|And marked with darkening leaves and boughs.
24869|Ah, whither would her steps be turned,
24869|And where the woodman's art had burned?
24869|Ah, whither would her steps be bent
24869|To turn her toil-worn heart once more,
24869|When all her hours were joy and peace,
24869|And all her hopes were set on store?
24869|Ah, let thy soul be comforted,
24869|Let trembling fancy still excuse
24869|The burden of a weary time
24869|That mars a saintlike life and use.
24869|Ah, if thy love were still the same
24869|That now I watch with toil and pain,
24869|That I could be for aid or flame,
24869|Could not my heart and bitterer gain.”
24869|And Lakshmaṇ to the forest came
24869|And told his tale with welcoming.
24869|He saw the tree where he was set
24869|With burning buds and leaves beset.
24869|He saw the tree where he was brought
24869|By Sítá of the glittering thought,
24869|And when the leaves were fallen, he
24869|Spoke of his lord the tallest be.
24869|"O Lakshmaṇ, I the deer will slay
24869|From thicket, cave, and mountain gray,
24869|Ere long shall I this forest seek,
24869|And Lakshmaṇ in the covert seek.
24869|O'er hill and wood the Vánar bands
24869|And watch the beasts of wood and sands.”
24869|He spoke: and Lakshmaṇ's love obeyed
24869|Nor did he speak as he was prayed.
# Sample 100:
38475|O Liberty, the patriot's sure defence!
38475|True to the man who fears a tyrant's eye,
38475|Preserve thy rights, and own his glorious cause,
38475|And yield the haughty title to a lie.
38475|No longer now on mean estate depend,
38475|And England owns thy sovereign vital force,
38475|And her best sons succeed to guard her laws,
38475|Or her best sons bestow a deedless course.
38475|Now, from that happy climate freedom's hope had birth,
38475|And made one day a milder country bleed,
38475|To the great cause that gave her aid is given,
38475|And to mankind one sure reward is even,
38475|Whilst I, perhaps, to distant climes must speed.
38475|To the same cause who has the cause to join?
38475|What foes against mankind may rise to arms,
38475|Boldly they fight, in actions of design,
38475|Yet all the same, and every day they charms.
38475|Ah, Washington! who can thy cause design?
38475|What can the nation do, or me, subdue,
38475|But still go on, in humbling folks admire!
38475|That we may praise thy conduct, that we fire,
38475|And for thy conduct many a hero dare,
38475|That we may rise, and cast the tyrants down,
38475|And tyrants fall, and fall the people crown!

Not bad.

These samples represent roughly top decile poem samples (~10 out of the first 100), at least by my selection.

& commenters highlighted a few more samples:

Thou know'st how Menoetiades the swift
Was dragged, of Hector and the fierce compeers
And Phrygian warriors. So, we will dispatch
Your bodies, then, yourselves to burn the ships
In sacrifice; with torches and with bells
To burn them, and with oxen to replace
Your gallant friends for ever. But I wish
That no man living has so long endured
The onset of his foes, as I have power
To burn or storm; for mighty Hector erst
Was slain, and now returns his safe return
My heart, why come you here alone?
The wild thing of my heart is grown
To be a thing,
Fairy, and wild, and fair, and whole
And they have seen the last light fail;
By day they kneel and pray;
But, still they turn and gaze upon
The face of God to-day.

And God is touched and weeps anew
For the lost souls around;
And sorrow turns their pale and blue,
And comfort is not found.

They have not mourned in the world of men,
But their hearts beat fast and sore,
And their eyes are filled with grief again,
And they cease to shed no tear.

And the old men stand at the bridge in tears,
And the old men stand and groan,
And the gaunt grey keepers by the cross
And the spent men hold the crown.

And their eyes are filled with tears,
And their staves are full of woe.
And no light brings them any cheer,
For the Lord of all is dead
Fair is the Lake, and bright the wood,
With many a flower-full glamour hung:
Fair are the banks; and soft the flood
With golden laughter of our tongue
How the clouds
Seem to me birds, birds in God's garden! I dare not!
The clouds are as a breath, the leaves are flakes of fire,
That clash i' the wind and lift themselves from higher!
In the dark the sun doth gleam,
And in the dark the moon doth seem
But now the evening is begun--
Gone is the sun upon the earth!
The silver moon doth like a cup
Of blood-red wine, and as that cup
Is drained of life, doth quench no drop.
What man will drink such wine?
There is no soul of earth or birth
Which man hath never known of earth.
There is no soul who doth not sit
And sing to it, and cry, "Drink!”
There is no soul whose feet are set
On youth's eternal paradise;
For all is a solemn harmony,
And all is a perpetual chant,
And all the world is a song of God.
There is no soul so wholly free
There comes a murmur low and sweet
As of far-off streams in a dream,
Or a murmur of many birds,
Or chime of little evening bells,
As of wedding-bells in the dells,
Soft, sweet and slow,
As of wedding belles that come and go.
A little green ribbon of lilies
By the door of my dear one's room,
A kiss on her cheek, and she whispers,
"I am the bride of the loveliest flower."
A moment we stand in the garden
Of dreams and things,
Dreaming of fairyland
And the fairy music there,
Sweet bells and dreams, and the fairy music,
The fairy songs of the air.

The top percentile of poems are probably quite good, especially with some light human editing to fix up the more glaring issues. To get a decent number of top percentile poems would require a lot of reading, but on the other hand, there is no reason why selecting or ranking poem samples could not itself be treated as a supervised learning task for retraining GPT-2-small-poetry on, by using selected/non-selected as labels and training to predict the probability of a given sample being selected, and then such a NN could be used to prioritize likely-good GPT-2-poetry poems (or any source of poetry) for human review (and, in a form of “active learning”, the results of the manual review can be fed back in as additional data to help discriminate between the best and the merely good samples).

GPT-2-poetry-prefix completions

Prompted samples can be done like this:

The downside of using the stock OA interactive prompt is that it returns on the first newline, so one either deletes newlines or uses a single line. Neither is good: a single line is hardly any context, while smashing many lines into a single super-long-line is dangerous because neither GPT-2 has ever seen poems formatted that way (only, perhaps, some prose that snuck in) and newlines have important semantic functions in poetry. So, to avoid either problem, I bypassed the interactive prompt entirely, and I modified the Python script to replace input (for taking 1 line of keyboard input) to instead read standard input (import sys; so I could simply pipe in multiple lines from files or from the copy-paste buffer using xclip -o.

The next issue in prompts is the question of the metadata; given that all the training data was properly labeled with origin and learning the meaning/associations was much of the point, it doesn’t make sense to not exploit this control in generation. If I was using authors, as with my previous char-RNN experiments, the prefix is simply whatever author one wants completions from, but in this case, it’s not quite so simple since we only have book IDs

If an author is already represented in the PG corpus, hypothetically one could look them up in it and see what IDs their poems were included under and use that, but that is a pain and doesn’t work for ones outside the corpus like Ginsberg. So, one could instead simply ask the model what prefix it thinks a prompt should use by feeding in the input several times and seeing what prefix it ‘hallucinates’ in the samples, and then adding that to the input for the real samples. If GPT-2-poetry-prefix consistently returns a specific prefix, then that is what it has learned and is useful scaffolding for the inputs; if it can’t do so consistently, then the prefixes aren’t useful for this particular input and it doesn’t matter.

So, to generate samples conditioned on relevant metadata, I pipe in the whole input unmodified several times, look at the generated samples for an ID, and if there is a consistent ID, then prefix it to the input and sample again several times.

Of course, now that everything is trained & I have a good input method, I want to see how GPT-2-poetry-prefix does on the same poems as GPT-2-small before!


First, “Howl”. Given that the Project Gutenberg corpus is entirely old poetry and wouldn’t include much in the vein of “Howl”, I didn’t expect this to be good. The finetuning would wipe out the knowledge of free verse.

Finding a good prefix was hard, also unsurprising—not much like it in the PG corpus! I ultimately had to settle for a “1997” prefix from a relatively free-verse sample for generating the 3 samples:

## Sample 1:
1997|Mona, gazing, voluptious! exhaling human blood.
1997|Bare, the sun, in a golden mist,
1997|Leans on the meadow; the clouds are gone;
1997|Yet the heaven of green mists around
1997|Cannot but give them a fresher light.
1997|The sun hath a juice that no time can stay--
1997|And a golden night-cap.
1997|How softly the quiet clouds go by:--
1997|One sees a star through the evening sky
1997|Shine in the valley.
1997|There is no sound but the stream that strays--
1997|The very air how soft, and oh!
1997|How the waters of the meadows flow,
1997|In the warm afternoon;
1997|And the gentle wind from the hills above
1997|Is a murmurous tune.
1997|Through the still wood the cicalas sing,
1997|And every blade grows young and wan:
1997|A man is carrying a heavy heart
1997|To save himself from the cold, and then--
1997|And the last of it comes.
1997|He makes his coat, and, as he goes,
1997|His garment of a color dies.
1997|It is the night-wind, blowing over and under,
1997|With its great peasings of thundering wonder and wonder.
1997|It is the night-wind, echoing and shaking,
1997|With its great peasings of wonder and wonder.
1997|The night-wind suddenly blows a rosy bud--
1997|It is the sound of the forest trees breaking out,
1997|And a great rumbling of branches, and hoarse, and heavy and strong,
1997|And it is a shrill voice, and clear;--and the wind that gives answer cometh out.
1997|it is the loudest the voice alone in the cry of the silence of the noises in the darkness and the sky, and the moon:--and the water in the little wind in the trees.
1997|sing of the raving, and the tree with its silence that followeth:
1997|And it too long and a quiet water and they that cry of the
1997|tree-playing sea, and the air and the silence.
1997|flashing of the sky.
## Sample 2:
1997|The thundering guns! the splintering masses! the thunderous mass! the splintering guns! A welcome!
1997|Gitcheers! Welcome! the river that from thee will pass:--the storm of flame burst!
1997|The fire that is thine will be there:
1997|Hear it, my Sun-god, breathe upon me;
1997|Be strong! We ask not one again.
1997|Thou, mother Earth, if this is done,
1997|That thou wouldst be for aye to-morrow,
1997|What is thy name? to what will follow
1997|The sun that to thee sang one breath,
1997|What is thy name? In what will follow
1997|The sun in darkness, breathless, fell?
1997|The sun must rise in a day's breaking:
1997|What is thy name? to what will follow
1997|The sun in the world's last spinning-wheel?
1997|There comes a word of a holy token
1997|From Jove, the cloudless king of rain:
1997|"What canst thou, then, that I should follow
1997|The sun in heaven as on earth again?"
1997|The Lord and the Lord are lords together:--
1997|The sun will rise in a day's faring!
1997|The cloud will darken all earthly matter;
1997|The sun will burst in a day's strong-wing.
1997|The earth will moan through the black night
1997|And utter a little scream behind:
1997|"What is thy name, then, to what wilt thou,
1997|Lulled in the silence by Jove of rain?"
1997|The Sun would sink in his blue heaven
1997|And be a torch toothed in the main;
1997|The earth would be a chamber filled once more
1997|And the voice, and the heavens rise of the sea.
1997|And break a voice of the music of the heavens and the rain.
1997|Erebell of that speaks the wind again, the thunder.
1997|And the day and the flowers of all things that sang, and ocean's breath of the sun,
1997|And thou and the morning, and all that sang thee and thunder, the rain and the rain and the rain.
1997|And thou, and all.
## Sample 3:
1997|The whole cathedral of the church; the human procession, _Lyral._ 4. It had hardly gone
1997|The temple, the Capitol, the nation, was still seated there, and the altar, the
1997|pulpit, the water-mill in which it was used by the Greeks, was a perpetual murmur and in
1997|the old church, or the great, or the small, or rather the less, would have gone on at
1997|The altar of God--the altar, which had not yet been raised to the new, or to the
1997|"Blessed are they, blessed, among those who went down! and now is my will so
1997|And they asked the priests and the virgins of God to give them a prayer, so the
1997|dwellers on the earth, in which they are thinking, "a little space is granted for
1997|ever. If he were his own son he is guilty of a wrong. But he is guilty of the loss of
1997|righteousness! If he were to die before his race, his race and parents have been
1997|And they answered with voices of joy, "Remember righteousness, and remember the
1997|Thescore, the obstinate, bad and bold, at the word of the priest, was crucified.
1997|And the pious fathers and the pious men had gone to tell their sons, and to tell
1997|Then the father replied, "I would ask not the grace of thy God to form the
1997|The son was seized with the thought of death.
1997|The priests, he received the gift with the body, and the priest at the same time threw him
1997|turban and the priest's mantle. This, however, was the ransom for the son received
1997|given, and the father increased family increased his blood. The priest departed; but his body was given, he went away with tears that a widow, he died. The sated the widow gave into the father in a
1997|And it was raised, after his body, for a sowing blood in compassion of him.
1997|inery. The venerable face was brought him into the house in all the funeral and buried.
1997|giveness. But before him. The priest, together with himself.
1997|The shepherd took the widow married is thus, the priest,

While they may be OK on their own and plausible as unconditional samples, they are disappointing as conditional completions, largely ignoring both the vocabulary & style. It would seem that the finetuning wiped out whatever it was GPT-2-small was using to generate its amusing “Howl” completions.


For “Ozymandias”, I fed it in a few times, and it seemed to like numeric IDs starting with ‘88’, so I used this as a prompt:

8820| I met a traveller from an antique land
8820| Who said: Two vast and trunkless legs of stone
8820| Stand in the desert... near them, on the sand,
8820| Half sunk, a shattered visage lies, whose frown,
8820| And wrinkled lip, and sneer of cold command,
8820| Tell that its sculptor well those passions read
8820| Which yet survive, stamped on these lifeless things,
8820| The hand that mocked them and the heart that fed;
8820| And on the pedestal these words appear:
8820| 'My name is Ozymandias, king of kings;
8820| Look on my works, ye Mighty, and despair!'
8820| Nothing beside remains. Round the decay
8820| Of that colossal wreck, boundless and bare
8820| The lone and level sands stretch far away.

Yielding (3 samples):

## Sample 1:
8820|... the last captive died of his own shall
8820|and the stateliest of towers was the king...
8820|And he entered the city.
8820|He sat ironing in his counting-house,
8820|Flinging his burdens from a hundred kings;
8820|(A queen had one, and her name was Pepban)
8820|And the king was the son of Pepban.
8820|Then he said: 'O son of Hernando!
8820|Not a crown of gold nor iron can he save,
8820|But he shall conquer by his own right hand,
8820|And the palace of the king be free and save,
8820|And the crown of the head be free and save,
8820|And the wealth of the world be free and save,
8820|And the wealth of the world be free and save,
8820|And the king be free and save.'
8820|And the seer took the gift. He received it:
8820|And he wrought a wonder in his brain, and died:
8820|But the seer, like a god, whose history
8820|Is unendowed, and expoundeth unto it,
8820|--So shall one tell the story of his glory....'
8820|And the seer surveyed his old hard case.
8820|But the king was old and rough, he died
8820|Of age and weakness; and the seer replied,
8820|'I will tell them to my tale, and they shall learn
8820|Who sing this song, and live like equal lives
8820|In the kingdom of the king.'
8820|Again the seer said, 'We will go and speak
8820|A gentler story in a palace: it shall keep
8820|These seers in silent wonder still and deep
8820|As sleep within his grave.'
8820|And the seer took the gold:
8820|'I have no doubt that I shall tell it out,
8820|And I knew that for me shall there be no fear,
8820|If I tell a lie or clear.
8820|But tell it forth,--the poet shall be clear,--
8820|For never any hero
## Sample 2:
8820| Alluding to the curious fable of the Aztec king, who, in his
8820| The palace called319iform; the people of the region of Bahr
8820| The palace called31920|--Rua, behold! a new king came to mingle these two
8820|_So that I wept beside the dead,
8820|And buried deep in thought I wept.
8820|I, like a desert in a glade,
8820|Keep wistful vigil all the night--
8820|And on the palace built a pile
8820|Of lamps that, kindled, slowly shed
8820|A ceaseless light upon the dead.
8820|On this I saw the shadows grow;
8820|And, dreaming there, I, too, lay down--
8820|The dead.  I wept.  I wept--upon
8820|A couch and saw the shadows too.
8820|And yet, indeed, I wept.  For such
8820|A cold to-morrow's light will gleam,
8820|Or such a flame toward one like this.
8820|Then shall the spirit's eyes grow dim;
8820|And in thy face she read my heart
8820|I sometimes read a thousand rhymes
8820|Of hopes, and fears, and hopes have part.
8820|A moment as I read thy thoughts,
8820|My heart was moved; I was athirst;
8820|Yet I was well content to be
8820|The slave of some great memory;
8820|Then, in this vision of the night,
8820|My heart grew very sore with grief,
8820|And I fell prone.  Oh, it was sweet,
8820|As I lay trembling near the dead.
8820|And, as the dead-man's face, that slept,
8820|Groping beneath the churchyard yestreen,
8820|I heard the murmur, soft and deep,
8820|Of a sweet spirit, wafted thin,
8820|That made the shadows seem to creep,
8820|And murmur, soft and low and mild--
8820|And watching, through the moonlight, where
## Sample 3:
8820| The ebbing tide bears only the sands
8820|And trees that wave their boughs to the western breeze--
8820|A broken landscape, where faint breezes waft
8820|The lingerers on their way to Indian seas--
8820|A land whose shores are sunset skies,
8820|Where the pale mists of evening slowly sail,
8820|And, as we steal, the rivers murmur strange
8820|To the low-murmured wind at fall of night,
8820|And to the distant rim of the far sky:--
8820|Here, silent as a solitary tomb,
8820|With all that life can purchase and wealth can lend,
8820|With youth's immortal spirit, thou, O Friend!
8820|Art not for me to give thyself to Art?
8820|To thee, who art the latest breath of lands,
8820|Whose fame has swept the world for which men sigh,
8820|I give my heart up with indignant hands,
8820|To thee, who art the only one I seek.
8820|For thee, alone with life and light impearled,
8820|I cast my eyes from out the vast of time
8820|Where nothing dwells but Destiny's brief stage,
8820|Unshrouded, till, in its frail waste of time,
8820|It shall re-risen and grow once more sublime.
8820|And then, within the bounds where all is strife,
8820|And death, and life, and love, are all my own,
8820|I take this heart up with indignant breath,
8820|That seems to harden with the weight of years,
8820|And with the bitter hardihood of tears,
8820|And with the heart that has no beat or fear,
8820|And with the hand that has no strength to bear
8820|Only the world's and with the heart that fears,
8820|And by its sorrows only known to keep
8820|The beautiful and glorious years we keep.
8820|Here are the little walls we meant to build.
8820|Here stands the doorway, opening to the East,
8820|The child-like likeness of a poet's child

Sample #2 is over-influenced by some prose footnotes/commentary which apparently were in PG, but the analogy of Ozymandias to Aztecs is a potentially fruitful one. And sample #3 here is a particularly good extension.

Essay on Criticism

Not clear what text exactly Scott Alexander used from Alexander Pope’s Essay, so I quoted the famous beginning section of Part 2. 3 samples strongly indicated Pope-like writing was associated with a prefix of ‘385’ (if not necessarily a full prefix) so I used 38511 for the following 3 samples:

## Sample 1:
38511|Pride even in numbers; wit's a kind pretence
38511|To something foreign still, but ne'er to sense;
38511|A constant waste of words, the world produces,
38511|A lazy sap, that nothing that's foreign to expels,
38511; that's foreign still unknown to the soul;
38511|A young, like, but in the world in motion,
38511|Obscending not, which smells all our own the worse than our own our own the soul's, and soul;
3851166|Like sense; which is like, but in all our whole.
38511|Which thus far more like, but in all things that's an excellence; and ne'er unchanged by which is folly's the worse, they give the worse maintained by which
38511|If sick of sense;
38511|Wholubil, or snug. ills, we know our own our first in sense the worse maintained between the worse, soon expired.
38511|Is blind and first blown on both being free from sense;
38511|From sense the worse maintained
38511|Wholuteness seems at once more used to sense the worse maintained by which
38511|Wholublime or enjoying sense; and first made to the worse, will's the worse maintained to sense;
38511|For sense; by which smells now discharged, and kept unseason'd from sense;
38511|Whose.  that's soon revived.  and then past the worse maintained, by birth to sense; by sense the worse, with weightyselves;
38511|Mankind by being all else barren; the worse maintained and last by birth to sense;
38511|Would cast, since built in nature lies from sense; for which smells and last, by repugither to sense;
38511|Whole; for our present, and first in life at all else to sense so long since built o'r by life to life, is soon revived by contact with heav' we know our own th
e worse maintained the worse it burns first made equal right;
38511|Is free.
3851166|Or dead: thus far more; who survey.
38511|Or wry's profuse and then dead: but what oft the worse maintained and next to life.
38511|From all; and
## Sample 2:
38511|There lies, that write the very best of all;
38511|For the lov'd fool, for those he courts and chokes,
38511|Is but a thorn inborn martyr in grief and sin,
38511|Who would all bawls and rattlantoms.
38511|Some hazels and isle from a thorn, or a starv'd for breaking hearts abh, to hogs;
38511|Or movel sooner writ, when by the starvels, or fombe.
38511|For men of any faultless wox and bribes.
38511|For wagers who should cut apart, and wak'd to make 'gin rights, for stink; for lamb, or chase; for lamb.
38511|Pounders or cast heel or a rope. For, for lamb, for lamb, for lamb or for lamb; for lamb, for lamb or starve.
38511|For no mean; for lamb, for sheep or for lamb, for lamb, lamb, for lamb, for lamb or for lamb, for lam, for mares.
38511; for lambs, a-heats. Suchley, for mares, for mares, for themselves.
38511. (for lambkins.
38511|mells; lam, lamb, lamb; lamb, lambkins; and other's for wares, lambkins; for struts; for sheep, lamb; or pricks, lamb, lambkins; for wer clothes; for mares: for sheep for lambkins; for goats for lamb.  for goats, lamb; for lambkins; and for moo: for hethers, for wark;
38511, lamb: for babes.
38511 unman; and for lamb.
38511; for lambkins; for mares; and the wox: for wheroat.
38511; for other goods, lamb. (let, frothriars, for spoles; a drum); for goats for lamb.
38511d:) for spoons, for spoons; for goats for whelornes, lamb, for mares: for syr trespass, for goats for mares: for lamb; for mares; for goats for sycam;
38511 theirs, for dog; for mawds.
38511; for whel
## Sample 3:
38511|They talk of constancy and faithless love,
38511|A seraph trembles at the specious glove;
38511|Nor in the rich confin'd relief of state,
38511|Find proud pretence, nor in the disdiscoveries of fate.
38511|For when misfortune makes choice remains the conduct's the prize half known,
38511|Can we secret soul without due, they fear of sense of more known.
38511|Some rise where' rights, they make it pays due.
38511861. Sense and judgment, as equal prize seem meanly, the reward the joy, as much possess the prize paid, as well reckon the prize we do not less dare not less keenly wise.
38511861|We see;
38511861|Of happy lovers ought, as well done, like a friend.
38511861|Know they ought, 'tis ev' is the other joy, as well worth a right;
38511861|The joy, as well might, as well may, as well may all is great.
38511861|Nor need of joys not as well maysters, as well as well may they give; but as little store; but as well as well as well may shewn, as much, as well we know, as well as well can be sure might prove, as well may well as well as well as well as well may view;
38511861|The mind: as well as well as well as well as much the fair as well as well as well as well as well as well as well may,
38511861, as thou woe.
38511861, when all.
38511861|Well done by the fair as well might rise, who merit thus prove by 'tis most of all are one;
38511861ly play the few.
38511861|There's well as well as well as well as well as well as well as well as well as well as well as well as well as not less admire;
38511861|Apostorably bright.
38511861|Well done by sight.
38511861, as well as well as by contact;
38511861|For they; the main aim

Alexander described his GPT-2-small sample from Pope:

It understands there should be line breaks, it understands the approximate correct length of a line of iambic pentameter, it understands how to talk like an overeducated 18th-century dandy—but it doesn’t appreciate rhyme or meter. In retrospect this isn’t surprising; GPT has no idea words sound like anything; it would be shocked to learn anyone uses language as anything other than text strings.

GPT-2-poetry-prefix still has “overeducated 18th-century dandy” down pat, but it manages to improve on the rhyming aspect: there’s quite a few rhyming lines in samples #2 & #3 (#2 seems to be screwed up by taking a digression into footnotes defining words and then bad sampling getting it trapped), like “pretence”/“sense”, “soul”/“whole”, “love”/“glove”, “state”/“Fate”, “bright”/“sight”, and a number of almost rhymes like “right”/“great”. One wonders if it’s learning by brute force and memorizing specific pairs of rhymes (although could there really be that many rhymes of “state”/“Fate” in even 3m lines of old poetry?), or if it’s doing something more equivalent to inferring the latent phonetics from the co-occurrence of byte-pairs? (That may sound unlikely, but word embeddings do many unlikely-sounding things with no more supervision than co-occurrence8.)

More concerningly, the samples are terrible. Pope’s poetry should be straightforward for GPT-2-poetry-prefix, as it follows standard meters and rhyme and relies on a classical vocabulary well-represented in the PG corpus. Why, then, are they so bad? I suspect this may reflect the corpus itself doing Pope a disservice. Pope’s inclusion in the corpus appears to consist of the following (grepping for “Alexander Pope”):

32190|The Works of Mr. ALEXANDER POPE. London: Printed by W.
32190|The Works of Mr. ALEXANDER POPE. Volume ii. London: Printed
32190|Letters of Mr. ALEXANDER POPE, and Several of his friends.
32190|The Works of Mr. ALEXANDER POPE, in Prose. Vol. ii. London:
32190|The Works of ALEXANDER POPE, ESQ.; vol. i. with explanatory

Checking PG entries and looking through the 32190 prefix, it starts:

32190|INTRODUCTION                                                 xv
32190|The Works of Mr. ALEXANDER POPE. London: Printed by W.
32190|BOWYER for BERNARD LINTOT, between the Temple Gates, 1717.
32190|This volume consists of all the acknowledged poems which Pope had
32190|The Works of Mr. ALEXANDER POPE. Volume ii. London: Printed
32190|by J. WRIGHT, for LAWTON GILLIVER, at Homer's Head in Fleet
32190|Letters of Mr. ALEXANDER POPE, and Several of his friends.
32190|London: Printed by J. WRIGHT for J. KNAPTON in Ludgate
32190|Street, L. GILLIVER in Fleet Street, J. BRINDLEY in New Bond
32190|Street, and R. DODSLEY in Pall-Mall, 1737. 4to and folio.
32190|The Works of Mr. ALEXANDER POPE, in Prose. Vol. ii. London:
32190|Printed for J. and P. KNAPTON, C. BATHURST, and R. DODSLEY,
32190|The Works of ALEXANDER POPE, ESQ.; vol. i. with explanatory
32190|Notes and Additions never before printed. London: Printed
32190|commenced printing his particular section of the octavos when the
32190|Quo desiderio veteres revocamus amores
32190|Atque olim amissas flemus amicitias.
32190|Nutrix mea fidelissima M. Beech, obiit 5 Novem. 1725, aet. 77.
32190|Edwardus Blunt, vir amicissimus obit, Aug. 1726.
32190|Francisc. Atterbury, Roffens Episcopus, vir omni scientia clarus,
32190|The fourth volume contains the Satires, with their Prologue,--the
32190|alterations. --_His Last Will and Testament._--WARBURTON.

This is perhaps not good training material for GPT-2-small-poetry/prefix and explains the bizarre degeneration—it is ‘expecting’ sudden random irruptions of largely-irrelevant prose such as introductions or footnote-annotations (rendered inline by PG’s text formatting). Other entries in the corpus will be more free of scholarly or prose apparatus. (In retrospect, a preprocessing step like dropping lines longer than ~60 characters might’ve been a good idea.)

8 Famous First Lines

The prefix trick doesn’t work on the 8 famous first lines nearly as well as it does with the long excerpts from “Howl” etc; I assume they are simply too short to home in on a relevant prefix. Nevertheless, I tried.

“Ulysses”, Lord Alfred Tennyson

“It little profits that an idle king,” yielded no consistency in prefixes, so I skipped adding one. 3 samples:

## Sample 1:
2428|And he who wears a crown, may die, but little lives.
2428|It was a pleasant day, when snugly shut,
2428|For many happy moons and many a sun
2428|Had lit the laughing waves with constant scarf,
2428|And heard the pealing waters on the run.
2428|'T was noontide; and with such a sky-lark flight
2428|The fleet white pieces flew in waves of sound;
2428|The silver stars, their fitful little light,
2428|Were beating on the liquid night around.
2428|Down went the cold, and all the dreariest blasts
2428|That ever chanced to blow, came mopping near:
2428|The first and fierce, the youngest, was our statesman here,
2428|Old Man-Autumnus!--An oak-headed oak!--
2428|The oak that in our court has grown so hard!
2428|"_Here am I_," answered he, "when I am dead,
2428|And round these sacred ashes I shall cling;
2428|And all the while my heart, unsatisfied,
2428|Holds but one clue,--and how must I be king?"
2428|How could he hope that a short space of day,
2428|And the warm blush of eve, and a fierce warning felt,
2428|Could prove the son of an old hunter, killed
2428|By the wolf's hounds in the wood! He crouched, and leant
2428|Against our shadowy robes; and the boughs bent low,
2428|Like the snow-flakes that cover the world below,
2428|Bent with a conscious motion. As the breeze
2428|Fluffs away, and the suns shivered and sighed,
2428|A shadowy motion, like a sigh, went by.
2428|It seemed a shadowy grief, and a mysterious
2428|Pestilence, like the voice of one who has sinned.
2428|At length, to his own chamber silently,
2428|We came, and gathering up the golden rim
2428|Of the great golden moon, looked down and wept.
2428|Then in a voice so soft, so
## Sample 2:
36287|Weary of wandering,
36287|Weary of loving,
36287|Can love a tawny cheek,
36287|Or a eye of ruby beaming through?
36287|My heart, why come you here alone?
36287|The wild thing of my heart is grown
36287|To be a thing,
36287|Fairy, and wild, and fair, and whole;
36287|With but the lion-heart to mate,
36287|And yet the heart to feel and know
36287|But in the wildest nightingale;
36287|The tongue you have for praise or blame,
36287|The heart that is more soft and tame
36287|Than the wild flower, this summer day,
36287|Shall never know your own dear way.
36287|Though cold and bleak your cheek I feel,
36287|Though bleak your mouth,
36287|Though cold your hand I would not steal,
36287|Oh, love, a thought beyond all words,
36287|Could I but win that word of birds,
36287|Could I but win that heart of yours!
36287|O sweet and tender is the lute,
36287|And soft the song the scarlet throat,
36287|When song-birds wake its wildwood lyres,
36287|And wild-bees through the glens go by.
36287|To you the wild-flower opens wide,
36287|To you one ripple stirs the tide,
36287|One long sweet longing seems to be
36287|A look of love made yesterday.
36287|To you the wild-flower bows its head,
36287|To you the wave is musical,
36287|Each kiss a kiss--but what of that?
36287|Ah, love, beyond your reach you keep
36287|All joys for you--ah, hear you me!
36287|Above your grave the winds are laid,
36287|Love lulls asleep within your shade;
36287|There is no rest for weary feet,
36287|The years are unfulfilled, I trow,
36287|And 'neath your sad and smiling eyes
36287|The sun tracks every path that lies
36287|Across your
## Sample 3:
1020|Who had too much of common human things,
1020|Being at peace with all he had on earth,
1020|Had something of a high and godly pleasure
1020|To do without him, since he had been
1020|In all the world so utterly,
1020|And that he had not far from his ambition,
1020|Had done no act for any human duty,
1020|Could not have felt his life's inheritance,
1020|If some good friend had lived to teach him thus
1020|His father's story of himself and country,
1020|And to restore his kingly name to men.
1020|O, the days that grow in my lament,
1020|And the nights that grow more still,
1020|When the earth lies awake,
1020|And the heaven it girdles,
1020|And the woods all shake,
1020|And the bright stars glitter
1020|Through the shadowy mist;
1020|When the night comes trailing her star-tinted hair,
1020|And the stars are the gray of our being,
1020|And the night wind is like some old sorrow
1020|Whose sweet touch is as of heaven,
1020|With its shadows and with its stars,
1020|Gathered in one place, is a child's face,
1020|A child's face made of golden sea foam:
1020|And the tears and the prayers that we utter
1020|Will be as those broken tears
1020|That fall upon bosoms, at the hour of their birth,
1020|With a sad and sorrowful melody,
1020|With a last, far-off singing of earth.
1020|Yet the night is a golden sleep,
1020|The day is like gold,
1020|And the stars go out alone
1020|Through the dark leaves and the darkness, alone.
1020|Night above and stars about,
1020|All our wanderings ended,
1020|Time above and darkness about,
1020|And the sun all hidden,--
1020|We are two two; the world is two.
1020|Howsoever the hours clap and clap,
1020|All the earth is happy,

“Sailing to Byzantium”, Yeats

“That is no country for old men.”, no consensus. 3 samples:

## Sample 1:
28591|I would to God that it all came to pass,
28591|That my old line of trouble and poverty
28591|Was left here desolate of my poor discontent,
28591|That the first cause of all my own heart's moan
28591|Was to be born in this sad world--a lone lone lone way?"
28591|I read it in my sonnet; his deeds were so brave,
28591|So kindly a husband, so cunning a guide;
28591|So tender a heart he went with his work,
28591|That his life never stinted its comforts at home.
28591|I read it in my sonnet--and he was so true,
28591|So loving my child, that it ever proved best,
28591|And though I was forced from his tender command,
28591|I felt I was growing up, when his voice, all tender, said,
28591|"If Love be unfaith, then he is the earth's father."
28591|I read in my sonnet--and he was so sweet,
28591|So eloquent was his look and his tones,
28591|That I smiled when I saw him, and ran forth to meet
28591|His father--and asked him in all things to meet.
28591|But some were too haughty, the most malgracious,
28591|To love their loved Father--the more to annoy;
28591|And some were too cold and abandoned, and when
28591|They saw how much reason has melted and flowed,
28591|They cried, "O Love, give us to love, give us fire,
28591|So we may die, as we will; O perfect, as dead,
28591|With the dead silence!" Then each took his own way,
28591|And, in its nature, an exquisite sorrow lay,
28591|Like love in a garden of fresh green;
28591|They were happy--too happy--by nobody knows!--
28591|The world, too, was happy by Roger and John,
28591|Though some at her heart cannot understand,
28591|Yet the whole world was happy on that first day.
28591|O thou who art so beautiful!
28591|What is the world without thee?
## Sample 2:
1365|O, there's a country, there's a town,
1365|And here's the place that I have stroll'd.
1365|See! there's a town!
1365|And here's the place, and here's the goal;
1365|And here's the flower for me and you--
1365|For you and me.
1365|Here's my little girl;
1365|And here's the spot that I call mine--
1365|"Come over here!"
1365|He's under the lilac-tree
1365|Where I loved so often to dwell;
1365|He's near his own tree-trunk,
1365|But here's his knee--
1365|And he's floating away
1365|To the roses in the churchyard,
1365|To the fields where I was born.
1365|Here's the lily-thorn, and here's
1365|"But where's the rose?"
1365|Here's a little lad;
1365|But a marigold he's brought
1365|Here's a ribbon at most:
1365|"Spring comes in the sun," says he,
1365|"And you dream of flowers gay
1365|On the hills of Arcady."
1365|That's the way he's trod;
1365|He shall be a king, and he
1365|Must have to sing--
1365|Ay, a king--a queen--a grave--
1365|Whose name was love, his heir--
1365|For the royal blood he gave
1365|At the royal marriage feast,
1365|And the crimson wine he drank
1365|At the royal marriage feast,
1365|And the crimson wine he drank
1365|At the royal marriage feast.
1365|They love to wake alive,
1365|To gaze at life and die;
1365|But they do not understand
1365|How many thousand voices call'd
1365|Out of the ages by the dead,
1365|Out of the ages by the living's voice.
1365|How many voices, and what else
1365|Did they call the living dead?
1365|All which of them was gone
## Sample 3:
1020|A time will come for tears and cries,
1020|And then I'll know her by their eyes
1020|I had her, but I cannot tell,
1020|And I am lonely in my cell
1020|Since the time the fair one came to me,
1020|I who loved her since my first year,
1020|Though in my youth she had been dead.
1020|I had grown quiet as the night,
1020|And I had won her tender sight;
1020|And the soft notes of her voice
1020|Would haunt me without joy or fear,
1020|Though the day had come.
1020|This breath in the air took the damp,
1020|And was gone to the damp,
1020|And was gone to the black,
1020|And was gone to the black,
1020|And was gone to the black,
1020|And was flown to the white,
1020|And was gone to the white,
1020|And was flown to the black,
1020|And was flown to the black,
1020|And was flown to the black,
1020|And was gone to the black,
1020|And was carried away in the mould
1020|With its all-written countersign.
1020|I thought the words would mean the fight,
1020|But the fighting face was not my sight,
1020|I thought the men would take a share
1020|Of pain and strife and weariness
1020|And leave me on the dusty street,
1020|Where I would meet that sorry man
1020|And tell him tales and talk to him;
1020|He said, "I'm sure I've done my best,
1020|But I'm not the man to meet the west."
1020|He said, "I'm going on our track,
1020|With my best friend here at the back,
1020|And my master, as I understand,
1020|And the rest of the country side,
1020|Where people go and people ride,
1020|And some end never to decide."
1020|"I can't," I ordered, "I can't!"
1020|He said again, "I say

Sonnet #29, Shakespeare

; no consensus, 3 samples:

## Sample 1:
28287|With heart and hand he went--
28287|The boy, like any lad of five,
28287|Was never seen to pair;
28287|For he ne'er met with a taste of grog;
28287|And now, howe'er his foes may say,
28287|I wish the boy would do all that he can,
28287|And so will merely go to bed;
28287|For, ere he goes to bed,
28287|He'll do most things, not more.
28287|He's gone to bed as readily;
28287|Then, before breakfast, he politely shuts
28287|The book, unless he's said,
28287|And talks of how the breeches blows,
28287|And how the shadows fall;
28287|And how a lady steals a cheese;
28287|And what a consternation comes to dress
28287|That baby sleeping there!
28287|He knows it is a drowsy child,
28287|And therefore it is very wild:
28287|For he ne'er goes to bed--'tis true--
28287|But he is there, and he is fast asleep.
28287|My dears, he says, if you can look
28287|In your writing, he'll just show you just
28287|How very fond you were of me,--
28287|And for how long I loved you so!
28287|And 'tis a very different thing
28287|He reads me, as he so loves me,
28287|Just to think and write of me.
28287|The moon's a very different thing
28287|I should prefer to read to him;
28287|The dogs do bark, the parrake buzz,
28287|And so do all upon me.
28287|One day there was a certain man
28287|Who said, "I love you better,"
28287|Because he had four teeth, they say,
28287|And still was most inquisitive.
28287|They sent him to a stylish drive,
28287|And, though they griev'd him greatly,
28287|They gave poor vulgar names--I trow--
28287|The names--and what do you think?
## Sample 2:
1852|He sees his wretched wife's fate, and dies.
1852|"And she, perhaps, may die in future time."
1852|In Moore's "Ballad of Trees and stones"
1852|A friend to Boston, now far known,
1852|In Boston town, "Is Alexis," sung
1852|A strain of triumph, and these words she had:
1852|"The city's honour, name, and wealth have I!
1852|"For this, a man of blood, toil, battle, strife,
1852|Mechanics labor, die, like this of life;
1852|The whole world's fame, the wind of it, the wave
1852|Of some great deep, that rolls to heaven or cave;"
1852|In that same song, by all the prophets sung,
1852|It is the song to tell the mighty throng
1852|Of those who for their country's honour take
1852|Life's journey to that sacred hill of Fame,
1852|When with great souls, that have been, and are still,
1852|The first great men, to do great things for ill;
1852|When, from no other world, the world's heart breaks,
1852|That noble man who, though he singly speaks,
1852|Has less of good to do than such a song,
1852|And lives in little more than we of him.
1852|What cause has he, this loss of friends and kindred?
1852|What hope of fortune? Wherefore has he perished?
1852|Why is he living on one man's estate?
1852|The poet sings. Where goes the soldier?
1852|The soldier dies. One man lives, all is one.
1852|Let him by no means lily stand alone,
1852|For one man is not worthy of it--all
1852|Is to be tasted by a thirsty sword,
1852|Which, while he lives, will drink up all he can.
1852|There's good in war, but better does he die.
1852|No, this, no more--that man is not the man.
1852|He may be used to giving freely what
1852|He gave, that he should dare
## Sample 3:
392|So many a noble man he scorns in fight,
392|Away he wanders, and his heart declines
392|To be among the dead, and live in death;
392|And now he meets with fear and trembling fear;
392|His courage fails, his faith is hardened, dead,
392|His courage fails, he cannot hope to appear
392|Forward he flies, now threatening danger dread,
392|He calls his lords, and charges them and flies;
392|In dust and ashes, as in sands that bee,
392|Now drives his foot, or in the grass that lies.
392|The trembling sailors, in their care to keep,
392|Now of their craft, and now of theirs surprise,
392|As if they wanted courage, speed, and sleep,
392|And all their thoughts on their audacious prize.
392|The watchful Franks at last had seen this feat,
392|Their flags down to their knees out-stretched they lay,
392|Their head all bare, their helmet on their feet,
392|Their bodies all of gold, their locks of gray,
392|Their bodies all so fair that death might deal
392|The blow, and break the shield, the hauberk too,
392|They drew their swords, and on their backs struck through.
392|After the first they flew in fierce career,
392|Like devils monsters come from hell and fire;
392|They changed their hue, their tunes, their forms, their cheer,
392|And drew their weapons; what their foes desire,
392|They turn and fly, and both their bodies spare.
392|The hosts, whose courage, arms, and arms, were fled,
392|Yet no dismay it had, when, with a spell,
392|The king's enemies were forced to stay,
392|That night each knight was used to aid them well.
392|But now the day drew on with dreadful ending,
392|The Franks, in fury, spurred them on their way,
392|And, for that great king's fall, had turned their backs,
392|And fled, before that host, to God and day.
392|To God or mistress, had they made request,
392|Save of his presence to those lords of rest,
392|That they would go, ere

“Invictus”, William Ernest Henley

; no consensus, 3 samples:

## Sample 1:
27139|And the stormy winds that blow
27139|All day long across the world
27139|I shall fly from earth below.
27139|"Oh, I am so weak and bowed!"--
27139|And the sun, in dazzling pride,
27139|Shook, and loosened as he heard
27139|All the turmoil of men's rights.
27139|In the town I should not be
27139|Wasted, wounded, and tormented;
27139|Made my spirit drunk with joy
27139|In the wonder of the sun.
27139|Oh, to see the sun shine through,
27139|And to hear the voice I knew,
27139|And to kneel because of old,
27139|And to speak, though it were gold!
27139|How it works, how it glows,
27139|In the sun, in the moonlight,
27139|And on earth, in the stream, in the odorous air,
27139|For the beauty of the moon.
27139|How it shines, how it glows,
27139|In the softness and the glow,
27139|And on earth, in the heaven, in the dusky air,
27139|Is the beauty that we know.
27139|Little, little, little sun,
27139|On your breast like the dawn of day,
27139|How I know about you always,
27139|With my heart and soul astray.
27139|Oh, to shine forever in your light,
27139|To shine forever in your beams,
27139|To shine forever in your brightness,
27139|To be shining, and to be!
27139|I knew the birds were going
27139|Out in the fair,
27139|Singing and working away,
27139|Going and staying and waiting
27139|Just as it might be.
27139|But the song of the children's singing
27139|Came every day to me
27139|Sweeter, and more than the singing
27139|That I may ever see.
27139|It rang so sweet, so clearly
27139|And far away,
27139|Making the angels, after
27139|Their work, but not a
## Sample 2:
38839|Where are the hours when we shall sleep?
38839|The night grows deeper, ruddy now,
38839|And on the east the morning light;
38839|No longer are the tears that weep,
38839|The noon-tide's drowsy murmuring
38839|That rises up through crimson leaves,
38839|As though the soul had fled away,
38839|To hide forever in the day.
38839|How sweet is the valley of Noodles!
38839|How quiet the little brook-heads there!
38839|What little bells by the cottages stood
38839|Tolled 'neath the heavy hours' flare!
38839|What tinkling bells the brook-folds heard
38839|As they crotoned by the millstone there?
38839|How soft the little brook-faces were
38839|Dimpled beside the cottage-door,
38839|And from the windows and out the trees
38839|The long-lost love-notes peeped once more.
38839|He came: the little brook he loved;
38839|And the brook beside his hearth he loved
38839|Where the crook whispered low, and the grasses
38839|That fringed its little patch of moss
38839|Beneath, the mossy water flowed.
38839|He came, and in God's blessed air
38839|He felt new life in that sweet air,
38839|New life and everlasting youth,
38839|And morning freshness, spring and sun.
38839|Weep not for me; our Father sent
38839|His little ones up to be content;
38839|As He had promised our poor ones
38839|We're glad to be thank'd like them;
38839|And let our taker daily say
38839|We're spared for our old love to-day.
38839|A rosy red rose is the rose
38839|And dear to me as children use;
38839|And though we sing it day by day,
38839|We know that mother's waiting here,
38839|And mother is the waiting-maid.
38839|God's plenty here, and He has sent
## Sample 3:
25953|The night that gives its glory,
25953|And the blue skies of the heavens;
25953|Sidney were worthless in that sight,
25953|All in a narrow space apart,
25953|I only had my life preserved,
25953|Left to myself to serve and guard,
25953|To guard and cherish, rest and guard,
25953|Of me the noblest was the king.
25953|Thus has my soul its homeward way,
25953|Thus has my soul its strength obtained,
25953|That what is dark may there appear,
25953|But evil in the light it feared,
25953|And in the light now burns my brain.
25953|With such a life, a wondrous life,
25953|Sounding through many realms and seas,
25953|The soul to millions makes at ease,
25953|And when it leaves the pleasant place,
25953|Binding in union is of grace.
25953|He who the body's life confers,
25953|From aught that is not, never slights,
25953|And when his body's life is spent
25953|The spirit cannot be content.
25953|I asked him what way he went,
25953|How many leagues he had not seen.
25953|The foolish wish I dared to own,
25953|To give the secret now I dare;
25953|I hid my foolish thought away,
25953|And lest the truth should be revealed,
25953|I thus began to test my will,
25953|And thus the answer was expressed:
25953|"I wish, O soul, that thou wouldst keep
25953|At present, in thyself at rest:
25953|But thou from sense hast naught to fear,
25953|And I my truth would not deny.
25953|And now by strength to hold thee fast,
25953|I do thee scorn, and I will give
25953|Henceforth my soul, and seek thy love
25953|As of the race of gods above."
25953|Then quickly I the thought requir'd,
25953|And thereupon this answer grew:
25953|"O soul, what wonder hath our Lord,

“Pioneers! O Pioneers!”, Walt Whitman

; no consensus, 3 samples:

## Sample 1:
31314|Do you think this summer would be just one?
31314|Did not all the gods in their thunder-storm,
31314|Say, would make the trees, temples, rocks, and tops, . .
31314|And now would they think these trees didn't know,
31314|But they couldn't be built, I think they do,
31314|And would like to live on in a greener place,
31314|And the gods have made it to be a good history;
31314|And we haven't the time, you see, to say.
31314|The woods, the fields, are green,
31314|And the flowers everywhere.
31314|The sun and the grass,
31314|And the birds and the fishing-trees,
31314|Are quiet everywhere.
31314|And there is the sun,
31314|And the flowers everywhere.
31314|The sun and the flowers,
31314|And the flowers everywhere.
31314|The sun and the flowers,
31314|And one after one,
31314|Are quiet all around.
31314|But there is an end.
31314|There is another place.
31314|The end of the road
31314|To which to go.
31314|The end of the road
31314|To which to go.
31314|Ah, why do you go
31314|Into the garden there into the forest,
31314|And into the wood, and into the village,
31314|And into the field?
31314|Why does the rain
31314|Make me uneasy?
31314|Do you see,
31314|Do you see?
31314|Listen to me,
31314|I am in a hurry,
31314|And away.
31314|There is a strange thing:
31314|I have that alone.
31314|It is not at all like I married:
31314|There is a new man,
31314|Who has just been married.
31314|I have a new hat,
31314|That's in my hat.
31314|I wish I could find out another,
31314|But then there is one . . .
31314|Oh, why do you
## Sample 2:
19|Let us sing this song of mine:
19|Where I am, there I'm,
19|Tell the mighty, mighty sounding,
19|Ocean's awful son of old,
19|In the islands of the blessed,
19|In the groves of Arcadian
19|In his cradle, cold and cold.
19|We, the Fairies, we the children,
19|We the islanders, the bold!
19|We are all that has been fashioned
19|In the wondrous dreams of old,--
19|We, the revelers, the giants,
19|We the children, and the bold!
19|O the wondrous song of battle!
19|O the spoils of men of might!
19|O the spoils of conquest, conquest,
19|Where the many are not quite.
19|By the walls of ancient stories,
19|By the marble-mantled wall;
19|By the chains of dread OENEUT,
19|And the marvels of the fall.
19|By the ramparts of the giants,
19|By the caverns of the deep,
19|By the graves of men immortal,
19|By the caverns of the deep,--
19|By the temples of the Morning,
19|By the temples of the Night,
19|Where the warriors and the giants
19|Met in vision met in fight,
19|And the dying captive maidens,
19|Sat in still and stately light.
19|Child of Earth, too fair for sorrow,
19|Mother of the Light of Life,
19|Fairest daughter of bright radiance,
19|We have sung thee a new song!
19|By the ramparts of the giants,
19|By the temples of the night!
19|By the peaks of Alleghany,
19|Where the eagle cleaves the sky,
19|Trojan prince of fire and glory,
19|Comes in might the ivory-headed:
19|By the palaceer Hippolyta,
19|And the brilliant dame who leads him
19|Where the women weep and smite them
19|On the great steed Aristagrus,
19|Sorrowing, binds his loins of golden
## Sample 3:
1165|Bring us back our olden treasures.
1165|In the small white house with the crumbs for food and fire,
1165|How you cursed the stars and drummers!  I hear you drumming.
1165|Are you getting sleepy-hearted?
1165|Are you writing toiling while night was ebbing?
1165|Do you yet look in the face so white?
1165|How your furrows, how you pined away!
1165|Up the ladder--who would fain be strong?
1165|Who would hunger after freedom long?
1165|Are you being hungry, child, and tired?
1165|Mother, mother, come blow me a song.
1165|Out of doors a man came and sat beside me:
1165|He was black, he was white, but I could not see him;
1165|He was robed in a sackbut of very little gold,
1165|And the words that he said were, "Hotel-fever!" . . .
1165|He said:
1165|He has seen Barra's vision.  He is not old;
1165|He is not fit for the desert.  He can find
1165|Sight and freedom, voice and heart and mind,
1165|And he has not learnt to forget . . . yet he
1165|Has a vision of Paradise.
1165|But, alas! the thing is over,
1165|And there is no chance can take me.  That is why
1165|I looked, he said.
1165|The place is filled with flowers,
1165|With curiosities and secret pain;
1165|One has a face like those of flowers,
1165|One has an accent like a bell.
1165|The small sad music of my days
1165|Moves on.  The grassy fields and lawns
1165|Are not more silent than the stones;
1165|But one face moves beneath the stars. . . .
1165|All this is very beautiful, perhaps;
1165|The hills and woods, the fields and meadows,
1165|The clouds and clouds and all the sky,
1165|The sea's sky and the hills' sky.

“The Love Song of J. Alfred Prufrock”, T.S. Eliot

“Let us go then, you and I,”; no consensus, 3 samples:

## Sample 1:
22142|When the summer comes again,
22142|When the birds on the sunny land
22142|Make the winter to come again,
22142|I may say that the happy hour
22142|When the harvest-time comes again,
22142|When the heart aches for the land of my love
22142|And the day is my heart's desire,
22142|When the harvest home is come
22142|And the days are my heart's desire."
22142|"Oh! what if we both would wander over the sea,
22142|Afar from the home on the lowland, and stray
22142|O'er the hillside and over the dale, as we stray
22142|O'er the hillside and over the moor, through the wood,
22142|By the light of the moon on the hillside and stray
22142|Till the golden mist o'er the landscape is gray.
22142|And often for me the olden log-house is seen,
22142|The cabin'd log and the swinging door,
22142|The house where I lay till the break of day
22142|Till the sun shone out and the shadows drew away,
22142|While the shadows still wandered o'er valley and hill,
22142|Till the heart stood still, till the hush of the hill
22142|Came o'er the meadow and wandered awhile,
22142|In the sweet early gloaming with autumn awhile,
22142|To lie in the light of the long autumn days,
22142|We two.
22142|The gray-haired woman that leaneth on my side
22142|And holdeth me fast in her arms,
22142|Hath bid me lie down 'neath the old oak tree
22142|That so thickly embosoms me.
22142|The greening spring came with its silent voice
22142|When the autumn leaves hang their green,
22142|And the winds from the woods whisper'd a strange
22142|remembrance of many a long vanished year,
22142|Till the gray shoon dropp'd 'neath the gray shoon's shade,
22142|'Mid the gray shoon's shade;
22142|But where is my home
## Sample 2:
30488|Over the plains and slopes, where the wild roses blow,
30488|And the low valleys and the brown hills meet the sea,
30488|And the winds hurry by with a cry, "We want you! we want you!"
30488|And we stand on the roadway, and walk in the rain,
30488|And weep for the dead that have gone from us--we are waiting,
30488|Waiting the word that the wind shatters over the hill.
30488|We have come back from the hills, from the plains where our
30488|blood was shed,
30488|From the fields where the grass was the ground and the
30488|We have come back from the valleys, we have come back
30488|from the hills!
30488|We have come back from the plains, from the plains where
30488|the grass grew green;
30488|And now that the voice of our winds shatters the
30488|world between.
30488|We have come back from the plains, have we come back
30488|to the graves,
30488|And the wind shatters the breath from the hurrying
30488|ship o'er the waves.
30488|We have come back from the plains, have we come back
30488|to the dead years,
30488|And the winds cry aloud, "Go up, and seek
30488|the faces of the gods!"
30488|We have come back from the plains, have we come back
30488|to the years,
30488|And the winds call out, "Go up, and seek the
30488|wilder world appears!"
30488|We have come back from the plains, we have come back
30488|from the days,
30488|And the swift leaves quiver, the bright leaves dance on
30488|our happy way;
30488|We have come back from the plains, we have come back
30488|to the dead years,--
30488|For the voices of men, or the gods' high
30488|heavenly joys!--
30488|We have come back from the plains, and
30488|from the long ago,
30488|Where in the cold and the gloom, on the
## Sample 3:
27494|To meet old friend Angelico's,
27494|And kiss the little one's beard.
27494|We shall have tea in the library;
27494|We shall have tea in town;
27494|By my good friend, Michael dissentund,
27494|I wish all you gentlemen!
27494|The sun has a face as fair as ever,
27494|As fair as ever was seen;
27494|As fair as ever was seen.
27494|There's a smile on his beautiful lips, my boy,
27494|As bright as the eyes of a lark;
27494|As bright as the sun at day-break gray, my boy,
27494|As bright as the eyes of a lark.
27494|There's a smile on his lips, my boy, my boy,
27494|As soft as the kisses he yields;
27494|As bright as the sun at noon-tide's end, my boy,
27494|As bright as the smile of a lark!
27494|Oh, there's something quieter than sleep, my boy,
27494|That twinkles in every eye.
27494|There's a smile on his lips, my boy, my boy,
27494|That shines like a lamp on the earth;
27494|As soft as the sun at noonday gray, my boy,
27494|As bright as the smile of a lark.
27494|A soft, pure love, that endures but ill, my boy,
27494|A soft Persuader that calms not nor charms,
27494|A king of the world-flowers and balm of the sea, my boy,
27494|A bright-hearted lover that seeks and finds, my boy,
27494|A rainbow of Heaven without a stain, my boy,
27494|A rainbow of Heaven without a stain.
27494|A sweet, pure love, that abides in a magical land,
27494|A kingdom-king, and a world-workeen band;
27494|A queen of the wildwood, a maid that is only one name,
27494|The bright-faced one calls her by name.
27494|A sweet, pure love, that all other names name, my boy,

Hamlet, William Shakespeare

“To be, or not to be: that is the question:”; some consistency, so prefix “1006”; 3 samples:

## Sample 1:
1006|Shall wretched mortals, for a certainty
1006|Of cowardice, from forth their secret places
1006|Take flight, and turn in view the point askance,
1006|From whence they can, on any object bent,
1006|So that the mirror may not abstain
1006|To 'GOD,' as in the sky grows colourd there
1006|The sunbeam.  If I had, I should prove
1006|The fountain's well-accustom'd form there.
1006|I should not be a word, that is not parting
1006|Betwixt the sun and moon, nor they, for sowing
1006|In opposite direction, doth import
1006|Their night and their moonshine to that shadowy land.
1006|Therefore into dark caves are we brought,
1006|Where underneath is alabaster, save
1006|Of dusky idol, in which shining moors
1006|'Cross-bows do us sometimes view.  Rumm, there,
1006|That consecrates the world, are seen below.
1006|How many stairs, as 't were, to o'erpass the threshold
1006|Of my triumph, when I did before thee,
1006|Enter'd, and were below, though light were not
1006|But in those hours to which this chant is not.
1006|For thereon is my mind alone intent,
1006|And this there be the singing souls, that wait
1006|Their Master's will.  If but e'en now thou choose
1006|To listen to this be melody sweet,
1006|Here may thy heart find rest.  If this thou hear,
1006|Thenceforth what argument there smote thy love;
1006|That, well perceiv'd of me in its ascent,
1006|Thou mayst at last the good from that have learn'd
1006|Of me made perfect, and of now prove o'ercome.
1006|That next I saw, which, hearing, thou shalt take
1006|For guide, arose from out my last embrace,
1006|As a good mother should, that her embrace
1006|Around each form Adams in her son,
1006|The same who
## Sample 2:
1006|Shall they not fear?--let one fear have his throat?
1006|No: for I see, and can perceive it not,
1006|That they, who were before me, were lamenting
1006|For the nature of the world, and in its light
1006|Have pity on their prison-gries, and therefore
1006|Spirits of low estate do not accuse.
1006|As oft along the still-perduing Nile
1006|Along the Indian isle the night-owl thronged,
1006|And with her breath had led them to the rivers,
1006|Thus each along the shore, in company
1006|With the fair consort of a lady, moved
1006|To supper with a working-place of eggs,
1006|Whereon all love was painted; and "O Lord!"
1006|The singer, as he finished singing, struck
1006|On the sea-shore at left and right below,
1006|Stay'd his steps, and thus in accents clear did:
1006|"Behold a wonder! behold a thing
1006|Which, whoso of mortal thing may tell,
1006|Hastes before him, dark and terrible,
1006|Of an eternal weal; and this accursed
1006|Foul thing, which may the doubting furnish try'd
1006|To feebor, I have set before him all,
1006|Who must to nothing turn.  But list: resound
1006|With me the patience, and the pity move."
1006|He then: "Now, sister!  think upon thyself;
1006|For how may I inform thee, if thou hearest,
1006|How soon and how slow time and means he takes?
1006|What world or language may at last he ask,
1006|Who to the world may'st give thee all things fair,
1006|Thou know'st, who hold'st it?"  Compassing that,
1006|And beck'ning the approach of his approach,
1006|I answer'd: "By new stars, and by George, spirit
1006|Of sign all cardinal virtues I'm assured,
1006|That wrong thy sight; but
## Sample 3:
1006|What became of thatch of yours
1006|That you sally forth so suddenly?'
1006|I did not know him; but he felt
1006|As freshening blood the air beneath: and tow'rds me
1006|He moved, and I towards him moved, and fond
1006|As father, and in words, and with the sweet
1006|And with the sweet comply'd; as he had will'd,
1006|I wept, and to his bosom lifting me
1006|Upward, and all the while distill'd my blood.
1006|Proceeding from the river, and I thence
1006|Shelter'd and still, I lifted up mine eyes
1006|Toward the sun upon the matin light,
1006|And looking, ever taking it in view,
1006|Again beheld it tow'rds the other bank,
1006|From those eternal Poubs where first it was.
1006|Unto the chasm precipitous, I found
1006|A narrow path and new to me expos'd,
1006|And to the left hand and the right mine eyes
1006|Directed, and between, on the sharp rocks
1006|Directed, and between, on each side me,
1006|And on the ground, were dark and cavern'd depths.
1006|Then to the right it spake, in semblance such
1006|As sometimes Indian women have beheld
1006|Saw in their dreams the latter realms of Hell:
1006|"Why partest from me with looks enchanted,
1006|"That farthest from thine eyes withdraw thee not?"
1006|I said it somewhat, and turn'd to the right,
1006|And when the chasm was gone, and I had reach'd
1006|The bottom, where the stairs were not beheld,
1006|Backward my eye alone its course directed,
1006|Directed to another care and deep,
1006|That in its bark it led me; and the sage
1006|To my recalling them returnd I not.
1006|"Of all that I have shown thee, since that gate
1006|We enter'd first, whose is the self-"

Romeo & Juliet, William Shakespeare

“Romeo, Romeo! Wherefore art thou Romeo?”; some consistency, with 1006 popping up again as a prefix (Shakespeare perhaps is memorable enough for GPT-2-poetry-prefix); 3 samples:

## Sample 1:
1006|If once thou see, the sport is vanity.
1006|But fix thine eyes beneath: the shameless'st thou.
1006|Ah! yet no better sight hath ever brought us
1006|To the destruction of Eve, from the rape
1006|Of the fair world, where wickedness lay hid.
1006|Ah! now before the good Augustus returns
1006|In daily deeds, let our pray'rs to the Furies
1006|Be changed; for that he longs counterfeited
1006|The world's adornment, which in books for us
1006|His saintly he still wears, who for us did
1006|Lays his beloved, us'd have thereby
1006|Been so enamoured of this roundelay."
1006|As it was spilt by him who severs the dry land
1006|And, each eve earlier than the setting sun,
1006|Wines underneath the rose that Tarquin leaves;
1006|So ravished I the more the fair, the more
1006|The great disdain, the pity, the long-sought,
1006|Toward the ribald nature of this law,
1006|Which the new world much sooner would be war
1006|With my sweet-will, the which the will hath oft
1006|Been usefully perverted.  I beheld
1006|Four mighty shades, and they besought for speech,
1006|But spake but slowly; whereto I re-turned,
1006|They showed me the grievous shame that clustered
1006|In their resplendence, who had all confounded
1006|With Adam, that Adam's fair abode
1006|Long time abode in peace.  "Remember," thus
1006|The teacher cried, "Adam, I the three."
1006|"And is this the true mettled nymph, that shine
1006|Thy twins now in th' ascent, as thou dost now
1006|Below?"  I said: and he, "My son, who know'st
1006|The art of song, and who but hear, may be
1006|Lest thou divide the loaves from five to six."
1006|Thereat the
## Sample 2:
1006|If truly the mad people of Raymond dread thee,
1006|Well hast thou 'scaped yourselves, they are already
1006|Each other's torment; and the wretches, mortals!
1006|Are to the shades of thy polluted ways."
1006|Whence she again replying, after us?"
1006|Purgator to the Red relates this; and Dante,
1006|"That spirit of the just hath spoken and said,
1006|'Purge now the mountain of the unquench'd, and make
1006|The sea of fire purge otherwhere he sees,
1006|Where Hood and other vain adventures 've led;
1006|Let the o'er-shadowing bridge ill-beseem'd
1006|Yoke up the mighty demons while they walk
1006|In different paths, and divers guides disentangle
1006|The tangled ways, so that no stork may turn them back;"
1006|And the good Master to me: "See thou findest
1006|E'en thus, how in the little one the race
1006|Is to be cumber'd with the broken blossom.
1006|That from this blow across theims of brightness
1006|Has not so virtue worried thee, it seems,
1006|That one with good intent in passing 'bove it
1006|Thou seest the harmonies of the mind,
1006|And therefore dost not comprehend them.
1006|That the disturbance which the sound invades
1006|Interpreteth not, maketh thy fancy
1006|Take in good hieroglyphs of everichoon;
1006|Thou seest that not only he who bids it
1006|Be, but that other who behind the hill
1006|Comes from step's shadow, comes in solitude;
1006|So that the evil one, who, in disdain
1006|Of creature freedom, 'bove the other waits,
1006|The same, who hoping in complacency
1006|To aid thee in thy loftier thoughts, like them
1006|I see now aids, which thee, thus overcome,
1006|With light returneth to thy mind from thee."
1006|Ill shapes that course which
## Sample 3:
1006|If thou no longer liv'st in the old age,
1006|To stop the running of the vein thou well;
1006|Thou by thy deeds hast purged away my youth,
1006|Strengthen'd by me in good deeds and old,
1006|And banish'd with my beauty from my song.
1006|O how much better were it I had Ruthven,
1006|With Beatrice and high excellence
1006|Of soul! to have companion'd such, who yields
1006|To her for intermingling of the world!
1006|But I see thee already clasp'd in thine,
1006|Not in thy only Heaven, but in the life
1006|To which thou crav'st it, and dost here renew it.
1006|This good companion, speak from whence it is
1006|That I should fear not, when the hour arrive
1006|That to your sighs a second time shall bring,
1006|Then with her grace, which that affection gave
1006|And your fair looks, which with her will I scan,
1006|Till my changed nature prove itself as yours."
1006|Whence her reply the thoughtful Sage imposed;
1006|"And I am He, whom thou didst long for, dead
1006|Cord de'il me not," he answer'd, "yet for thee
1006| hamm'st me.  Heart of Potentate was I,
1006|And life was in the willow.  That the words
1006|Which now I hear, shall marvel thee, wert thou
1006|A dearth in love.  Ah, how I long, long to
1006|For it am pain and grief!  that knowest well
1006|My soul's fair metes on earth!"  His sovran speech
1006|Bespake the meek thanks of the high Lord.
1006|I do not over 8 oceans turn," he then
1006|Began, "if new device can be of late
1006|Received in thee, tell us, who thou wast, and by
1006|The heels of th' mountain, times such words to me."
1006|"That I frizzano elsewhere is

“Jabberwocky”, Lewis Carroll

Upon request, I generated 100 samples of Lewis Carroll’s . Examining preliminary samples, the closest prefix was #24650, corresponding to , Wells 1899, an anthology of humorous children’s verse (which makes sense). “Jabberwocky” itself does not appear in the PG corpus but the “Jabberwock” is mentioned in one of the poems in Wells 1899, the acrostic poem “An Alphabet Zoo”, so, close enough.

Some samples:

24560|He found a foxy in the brake,
24560|A cunning fox of scarlet dye,
24560|And from that foxy followed make
24560|The scrawny fox in glee.
24560|He followed with his dam and horn
24560|To where the river-water runs,
24560|And as his living current on
24560|The river-water likes him up
24560|A mighty rocky heifer heaves,
24560|And in a single field, or twain,
24560|Shows like the yellow corn;
24560|And when the wind doth blow, so too
24560|Low in his bottom lies his head,
24560|And in the grass leaps up again,
24560|In fearful freedom unbetrayed.
24560|And while the sword-strokes gushed and ran
24560|A-gathering like a hurricane,
24560|The wildest war-cries, loud and shrill,
24560|Came surging through the hills.
24560|At times a wild yell rose in the air,
24560|From steep to steep, from moor to moor,
24560|A scream of horror and despair,
24560|Of death approaching neathernock,
24560|Of dying embers in the pore.
24560|"Thy shield, my son! thy sabre bright!
24560|My hands! thy lips! thy lips of flame!
24560|O hasten! I have done the same.”
24560|It made the blood tingle and squall
24560|To hear that wounded warrior's cry;
24560|It made the life blood quickly fall
24560|On the proud swine that fought and died---
24560|"Say on! say on! thou wounded swine,
24560|Say on! thou hewer of the sanguine!
24560|Say on! thou hewer of the sanguine!
24560|Say on! thou hewer of the sanguine!
24560|Say on! say on! that pierced youth
24560|Who crossed the heath in days of yore,
24560|And when he crossed it, "Say on!”
24560|It made the blood tingle and squall
24560|To hear that wounded warrior's cry;
24560|It made the life blood quickly quail,
24560|To hear that death did soon draw nigh.
24560|It made the blood tingle and squall,
24560|To hear that death did soon draw nigh.
24560|"Say on! say on! thou wounded swine,
24560|Say on! thou hewer of the sanguine!
24560|Thou art not doomed to fetch a pail,
24560|'Tis not the ebbing blood so drain.”
24560|And then it made the blood tingle,
24560|To hear that agony within:
24560|"Thou liest
24560|"It is morn!” said the clover-bush,
24560|As the sun came up the sky;
24560|"It is morn!” said the cattle-cock,
24560|As it drifted by.
24560|Then loud cried the blackbird, and forth
24560|On the moor the clamorer fell;
24560|"The rusty ruts!” he cried, "is morn!”
24560|And he opened his little cage,
24560|And went to his morning cell.
24560|Two ravening birds sat at his feet,
24560|One flew away and one fell,
24560|The blackbird with a shining face,
24560|Sang these grand welcomes to his race.
24560|"Who is the Blackbird?” they said,
24560|"Singing high, sing low, sing low,
24560|Sings the song for the giver and light?”
24560|"I am the Blackbird---a joy befall
24560|To the young in the woods so fair.
24560|I am the wild-duck, which roams at will
24560|Up and down in the gulphyry rill.
24560|I, the giver of rivers fair.
24560|I, the joy-devoted mole,
24560|I, the joy-devoted mole.
24560|I, the humming-bird, which sings so sweet
24560|I, the giver of rivers fair.
24560|I, the bird-cail'd eggs, with a silver sound,
24560|I, the joy-devoted mole.
24560|I, the giver of goodly things,
24560|I, the swift-wing'd eagle, I,
24560|I, the joy-devoted mole.
24560|From the sunny, sunny south,
24560|From the sunny south,
24560|The swarm departed,
24560|But woe to every wicked wight
24560|That ever them befel!
24560|"O shame to every wicked wight
24560|That ever them beguile!”


In May 2019, , which increases the parameter count from 117 million to 335 million, an increase of almost 3x. The 345M model has increased layer depth & more attention heads but apparently similar window size; as such, while it may not be much more able to maintain coherency across long samples, its coherency & quality should be superior to 117M within each window, as it can absorb more knowledge into its parameters & the increased depth may allow for more ‘thinking’ at each step.

The regular text samples from the 345M model struck me as somewhat subtly but noticeably higher-quality than 117M, so while I was hoping someone would supersede GPT-2 entirely by releasing a more advanced model (like a large-scale Transformer XL or Universal Transformer, or even newer models like the which marries bidirectional & unidirectional Transformers), I decided to train GPT-2-345M on the PG corpus to compare it with 117M.


This proved more difficult than 117M. The 117M model was already large, at 480MB for the whole, so making it 3x larger bloats it to 1.4GB on disk; and the VRAM use on a GPU is even worse: with 117M, a training minibatch of n=2 could barely fit on a 1080ti’s 11GB, but at 345M, n<1! The main culprit seems to be the self-attention layers, as regular self-attention scales more than linearly, so GPU VRAM gets eaten up fast, and apparently 16GB might not have been enough for 345M either. While I have enough system RAM to train 345M without any tricks, my Threadripper CPU is still ~14x slower than a 1080ti, and if one guesses that 345M takes 3x longer to train than 117M, and 117M takes 1–2 days, and CPU is 14x slower, then that’s <84 days for the poetry finetuning, which would not be fun.

To solve this, nshepperd extended his GPT-2 training codebase to employ a technique OpenAI helped introduce (and presumably used in training GPT-2, although the GPT-2 paper is silent on the details): . Gradient checkpointing is a space-time tradeoff which throws away some of the intermediate states of a NN, potentially greatly reducing total VRAM use, but at the cost of some slowdown when those intermediate states need to be recomputed for doing the backpropagation; the slowdown, fortunately, turns out to be fairly modest.

The downside of gradient checkpointing is that for 345M, it is still not memory-efficient enough to train it just like 117M—the self-attention layers checkpoint nicely (as the remarks9 apropos of needing extremely wide Transformer windows to accomplish ), but it’s not enough, due to the giant word/BPE-embedding, which blows out RAM usage. (Although it’s possible nshepperd didn’t implement gradient checkpointing quite right for GPT-2, as the OpenAI papers don’t mention any difficulties related to the embedding or using gradient checkpointing.) His initial solution was to simply disable training of the embedding and train only the Transformer layers, reasoning that the generic English embedding probably wouldn’t need to be trained that much as the Transformer layers are where the real work is done; much later, it occurred to us that the Adam SGD optimizer was part of the memory problem, as, being an adaptive momentum-based SGD optimizer, it must store a mean/variance for every parameter to adjust its updates per-parameter, which greatly increases memory use (and which gradient checkpointing does nothing about); when we switched to simple SGD, that freed up enough RAM to re-enable the embedding. This is important in part because the learning rate for Adam & SGD differs by orders of magnitude: a LR <0.01 seemed good for me for SGD, but Adam wanted a LR more like 0.00001. With n=1 minibatches, the training loss is extremely noisy and it is difficult to see the impact of any hyperparameter changes for the usual hand-tuning, so nshepperd also implemented a simple ‘validation loss’ function, which was helpful toward the end.

So, the upshot seems to be that 117M can be trained end-to-end with Adam on a commodity GPU in 11GB VRAM; and 345M must be trained with gradient checkpointing, and one must choose between either fancy SGD optimizers or full end-to-end training including the embedding; and 744M (released 20 August 2019) can’t be trained at all. Toward the end, I switched from Adam+Transformer-only to SGD+all, and this seemed to drop my 345M-poetry validation loss by ~0.01 to a final 1.915 (which is not nothing, so perhaps the embedding did need some adjusting for a more poetic vocabulary).

In total, I trained 345M-poetry for 815326 steps (minibatch n=1), with an Adam LR ~ 0.00001 and SGD LR ~ 0.001, over ~7 days (4–13 May 2019) on 1 Nvidia 1080ti; the necessary training time, with the benefit of hindsight, was probably closer to 3 wallclock days. 345M-poetry converged to a final loss of 1.915, an improvement of ~0.1 over 117M’s ~2 loss (so, in some objective sense, which is indirectly related to generated poetry quality, one could say that 345M is 5% better than 117M). I had expected somewhat more quantitatively, so I wonder if more aggressive training methods like cyclic learning rates+SWA would have worked if they were implemented in this codebase & I had the patience to wait a week or two for multiple cycles? In any case:


Training samples

[285999 | 5982.87] loss=2.07 avg=2.29
Saving checkpoint/run1/model-286000
Generating samples...
======== SAMPLE 1 ========
1170|Thy soul, thy very soul is burning!
1170|I look, the old-time woman I love
1170|Comes in for a kiss.  The room returns
1170|A ghost of its faded glow the same,
1170|But the old woman is gone as before;
1170|The moon has come down with a veil,
1170|And the old woman sits by my side
1170|Like a new moon rising from my grave!
1170|One day, I saw a bird full ripe:
1170|And a little flower in my garden
1170|Was the signboard's sign of peace to be.
1170|Like unto one, it told of the spirit
1170|Who dwells with all things and can see them not.
1170|The birds of all colors sang freely,
1170|And the flowers of every hue and scent
1170|Ran on like a dance up heaven's stair,
1170|With songs and flutterings of the blue and red,
1170|And sunbeams, glittering the branches overhead,
1170|And the wild sunbeam streaming upon them.
1170|And my soul was filled with a sweet desire,
1170|A need that was strange and divinely dear;
1170|And the birds sang so that the day was gone;
1170|And the flowers came full when night was near,
1170|With full hearts of light and fragrance blown,
1170|And the flowers and the birds would sing again
1170|Toward the evening that would bring them near.
1170|And the sunbeam was white as the sea,
1170|The flower was white as the evening skies,
1170|The star on the rainbow was gay and bright,
1170|And the bird of fire was a flutter of wings,
1170|And the flower and the bird would sing once again--
1170|Till I knew my soul was full of its sweetest dreams.
1171|I see how the leaves of the trees fall
1171|In the sunlight of morning;
1171|And in many a wood, with a thousand sounds
1171|And a thousand eyes, a thousand suns to behold,--
1171|I see how the leaves of the trees fall.
1171|How they fall in summer on the slopes
1171|And the meadows and the streams and streams!
1171|How they fall in the winter, soft and slow,
1171|How they fall in spring!
1171|Yet this fall must ever be beautiful;
1171|I think of the leaves, I hear them speak,
1171|The stars in heaven see them pass and pass,
1171|The rains are silent with them in their fall.
1171|The leaves of the trees are lightening and falling;
1171|The leaves of the trees are falling;
1171|And the leaves of the trees are all of their leaves
1171|Fallen on the fields,
1171|And they fall, and they fall on the fields,
1171|And I see how the leaves of the trees fall.
1171|The stars of the heavens look above,
1171|And the leaves of the trees are falling and falling,
1171|And bright winds sing,
1171|As in gladness, softly, soft and low,
1171|Loud and low;
1171|And in every star--what a wonder is it!
1171|The leaves of the trees are falling on the plain,
1171|And light-begotten clouds are lifting up the leaves,
1171|That fall like drops of rain.
1171|They fall on the flowers; and a thousand roses,
1171|And a hundred white lilies,
1171|And a thousand roses, and a hundred blossoms,
1171|And a thousand white lilies.
1171|They fall on the flowers, and droop and decline,
1171|And sleep in the sun.
1171|The leaves fall on the grass with an eddy they weave,
1171|And a thousand golden lilies
1171|Dream of the night of the leaves that was swept away.
1171|Their dreams are done.
1171|The leaves fall on the fields. They sleep in the sun,
1171|And the flowers shall be green,
1171|For the leaves of the trees are falling in the sun,
1171|With their rich lights of blue.
1171|I go to the door to-night,
1171|And look in the

[286000 | 6003.19] loss=1.03 avg=2.28
[347999 | 52462.43] loss=1.98 avg=2.06
Saving checkpoint/run1/model-348000
Generating samples...
======== SAMPLE 1 ========
 closed, or wearily
24869|Lest some foul fiend should be o'erthrown,
24869|Our hosts of fiends in wild rage we
24869|Began with furious rage to fight.
24869|We met them at the ramparts high,
24869|In numbers few and in strength not few:
24869|Whence we, our life like water spent,
24869|Abandoned the walls and basely fled,
24869|And the great flood swept like a raging sea,
24869|That o'er the deep ocean's bound
24869|Comes rushing on a torrent sea.
24869|And now the fiend's fury we met,
24869|And, like an eaglet chased by an eagle,
24869|We met, O King, with our lives in stake,
24869|As if our lives were forfeit by the foe:
24869|For in the raging tide that swept
24869|Our spirits from the wall and low,
24869|We lost and were scatheless once more.
24869|No foe may stand the fury of the king
24869|Who comes so fierce and bold to fight.
24869|And ere night's dark clouds have passed away,
24869|Our fleet will fly, and, safe return,
24869|We shall be in our country won.
24869|So will he take his own, O King,
24869|As some strong bull that fights in sport.
24869|The fiercest foe we ever saw
24869|Has strength to fight, has courage long,
24869|Is he whose flesh is fiercest, bold,
24869|Or who has lost his princely sway.”
24869|They heard with trembling hearts and fear
24869|The angry speech of Rávaṇ sad:
24869|In trembling hearts and fearful fear
24869|The brother of the King of Thieves.
24869|His anger at her words they felt,
24869|And many tears they hurriedly shed:
24869|The king who ne'er to others shed
24869|His rancour of unwholesome truth.
24869|Like some vast crocodile or snake,
24869|Like the vast crocodile or snakes,
24869|He who his pride of power would show,
24869|His brother whom he scorned and belied,(856)
24869|Saw the sad news with many a sigh
24869|To Ráma by that voice revealed,
24869|And thus he spoke, most pitiful:
24869|"If thou wilt fight with Ráma, then
24869|The giant King, thy vassal, may
24869|This city all take securely in:
24869|But thou with faithful Ráma fight.
24869|'Twere well that you the city bring,
24869|Our lives in danger should ye die.
24869|To-day and to the morrow, he,
24869|Great king of giants, may he meet,
24869|The giant king I know who stole
24869|From thee thy darling Lakshmaṇ, when
24869|Our fathers on the day of vengeance
24869|Saw thee in this city slain.
24869|To-day my son is born to thee,
24869|To-morrow to the King may he
24869|Thy Ráma and thy Vritra slay,
24869|And thy own heart-sick brother, too.
24869|To slay my brother thou hast vowed
24869|This day and day in turn to dwell.
24869|Why, Ráma, is the world thus full
24869|Of fierce wrath, full of deafening noise?
24869|Thy brother's life and thee's, I ween,
24869|Lie evermore in my eyes.
24869|For all a life like this can yield
24869|Is precious little: thou must play
24869|With thy own brother, and a prey
24869|To thy most furious foe consent,
24869|If Lakshmaṇ and thy brother die.
24869|Now I would fain the King of Thieves
24869|A thousand worlds in strife destroy,
24869|And all the race of fiends who dwell
24869|Deep in the wood, a million moons.
24869|I would not seek retribution here
24869|For thine own hand upon the dead:

[348000 | 52483.09] loss=2.75 avg=2.06
[531999 | 9443.46] loss=2.09 avg=1.97
Generating samples...
======== SAMPLE 1 ========
 better his life and wealth to him
2136|I will set the gold in store.”
2136|He spake no word, but bowed his gray head low
2136|And left him swiftly, and he came
2136|Through the great doors his sons to see.
2136|And there within the court his sons brought
2136|The golden treasure, all bound in gold,
2136|The precious gems and the wise gems round
2136|They held, and bore them back and brought
2136|To King Olaf, and from them cast
2136|As gifts before the king his gold.
2136|And as he heard the sons bring in the gifts
2136|He spake unto King Olaf,
2136|"My mother, look ye here and see
2136|What hap lies layed before me here.”
2136|On the ground the King his mother spake
2136|From the dark wall, where her breast showed white
2136|With misgiving and dread:
2136|"Nay, but this man's heart that did breed
2136|Fell full of bitter grief as mine shall fall,
2136|And grief is for the day that comes,
2136|And sorrow is for heart to know.”
2136|"Nay, O King, to-day is good and fine
2136|Withal as glad and long to be;
2136|But to-morrow may bring mickle pain
2136|Withal for grief,” the king replied.
2136|"And shall be.”  "That will I say.”
2136|Then rose and said, "My friend, I stand
2136|Before the king in glory now:
2136|"The gold now lies at my royal hand,
2136|The gems of my folk's great king there be:
2136|And from my mother, the dark woman, bear,
2136|The dark woman, the great lord's daughter fair
2136|And mother of our good king and lord,
2136|This golden thing:
2136|This ruby and this chrysolite,
2136|Gleaming bright with golden light.”
2136|And Olaf spake, for quickly he spake
2136|Sooth and plain, in full or hidden wise,
2136|And as the king in council him eyed
2136|Stood to speak, saying, "Be it so, now,
2136|Here is for mourning and for mourning sake
2136|A costly gift for our father's sake.”
2136|"Goodly gift as I ween,” the king rejoin'd,
2136|"And I will take it all, I ween,
2136|Than for grief or for sorrow some wise need.”
2136|And so spake the good knight Grettir, and said,
2136|"My son, let the gift-bag now bring too,
2136|With silver wrought in gold and satin wrought
2136|And gold, and then our lord and master true
2136|May go in as wont:
2136|"And we will bring it,” the king replied, "with speed.”
2136|And so he took the golden gift and laid
2136|The ruby and the chrysolite by him,
2136|Which the cold hand that laid it must give
2136|And let him lay it next before him
2136|Behind the arm to bring it to the king.
2136|And straight the two princes went to the king,
2136|But went in silence, gazing at the golden ring
2136|Unto the lord of gems, whose shining face
2136|With a low voice them said, "Why dost thou stand
2136|And look all gleaming in the kingly door
2136|Where kings and heroes pass?
2136|"I knew not that thy heart so glowed with hate
2136|As to look on me, though I stood there high:
2136|Now know I nought,” the warriors said, "dear king,
2136|Thy great heart's wrath shall not thee displease.
2136|"Not thou thine anger or thy great displeasure:
2136|For now at last with death thy father dies,
2136|And in thee as in his sons doth the will

[532000 | 9459.46] loss=1.31 avg=1.97
[534999 | 1390.41] loss=2.18 avg=1.89
Saving checkpoint/run1/model-535000
Generating samples...
======== SAMPLE 1 ========
1728|the ground: the other two with their hands each on his
1728|sleeve laid flat against his face, and strove to cover
1728|himself with the cloak. Then each was moved by pity, and
1728|clothed him in the cloak. And the hero Alcinoüs rose
1728|straightway at his father, and gave him his spear, and spake
1728|in his turn to Alcinoüs, saying:
1728|'Alcinoüs, no longer endure to sit, as though a
1728|mere man, and let the great Menelaus come in; but come in
1728|and see the son of Nestor, the Achaean hero; and ask him
1728|all about thy deeds and thy father's house, whether to have
1728|stayed, or to have returned to thine home. Say, is he still
1728|living, or hath he now gone down to the house of Hades
1728|gods to abide by them?'
1728|Then the noble son of Nestor answered him, saying: 'Father,
1728|I was not yet the son of Menelaus, when a god set me
1728|here of mortal man; yet he will not lay an hand on me,
1728|unless I too go down to the house of Hades, through fear
1728|of Zeus, lest I repay the wrong that He hath done. My noble
1728|son has gone with thee to the halls of godlike
1728|Eurymachus, and hath vowed thee unto death with a spear,
1728|on account of thy evil counsels; and in a fit of anger
1728|he promised to slay thee by the hands, on pain of death.'
1728|Then Antinous, the son of Nestor, answered him and said:
1728|'Yea now, my son, I will go thy house and seek thy
1728|father, that he may tell thee all aright.'
1728|Thus he spake, and with him went forth the noble
1728|son of Nestor. Then Alcinoüs and the other Achaeans
1728|greeted the king, and each one offered gifts. But he did
1728|not speak to mortal man about his ransoming of the
1728|gods: as for the sons of Nestor, he made them wait a long
1728|time at the ships, for he was fain for gifts in form of
1728|merry or gold, and a goodly store of raiment, and himself
1728|a well-beloved son. In like manner he minded that the
1728|children of men should offer gifts for his sake, and
1728|ponder his counsel; and the other captains and counsellors
1728|would be listening in their halls.
1728|Now the noble son of Nestor had gone down to be dead,
1728|for his house was near, and the people thronged that way about
1728|him, and at the first dawn they found him, sitting
1728|at the entrance of his hall, where he had left his golden
1728|and he looked great tears upon their faces. Then his spirit
1728|came not again to his own body with him.
1728|So long as he dwelt beneath the waves of the dark sea,
1728|some god had nursed him in the dark, but as soon as the
1728|winds swept him from the sea to the land, he fled to the
1728|tomb of Zeus. There a great swan lay in wait for him, the
1728|lordly Erystus, grandson of old Oceanus, and was first
1728|to woo him, being the first who came to woo a maiden,
1728|for surely there was none else in the wide deep to
1728|prevent him. So the lordliest son of Nestor was minded to
1728|offer him a goodly fair gift of goodly gold and bronze,
1728|and he did so, and the gift was the best, the fairest of
1728|all. And the king Alcinous and the other chieftains and
1728|favored of goddesses then brought him to the house of
1728|Zeus, who is godlike and lord of generous souls. And
1728|the blessed gods showed great joy of the deeds they
1728|had done in the sight of all. As for him he lay lying
[659999 | 5522.96] loss=2.36 avg=1.90
Saving checkpoint/run1/model-660000
Generating samples...
======== SAMPLE 1 ========
In vain: there is one true knight that we
1728|shall meet, if we seek it so, and there will be
1728|no need for anything."
1728|So he spake, and stirred the blood of Thrace, where
1728|the thick-sown reek of the battle had made a
1728|great pond, and the blood-drinkers were all weary
1728|of battle for their wine-youth, which they had got in
1728|their hands. And now they set fire to their braziers
1728|and went to bed, and their eyes still looked toward
1728|Ithaca, for they thought to see the haven called
1728|Lacedaemon, and the good news of Agamemnon.
1728|Now when they were come to their place, they feasted
1728|on plenty, and by great signs the Trojans led
1728|themselves about the gates, and when they had put
1728|everything of weight in, they took their stand
1728|straight on them, smote on their helmets with their
1728|hoofs, and the loud clamour of the fight was made
1728|heard and known; so there were great confusion and
1728|revenge in their breast, and the stout men
1728|thrust down their armour from them and dashed
1728|them upon the ground. Thus they fought in the
1728|thick of the battle. But the valiant Dardanian
1728|strong in fight found his strong men among the horses
1728|of the foremost, and caught them up in one. And he
1728|sitting by those heroes, put to their mouth a
1728|salt, and he sprinkled them with the drops of
1728|grapes, and spake and hailed them:
1728|'Friends, it is meet, O friends, that we should show ourselves
1728|good men, who know not warlike service by a name
1728|otherwise than this, a mind of mortal men: for war
1728|shows weakness in the hearts of men. Let us be men,
1728|and fight with heart and hand. As for this word of
1728|sorrow that ye bear after so long sorrow, know
1728|none the less that it is the will of the gods:
1728|we have seen great evils in our father's house, and
1728|this very day the people have perished for a
1728|little, for there was no man in them save but three
1728|old brothers, men of renown in the halls, yet they
1728|wrenched apart themselves and went their ways as wolves and
1728|foxes. And the daughter of Poppaea bare them sons,
1728|And the lord of the Credans, a man of evil mind,
1728|came at the fourth year to Mantinea: one came in the
1728|twentieth year, and begat seven sons, and he was
1728|the lord of the city of Mantinea, where he made an
1728|immediately for himself an offering from his own substance.
1728|Therefore we will not fight with these people, but let
1728|them take their way across the sea from the land, and
1728|the old man's tale he answered with bitter mouth:
1728|'"If this will not win ye to the passage of the
1728|sons, then bid your men stand guard at once upon the
1728|shield-wall, lest that too I take the sons of my
1728|father and sons-in-law in the house and see the
1728|evil deed done in all my flesh. As for me, I do not
1728|believe that I am of any worth whatsoever, for ever as now
1728|I am lying under the earth with bare shoulders,
1728|having lost even my manhood and the strength of my
1728|hands, and ever through the days of my life I am without a
1728|strength, and ever in the counsels of my heart I was
1728|Then wise Telemachus answered him, saying: 'Stranger,
1728|not till all these things have come to a pass and
1728|the people of the Achaeans make great feast and
1728|have a banquet, then wilt thou boast to thee of thy own
1728|courage and thine own strength in going and bringing
1728|them by strong armament over the sea,
[656999 | 2795.58] loss=2.49 avg=1.85
Generating samples...
======== SAMPLE 1 ========
 hope the night shall cease before the day.
27441|As the last year lies at last,
27441|And the last year's sun
27441|Ascends the hill-tops once seen in;
27441|So th' last year dies, and shall not die;
27441|For the day is spring's and spring's heir.
27441|When the last year sleeps,
27441|When the last year's day
27441|Darkens the hills,
27441|When thy light is no more, but only thy shade,
27441|Then will love make the hills and valleys glad.
27441|When the last year lies at peace
27441|And the last year's sun,
27441|Shines, as it before, upon thine altars alt,
27441|Then to thee, O God, th' offering will be offered.
27441|To the music of the spring-time we come in,
27441|We love to trace old Ocean's flow
27441|'Gainst the banks of every shallow lea,
27441|Or watch the rolling green of every hillock green
27441|Like gold in the dawning;
27441|We love to hear the sea run screaming and laughing--
27441|We love to hear the deep grow wilder and wilder--
27441|We love to hear the rolling of the sea-waves
27441|Crashing through all the coasts of the world.
27441|We love to watch the birds and the rising sun,
27441|Like a great stream through an isle of gold;
27441|We love to see the birds in their lovely green,
27441|And a sunbeam in their shining;
27441|We love to hear the sun beating, beating--
27441|It's a joy to be young when you're old!
27441|The spring-time's in its rosy prime.
27441|All the birds are flying;
27441|We're happy, and we love to be young, in summer too!
27441|'Twill come to us still, as old as we,
27441|To sing our story ere we sleep,
27441|And wake, and find the old world bright in the morning!
27441|_The_ WORLD, we know, is passing away,
27441|With all its pleasures and its woes:
27441|But one thing, sure, must remain:
27441|Our story, on the morrow, we'll tell.
27441|On the morning of the morrow I, the sea-king, lay in a lonely deep,
27441|And heard a strange old song;
27441|A sound like thunder came from the heavens, and it seemed the old sea
27441|The night has long been cold and dark and dead,
27441|Yet a ghost-like light gleams from the eyes of the stars,
27441|And a strange old song I heard.
27441|The starry heavens grow dim;
27441|The waves are foaming white and red;
27441|But I, in a lonely, weary way,
27441|Hear the great waves roar and roar,
27441|And the old sea-song still sounds on
27441|In the lonely hours of the night.
27441|Sweet-voiced as the misty-colored birds,
27441|The wind on my beach has been singing;
27441|Ah! who is this that sings of the sea?
27441|What are the songs of the sea
27441|That I have heard afar on the shore?
27441|'Tis a ghostly thing the winds are singing:
27441|'Tis a strange old song; O who is this that sings?
27441|It is many a summer o'er the waters
27441|After the sun has sunk,
27441|The sea-bird, high in heaven,
27441|Has never been seen by mortal eye;
27441|But he has been, and it has been well
27441|The waters passed, so he would stay.
27441|With the leaves of the forest all glisten,
27441|And the star of the sky above
27441|Looks lustily with his golden bars,
27441|And the trees are bending low;
27441|He brings the night a blessing, and he makes it clear,
27441|He lulls the weary stars to slumber,
27441|And he lulls the waters under till dawn may keep
27441|The dawning of the day.
Generating samples...
======== SAMPLE 1 ========
 let you know
9578|What God will never do to you in trouble.
9578|If it's through some brave, stalwart soldier-man
9578|You strive to follow, out-fighting him will spring
9578|Like the break in a wall the water makes
9578|Just when it's nearly over!
9578|A soldier's a soldier for life, and must
9578|Rise up on either hand, firm and growing,
9578|To the work he's called to do, unshaken!
9578|Your country calls you to it; rise up and do
9578|Whatever it asks of you.
9578|It may not be great fighting; may be home drubbing;
9578|May be a life of disgrace and shame;
9578|But never, never, last of any man who strives
9578|To do God's will for country or God's will for home.
9578|Your heart is in it; up and do your work!
9578|God bless you, soldier!
9578|And when the shadows lengthen
9578|And it's very dark inside,
9578|And you hear the droning of traffic in the street,
9578|And the wet fog of the night is about you,
9578|Remember that you are heroes and that your work
9578|Is never done well enough;
9578|That your battles of yore are not lost though you grope
9578|Wildly at your work, nor yet finished nor done;
9578|God bless you, soldier!
9578|Never let your fame be counted
9578|A victory or a defeat!
9578|God loves all great and mighty men,
9578|Not great or mighty in the least,
9578|And He knows that among the sons of men
9578|There are some who cannot reach Him!
9578|"He hath called his seer."
9578|Now that I've tried, God rest my soul!
9578|My feeble prayer failed of a dose.
9578|You'll find a lot of soldier-people, I'm told,
9578|Who'd like to be really great, and they'll try.
9578|I had a vision of a vision once
9578|Of two white boys, one who wore a hood,
9578|And one who, more like a black-hooded ghost,
9578|Held up in his hand the stick of wood!
9578|A soldier boy from the house before me
9578|Stepped forward in his work-shirt white,
9578|And, holding up my old rifle and my old butt-stock,
9578|He turned the wrench of the safety down!
9578|"You'll never," said the seer, as I stood at attention,
9578|"See such a rifle anyhow!"
9578|And that was a vision false and dim,
9578|An apparition of the past,
9578|Of a gray boy from the village of my childhood,
9578|And of my boyhood at home and here,
9578|Who had turned the wrench of the safety to decide
9578|Whether from that dark future or no,
9578|The rifle I would choose at the other's risk.
9578|Not much longer I had written to Colonel Taylor
9578|In the hope of getting prompt reply;
9578|But I found in Colonel Taylor's file a page or two
9578|Where I could search; and there myself said he,
9578|"If a black boy from the village of my childhood
9578|Were to go down into the world to-day,
9578|Would you send up your heart to him? And then
9578|Imagine how the earth would hear and bow,
9578|And smile on you and help you up, and set you free,
9578|And then send up its pitying bosom to you!"
9578|The sun sinks mournful down on the dying of day,
9578|And the leaves are turning into night;
9578|But let not your heart fail
9578|To mourn forever the boy,
9578|Who died before his birthday,
9578|For he dreamed of a great future, too,
9578|And dreamed that the world was his to share.
9578|He dreamed that his step was on the way
9578|To that bright and glorious thing,
9578|Where all who are gifted, gifted, shall go,
9578|To the far, far shore,

Calculating validation loss...
100%|███████████████| 40/40 [00:17<00:00,  2.34it/s]
[658000 | 3737.42] validation loss = 1.93

Random Samples

Testing 345M-poetry, a slightly higher temperature felt warranted, so to generate 5000 random poetry samples:

I also generated 500 conditional samples for Yeats’s “The Second Coming”.

Reading through training & random samples, they feel noticeably more coherent; it feels easier to extract meaningful subsections which form reasonable poems. (In particular, the pastiches of classical epics or Dante have gotten remarkably good.)

Some further samples:

======================================== SAMPLE 530 ========================================
16265|In their little room with the door ajar
16265|And the candle hanging on the wall ajar,
16265|I have come across the word "Rise"
16265|With a face as grave and flat as you please.
16265|The one thing I remember of "Rise"
16265|Is the way it makes you feel--so bad, so bad.
16265|And I've come across many words to-night
16265|That are so like "Rise"--so like--so vague, so vague.
16265|"Elegance," and "Artistic Vigour,"
16265|But "Rise" is far above the rest,
16265|And I cannot hear--or see--the word,
16265|I will just stop here (I'll stop if I can).
16265|If you don't know what "Rise" means, try.
16265|"Rise" says--but don't ask.
16265|Well, I believe that I do--
16265|And if you don't know what "Rise" means
16265|You won't either!
16265|It's when the sun makes his round
16265|On clouds a-shine;
16265|And the little stars come out
16265|From under the silver dawn
16265|To shine about me so.
16265|When the sun puts out his light,
16265|And closes his earth-door,
16265|And with shadows blackens the blue
16265|Of heaven's blue;
16265|When all the stars I own
16265|Have put their clubs in bed,
16265|And I lie just like a log
16265|In the deep, still earth;
16265|When I don't hear my mother
16265|Any more in the hall;
16265|When the cat is out of the kitchen,
16265|And my brothers and I
16265|Are up to something unsaid
16265|That we've long been keeping--
16265|I have been like to catch a cold
16265|For weeks beyond reach.
16265|And I know that this is so--
16265|And the fever that wrings
16265|My bones is a cunning one
16265|That will not go well
16265|With the morning.
16265|And when I feel all warm and well
16265|And young again,
16265|I have a thought--and I say it--
16265|Of the night that's come.
16265|'Tisn't much: a summer's night--
======================================== SAMPLE 570 ========================================
18007|With a face of a smile, and a mouth that was redder than wine,
18007|And a hand, and a mouth that was whiter than snow,
18007|And a voice that was softer than dew of the morn!
18007|And I leaned my face between his, and I held him--I
18007|Holded him, and I kissed him, and--ah, there's a stain!
18007|I held him, to prove him faithful, while I dreamed
18007|Of a face of a smile, of a mouth that was redder than wine,
18007|Of a hand, of a mouth that was whiter than snow,
18007|Of a face, of a mouth that was blacker than black.
18007|But all in vain; and the moon, the moon, the moon
18007|Came a-drifting, like a careless sister, and stole
18007|Away from the village, and left them aghast,
18007|When they found him at last in the house of his wife--
18007|Sick with love, with a heart that was blacker than black.
18007|One moment as he wept, and then, like a priest,
18007|He stood weeping, with a pallor on his face,
18007|He stood with his head bowed, and prayed, and was silent;
18007|Then he turned to his wife and said, "Wife," and I
18007|Shuddered--and kissed her, and said a poor thing, and bad;
18007|And--there's a stain!
18007|And then came the moon that came so seldom this year,
18007|And vanished so swiftly that I forgot to say "Good-night."
18007|And she answered, "Kiss me, my sweet; it is time."
18007|And she rose and went to the door; and then--I cried,
18007|And--there's a stain!
18007|Then I thought that I had forgotten all,
18007|All that I had felt and done and said
18007|For I had little to lose, and I had not much to say;
18007|But I knew what did come after I turned in the door,
18007|And that is, that the stains will never go away.
18007|What matters it whether the moon come again,
18007|Or the sun rise, or the little birds sing;
18007|That the little things that I have known and done
18007|Will hurt me still, and stain my lips with black.
18007|The moon is low, and from the windows, low,
18007|The little stars wheel in their canteens,
18007|Ripening for a night when the world is gray,
18007|In the west, far, long ago;
18007|Like flowers, they come, the stars of midnight, bright,
18007|Like flowers, in a cloud of gold and snow.
18007|The stars of midnight hang like ripples of wine;
18007|The flowers have bloomed and vanished away,
18007|Too early to be precious to us all.
18007|But the things I love the best, they have not died,
18007|And shine through the twilight, gold and red,
18007|Too late for our love, too soon for our tears;
18007|So dark and drear for their tender glow,
18007|But yet they are bright, for they will not die.
18007|In the land of shadows I will seek
18007|The star of night that shone above her urn;
18007|And she will laugh in the darkness again,
18007|When I have been gone with a heavy heart,
18007|And she will smile in her lonely home on the shore,
18007|When I shall come no more.
18007|Out of the West, out of the night,
18007|A star has come.
18007|He has rolled in glory before
18007|My soul and I.
18007|I think I have felt him before,
18007|But I never was human.
18007|Now the stars are turning into night,
18007|And the clouds are brightening.
18007|For I never was half so fair
18007|As he now is;
18007|And I never shall see him, only,
18007|Till our bodies meet, heart to heart,
18007|In the bosks of love.
18007|When all is
======================================== SAMPLE 610 ========================================
29700|Hark! from those shadowy depths thy voice
29700|Mournfully echoes, "AUTH".
29700|That was the night when from her fane
29700|Mangu Damoneo, her priest,
29700|Sang by moonlight in the glade
29700|Of her garden; but, alas,
29700|Her lovely form was gone.
29700|The night of sudden twilight,
29700|Of sudden funeral, laid
29700|The flowers of her beauty by,
29700|And left her lying dead.
29700|The mourners, with no pity
29700|For that fair maid, who had died
29700|So young and sweetly, they
29700|Drew from the crowded bier.
29700|Now the long twilight mourns
29700|In dark convulsions slow,
29700|The sun, whose brightness, when it climbs
29700|To meditate its last,
29700|Turns the great globe of heaven
29700|To a vast blackness; and the moon,
29700|In the blue distance lost,
29700|Waves her orb all palely pale
29700|O'er earth and ocean's bed,
29700|While, at her last sad funeral,
29700|The waters are still.
29700|"Where, O, where," the poet cries,--
29700|"Where is she, whom so fair,
29700|So pure a form has laid,
29700|Whose heart was true, and tender,
29700|And soft as summer air!"
29700|Yet there is sorrow in the words;
29700|For in the middle space,
29700|The grave is silent, but the sea
29700|Is louder still than I.
29700|The poets write how fair
29700|She lay in moulded tomb,
29700|A shape of beauty, soft of feature,--
29700|Yet still she's dead!
29700|I look to distant lands,
29700|Yet still I see her face,--
29700|How fair she lies in tomb!
29700|And all that Nature left,
29700|Which never came to pass,
29700|Is on her cold dead breast
29700|That cold dead face seems to look,
29700|As if his own it were.
29700|From sea to sea he sailed,
29700|And then upon the shore,
29700|The man-child found his mother's breast,
29700|The mother's bosom bare.
29700|On many a foreign shore,
29700|When sorrowed he passed along,
29700|The deep-embattled ocean cried,
29700|"A bitter heart, I've had!"
29700|The storms and the tempests of his youth
29700|He turned upon himself alone;
29700|And though the storm and the tempest had
29700|Their rage, he had passed with the wind,
29700|With God,--his only sign;
29700|And, ere the cloud of the tempest blew,
29700|His soul was with the world at play.
29700|He looked to the stars, and the stars smiled,
29700|And the moon in the heaven looked;
29700|And, as he looked, he beheld her light,
29700|And all the heaven smiled with him.
29700|When winds and tempests fly,
29700|When floods and fires fail,
29700|As their wake doth meadow and fen,
29700|'Tis the man-child's heart that craves.
29700|And I--I shall be bound,
29700|With the hoary-headed, strong, old,
29700|To earth, and the graves of the dead,
29700|Whose feet are mowed down, as they lie;
29700|And I shall rest my weary head,
29700|In the silence of Eternity,
29700|In the peaceful arms of God.

Here is a ‘failed’ example, where GPT-2-345M-poetry imitates the scholarly apparatus that unfortunately contaminates the PG poetry corpus; it is quite plausible-sounding, even including plausible-looking Latin:

======================================== SAMPLE 620 ========================================
38566|and of the two main branches of the lyric poets, he is
38566|of the first. This latter is more like a novel, the former
38566|a mere exercise in poetic technique. It is true that some of the
38566|longer poems are only in two or three of the seven acts,
38566|(as 'Cytherea and Amphithoüs', 'Odysseus and the Daughters of Pose)
38566|The metre of the whole of the tragedies is at most
38566|two or three distinct feet, two or three acts, or perhaps
38566|no more than one line of one act. But there is always no
38566|clear, distinct point of passage, and the metre may change
38566|after the action has begun, and in doing this the danger of
38566|distinction. As regards Lucan, we might as well look to one of
38566|the last days of the Republic, the 'Ante-ratu' of The Deed
38566|(cl. 27)--'Ad locum triplice trieter hoc sibi sibi dicas, etc.
38566|Præestus huic erat precatus in sæpe Lucania per Bettini
38566|Ergo sæpe diadema'--quoting an asp of the same name as the
38566|antagonist,--'Ad mihi quærent horribilem fratres' being a better
38566|means than the one named 'Hæcam.' And thence we can infer from this
38566|that the 'Ante-ratu' of the other tragic poems was a regular and
38566|accidental epigram of Lucan himself,--the kind of thing which
38566|Sextus in his Annales, 65 says of the poets of A.D.
38566|Mihi sæcula, quærent Hæc morbis,
38566|Nec coelo mutum modo vel fratris
38566|Rara ferus.
38566|If it is right to treat of the tragic poets of a later period, it
38566|should not surprise us that Lucan is so honoured in our day. Already
38566|GREEK, and still more after that, in the days of A.D. 60 and A.D.
======================================== SAMPLE 670 ========================================
36773|But not with the first men of Greece;
36773|Not so much the Greek itself
36773|As what this Greek gave and took from him
36773|His power to change what the past had been.
36773|The past he made more perfect than it was;
36773|Not for the beauty of the things,
36773|Not for the new things it is, and strange;
36773|But for an inner sense and power,
36773|The force of a spirit to penetrate
36773|Truly into Nature; the power of Nature's mind.
36773|But these are, in spite of words, a vain,
36773|A futile spectacle!
36773|For what were all
36773|But scenes of dream
36773|And scenes of glass,
36773|Till that which best is is at last all?
36773|The image of the true?
36773|Not this, as of old; not this, as of old
36773|When, leaning on his pike the Spartan king
36773|Spoke to his multitude like one man alone.
36773|Nor is it so with me; since naught can prove
36773|My nature's image; all I feel is known
36773|And knows what it is: the image of man;
36773|The body; the sense; the mind; the will; the blood;
36773|And so goes ever on to the dark heart of things.
36773|But this image, as the great natural bulk
36773|Of all the world, and the universal frame
36773|And home of all things, is a wordless sound,
36773|A little, but a strong one, heard and seen
36773|In the unspeakable calm of the supreme;
36773|That, like one man's word of it, "A little more."
36773|The soul shall know
36773|How, like the world of things,
36773|And like the thoughts,
36773|Like these our thoughts, our thoughts are many,
36773|Like the world of things,
36773|And like the sounds,
36773|Sounds of life heard and unseen,
36773|Like the world of sounds,
36773|Sounds of thought and thought, they are the same!
36773|Though what we think is like all things, like we,
36773|Though we be like one thing, not like all.
36773|This is the meaning of life.
36773|For each is like to him, like all;
36773|Though all like him are like to him, not all.
36773|Not like, but like how, and with these two
36773|The earth is like most in the sense of being strong,
36773|The rocks and the sky, sea and shore,
36773|And air and water, and all things like to blood.
36773|_With many a rose for the dead_
36773|_Is laid the flower of mine._
36773|_The flowers, the rose, the sea, the air, the water, the land._
36773|'For we are the pictures of our lives, and our ways
36773|Make up a universe.
36773|'We live life out in the open,
36773|And leave the rest
36773|To fancy._
36773|_The mind-for-life of the sea._
36773|'The mind-for-life of the sea
36773|Takes place for us, for us
36773|The little seas, the little woods,
36773|The little leaves, and the little birds,
36773|The little fishes, the little things that live
36773|The round world over,
36773|And love us with its love of the things we see;
36773|And the mind-for-life of the sea
36773|Is our outward life,
36773|And we would leave it undone, undone,
36773|Right, by ourselves,
36773|For any one to do it for us,
36773|Without any help,
36773|For any one to bear such pain,
36773|That we could never be content,
36773|For any one to stay,
36773|As we have been,
36773|And go on being, and on being.'
======================================== SAMPLE 740 ========================================
19385|"While his auld foot beats thae ances,
19385|Ah! daur ye bee at the auld bar?
19385|Ah! daur ye bee at the auld bar?,
19385|An' if ye have ten min'us ances,
19385|Daur ye be at the auld bar?
19385|There's a cauld, dingy bar,
19385|An' ae blythe young lady
19385|Was a' in his auld house,
19385|He had a fondness for her,
19385|She was his auld ha'."
19385|"I am a king o' the Irish clover,
19385|An' nae sic clogs o' gowlds I ken;
19385|But, whisht! I'm wae to hear men bleedin'
19385|At my reign o' the Irish clover,
19385|An' to see women's looves like wee diz,
19385|When the land that I rule is the land o' clover!
19385|I hae a wife an' a daughter fu' o' age,
19385|She lives down in the clover ha'--
19385|The land that I rule is the land o' clover."
19385|Oh, I was sae blest to leave the play,
19385|That, like a ghaist, rusts awa!
19385|On a windy hill, I heard the snowdrift sweep,
19385|But, oh! how gleyerfu' it was to die!
19385|I set me down upon the heather, where
19385|I heard the wild bee as it swarm;
19385|While the linties all were on me blithe and bonny,
19385|In a bonnie ring I lay.
19385|I slept the sleep that fyfu' the night end,
19385|I thought that I was left alone,
19385|But in a swoon I see the light is glint,
19385|I saw the light was west.
19385|My gowld was yellow, my heart was saft,
19385|My hopes o' long ago were fled--
19385|But lo! ilk bird's sweet singing on,
19385|I heard it far and near.
19385|I think upon my past, the while I'm casting
19385|Ae thought o' Heaven above me, yet
19385|I'm ever glad to be gane!--
19385|The land that I rule is the land o' clover!
19385|'Tis a saft breeze that gies me a kiss--
19385|'Tis a saft, fresh caftinge on my love,
19385|That waves upon my face,
19385|As gladness I do kiss, and he kiss me,
19385|Though he be sae slow.
19385|I canna get used to a kiss from your lips,
19385|But I'm wae to think that I'm blest,
19385|Whar the winds are blowing, and the rain's in bar;
19385|That they may be a-kissin' and a-kissin' at the same.
19385|The gowan-stalks grew high in the heather,
19385|The kye lay snug in their beds;
19385|And ne'er in the days o' our childhood
19385|Were we wint o' such joy;
19385|For, as life and its pleasures might come annee,
19385|So we fain was we to lie where the heather lay.
19385|Then he leaned o'er the heugh, and he laughed a hearty laugh,
19385|And the kye looked aghast, and the deil look'd asker anither,
19385|For their heads were sune to change.
19385|But the gowan-stalks they turn'd in the air,
19385|And the kye wad sieve awa',
19385|For, like a saut bird, O, saut were the tears
19385|That we shed as we fain had we seen, a wild bee on,
19385|But what could be a bee on!
19385|And sair, sae we sieve awa!
19385|And sair, sae we sieve awa!
19385|And we fain would lie where the heather lies, close

Ganbare, GPT-2-chan:

======================================== SAMPLE 800 ========================================
27669|We never say "Thank you"
27669|(Except when it's very late!)
27669|But he's the kind that would go
27669|And do his very best,
27669|If you'd give him a hand
27669|If he only were mine.
27669|_And sometimes when we sit around the fire
27669|We talk of a thing or two:
27669|Of the boy that was mine when I was young--
27669|(No matter!)--we talk of the boy we used to know,
27669|When he played in the wood,
27669|When he went a bowing round,
27669|In the golden olden time;
27669|He was just another boy._
27669|It's a fine old wood and I've tried to walk
27669|Through the boughs above a few leaves,
27669|And I've looked where there's little green places,
27669|To the shade of a great blue sky.
27669|There's the sea, there's the meadows, the white road and yellow sheep,
27669|And there's the old, old, old road where I met him last._
27669|'Twas a wood that he liked; it seemed just green and fair;
27669|And he liked the boughs above the great blue sky.
27669|'Twas a field we used to pass, now he is far away;
27669|And he loves the meadows, the white road and yellow sheep.
27669|And he loves the field when he is dead, but he'd rather be
27669|With his little boy there in the woodland far away.
27669|When he's gone
27669|He's a rich young boy sitting in his mother's lap
27669|And that is why he'll not forget
27669|To kiss and to hug me when he's gone._
27669|_Oh! would you'd come back, this would-be friend,
27669|And we'd talk of a time when it hadn't been too hot
27669|Since we all set out; and how we'd walk
27669|Along the road again, where he'd left us.
27669|And when we'd reached him in the valley he'd say,
27669|With a smile, "Good-by, my dear, good-by!"
27669|And look at me with his dimpled smile;
27669|And I think we'd talk and joke about such things.
27669|And maybe we'd play at our old home game,
27669|If you'd come back
27669|And help to share in our old home joy.
27669|And when you are back
27669|It's just that we two can walk,
27669|Along the road again:
27669|We can watch the sheep and the grasses grow,
27669|And tell each other stories and listen to the wind
27669|Crying to the woods and calling in the trees
27669|Through the blue day, "Old dear, old dear!"_
27669|_As we were riding down that lane of blue,
27669|O'er the brown hill's crest;
27669|And there in the valley between our feet
27669|There stood a house._
27669|'Twas a tiny, simple-minded place,
27669|With a fence, and railings, and railings more,
27669|That made us feel so small.
27669|It could only be called 'Half Price' Town.
27669|There were houses up and down,
27669|In the distance we could scarcely see,
27669|To the horizon's end.
27669|_We sat, in the shade of a timber tree;
27669|Our eyes half closed, and our faces bent;
27669|And with a heavy heart we thought of home,
27669|And of the woman who'd died asleep,
27669|In the house we had grown to cherish.
27669|And the voice was quiet again;
27669|In the silence of the night,
27669|In the light of the moon and stars,
27669|And the night-wind we could feel
27669|The pulses of the sleeping and dying,
27669|In the quiet, moonless house.

This is a peculiar one; it starts as a satirical poem but I can’t make out what it is trying to switch to partway:

======================================== SAMPLE 950 ========================================
1279|The King of Wales is a very clever fellow;
1279|He's sly as a cat, and a thief as well,
1279|He's like to set round a lamb a score,
1279|To take and turn round, and a-mimicking her tone.
1279|But to crown my Lord of Leven's min' fun,
1279|He's just like a dunce in w'at he can do;
1279|He's all in a twirl if he's going to be tail,
1279|For the man that heaps things at Lord Brougham's feet.
1279|Lord of the Whinnybone and the Hooplebooroo!
1279|You're a wit all as fine as Sir Thomas Suckly,
1279|And I'll venture to say, in my noddle,
1279|That when all the rest are out to piss,
1279|Then your Lord of Leven is just the man to go.
1279|Now, the best and the brightest of our ancestors,
1279|Who bore the race formerly known as Bow-wow-wow,
1279|Were all of one gender, and of one sex,
1279|And yet they were men of many other breeds,
1279|And more and more there was variation in their breed.
1279|For example, of the male, who was black and of white,
1279|There was also a fair breed of females all over the country;
1279|And yet 'tis a fact that the best men of the day
1279|Were all of one sex and of one species,
1279|And yet they were men of many hundreds thro many centuries.
1279|So there never was anything like it--there never was,
1279|In any age or clime, a breed like to the first:
1279|All were of one sex and of one sex combination--
1279|So, you see, they were men of many hundreds,
1279|That were all of one sex, and of many thousands,
1279|Whose species combination was myriad times more numerous than yours.
1279|In every locality from Cornwall to the Cape,
1279|There were divers sorts of males, and divers sorts of females,
1279|In every locality and every rank and class;
1279|For example, of the first five generations,
1279|There were divers sorts of weds, and divers sorts of brides;
1279|In every locality, and every rank and class,
1279|There were divers sorts of housings, and divers sorts of shoes.
1279|And therefore in every locality and rank and class,
1279|There was variation, combination, and change of occupation:
1279|There were divers sorts of women, and divers sorts of men,
1279|In every county and every town and town-house too.
1279|The first five generations were all of one sex;
1279|And this was probably why before the time of the fifth,
1279|There were divers sorts of brides, and divers sorts of men;
1279|For example, in the first five generations,
1279|The wife was the opposite of the husband--most probably;
1279|And the husband was usually a bachelor, or a man,
1279|With another wife, and the same sex as his wife,
1279|In every county and every town and town-house too.
1279|In every county and every town and town-house too,
1279|There were divers variations of dress and apparel;
1279|Some were dressed in crape, and crape-cloths and bonnets,
1279|Some in kirtles, and some in hose and hose-at-heel;
1279|Some in high-heels, and some low-heels, and some hose-nigh-loose,
1279|Some in high-heels, and some in hose-nigh-loose;
1279|And some wore noddles trimmed with blue or white,
1279|And some with knobs, and some with rings, and some with rings;
1279|And therefore there was plenty of noddles for all ranks and classes.
1279|The last five generations (which I shall say was one
1279|generation before the last), were all somewhat further off
1279|than the last, and thus left me to interpret
1279|The names of the divers divers divers generations.
1279|A woman in red and white was as happy as a girl;

This one I think must be a mix of and the (but if a wizard offers you rainbow-colorful draughts of rum strained through his magic red beard, I suggest declining in the interests of hygiene):

======================================== SAMPLE 1060 ========================================
5185|"To the fish-lake of Pohyola,
5185|To the lake of honeyed Hiisi;
5185|Weep Iwi, weep upon Lempo,
5185|On the blue rock of Hiisi;
5185|Weep the god whose tears enrich;
5185|Weep the hero who subdued
5185|By his iron-handed arms
5185|This enfolding water-brook,
5185|By his magic hurl the rock
5185|To the upper deeps of Hiisi!"
5185|On the floor of clay they throw it,
5185|On the stone on which they write it,
5185|Spake these words in magic measures:
5185|"Lo-ye-Yo-Ru-Raka, son of Hiawatha,
5185|Take this magic measure,
5185|Words of ancient prowess!
5185|Put your senses into it,
5185|Take these mental forms into it,
5185|Turn to toe the enchanted drink
5185|Of the ancient, good, and wise Wabun!"
5185|As they twirl the mental measure,
5185|As they pour the mental formulae,
5185|Words of magic potency
5185|From the wizard's magic red beard
5185|Stream into the magic waters,
5185|Rainbow-colorful draughts of rum
5185|From the six-pending arts of brewing.
5185|Thus at midnight, hour of darkness,
5185|From the six-pending arts of brewing,
5185|Munchausen' of the spirit SAMURAI,
5185|Bitter his bitter sorrowful experiences,
5185|Tears flow to see the brewing measures,
5185|Casting him dark as night to HIAWASH,
5185|To the dancing-bower of Winansi,
5185|To the palace of the Song-element,
5185|To the home of ancient Wabun.
5185|There he sees the wizard fire-breather,
5185|There he eats the fruit of fire-fruit,
5185|Warm the water in the blue-smoke,
5185|Honey of the fire-flies, Sahwa;
5185|Nevermore to rise from Sahri
5185|To the glittering skies of Hiawatha,
5185|To the shining islands in the ocean,
5185|To his home in Winansi's honey-lands.
5185|True he sings no more of former loves,
5185|Sings no more of former partnerships;
5185|Beauty and truth the singer forsakes,
5185|Joys and dreams of CHEWBEL tree-top hollow,
5185|Thus again he sings to little honey-paws,
5185|Sings but one sweet thing, one only,
5185|Sings of the honey-pastoral flowers,
5185|Sings not of the meadows golden-rod,
5185|Nor the soft, corn-fields resting in slumbers.
5185|Young again the wizard sings of combing
5185|Sunny downs and forests of awaking,
5185|Of the golden balls in linden-groves,
5185|In the nooks of meadow-lands adorned
5185|With the purple blossoms of the ash,
5185|And the golden globes of chrysanthemum.
5185|As he sings, he flays and bleaches piarmate
5185|Peppers, onions, garlic, and ears of corn,
5185|For a meal to make his night's banquet.
5185|Straightway Hiawatha asks the artist,
5185|Places in his basket heaps of grain,
5185|Hangs himself to earth in alley lowly,
5185|Lays his hands and knees upon the rafters,
5185|Drops his baskets of grain into annihilation,
5185|Sings in low, third parts, low alto-dingule.
5185|Finally he hastens to the stable,
5185|Finds the black-frost of the winter gathering,
5185|In the center of high rafters framing;
5185|In the center a famine-gathered mass,
5185|In the center a man and his banditti,
5185|In the rafters great store of grain reaped from snow-sledge.
5185|Homeward goes the artist, homeward.
======================================== SAMPLE 1420 ========================================
1381|The loon of the moon!
1381|In the night, to me,
1381|The air was full of stars;
1381|The wild bird piped upon the branch,
1381|And the blackbird, tuned his song
1381|To the sweetest note
1381|That ever a lark knew!
1381|And I heard it ring, above the sea,
1381|In the land of the dreaming sea,
1381|In the land of the dreaming stars!
1381|A star is born when the summer moon
1381|Is full in the sky:  she wets her tears
1381|To kiss the star and sigh to hear
1381|One sigh from a thousand stars.
1381|A star is born when a nightingale
1381|Sings in a blue-bell tree; and she
1381|With such a song and such a tune
1381|Fills all the fragrant heart of the wood
1381|With such a fragrance as is there,
1381|And she sings in the heart of it.
1381|A star is born when on the wildest seas
1381|An Indian maiden dances alone,
1381|Pale as a ghost in the moon's eclipse,
1381|And the stars and the wind sing from the foam;
1381|And as the moon is dimmed in the heaving deeps,
1381|The heart of the maiden lies high and free,
1381|Shaking her curls in the beating breeze,
1381|And her eyes are bright with ecstasy.
1381|A star is born when the wind-flower sings,
1381|Or the hawthorn bathes in dew;
1381|And the nightingale sings like a bird in the sky,
1381|And all the stars and the nightingale ring,
1381|And the heart of the wind-flower sings.
1381|A star is born when the snow-white swan
1381|Drops her white soul to the river:
1381|And the winds and the water sing with glee,
1381|And the white swan lifts her face to the sun,
1381|And the stars sing out in the glory of day:
1381|And the heart of the swan is happy in heaven,
1381|And the heart of the river sings in the sun!

An alt-history where Germany won WWI:

======================================== SAMPLE 1550 ========================================
937|And, as you will, I'll be bound to you,
937|And always be and never be forgot.
937|We were a little band of heroes then,
937|With our flags all waving by our side,
937|And our hearts all set on some deeds of fame,
937|And our eyes all full of good-will light,
937|When the Kaiser came along to help them.
937|He was a mighty thing of might,
937|And when with us our flag all flowed,
937|When all men called each other comrade,
937|When the earth was made for happiness,
937|When freedom's dawn was dawning on all lands,
937|Our lives were all to some great cause done
937|We might not be wholly silent there,
937|For our hearts all yearned to join with 'em,
937|And we could not all go quietly by
937|But stood to fight the battle hard to the end;
937|And so, long as we had hearts to move,
937|The Kaiser, to keep back our strong men,
937|Was always there to tell us that we were called
937|To fight for liberty -- and all were we:
937|And every man must be a Kaiser
937|When our glorious flag we salute.
937|And now that there's a Kaiser in town,
937|The hearts we had at first to thank
937|For the end of the old war we went on --
937|To keep back the strong men to stand by us;
937|We are so proud to hold in scorn
937|The Kaiser's warning words back.
937|So when you hear our cheering chorus,
937|Look down in the faces of your foes.
937|And when you're called to your father's home,
937|Oh, just smile -- and let him go by:
937|It was the Kaiser's own call
937|That your country was called to give.

Tao Te Ching

The (TTC) is a famously enigmatic text, written in a difficult style in an more difficult language, and because of the challenge, has attracted many highly-varied translations.

compiled in one text (108Kb/19k words), and compiled per-chapter translations of the rest of the TTC as well (4.8M/870k words). Using TTC snippets to prompt GPT-2-345M didn’t produce good results, so he asked me to train a GPT-2-345M on the TTC to see what it did.

I converted the first corpus to Unix text format, replaced various escaped character entities with their Unicode equivalents, and replaced double newlines with single lines, and trained the original GPT-2-345M (I was unsure if using one of the poetry models would help) with the usual settings for ~6 GPU-hours, at which point it had reached a loss of ~1.8 and I began to worry about overfitting & stopped.

I generated the usual ~1k unconditional samples:

Some training samples:

[12999 | 10468.36] loss=2.03 avg=1.96
Saving checkpoint/taotehching/model-13000
Generating samples...
======== SAMPLE 1 ========
 can find a way to rid yourself of these issues.
Forced to see them (the spiritual essence), there we receive freedom.
Free from spirituality, we receive nothing but desire.
These are not separate things. They are one in common. Together they are the Mystery.
The Mystery of the World is all illusion.
Everything we see is but a small part of the Mystery.
The true mystery is beyond understanding.
The mere awareness of the world is a mere shell.
By being aware we become a turd.
Those who try to hide the true inner essence will become trapped in the shell.
Those who hide their inner essences are free to become free.
These two pairs of differentiations arise from the same source: nothingness and everythingness.
Their differences are mysterious, but still they are infinitely subtle.
Beyond spirituality, they are the portal to all wonder.
The words we use to express the nature of these differences are not the same as the words we use to express the nature of other differences.
The same name may be applied to different things.
As a mere passing fable, it is the boundary of mysteries.
The real is infinite, but it has limits. The empty of the empty is the gate to all spirituality.
The name that we use to explain it, is not the name of the real.
It is the Mystery of all things; as "the ground of the profundities of the heavens and the abode of the ten thousand names.
The profound subtleties are discovered from the deepest minutiae.
The Uncomprehending is the gate to all mystery.
The unsearchable is the door of all knowledge.

The Tao that can be spoken is not the eternal Tao. The name that can be named is not the eternal name.
The nameless is the beginning of heaven and earth; it is the mother of all non-being.
The named is the mother of all things. From the named it is known what is above and what is under.
Words are ways of naming; words are names for things.
There is nothing beyond the limits of the mind and thoughts; there is something beyond the limits of the mind.
This vision that can be voiced in words is not eternal vision; this that can be voiced in words is not eternal words.
The indescribable is the essence of heaven and earth; it is the Mother of all things.
Exclamatory to the absolute is the threshold of all comprehension.
Descriptive to the absolute is the gateway to all differences.
Non-being is the limit of what is known. Exclamatory to the absolute is the threshold of all transcendence.
Non-being is the origin of heaven and earth. Exclamatory to the absolute is the gate to all diversity.
The name of the universe is not the true name of the nameless.
The Nameless is the gate to all mysteries.
Having a body we see, The body is the door of all wonder.
Hence, The unmanifest is the gateway to every wonder.
Names are ways of naming; Names are limits of names.
Unmanifest is the gate to all things.
The unmanifest is the mother of all things.
Nameless it is the mystery of the universe.
Existence is the domain of all that is.
Obscure is the domain of all that is known.
"It" is the beginning of heaven and earth.
The manifest is the Mother of all things.
This unity, Is the beginning of the universe.
Non-being is the Mother of all things. Non-being is the Mother of all things.
It is the mystery of the universe.
All things are (A Real) if they are perceived as a number, or unity, The mystery is the key to all things.
Names that can be names are not real. Names that cannot be names are real. Nothingness and all its aspects can be perceived as the subtlety of the mind.
The Namelessness or totality is the essence of all things.
Names that can be names are not real. Names that can not be names are real.
Nothingness is the Mother of all Things.
Nothingness is the mystery of all things.
Non-being is the Mother of all things. Non-being is the secret of the universe.
The Teaching which we can name is not the Mystery we can observe. The name that we can give is not the name we can give; but the Name that we can call is the Name that we can see.
The way that we can see is not the Way of the Past; it is the Way of the Past that is manifest, But without lasting, though enduring, we shall see the Unseen.
This we call the Mind, The Mother of the Past and of all Past: It is the gate

Calculating validation loss...
100%|███████████████████████| 40/40 [00:16<00:00,  2.45it/s]
[13000 | 10503.37] validation loss = 1.81
[13000 | 10504.22] loss=2.17 avg=1.96
[15099 | 12666.34] loss=1.95 avg=1.78
Generating samples...
======== SAMPLE 1 ========

The maiden-maiden is the Mother of the Serpent,
The mother of all things, is called Mother,
Who has been named and is called mother, has no share.
He who has no name conceives
He who has an idea conceives
These two, existing without separation,
Have their root in the same source; In the same sort of source,
The One and the Being of the same and both differing in name,
Reaching out from the same source, Dissolves both.
Through these two there arises within all things a measure.
From these two measure and measure, They are called the World-Dao.
So, just as Nameless, let us look upon the mysterious depths of existence;
As Manifest, the sight of its manifestations.
To rid ourselves of desire, let us view its nameless outer form;
As we perceive its unmanifest core, we see its pure Inner Light.
This is the mystery hidden in the entire mystery.
As for the world, Nameless we see its beginning.
As Manifest we see its progression.
That which we are yet not filled with, Is filled with fire.
The fire of all-pervading Truth, Wherece from without can spring, From which flow all those things that are known.
Truly alone, viewing the Unconstructed, I see a mystery deep within the Unconstructed.
That which is Unconstructed is but the beginning of great things;
That which is Named is but the gateway to great things.
It is by working that the Heavenly Way may become manifest,
That through an effort that We may see its full extent; That is, We seek to see the hidden parts, We may see its mysteries,
that is, We must see the Unconstructed.
In that which is unobstructed, the dawn of all things is brought.
In that which is filled with things, the darkness of all things.
These two emerge and separate, Yet together, All things are called darkness and light.
They are both of One Origin, Which is the Gate to all Profoundness.

The Nameless is the door to Mystery,
The Manifest is to a Reasoned Way the gate to all hidden Mystery.

It is from mysteries which we shall not know, That we may approach to the Eternal, In order that we may obtain the Path of Life.

It is not always the case that, While nothing is experienced, The Essential Principle itself is hidden.
This state of Nothingness is the gateway to the Manifestation of all that is Eternal.
It is the origin of that which is lasting, Therefore it is the Emptiness which is the Gateway to the Finality.
That which is without beginning, It is the gate to all the subtleties of the Eternal;
That which has an origin, Being a process, Is the source of all the subtleties of the Eternal.
If you would know the Divine mystery, Become a Deserter. Life is a spring from a Deeper Mystery.

The unvarying Manifest is the Source of the Unvarying;
That which has a name is the mode of manifestation;
That which is unnamed is the mode of existence.

Unity is source of Manifest, Non-manifest the mother of the manifold.
Thus to watch, to wonder; to have desires and desires; All things, Manifest and non-manifest, To return whence to seek its way.

Tribulation is its mother.
The mystery of the Tao is the Body of the Tao.
The Essence of Tao is Tao, Tao the Cosmic Universe;
All non-being Is its mother.
Non-being is its mother Because it has the Secret Principle hidden,
Though without Secret Principle It is the Mother of all.
That Which Is Deeper Goes Before Deeper,
And That Which Is Secret Goes Before Secret;
They Are Called the One and the Mother of All Things.
The Name Is from Mystery,The Creature from Form,The Origins of all things
To perceive the Secret, Become a deserter
Desire Returns from the Secret: It is its Mother The Secret of all existence.
These words:
"Empty Words"
To see them Is to see the Secret,
To understand The Secret Is to understand The Origin
Its Dynam is Manifestness (Of all)
That is Its Mother (Of all)
That which is empty
Remain a Mystery
Beyond Mystery
Beyond Proserely
With all mystery in Its Depth
Has the Serpent walked
To understand Its Secret
Has become a Serpent
As for that which has a name It is the beginning of All Things
The One without a name has originated All things.

What arose is called the Essence of All Things.
These two that were separate
With opposite names But Ever Dissolving Is the

[15100 | 12682.90] loss=1.69 avg=1.78

I then began training it on the full TTC corpus, which was split into per-chapter files. Remembering the problems with run-on poetry, I added the <|endoftext|> markers to the end of each file. (It would be better to add that to the end of each translation, but HW didn’t include any delimiters for each translation, and doing so manually would be too much work.)

This required substantially more training time.

Some training samples:

[88999 | 6150.00] loss=2.33 avg=2.44
Saving checkpoint/taotehching/model-89000
Generating samples...
======== SAMPLE 1 ========
 Tao Tao's Way is universal. It covers all lands. Therefore those who realize it can accomplish all under Heaven. Therefore he who maintains the Tao, can be believed to know the Universe.

There is an ocean and a river, and all beneath the heavens aspire to be carried by them. The people of the Tao country follow these rivers and streams, as they may be carried. The people of the man of the Tao country follow the rivers and streams, as they may be guided; and the people of the man of the Tao country follow the rivers and streams, as they may be measured. All under heaven strive to be carried by the Tao Way, as they might be carried by Heaven. The great man, when he knows the Tao Way, can achieve all under Heaven. When a man who has completed the Tao is believed to know the Tao, this is shown by his ability to govern.

All Tao is infinite; to people like it, it is the very mother of all things. That which flows with the Tao is universal; that which opposes it is not. The Great King possesses the Tao Way, so that he can hold everything under heaven. To know Tao, one must be a man of Tao. To know the Great King is the same as to know Heaven. The Great King who understands Tao can hold everything under heaven. A man of Tao must know everything under Heaven, just like he knows Heaven.

True and Eternal Tao Tao can take care of all things that exist, without needing a reason. The person who is able to hear Tao is capable of capturing everything under heaven. Having enough listening ability, the one who can take care of Tao Is able to know all things in the universe. Having enough being able to listen to Tao, The one able to know everything under heaven is able to govern all under heaven. Thus he who is able to control Tao is the one who will be able to control the universe. Whoever knows Tao can grasp everything under heaven. The person of Tao has no limitations, therefore he is able to grasp everything under heaven.

By Tao I can all things be fulfilled, Without needing to get into details. All things call to the Tao Way Without needing to tell the other Tao people to be fit. I know the Tao Way And all things flow with me. If you're also able to think, You can capture the whole Tao Way. Knowing all things That come to you Can help you to know all things that come to others. When one can know Tao You can also master Tao. When the Master knows Tao He can master the Tao Way. And if you are able to master the Tao You can also take all things into your own hands. You don't have to control Tao or all things.

Under Tao, Everything is fulfilled. Tao never requires a reason for doing something. All things— even beasts and plants— seek out Tao. But because of Tao's inner being, All things have this one thing to do with it. Knowing the Tao gives you the inner clarity of Tao. But to have the inner clarity you also have to know that you are a human being. So one who is a Taoist learns that he is a human being. The Master of Tao learned that he was a human being. The Taoist who understands the Tao is not burdened by it; He has the inner clarity he needs.

The Way of Tao: All things call to it, Through but one source. Everything that desires the Tao Must either reject or destroy itself. The Great Thinker has Tao, And can guide all under Heaven. To have the Tao Knowledge is to know the mysterious Source. Knowing the mystery reveals the subtle power of Tao. The Supreme Man can conquer all under Heaven. If he knows Tao, he can be truly the Master of All. To know Tao is to understand the mysterious nature of Heaven. If you know Tao, you also can serve Heaven. The Great Thinker has Tao to serve Heaven, And the Taoist who has Tao to serve Heaven is equal to all things.

All things around a "Way of Tao" “says to its people, “everything”s the natural way,”‘ and everything that should reject it’s it alone. Knowing everything about Tao, people accept Tao, and everything under heaven has Tao in it. To know Tao is to know all things. To know Tao is to know that Tao itself is all things, and the Master of Tao can know all things under heaven. Therefore, he who understands Tao can serve Tao.

Tao was born of the Tao Way It's there for you But not through words, and it's everywhere All things call to it, But not through its Tao Words will not accomplish the feat. You cannot hold on to Tao's place without grasping the primeval essence of this force of nature. Know Tao, or your Tao will know the universe It's how you grasp the primeval essence that matters Don't hold Tao hostage and force it to shape

Calculating validation loss...
100%|██████████████████████| 40/40 [00:16<00:00,  2.36it/s]
[89000 | 6186.11] validation loss = 2.44
[89000 | 6186.98] loss=2.34 avg=2.44
[91999 | 8892.62] loss=2.22 avg=2.42
Saving checkpoint/taotehching/model-92000
Generating samples...
======== SAMPLE 1 ========
 eyes, nothingness. Therefore, he who observes the Dao conceives at his door a room, the ground a mat of grasses, the great thing a hole. He finds no traces of the dwelling of the Sage. He finds in the dwelling of the Sage the Mother, the mother of all. The Sage Mother produces the mother of all. Thus, the Sage governs all things without using arms. He governs them without using the tools of the world. It is he who is the mother of all, and therefore of all things.

To know the Tao is the most important thing in the world. The minute, mysterious details of Tao are what people value. To know the Tao and have a good memory is more important than learning the world. For when the world was simple, The small details were as important as the great details. Nowadays, we are taught to study all things well, but then forget to know what the Tao is. We learn the Tao and love to be mysterious, but when we don't know what the Tao is, we can't be specific. For when we know what the Tao is, it's as if we have a place where all things are and born from. It is the way of Tao.

"The Tao" (truth) is the greatest attribute on the whole world; The minute details of the Tao are important. One who knows Tao enjoys quietude, while one who does not know Tao is restrained. One who is called "taoist" desires the dark side and the subtle, while one who is not called "taoist" wishes to be subtle. They are not the same. Therefore, those who call themselves "Taoists" are not the same as those who are not "Taoists." They who live the way of the Tao do not have weapons, they do not wage war and use the tools of men. In the matter of life, they are different. Thus they are different. That is why they live in the world.

The greater part of humanity cherishes and follows the Tao in common with Tao, even the great and glorious antiquity. This is the reason why people respect the Tao and live in the Tao. Tao is the essence of mankind, or rather the essence of all. It is not the essence of the universe. Only the Son (the Existence) knows the nature of the universe, and the former also what the latter exists. As the Son knows the nature of the universe, the latter also is known. Since that is the reason why the Son has only one nature, and that the latter the other, therefore the Son is distinct from the former in both respect. Therefore, the Son rules the universe and the universe rules the Son. The universe is understood of, by, and for the Son, and he serves the universe with his own hand.

The Tao (the Eternal) is the mother of all Things. Only those who grasp and truly live the Tao can be truly happy. Those who ignore the Tao (the Eternal) turn into chaos. Those who hold fast to belief in false beliefs deprive all beings of happiness. Those who glorify existing things end up in misery. The Tao's path is tranquil; Its ways are pure. It can act with the Tao without deceit. It can behave with the Tao without weapons. It can act with the Tao without instruments. This is what it is like.

To know the Tao is to be the mother of all things, As minute as the minute details, True as being whole. Those who know Tao love not to be ignorant; Those who do not love Tao do not know it; Hence, they cannot be good. Those who keep faith in the Tao get the mystical, Those who no longer believe in the Tao Are called demons. Those who trust the Tao Are bound to bless them, Those who love the Tao are good, Because the Tao is good. The Tao is the mother of all, Which is in all but name.

To know the Tao is to have the mother of all things. Nowadays, the tiny details are more important than the great details. Nowadays, we are surrounded by thoughts and desires. Those who do not live on the Tao have no desires. Those who are good with the Tao see everything in a whole. This is why the Tao is the mother of all things. Because it lives for all things, it does not become a weapon. Because it makes people good with the Tao, they are not evil. Because the Tao is good, it always acts as though it were able to act on its own. Thus it is all-pervading. Everyone lives like this because it always acts according to its own nature.

"To know the Dao is to be the mother of all things. Simple and dark, The depths of Tao resemble life. Who can have true purity without knowing Tao? Those who live in the Dao gain the child, Those who do not live in the Dao suffer in the nurture.

Calculating validation loss...
100%|████████████████████████| 40/40 [00:16<00:00,  2.35it/s]
[92000 | 8928.83] validation loss = 2.44
[92000 | 8929.71] loss=3.07 avg=2.43

Hiræth Weltschmerz was able to improve the TTC training dataset by providing the first 41 chapters with the original newlines/linebreaks, and separate translations separated by a blank line, so I replaced blank lines with <|endoftext|> and trained that for ~24 GPU-hours to a final loss of ~2.10. The results read much more poetically, I felt.

For this final TTC-with-linebreaks, I uploaded the model & generated 1000 random samples as usual, but I also generated per-chapter samples. For per-chapter samples, I used csplit to split each file/chapter into separate verses/translations, selected a random 10 of them, and then generated 10 random completions of each one (100 total per chapter):

The idea there is that one can write one’s own Tao of GPT-2, going chapter by chapter: select some of the chapter 1 prompted conditional sample completions to create a new chapter 1, and so on, in a way which would be difficult to do with just random unconditional samples.



Subjectively, the output shows a lot of poetry knowledge, much better than the char-RNN samples. There’s rhyming, themes are continued for shockingly long passages compared to char-RNN, and there are many passages I feel could inspire a poet or even be cleaned up a little to be passable poems on their own. Adding the metadata did help—GPT-2-poetry is worse than GPT-2-poetry-prefix. Some of the ones I liked most are (first lines) ‘We never say “Thank you”’, ‘Thy soul, thy very soul is burning!’, ‘“It is morn!” said the clover-bush’, ‘And they have seen the last light fail’, ‘There comes a murmur low and sweet’, and probably the best is ‘The sun is gone, and the night is late’.

Is GPT-2-poetry-prefix better than GPT-2-small at poetry completions (since GPT-2-small will probably hardly ever generate poetry without a prompt)? Probably, with exceptions. “Howl” is far worse, but that is for good reason related to the oldness of the PG corpus; if anyone could assemble an equally large corpus of more recent poetry, I’d expect GPT-2-small finetuning to produce better completions. The Pope samples from GPT-2-poetry-prefix are clearly better (before diverging into prose). I would argue that the Shelley samples are somewhat better. And the 8 famous line completions are overall of much higher poetic quality (several of the GPT-2-small completions are just prose, unsurprisingly).

So, if one is looking for poetry completions in an old-fashioned vein, it delivers, but at the cost of flexibility like more prose-like (and hence contemporary) poems. This is an expected and fixable problem, and overall, I consider GPT-2-poetry-prefix to be successful as a poem generator & better than my previous char-RNNs.


Nor is this near the limit for Transformer-based poetry generation, as there are many possible improvements which could be made, all of which I’d expect to deliver substantial gains:

  • make it bigger:

    • bigger NN models: these results use the publicly-released GPT-2-small, which delivers inferior results on all tasks compared to the unreleased GPT-2-large: the samples generated by OpenAI & associates from GPT-2-large are much better than GPT-2-small samples, indicating that simply scaling up continues to deliver gains. Nor did the various GPT-2 model sizes appear to reach any natural limit with GPT-2-large, indicating that the Transformer NNs can be increased much further before hitting zero marginal gains. (This is consistent with other large-scale NN research, particularly on CNNs where even billions of images can be usefully trained upon.)

      OpenAI’s (February 2019) that OpenAI intends to keep scaling GPT-2-large up with aspirations of training 10–1000x larger models (although it’s unclear what these factors refer to: data corpus size, model parameters, window width, layer depth, or all of the above). Given that GPT-2-small delivers such qualitatively inferior results to the ~10x bigger GPT-2-large, and if GPT-2-large is equivalently qualitatively inferior to a hypothetical 10x bigger ‘GPT-2-huge’ and a 1000x bigger still ‘GPT-2-enormous’ is possible, the quality leap from GPT-2-small poetry to a hypothetical ‘GPT-2-enormous’ would be staggering.

    • better NN models (which will probably need to be bigger): , adding recurrency like Transformer-XL or or or more attention heads or more layers or external memory functions or on-the-fly adaptation; there are many possibilities here. (The prefix can be seen as an extremely crude kind of recurrency or memory, and helped a lot; how much more so a real memory?)

    • more & better data: quantity-wise, the PG corpus is barely a tenth of a gigabyte and exhibits many enormous omissions—all of modern poetry, for example, not to mention most foreign poetry, or non-English poetry as a whole (why not a multi-lingual GPT-2 if sufficiently large? neural machine translation approaches improve the more languages they have access to, why not regular language generation?). There are many places additional poetry could be obtained from, such as WikiSource, Poetry Foundation, Libgen, or the Internet in general (perhaps write a poetry-detector Transformer to search through a dump like Common Crawl for poetry?). Quality-wise, the PG corpus is good but still has a number of flaws: a lot of prose, just enough non-English poetry to screw things up (especially Latin), mostly pre-1923 poetry, & minimal metadata (ideally, poems would be individual units rather than book-length streams, and metadata like author would be available to use in prefixes)

  • generate smarter

    • using a better sampling strategy than top-k, like (but curiously, not —beam search gives substantial improvements on what the nucleus sampling authors call “closed” text generation tasks like translation, but while , it damages results badly the wider the beam, and gives particularly bad results on GPT-2; that beam search can work in contexts with heavy constraints, like being constrained to generate rhyming lines or explicit repetition penalties)

      Note: nucleus sampling has been implemented in nshepperd’s Tensorflow & Hugging Face’s PyTorch GPT-2 sampling code.

    • use tree search methods: any deep, thorough, search inevitably becomes a tree; tree searches are useful for enabling kinds of ‘backtracking’ and ‘revision’ or ‘changing its mind’ about multiple possible variants of a poem, as opposed to the usual sampling approaches which tend to commit to each word and force all-or-nothing choices

  • train better, by fixing the loss (eg 11), or switching to the RL setting to directly maximize generation quality:

    • adding global end-to-end losses, which enable training to optimize non-differentiable properties rather than easy (but partially irrelevant ones like predictive losses such as cross-entropy in prediction of the next word). For example, rules defining acceptable meter or rhyme use or penalizing total repetition—these cannot be done via the normal training because no individual discrete word is responsible and parameters cannot be smoothly adjusted to decrease/increase a global property like ‘rhymes’ which is the result of all words considered together as a whole. (This sort of RL loss has been employed in other natural language tasks like machine translation, where metrics like predictive loss do not map onto the desired goal of semantically-correct translation, and word-by-word generation of translations yields similar issues as here, but there are metrics like BLEU or ROUGE or grammar checkers which provide a crude measure of global quality. RL approaches have .)

    • using subjective quality-based losses, like :

      instead of training a NN to predict individual next-characters as accurately as possible or imitate a text corpus as well as possible, we really just want them to predict good next-characters to write text as well as possible—which is not the same thing at all, any more than accurately predicting a human Go player’s next move on average is the same thing as playing Go superhumanly well.

      This encourages more global coherency, more thematic progressions, use of rare words when appropriate, surprising subversions or twists which work well when tried but don’t appear in the original corpus, learning esthetics, and so on. If it works and the new GPT-2-poetry is able to successfully produce new poems which consistently get the top score from the critic and no further improvement is happening, then you simply read a bunch of its new poems, pick which one in each pair you like, retrain the critic on the expanded dataset to detect the remaining flaws in the ones you disliked, and then keep training GPT-2-poetry to avoid generating the ones you disliked & generate more poems like the ones you liked. Repeat with many cycles, and it should generate excellent poems while avoiding all the flaws of crude likelihood training and even cruder top-k sampling which hobble GPT-2-poetry right now. Even better, you could create a website to crowdsource the rankings to keep it training 24/7 and improving indefinitely.

    • using “expert iteration” architectures like AlphaZero to do much more sophisticated search over possible poems, creating an iterative bootstrap

    • adding creativity losses along the lines of , Elgammal et al 2017, where updating GANs encourage diversity

      • one could attempt to invent new styles of poetry by taking inspiration from evolutionary methods, such as the “Population-Based Training” variant employed in which created diversity by deliberately scrambling the ‘rules’ for each lineage of agents. The “AlphaStar League” used a population of multiple NNs, each forced to specialize in using a particular unit or rewarded for achieving particular goals like defeating a specific NN (rather than winning in general). The AlphaStar League was credited for forcing the overall AlphaStar population to explore strategies reliant on particular kinds of units and figuring out counter-strategies to successful ones. Something similar could be done with poetry rules: train many different agents, each given a specific rhyme scene or meter or vocabulary for their reward function, and in preference-learning approaches, the best poems can be provided to human critics for rating & improving the NN critic. Potentially exciting new combos could emerge as producing the best poems as rated by the humans.

Given that GPT-2-small is far from the state of the art as of February 2019, and hardware & generative NN research is advancing rapidly, it will be exciting to see what sort of poetry can be generated given another 4 years!


Bradley-Terry Preference Learning

introduced a deep reinforcement learning architecture for learning “I know it when I see it” subjectively-defined reward functions from human feedback: a human makes comparisons of actions/datapoints/episodes to select the ‘better’ one, a NN is trained to predict the better one based on these comparisons, and another NN is RL-trained based on the predicted comparisons interpreted as a reward. Since the human is unable to write down a conventional reward function in software, the predictor NN (analogous to a Discriminator in a GAN or a ‘critic’ in actor-critic RL) learns the reward function by example, and then the RL agent NN (analogous to a Generator in a GAN) learns by trial-and-error what sequences will optimize this complex reward function, and the human feedback provides additional guidance on new parts of the problem as the pair of NNs bootstrap into better performance. This is demonstrated on video game or robotic-style simulations, but appears equally applicable to other sequence problems where reward functions are impossible to write and existing losses like maximum likelihood are imperfect for generation (such as music or poetry composition).

As originally framed, the predictor merely does comparisons, receiving & providing binary feedback. This is justified as being implicitly equivalent to a standard pair-comparison/competition model, the (akin to the famous ELO), where each datapoint has a latent variable on a common cardinal scale (often, like a , scaled to for convenience), producing a total order which efficiently extracts all possible information from the comparisons.

I suggest that this is not necessarily the case, as examples from GANs indicate that such a preference-learning architecture may be learning something odder (such as memorizing comparisons), and that the architecture could be improved by removing the implicitness of the B-T ranking, computing the B-T rankings directly (which can be done even with non-overlapping comparisons by using a Bayesian model with priors and using covariates such as the predictor’s own estimates), thereby providing absolute quality scores for correctness of comparisons, more efficient regression, RL rewards, and meaningful interpretable scores for downstream uses.

The motivation for the double-critic architecture is that the data being collected from humans is pairwise, and so one trains the critic to predict comparisons. This outside training loop then has an inner G/agent training loop etc. The double training loop is necessary to collect ratings from brand new areas of statespace that the G/agent can newly access, but also, GAN-style, to avoid the D/critic from being too powerful and saturating loss. (The original Christiano implementation is used to avoid problems, with only the most recent n=3000 comparisons are stored.)

But, just because the input is pairwise doesn’t mean that the output must also be pairwise. It could instead be a scalar indicating global rank, with the D/critic performing regression. GANs and DRL are closely connected (/), and in both fields, a richer reward signal is always better, allowing for stabler faster training to better final performance. And a global rank is more informative than a comparison.

Full Bradley-Terry Training

A Bradley-Terry (B-T) model is simple and easy to estimate on even large samples12, and can easily produce cardinal rankings. Each datapoint gets an estimated cardinal ranking in standard deviations of a hypothetical latent Gaussian. The D/critic then is trained to do regression from a single input to the estimated latent variable of quality.

So the new loop would look like this:

  1. run off-the-shelf B-T over a dataset of comparisons of datapoints

  2. extract the estimated latent variables for each datapoint

  3. until convergence, supervised training of a D/critic NN to predict the latent for each datapoint

  4. until convergence, RL training of a G/agent NN with the D/critic NN

  5. extract n new datapoints from the trained G/agent NN and add to the dataset

  6. run B-T over the augmented dataset

  7. ask the oracle for a rating of the m datapoints with the largest posterior uncertainty or some proxy thereof like standard error (which will usually be the new datapoints)

    • active sampling or bandit algorithms can be used to maximize the informativeness

Is Preference Learning a Bradley-Terry Model?

What’s the difference?

By using only comparisons, each predictor training is less meaningful, and even if the predicted variable is still results of comparisons, not fitting a B-T model means that one can’t train on comparisons between all datapoints (since one needs the B-T model to predict, based on the global ranking, what the outcome would be).

It is also unclear that the preference learning architecture is implicitly estimating a B-T model. (I am not familiar with any paired-comparisons approaches which optimize ML models to predict a fixed set of comparisons, or work purely on disconnected comparisons.) Because no global ranking is ever constructed, no comparisons can be trained on other than the exact ones that the human made, and that may not be enough training signal to force inferring a global ranking, rather than merely learning locally-consistent pairwise comparisons which are nevertheless globally inconsistent (with cycles like rock-paper-scissors). The predictor may be learning something much simpler, such as which distinguish within each fixed pair, but without learning what we thought it was learning—generalizable quality features which allow a meaningful global ranking across all pairs.

In a GAN, you have real and fake datapoints being compared; the D attempts to regress the probability of each point being a winner/loser, so to speak, producing a log probability (in the original formulation); does D learn generic features of quality or realism? Apparently not because even a ; and in my anime face GAN work, when I use a well-trained StyleGAN D to rank real data, the rankings are strange, with outliers ranked both low & high, suggesting that garbage data can be ranked extremely confidently by the D simply because it could easily memorize them as outliers. So, we have a case where a D/critic is trained on comparison data from an oracle (real vs fake), is useful for training, outputs a variable which look exactly like an ELO and even has an ELO-like theoretical interpretation—and is completely ungeneralizable and not learning anything remotely like a cardinal score or even a transformation thereof like an ELO. What is going on? Apparently the D is memorizing real datapoints, and pushing the G away from them and toward nearby potential datapoints.

Why can’t this be the same thing for the preference-learning D? It is given a small dataset consisting of fixed pairs of good/bad datapoints, and it memorizes bad datapoints within a fixed pair, latching on to some feature or other (possibly important features, but they could also be the ‘non-robust features’ involved in adversarial learning) in order to memorize just within that pair (if it can overfit…), and this then pushes the G away from trajectories that look like bad datapoints, producing useful training just like in a GAN.

This would be consistent with the paper’s reported success, but would have a different interpretation: the D is not learning any generic quality metric, is not implicitly ranking all datapoints on a common scale of reward, and is not equivalent to a B-T. It is merely memorizing some data points or some ungeneralizable non-robust features which happen to let it distinguish within the pairs. As such, it can’t provide a stable ranking within or across iterations or datasets, and its feedback is of limited value (since once the G/agent has moved sufficiently far away from the penalized memorized datapoints, that no longer provides a training signal for more improvement and new relatively-bad datapoints must be learned and penalized).

Active Learning

As implemented, preference learning is (potentially, assuming it’s equivalent to B-T) more sample-efficient than a naive B-T: each data point appears once in a unique comparison (rather than in multiple comparisons with multiple other datapoints), and so each comparison is potentially maximally efficient (in the sense that each additional comparison involving a datapoint provides the predictor less information than the first one did). A naive B-T, like the usual frequentist implementation, requires multiple comparisons to connect all datapoints via a chain of comparisons, and may be undefined if any datapoints are ‘unconnected’.

A Bayesian B-T model mitigates this by having priors on any new datapoint, which provides a meaningful estimate without few or no comparisons. (With no comparisons, the posterior mean is simply the prior mean, presumably something like 0.) The estimates aren’t informative, but they are well-defined and can be used for sampling strategies.

The lack of comparisons can be fixed partly by using covariates. There are two particularly relevant covariates which could be used:

  1. the predictor’s own ratings of each datapoint:

    Since the predictor should be able to reach high accuracy, its estimate before any comparisons should be quite accurate and reduce the posterior uncertainty considerably (despite having no comparisons). This can be particularly useful for a sampling strategy because it can help discard samples which are estimated as low quality and not informative about the best samples that we want to reach.

  2. the current iteration

    Since we hope the generator/agent is also improving, the iteration a datapoint was generated from is relevant: early datapoints should be bad, intermediate datapoints should be medium, and recent datapoints should be the best. The first few comparisons inside a batch give a strong indication how good the batch is overall, and the quality can also be extrapolated from earlier iterations by fitting a progress curve (like a log or spline).

An example of a sampling algorithm would be best-arm racing algorithms. Since in this scenario, we’re trying to teach the NN to generate the best datapoints, we don’t value variance reduction elsewhere, we want certainty about the best datapoints in order to penalize the NN for generating any inferior datapoints. A simple posterior sampling racing algorithm for B-T might goes like this:

  1. take the arm/datapoint with the highest posterior mean ranking, which is estimated to be the best;
  2. draw a ranking from the posterior of every other datapoint;
  3. compare the best known datapoint with the highest posterior-sample;
  4. update.

This explores datapoints based on their remaining posterior probability of being the best. (I used this once to ) This can be applied to the k best datapoints for batch evaluation etc.

So a training loop could go like, begin iteration #11 by generating 1000 new samples from iteration #10’s G/agent model; score each with the D/critic; insert the 1000 into the dataset with their estimated score and iteration=10 covariate; do the B-T regression with Comparison[NA1][NA2] ~ iteration1+criticEstimate1 - iteration2+criticEstimate2 (pseudocode) to estimate posterior distributions of estimates for all datapoints (missingness of comparisons doesn’t matter, the model can still be fit); run the racing algorithm, finding that new sample #551 has a critic score of +5SD, giving a posterior estimate exceeding all other datapoints (despite not having been ever compared yet), and that new sample #998 get picked by posterior sampling; ask the user to compare #551 and #998; record the result; refit the B-T for an updated ranking; retrain the D/critic; retrain the G/agent; begin iteration #12 etc.

We efficiently home in on the best datapoints without necessarily requiring any ‘redundant’ comparisons, while providing informative stable cardinal rankings for the D/critic based on an ordering of the entire dataset, enabling it to provide more meaningful rewards to the G/agent. To the extent that we engage in ‘redundant’ comparisons, unlike the preference learning approach, those comparisons must have been necessary.

It’s an adaptive procedure so it’s hard to say exactly how it would differ from preference learning. Depending on how much the G improves each iteration, and how accurate the D is, and thus how much posterior overlap there is between different batches and different datapoints within each batch, it could look a lot like the current heuristic approach of doing only unique comparisons once within a batch and throwing away+never-comparing with prior batches, or it could look quite different, and change with each iteration as necessary:

  • If the G improves relatively slowly, so there’s a great deal of overlap between successive batches, and/or the D is only weakly correlated with measured rankings, then the procedure might need to sample a lot of comparisons between old/new batches in order to improve estimates of the progress curve and all datapoints within the new batch, and it might want many comparisons toward the tail of highest-ranked datapoints (which is not a bad thing because that’s where we should prioritize improvements, since that’s where the G is moving towards, and it’s less important to estimate accurately less-highly-ranked datapoints).
  • If the G or Ds are intermediate, I think the dynamics might look more like sampling mostly pairs within the new batch, mostly unique comparisons, and a few comparisons with old batches to finetune the mean of the new batch.
  • If the D + G progresses so rapidly such that rankings don’t overlap at all a priori, then few or no comparisons with the old batches are necessary: the D covariate predicted-rankings eliminate most of the posterior uncertainty despite no comparisons being available, and the G progress means that the old datapoints (while still useful for G training in teaching it the full spectrum of datapoints) are unlikely to be anywhere near the best datapoints and so aren’t worth measuring more accurately, so comparisons focus on the most uncertain pairs in the new batch.

Advantages & Disadvantages

This could have a lot of advantages:

  1. the D/critic NN is conceptually simplified: instead of 3-way classification on a double input corresponding to an implicit global ranking, it is just a single input for regression on a quality score

  2. more memory-efficient: before, a double input takes up memory, even with tied weights, only to yield a single comparison; in the same space, 2 regression models could be run, each with a different input + target quality rating. If, to save memory (critical with GPT-2), a single input is used instead, now there must be two separate passes for each input, and each pass merely trains one-half of the comparison.

    This could be particularly useful if one tries to use a large Transformer model like GPT-2-345M where memory consumption becomes a serious barrier to running it at all… (At 345M, we’re down to n=1 minibatches.)

  3. more D data-efficient: many comparisons will be useless, or for a given pair, they will quickly cease to be informative; a quality rating is informative regardless of what might’ve been used as a comparison, providing richer feedback on each input (analogous to AlphaZero switching to a regression target)

    • possibly better ‘off-policy’ learning: related to saturating, a D/critic trained from a corpus (eg initializing a D/critic by taking a dataset of real and GPT-2-generated poems, and labeling all comparisons as victory for the human poem) might destroy G/agent training if it provides only comparison feedback
    • better value function/reward signal for any other approach leveraging the NNs (like MCTS over a tree of sequences), too
    • humans or other datasets can supply cardinal ratings directly when those are available
  4. possibly more D training-efficient: by training comparisons, the D/critic must, implicitly, be learning an equivalent to a quality rating, in order to provide accurate predictions of a human comparison of all possible pairs—but it does so in an indirect fashion

  5. G gets more data & training efficient: a richer reward signal for each sample will of course be quite useful for the G; instead of saturating (there is intrinsically not much information in comparisons, and moving from, say, 99.99% to 99.999% is not helpful)

  6. the quality variable provides an objective loss for understanding training progress (useful for tasks which don’t have them, like poetry generation!), which is also interpretable and could be useful outside of the task (eg ranking poems for recommendation or data-cleaning)

    • for example, one could get insight into a trained G/agent by generating a number of samples, and ranking them
  7. enables active learning via B-T posterior uncertainty without any need to extract uncertainty estimates of any kind from the D/critic NN; human ratings can be acquired more efficiently, or datapoints selectively pulled from a large dataset (eg imagine a huge dump of poems from Project Gutenberg or elsewhere, of wildly varying quality—with a regression style D/critic NN, you can do a single pass over it with the D/critic NN to select the k% highest poems, use the estimate as a pseudo-datapoint, insert into B-T, and ask humans for the most informative comparisons; with a comparison D/critic NN, how to import usefully a large unlabeled corpus is harder to see)

The main downsides I can see:

  • the latent variables are not necessarily 100% stable, as the whole distribution can drift. The B-T estimates a distribution arbitrarily defined as ; if the B-T sees only very selected datapoints at the beginning, it might be that after G/agent trains enough, the B-T step would be looking at datapoints which are much better than a mean of 0, so there might be new datapoints all the way out at (what used to be) +100SDs, say. This then leads to the B-T estimate the next cycle shifting the mean/SD to restore the conventional . So the regression target for the D/critic’s predictions of old datapoints may gradually shift over time, precisely because the richer latent variables don’t saturate the way simple pairwise comparisons would. I believe this would be a minor problem easily solved by training the D/critic NN each iteration, which is necessary just to handle novel datapoints anyway; since improvements will be small each iteration, the retraining should be easily able to keep up.
  • (frequentist) B-T might require more comparisons in order to infer any total order: a datapoint has to be compared with other datapoints which themselves have comparisons if it is to be globally ranked at all, while a comparison D/critic can work with two entirely disjoint sets of comparisons which don’t overlap. (This can be avoided by using priors & covariates in a Bayesian B-T model.)

All in all, I think this version of preference could be simpler, easier to implement, and train faster. The potentially better sampling is nice, but my guess is that the D providing richer feedback (for both the G and downstream users) is the biggest advantage of this approach—a comparison is a bit, and a bit is worth only a little bit.

  1. A Transformer is a considerably different architecture than an RNN, and is not that easy to explain, as it uses multiple convolutions to implement “attention”, allowing flexible internal control flow, over a large but finite input window, without any recurrency or hidden state or LSTM units necessary. For increasingly-technical explanations, see:

  2. 774M requires changes to nshepperd’s checkpointing, specifically, removing the layer == 10 restriction in, and letting the checkpointing code checkpoint as much as possible, which enables training minibatches n≤10 on my 2x1080tis.↩︎

  3. It would require either high-end GPUs with >=16GB VRAM, or TPU instances (which were used to train it). GPT-2-1.5b can’t be trained on my 1080tis with either the nshepperd codebase or Shawn Presser’s fork, although Presser has a which can train it.↩︎

  4. Other examples of finetuning are , nshepperd’s unpublished Linux kernel C source code & IRC-log training, and . And, while it doesn’t use GPT-2-small, too good to not mention is .↩︎

  5. For example, 2 people finetuned GPT-2-small on an IRC channel’s logs, getting losses of 1.95 & 2.3; why was the latter’s loss 18% worse compared to the former when they were using the same IRC channel, GPT-2-small pretrained model, training codebase, & had both apparently converged? Because. while the IRC channel was the same, they used different IRC clients which had different IRC log formatting conventions—the former’s logs had the full timestamp prefixed to each line, and the latter didn’t. Said timestamps made up ~20 characters of ~110 character lines, or, ~18% of each line! So the models were performing identically on the content that mattered, and the much lower loss was simply because of near-perfect prediction of the highly-repetitive & predictable timestamps on every line. (Indeed, given the limited window of GPT-2-small, arguably the model with the worse loss would be better in terms of generating fun coherent samples.)↩︎

  6. I have 2 GPUs but nshepperd’s code does not (yet) support multi-GPU training easily. Some support using Horovod for multi-GPU has been added but I cannot vouch for it. ↩︎

  7. One might worry that by taking up space in the model’s limited context ‘window’ of inputs, because the Transformer has no hidden state or ‘memory’, such inline metadata would be a bad thing as it will push real words out of the context window, thereby degrading quality and making it even more incoherent & rambling.

    But on the other hand, if it does learn to associate specific IDs with genres/topics, then repetition of the inline metadata serves as a ‘mnemonic’ for global information which is available to all subsequent iterations of the model, serving as a crude memory itself.

    For example, if Homeric pastiche has ID #16452, then as long as the final iteration of the model overlaps for just the ID with the first iteration of model during sampling and both see “16452”, all models will be able to consistently agree on generating Homeric pastiche rather than some other pastiche because they all see the same ID somewhere in their context window & that guides their generation.↩︎

  8. has collated some of the results:

    • All U.S. presidents and Russian leaders in temporal order, where the order was not specified in the documents used; also, all tennis champions in international competitions over the years. So, temporal order can be extracted.
    • The longitude and latitude of cities in the U.S. and Europe, along with their relative distances.
    • The relative size of many kinds of objects, like cars, elephants, humans, houses, and so on—which object is larger than which others.
    • The exact sizes of many objects in meters, with reasonably small error. For example, it might say the dimensions of a windshield are about 1.4 meters by 1 meter.
    • Which kinds of animals are dangerous, which are not; and which kinds of objects (e.g. “fire”) are dangerous, and which are not (e.g. “water”).
    • Which animals are smarter than which other ones; which animals are faster, which are slower; which animals are heavy, which are light; which animals live in water, which do not.
    • Which cities cause arousal (e.g. “fun”, “exciting”), and which do not; which are expensive, which are not; which are dangerous, which are not; which are religious, which are not; which are large, which are not; which are hot, which are not; which are wealthy, which are not; which have a recognized intellectual culture, which do not.
    • Which kinds of clothes are appropriate for different age groups; which kinds cause emotional arousal; which kinds are expensive; which kinds are appropriate for different sexes; which kinds you expect to find in different locations; which kinds are associated with wealth; which are not appropriate for hot weather, which are; same for cold weather.
    • Qualities of mythological creatures—like the ones for animals.
    • Qualities of professions and professionals: age, arousal, danger, gender, intelligence, location, valence, wealth.
    • Qualities of sports and sportsmen / women: arousal, danger, gender, intelligence, location, speed, wealth.
    • Qualities of states: cost, intelligence, political, religiosity, size, temperature, wealth.
    • Qualities of types of weather and weather phenomena (e.g. “tornado”): danger, temperature, wetness.
    • Physical properties of objects, such as rigidness and strength. Probably also includes transparency, softness, hardness, round, prickly, angular, and so on.
    • Relations like whole-and-part, and relative locations of a part within an object: e.g. hand is connected to arm, arm is connected to shoulder, shoulder is connected to neck, neck is connected to head.
    • Properties of countries and cities: geolocation, GDP, GNI per-capita, CO2 emissions per-capita, fertility rate, amount of internet use, calling code, military expenditure, life expectancy, energy use, population, places imported from, how long they’ve had a national anthem, kinds of sports, GDP growth, crime rate, and so on.
    • Binary attributes of countries and cities: continent, time zones, contained-by (which regions contain which countries; which countries contain which cities; which boroughs are contained in which cities; and perhaps even relations between the boroughs—which border which others, how they are shaped, and how large they are), language, high or low crime?, military conflicts, athletes, medals won, organizations founded, schools founded, companies founded, weather, type of government, officials, and many more.
    • It’s even possible to predict the qualities of objects not in the training corpus, using something called the

    [Some references:]

  9. Child et al 2019:

    We also introduce (a) a variation on architecture and initialization to train deeper networks, (b) the recomputation of attention matrices to save memory, and (c) fast attention kernels for training. We call networks with these changes “Sparse Transformers”, and show they can model sequences tens of thousands of timesteps long using hundreds of layers. We use the same architecture to model images, audio, and text from raw bytes, setting a new state of the art for density modeling of Enwik8, CIFAR-10, and ImageNet-64. We generate unconditional samples that demonstrate global coherence and great diversity, and show it is possible in principle to use self-attention to model sequences of length one million or more.

    5.4. Saving memory by recomputing attention weights

    Gradient checkpointing has been shown to be effective in reducing the memory requirements of training deep neural networks (), (). It is worth noting, however, that this technique is particularly effective for self-attention layers when long sequences are processed, as memory usage is high for these layers relative to the cost of computing them.Using recomputation alone, we are able to train dense attention networks with hundreds of layers on sequence lengths of 16,384, which would be infeasible on modern hardware otherwise. In our experiments, we recompute the attention and feed-forward blocks during the backwards pass.

    …For each sequence length, we attempted to train the largest model which could entirely fit into 16GB V100 accelerators without model parallelism. Overall, we found that increasing the sequence length by a factor of 4 requires a reduction in model capacity of approximately . Thus we found we could use factorized self-attention on sequences over 1 million timesteps long, albeit with extremely few parameters (3 million).

  10. Because it is so large, I am providing an alternate rsync mirror: rsync --verbose rsync:// ./↩︎

  11. also demonstrates problems with likelihood decoding strategies in GPT-2 for story generation.↩︎

  12. eg my simple , just re-estimates the entire B-T model each interaction, rather than attempt any caching or incremental updating to a stored model, because it takes a fraction of a second to fit. A fully Bayesian model can be fit via MCMC in a few seconds, which is negligible in a DRL context.↩︎