GPT 2 Poetry

Some of the GPT-2 based tools available publicly can generate decent examples with the right prompt. However I wanted to see whether a finetuned model might do a little better. 

I collected a corpus of traditional poetry from PoetryDB, together with this corpus. After finetuning and taking samples at various different stages, my impression is that a finetuned model (in this case the GPT 2 medium size, 355M) can indeed generate poetry that holds its own against the default models', and in some cases may well be an improvement. A well crafted prompt is important though, and it helps to take into account the corpus it was trained on.

Decent snippets of poetry, in my experience, come at a frequency of less than 10% - probably somewhere between 5-10%. It definitely pays to try different prompts. It is clear that the model is willing to try and imitate a particular style, but the effects often wear off as the text generation grows longer. After a while it simply reverts to its comfort zone of country meadows, gurgling streams, and singing skylarks.

I enjoyed playing around with Emily Dickinson's style, because it is so distinctive. After just 12000 steps of training, when average loss was still quite high, I got these (my prompt in bold):

Sample 1

Here is a poem about life on Mars that is written in the terse style of Emily Dickinson:
Bereft I -- astronaut
Idle eyes -- earth
The alien things of Ancestors
Grainy melodies --
Silent majorities -- distant society
As a normal human--
Sadness and my fantasy --
A fear of annihilation --
The peace of carnival at dusk --
Strange adventures -- secrets hidden
Ghetto art-wars --
Dark powers from Russia's mausoleum
Postcards of Mars --
As fantasy, an exoticist
Conquer as I wish!
Peace is the enemy!
Peace the enemy!
There are precious elements
Of war --
Stones of battle -- war --
Air of steel -- shipyard wars
The enemy's sacred hideout
Among the bloodstained bone
All trails left in ruins
Go to the enemy --
I am closer to Heaven than I am to Earth
To Heaven than to a shell --
I am closer to Heaven than I am to death
To Heaven than to a mirror --
To a family --
The stranger to a bank --
The broken wave to which you come --
The little town behind --
The stranger with a nightmare
So clever it ends every day.

Sample 2

Here is a poem about life on Mars that is written in the terse style of Emily Dickinson:
Bereft I -- astronaut
Idle eyes -- earth
The alien things of Ancestors
Grainy melodies --
Silent majorities -- distant society
A wordless rebellion --
Housebound -- stricken with hope --
Cul-de-sac -- highway music
Psychoanalysis
Peace through gun violence
Offshoot of modernity --
Strenuous original genius
Raindrops -- cloud-flowers
Earth's marvelous eagle flight --
Fireflies -- find in poetry
Joy in example --
Astronauts lift alone
Lost souls (Fireflies)
The sole unknown genius of wisdom
The survival of the flying saucers
And humans
Further testing of binary
Death's voice -- death's may
Peace through gun violence
On Mars -- last challenge
On Mars -- the one.
And beyond a wilderness
Involuntary woods -- earth --
Under-written by humans
Animal culture -- obsession --
Dismantling your weakness
Unsung heroism --
Immortality

These examples appeared to be of decent quality despite training loss still being quite high. They probably represent a middle ground between the original model, and the new one, so that much of the model is still biased towards the original text (more prose-like) it had been trained on at OpenAI. Although not shown here, many samples evidenced prose-like sections.

Fast forward to a new version of the model at 33000 training steps with average loss around 0.5, and the poetry is more typically "poetic", but seems to be less versatile overall. This confirms the notion that it has taken on board more of the new training material (poetry), and is less influenced by the wide corpus it started with:

Sample 3

Here is a poem about life on Mars that is written in the terse style of Emily Dickinson:

Bereft I -- astronaut
Idle eyes -- earth
The alien things of Ancestors
Grainy melodies --
Silent majorities -- distant society certainly --
Nocturnal winds -- while the stars are alive --
The tribes of strangers -- and the long-term ill --
And the strange sudden malcontent
That got in, and got out, and didn't rest --
And the antitype -- that never stops --

The bonds of love -- the husband's --
The chieftain's -- and the subject's --
And the stranger in the asylum
The sportive puppy-vassal
Rests with the deer, and the hound with the hound
Stirms with the dwarf-father in the cave.

As the crow's wings grow dully broad
And the green-breasted dove
Eclipses the sun to land,
As the wild cherry-blossoms blow
And grow in the wind,
As the starlight fades
And spreads its feathery arms
And fades away in the flower,
As the blaze of the wild cherry-blossoms
Fades and leaves in the wind --

Sample 4

Here is a poem about the harsh and lonely life on the planet Mars. It is written in the terse style of Emily Dickinson:

Bereft I -- astronaut
Idle eyes -- earth
The alien things of Ancestors
Grainy melodies --
Silent majorities -- distant society
Alone -- in the vastness of the universe --
Settling there -- far away from the nearest
Humane -- or human -- aggressor --
Till I close my -- orbit -- in the moon --
Then out of the night -- into the day --

My career is not -- not yet --
My work is not yet --
The imagination's got my leave
To taste the dulness and thraldom
Of the little things.

The morning's the night's turn and the dreary
Dinner is between --
The fire shall burn forth --

The earth's shade is the lamp's boding light
And the moon's face is thaw --
But his cup, my heart insists,
That water's the wrong drink.

The sky's a slumber -- the north wind
Stirs -- and so, the sky.
For, with this world's uses to come,
A world's house is but a sky.

The sun's a-rising -- the stars,
Spangling the silvery day
With a tapestry of rising light --
The wrong house for me, I hanker

I tried a number of different prompts. Here's something a little like John Ashbery's:

Sample 5

Here is a poem about life on the planet Mars. It is written in the style of John Ashbery:

The rover will roll by later.
For now, the rockworker pauses on red ground,
fills in the time of day, the rocks' location,
leaves a note for the nightworker: "What the hell
are we going to do? I mean ...
The water is already drying up."
Rain on earth, as it does in heaven.

For those who come here, the note is of less importance, the rain. Once, once, indeed.

That voice is drowned.

And now at length the deed is done.

The morning-color comes on high;
The evening-color comes on high.

All the hills around Steep-Foot's bed
Look at the cloud-burthen coming down
From the very, very edge of last night's storms.

How the rock-bearers throw up their spears
Before the first bow of the rising moon!
Pleasant was the task to reach the brink.

Of yellow sapphire, and its rim of gleaming steel,
Is the price we pay for beauty this morn.
Lo, the sun shines, but the clouds are gray,
And the moon is on the big side;
For she rises like a volcano's peak
With her bright, lustreful head,
And her soft glossy arm is bare and soft
Like the hand that slips and slips again.

Traditional poetry is definitely the finetuned model's preferred mode of poetry generation. It seems to reel them off with ease, and the percentage of decent snippets come at a higher rate. It's not that it always makes perfect sense, but it's easy to see how even just a little tweaking can turn it into something quite readable. Here's one example:

Sample 6

I wandered lonely as a toad,
Till, stung by the keenness of my woe,
I half grew to fancy I was
A fowl of another kind,
And yet I could not despise
My own flesh:--a dreamy note
To my more melancholy heart
Said, 'Eat not, weep not; this is pity.'

'Sad was the agony, when
That hoary Flandric tree,--
Pity me! I have none,--
Sore as I am, with barren beak,
And I would die like a wood-dove,--
One moment made me think I was
With all the rest, a bugling wight,
Free from all misery, save fear.'

'Sad was the agony, when
One was clapt in the dark blue air,
And strode to the light of morning;
Sad was the agony, when
Saw a beam of light on a bed
Of many a fluttering flower,
And one was silvery in a ball;
And a sound of music came bright,
Cloven through and through, and far away
It stirr'd the chamber. Sad was the agony
Of listening to that light sound--
It did its work in silence and in pain.--

'So now I was old, and had to die;
Some fadeless bird from the glen,
A day had killed it,

Or another:

Sample 7

I wandered lonely as a toad,
Till, bored, I fell into the water;
For well I know those raggedy old stairs,
Worn by foot, and faded for good:
They know not that I was a Tot,
A "kingdom, or house," from out their memory;--
Never mind! Let darkness in again
Re-illustrate the stupidity of youth.

I've shared the checkpoint at 33000 steps of training along with a Notebook that can be opened in Colab (edit 2020-10-29: updated link to point to new aitextgen version of the notebook).