Sunday, May 03, 2020

BERT for Poetry

After the fun I had with the BERT Summariser and short stories, I decided to turn the trick to poetry. If anything the results are even better.

Here are five examples, starting with The Wasteland, which perfectly illustrates BERT's ability to find continuity.

The Wasteland, by T.S. Eliot


I. The Burial of the Dead

April is the cruellest month, breeding
Lilacs out of the dead land, mixing
Memory and desire, stirring
Dull roots with spring rain.
If you see dear Mrs. Equitone,
Tell her I bring the horoscope myself:
One must be so careful these days.
When Lil’s husband got demobbed, I said—
I didn’t mince my words, I said to her myself,
HURRY UP PLEASE ITS TIME
Now Albert’s coming back, make yourself a bit smart.
But at my back from time to time I hear
The sound of horns and motors, which shall bring
Sweeney to Mrs. Porter in the spring.
In this decayed hole among the mountains
In the faint moonlight, the grass is singing
Over the tumbled graves, about the chapel
There is the empty chapel, only the wind’s home.


The Sonnets, by William Shakespeare


When forty winters shall besiege thy brow,
And dig deep trenches in thy beauty's field,
Thy youth's proud livery so gazed on now,
Will be a tatter'd weed of small worth held:
Then being asked, where all thy beauty lies,
Where all the treasure of thy lusty days;
To say, within thine own deep sunken eyes,
Were an all-eating shame, and thriftless praise.
Thy outward thus with outward praise is crown'd;
But those same tongues, that give thee so thine own,
In other accents do this praise confound
By seeing farther than the eye hath shown.
For I am shamed by that which I bring forth,
And so should you, to love things nothing worth.
So that eternal love in love's fresh case,
Weighs not the dust and injury of age,
Nor gives to necessary wrinkles place,
But makes antiquity for aye his page;
Finding the first conceit of love there bred,
Where time and outward form would show it dead.
Never believe though in my nature reign'd,
All frailties that besiege all kinds of blood,
That it could so preposterously be stain'd,
To leave for nothing all thy sum of good;
For nothing this wide universe I call,
Save thou, my rose, in it thou art my all.
Thence comes it that my name receives a brand,
And almost thence my nature is subdu'd
To what it works in, like the dyer's hand:
Pity me, then, and wish I were renew'd;
Whilst, like a willing patient, I will drink,
Potions of eisel 'gainst my strong infection;
No bitterness that I will bitter think,
Nor double penance, to correct correction.
Poor soul, the centre of my sinful earth,
My sinful earth these rebel powers array,
Why dost thou pine within and suffer dearth,
Painting thy outward walls so costly gay?
Then soul, live thou upon thy servant's loss,
And let that pine to aggravate thy store;
Buy terms divine in selling hours of dross;
Within be fed, without be rich no more:
So shall thou feed on Death, that feeds on men,
And Death once dead, there's no more dying then.


Poems by Elizabeth Barrett Browning


We sow the glebe, we reap the corn,
We build the house where we may rest,
And then, at moments, suddenly,
We look up to the great wide sky,
Inquiring wherefore we were born…
For earnest or for jest? Ere I answered he was gone,
And none was left to love in all the world.
A THOUGHT ay like a flower upon mine heart,
And drew around it other thoughts like bees
For multitude and thirst of sweetnesses;
Whereat rejoicing, I desired the art
Of the Greek whistler, who to wharf and mart
Could lure those insect swarms from orange-trees
That I might hive with me such thoughts and please
My soul so, always.
Let them feel that this cold metallic motion
Is not all the life God fashions or reveals:
Let them prove their living souls against the notion
That they live in you, or under you, O wheels!
If He heard us, He would surely
(For they call Him good and mild)
Answer, smiling down the steep world very purely,
'Come and rest with me, my child.'
They look up with their pale and sunken faces,
And their look is dread to see,
For they mind you of their angels in high places,
With eyes turned on Deity;—
"How long," they say, "how long, O cruel nation,
Will you stand, to move the world, on a child's heart,—
Stifle down with a mailed heel its palpitation,
And tread onward to your throne amid the mart?
How there you sat in summer-time,
May yet be in your mind;
And how you heard the green woods sing
Beneath the freshening wind. Not as the conqueror comes,
They, the true-hearted, came;
Not with the roll of the stirring drums,
And the trumpet that sings of fame;
Not as the flying come,
In silence and in fear, -
They shook the depths of the desert's gloom
With their hymns of lofty cheer.
EXPERIENCE, like a pale musician, holds
A dulcimer of patience in his hand,
Whence harmonies, we cannot understand,
Of God; will in his worlds, the strain unfolds
In sad-perplexed minors: deathly colds
Fall on us while we hear, and countermand
Our sanguine heart back from the fancyland
With nightingales in visionary wolds.


The Rape of the Lock, by Alexander Pope


'Nolueram, Belinda, tuos violare capillos;
 Sed juvat, hoc precibus me tribuisse tuis.'
If e'er one vision touch thy infant thought,
Of all the nurse and all the priest have taught;
Of airy elves by moonlight shadows seen,
The silver token, and the circled green,
Or virgins visited by angel-powers,
With golden crowns and wreaths of heavenly flowers;
Hear and believe! Unnumber'd treasures ope at once, and here
The various offerings of the world appear;
From each she nicely culls with curious toil,
And decks the goddess with the glittering spoil.
Then prostrate falls, and begs with ardent eyes
Soon to obtain, and long possess the prize:
The powers gave ear, and granted half his prayer,
The rest, the winds dispersed in empty air.
to your charge repair:
The fluttering fan be Zephyretta's care;
The drops to thee, Brillante, we consign;
And, Momentilla, let the watch be thine;
Do thou, Crispissa, tend her favourite lock;
Ariel himself shall be the guard of Shock.
Hither the heroes and the nymphs resort,
To taste awhile the pleasures of a court;
In various talk the instructive hours they pass'd,
Who gave the ball, or paid the visit last;
One speaks the glory of the British Queen,
And one describes a charming Indian screen;
A third interprets motions, looks, and eyes;
At every word a reputation dies.
At this, the blood the virgin's cheek forsook,
A livid paleness spreads o'er all her look;
She sees, and trembles at the approaching ill,
Just in the jaws of ruin, and Codille.
The Gnome rejoicing bears her gifts away,
Spreads his black wings, and slowly mounts to day.
No common weapons in their hands are found,
Like gods they fight, nor dread a mortal wound.


Poems by Henry Wadsworth Longfellow


Filled is Life's goblet to the brim;
And though my eyes with tears are dim,
I see its sparkling bubbles swim,
And chant a melancholy hymn
With solemn voice and slow lines
Listen my children and you shall hear
Of the midnight ride of Paul Revere,
On the eighteenth of April, in Seventy-five;
Hardly a man is now alive
Who remembers that famous day and year.
Birds of passage sailed through the leaden air, from the ice-bound,
Desolate northern bays to the shores of tropical islands,
Harvests were gathered in; and wild with the winds of September
Wrestled the trees of the forest, as Jacob of old with the angel.
Then, as the night descended, the herds returned from their pastures;
Sweet was the moist still air with the odor of milk from their udders;
Lowing they waited, and long, at the well-known bars of the farm-yard,--
Waited and looked in vain for the voice and the hand of the milkmaid.
From the red stone of the quarry
With his hand he broke a fragment,
Moulded it into a pipe-head,
Shaped and fashioned it with figures;
From the margin of the river
Took a long reed for a pipe-stem,
With its dark green leaves upon it;
Filled the pipe with bark of willow,
With the bark of the red willow;
Breathed upon the neighboring forest,
Made its great boughs chafe together,
Till in flame they burst and kindled;
And erect upon the mountains,
Gitche Manito, the mighty,
Smoked the calumet, the Peace-Pipe,
As a signal to the nations. The worthy pastor --
The shepherd of that wandering flock,
That has the ocean for its wold,
That has the vessel for its fold,
Leaping ever from rock to rock --
Spake, with accents mild and clear,
Words of warning, words of cheer,
But tedious to the bridegroom's ear.
I tell the mariner when to sail the seas;
I waft o'er all the land from far away
The breath and bloom of the Hesperides,
My birthplace. One mass of shade,
The elm-trees drop their curtains down;
By palace, park, and colonnade
I walk as in a foreign town.

Saturday, May 02, 2020

BERT for Short Short Stories

As a creative writer I'm always on the lookout for new developments in NLP and language modelling. With the advent of the new Age of Machine Learning there was a lot of promise that creative breakthroughs might be around the corner. There was an early burst with the work of people like Ross Goodwin's Sunspring in 2016, and Botnik Studios' Harry Potter and the Portrait of What Looked Like a Large Pile of Ash in 2017.

However this momentum appears to have stalled more recently, and the most interesting AI collaborations have been in the visual arts instead, highlighted by Obvious' auctioned Portrait of Edmond Belamy, but even more so by the avant-garde work of serious artists like Mario Klingemann.

With incredible language models like GPT-2 and XLNet now openly available, it is disappointing to note a comparative lack of collaboration between creative writing and these advances in AI. Is it perhaps a case of more not really being better, when it comes to language generation? Like that scene in The Matrix Reloaded where the CGI was amazing for its time, but not quite convincing enough to carry the story.

But predictive generation isn't the only NLP game going at the moment, and BERT is another model that has garnered a lot of interest. In short, its relative success in language understanding has made it suitable for various related tasks.

One such task is text summarisation. I recently discovered the Bert Extractive Summarizer, which makes this incredibly easy to do (there's an online version you can try out - although it has some limitations). I decided to play with a selection of famous short stories, and the results are quite fun - a bit like micro stories in their own right.

Here are 5 examples. Some of the originally longer stories required a smaller ratio than the default (0.2).

The Garden Party, by Katherine Mansfield


They could not have had a more perfect day for a garden-party if they had ordered it. "That's right, miss," said the tallest of the men, a lanky, freckled fellow, and he shifted his tool-bag, knocked back his straw hat and smiled down at her. " Laura's upbringing made her wonder for a moment whether it was quite respectful of a workman to talk to her of bangs slap in the eye. She crouched down as if to warm herself at that blaze of lilies; she felt they were in her fingers, on her lips, growing in her breast. Laura caught hold of her sister's sleeve and dragged her through the kitchen to the other side of the green baize door. If some one had died there normally - and I can't understand how they keep alive in those poky little holes - we should still be having our party, shouldn't we?" "I don't understand," said Laura, and she walked quickly out of the room into her own bedroom. What did garden-parties and baskets and lace frocks matter to him?

The Mask of the Red Death, by Edgar Allan Poe


Ratio : 0.2

The "Red Death" had long devastated the country. Blood was its Avatar and its seal--the redness and the horror of blood. The external world could take care of itself. The prince had provided all the appliances of pleasure. There were buffoons, there were improvisatori, there were ballet-dancers, there were musicians, there was Beauty, there was wine. There was a sharp turn at every twenty or thirty yards, and at each turn a novel effect. The second chamber was purple in its ornaments and tapestries, and here the panes were purple. The seventh apartment was closely shrouded in black velvet tapestries that hung all over the ceiling and down the walls, falling in heavy folds upon a carpet of the same material and hue. There was no light of any kind emanating from lamp or candle within the suite of chambers. But in the corridors that followed the suite, there stood, opposite to each window, a heavy tripod, bearing a brazier of fire, that projected its rays through the tinted glass and so glaringly illumined the room. But when the echoes had fully ceased, a light laughter at once pervaded the assembly; the musicians looked at each other and smiled as if at their own nervousness and folly, and made whispering vows, each to the other, that the next chiming of the clock should produce in them no similar emotion; and then, after the lapse of sixty minutes, (which embrace three thousand and six hundred seconds of the Time that flies,) there came yet another chiming of the clock, and then were the same disconcert and tremulousness and meditation as before. It was necessary to hear and see and touch him to be _sure_ that he was not. To and fro in the seven chambers there stalked, in fact, a multitude of dreams. The dreams are stiff-frozen as they stand. And the rumour of this new presence having spread itself whisperingly around, there arose at length from the whole company a buzz, or murmur, expressive of disapprobation and surprise--then, finally, of terror, of horror, and of disgust. The whole company, indeed, seemed now deeply to feel that in the costume and bearing of the stranger neither wit nor propriety existed. And the life of the ebony clock went out with that of the last of the gay.

Ratio : 0.1

The "Red Death" had long devastated the country. No pestilence had ever been so fatal, or so hideous. The external world could take care of itself. In the meantime it was folly to grieve, or to think. The second chamber was purple in its ornaments and tapestries, and here the panes were purple. There were arabesque figures with unsuited limbs and appointments. And the rumour of this new presence having spread itself whisperingly around, there arose at length from the whole company a buzz, or murmur, expressive of disapprobation and surprise--then, finally, of terror, of horror, and of disgust. There was a sharp cry--and the dagger dropped gleaming upon the sable carpet, upon which, instantly afterwards, fell prostrate in death the Prince Prospero. And Darkness and Decay and the Red Death held illimitable dominion over all.

The Darling, by Anton Chekhov


Olenka, the daughter of the retired collegiate assessor, Plemyanniakov, was sitting in her back porch, lost in thought. They want a clown; what they ask for is vulgarity. In the evenings and at night she could hear the band playing, and the crackling and banging of fireworks, and it seemed to her that it was Kukin struggling with his destiny, storming the entrenchments of his chief foe, the indifferent public; there was a sweet thrill at her heart, she had no desire to sleep, and when he returned home at day-break, she tapped softly at her bedroom window, and showing him only her face and one shoulder through the curtain, she gave him a friendly smile. "AWAITING IMMATE INSTRUCTIONS FUFUNERAL TUESDAY." "Vassitchka and I have no time to go to theatres," she would answer sedately.  Little by little the town grew in all directions. "I have resigned my post, and have come to settle down and try my luck on my own account. Besides, it's time for my boy to go to school."

The Haunted House, by Virginia Woolf


Whatever hour you woke there was a door shutting. And then, tired of reading, one might rise and see for oneself, the house all empty, the doors standing open, only the wood pigeons bubbling with content and the hum of the threshing machine sounding from the farm. The windowpanes reflected apples, reflected roses; all the leaves were green in the glass. If they moved in the drawing room, the apple only turned its yellow side. "Safe, safe, safe," the pulse of the house beat gladly. But the beam of the lamp falls straight from the window. Wild beams of moonlight cross both floor and wall, and, meeting, stain the faces bent; the faces pondering; the faces that search the sleepers and seek their hidden joy.

The Kiss, by Guy de Maupassant


My Little Darling: So you are crying from morning until night and from night until morning, because your husband leaves you; you do not know what to do and so you ask your old aunt for advice; you must consider her quite an expert. You say that you are all attention, love, kisses and caresses for him. Perhaps that is the very trouble; I think you kiss him too much. To tell the history of Love from the beginning of the world would be to tell the history of man himself: Everything springs from it, the arts, great events, customs, wars, the overthrow of empires. A preface which can always be read over again, whereas one cannot always read over the book. One caress alone gives this deep sensation of two beings welded into one --it is the kiss. Therefore, my dear, the kiss is our strongest weapon, but we must take care not to dull it. After describing the expectancy of a lover, waiting in a room one winter's evening, his anxiety, his nervous impatience, the terrible fear of not seeing her, he describes the arrival of the beloved woman, who at last enters hurriedly, out of breath, bringing with her part of the winter breeze, and he exclaims: Oh! The taste of the kisses first snatched through the veil. Therefore, the value of this caress being entirely a matter of convention, we must be careful not to abuse it. Well, my dear, I have several times noticed that you are very clumsy. You had been paying no attention to it, and it was almost out. Then when you freed him, you began to grumble: "How badly you kiss!"

On the whole the effect is interesting and often pleasing. The digests retain the language, which in the originals are unfailingly elegant, and often a discernable strain of their meaning too. A digest, prosaic as it may seem, is creative in its own way. Synthesis and understanding requires a path through the heart of a text. This tends to stand in opposition to novelty, but they can also form two parts of a larger storytelling process.

What if we combined them to come up with something new?