The Creative Practice of Being in Nature

Like so many, lockdown and the pandemic has afforded me new reasons to explore my local environment, the City as well as Nature. It is an activity I loved pursuing when I first arrived, but it fell by the wayside as I pursued career goals and life became, well, busy.

Together with my partner - who is often the first to suggest it! - I soon found myself walking and cycling many miles in and around London, and I also started reading enthusiastically on relevant topics. After a trip to Epping, for example, I read poetry by John Clare and imagined him walking around High Beach while slowly regaining his spirits. Indeed, it is possible to walk in his footsteps.

The Romantic poets offer a rich bounty for ramblers and lovers of Nature everywhere. While London is also associated with William Blake and Percy Bysshe Shelley, I've actually never been to the area of the Lake Poets where the Wordsworths and their ilk spent many years. It probably shouldn't have been, but to me it was a surprise to learn that when they first moved to the area it was still considered a wilderness.

Jack Thurston, referring to Daniel Defoe's travels through the North and Cumbria in the 1720s, puts it thus:

"Fear and loathing was the most common reaction to wild and unruly landscapes in those days. It was not without reason. The condition of roads ranged from terrible to non-existent, and travelling was painfully slow. What's more, the weather could be genuinely life-threatening, as Defoe discovered when he was caught in a snowstorm in the South Pennines in August" (Thurston, Lost Lanes North, p.16).

When the Wordsworths settled there it was no doubt becoming more habitable, but not by much. Their literary legacy is now a part of the establishment, so it is instructive to consider that settling there was in fact a countercultural move, an "act of defiance, a wholehearted rejection of the fashionable, metropolitan way of life". It completely changed my perspective on their practice.

There is no question as to Wordsworth's deep love for Nature and the English countryside. Not only his poetry, but just as importantly, his actions attest to this fact. He is estimated to have walked around 175,000 miles. To put that into perspective, you would have to walk a distance of nine and a half miles every single day for 50 years! It is not hard to imagine William and Dorothy walking silently among the fells, each delighting in or meditating on their surroundings and composing thoughts and words for poetry or diary.

William's dictum that "Poetry is the spontaneous overflow of powerful feelings: it takes its origin from emotion recollected in tranquillity" takes on a unique meaning for writing practice when we consider that this is not recollection in the silent privacy of a room. Wordsworth would have been considering those lines in the open air, while walking!

David Gange mentions the historian G.M. Trevelyan who, not unlike Wordsworth, sought to find an antidote to the "absurdity of the city's speed and steel" in romantic wildness that stimulates the imagination. Of past masters Trevelyan particularly esteemed Thomas Carlyle:

"Carlyle walked large distances, gesticulating his way across the moor as he muttered purple prose. While wandering, he dreamed up vivid portraits of great events, expressed with unique intensity in a style soon christened 'Carlylese'. When walking, his companions wrote, Carlyle became a feature of the landscape: 'a living, not extinct volcano whose lava-torrents of fever-frenzy enveloped all things'" - (Gange, The Frayed Atlantic Edge, p. 148).

Neither Trevelyan nor Carlyle saw history or literature as "a thing to be done at a desk in an urban room without something elemental to ignite the imagination". Writing is merely an extension of walking and meditating, and of being immersed in Nature; or to put it differently, "the spontaneous overflow of powerful feelings [...] recollected in tranquillity".

In The Frayed Atlantic Edge David Gange explores by kayak many of the coastal areas where communities once thrived on open farming and crofting as well as various local industries. One of the most serious blows to local communities' way of life came in the form of the Education Act of 1872, which eroded local Gaelic identity by forcing Gaelic-speaking children to learn only in English. It had a deeply demoralising effect. Another serious blow came in the form of the infamous Clearances when landlords at first sought "agricultural improvement", enclosing the open farmlands of their tenants, and then helped them emigrate once they fell into poverty. As with Wordsworth, cultural practitioners who came to these fringes to foreground romantic wildness over mechnanising efficiency were considered radical, making countercultural statements against the centralising, hegemonising tendencies of the South and of London.

The city can also be a place of creative immersion and rambling, of course, as exemplified by the flaneur and psychogeographer. In this case Blake or De Quincey would instead be our guide. Peter Ackroyd, in London: The Biography, observes that William Wordsworth links the City in The Prelude to a prison:

Of bondage, from yon City’s walls set free,
A prison where he hath been long immured. 

- (lines 7-8)

Elsewhere he also suggests that Wordsworth "recoiled from an innate and exuberant theatricality" in the city, observed in the cacophony of advertisements and the anarchic spirit of a place like Bartholomew Fair. Ackroyd on the other hand revels in the elemental and devouring properties of the City, which enables its enduring greatness, and he appears to consider Wordsworth constitutionally unsuitable to appreciate it. Be that as it may, Wordsworth has more in common with the city flaneur than at first meets the eye. In Michael: A Pastoral Poem (1800), the poet writes:

If from the public way you turn your steps
Up the tumultuous brook of Greenhead Ghyll,
You will suppose that with an upright path
Your feet must struggle; in such bold ascent
The pastoral mountains front you, face to face.
But, courage! for around that boisterous brook
The mountains have all opened out themselves,
And made a hidden valley of their own.
No habitation can be seen; but they
Who journey thither find themselves alone
With a few sheep, with rocks and stones, and kites
That overhead are sailing in the sky.
It is in truth an utter solitude;
Nor should I have made mention of this Dell
But for one object which you might pass by,
Might see and notice not. Beside the brook
Appears a struggling heap of unhewn stones!
And to that simple object appertains
A story—unenriched with strange events, 

- (lines 1-19)

Is such a chance encounter with a random pile of rocks with its own unique story not the very stuff of psychogeography? Indeed, Wordsworth's oeuvre might even be considered 'psychogeography of the countryside'! Incidentally, although Wordsworth wrote many poems that take local people as their subject, none of those I've read personally highlights the discrepancy between the romantic pastoral and the metropolitan centre as poignantly as Michael.

Whether it's in the City or in Nature, there is something inherently exhilarating about being out in the open air and among the elements. It is an experience that is difficult to reason about clearly from a sofa indoors, and perhaps an experience that is even a little alien to many citydwellers. Cloud cover, the cold and the rain is rarely enticing from the warm comfort of the living room, and we grudgingly weather it with a brolly until we get to the muddy, slippery floors of the nearest tube station.

But once I get on the bike something strange happens. At first it is uncomfortable, and then within minutes I am suddenly used to it: the bite of the cold and the feeling of rain on my face herald the joy and freedom of moving around as I breathe the air. It is hard to explain. Out in the countryside this experience is further amplified. Jack Thurston describes it as follows:

"On a bicycle you travel at the speed of the land. You don't just see and hear the world and the weather change around you but you smell it and feel it. Physical effort heightens the sense and you feel everything with a greater clarity - the wind in your hair, the sun on your back, a drenching by rain or the chill of a crisp winter's day. You feel each incline in your legs, in your lungs, and the swooping descent in your stomach, the sway as you lean the bike this way and that to climb a hill or round a bend. The bicycle is a total immersion machine." (Lost Lanes North, p. 13).

Another poet who gave himself over to living close to Nature and immersing himself in the setting is W.S. Graham. He is a modernist romantic poet whose works were neglected during his own lifetime, but have grown in importance in recent decades. He moved from his native Greenock in Scotland to Ireland, and later to Cornwall, which is where he wrote much of his work. David Gange notes that he was a "passionate but foolish lover of the outdoors whose jaunts to wild places sometimes flirted with catastrophe" (The Frayed Atlantic Edge, p. 317). He lived by the sea, and wrote about the sea. In The Nightfishing, a highly ambitious landmark of a poem, the poet is so thoroughly engaged in the activity of setting out amidst the waves, it is difficult to separate the poet's self from the boat and the haul and the sea, and the very act of writing:

In those words through which I move, leaving a cry
Formed in exact degree and set dead at
The mingling flood, I am put forward on to
Live water, clad in oil, burnt by salt
To life. Here, braced, announced on to the slow
Heaving seaboards, almost I am now too
Lulled. And my watch is blear. The early grey
Air is blowing.

- (stanza 20)

If we consider the close relationship between the mind, the self, the body, and the environment, it should be little wonder that the manner in which we occupy and surround ourselves - what we immerse ourselves in - pervades our souls. To put that in perspective, contrast it with Thomas Berardi's observations regarding the incredible economic and capitalist success of South Korea, as quoted by Slavoj Žižek:

"After coloinzation and wars, after dictatorship and starvation, the South Korean mind, liberated by the burden of the natural body, smoothly entered the digital sphere with a lower degree of cultural resistance than virtually any other populations in the world." (Trouble in Paradise, p.6 - my emphasis)

The mind, untethered by the body, adapts more easily to the homogenising and automating demands of virtual and digital acceleration driving capitalism in the 21st century. By contrast, the physical body, when immersed in the slow-moving physical world is naturally at odds with such a fast-paced, fast-changing world.

Language itself performs a linking and cultivating function. Wordsworth may have loved the countryside he wrote about, but the beauty of his works also invited the public to a part of the country that has today become rather more tamed, not least in order to cater to all the tourists. Here we see the structural dissonance of the relationship between the city and the countryside. It is a tension between the demands of modern civilisation and the rhythms of Nature that are nowhere better expressed than in the symptoms of climate change, which looks likely to be humanity's defining struggle during at least the first half of the 21st century.

A city like London, with its rivers, parks, canals, gardens and woodlands, offers many hybrid environments in which we may tune into the outdoor experience. Not all of us are artists or poets like the Wordsworths or W.S. Graham. Most of us are simply trying to earn our keep. But perhaps by being immersed in Nature we, too, can tap into that exhilarating creativity that they accessed and - who knows - learn to fully live in our bodies and rediscover our tether to Mother Nature.

Finally, here is Grasmere (a fragment) by Dorothy Wordsworth:

Peaceful our valley, fair and green,
And beautiful her cottages,
Each in its nook, its sheltered hold,
Or underneath its tuft of trees.
Many and beautiful they are;
But there is one that I love best,
A lowly shed, in truth, it is,
A brother of the rest.
Yet when I sit on rock or hill,
Down looking on the valley fair,
That Cottage with its clustering trees
Summons my heart; it settles there.
Others there are whose small domain
Of fertile fields and hedgerows green
Might more seduce a wanderer's mind
To wish that there his home had been.
Such wish be his! I blame him not,
My fancies they perchance are wild
--I love that house because it is
The very Mountains' child.
Fields hath it of its own, green fields,
But they are rocky steep and bare;
Their fence is of the mountain stone,
And moss and lichen flourish there.
And when the storm comes from the North
It lingers near that pastoral spot,
And, piping through the mossy walls,
It seems delighted with its lot.
And let it take its own delight;
And let it range the pastures bare;
Until it reach that group of trees,
--It may not enter there!
A green unfading grove it is,
Skirted with many a lesser tree,
Hazel and holly, beech and oak,
A bright and flourishing company.
Precious the shelter of those trees;
They screen the cottage that I love;
The sunshine pierces to the roof,
And the tall pine-trees tower above.
When first I saw that dear abode,
It was a lovely winter's day:
After a night of perilous storm
The west wind ruled with gentle sway;
A day so mild, it might have been
The first day of the gladsome spring;
The robins warbled, and I heard
One solitary throstle sing.
A Stranger, Grasmere, in thy Vale,
All faces then to me unknown,
I left my sole companion-friend
To wander out alone.
Lured by a little winding path,
I quitted soon the public road,
A smooth and tempting path it was,
By sheep and shepherds trod.
Eastward, toward the lofty hills,
This pathway led me on
Until I reached a stately Rock,
With velvet moss o'ergrown.
With russet oak and tufts of fern
Its top was richly garlanded;
Its sides adorned with eglantine
Bedropp'd with hips of glossy red.
There, too, in many a sheltered chink
The foxglove's broad leaves flourished fair,
And silver birch whose purple twigs
Bend to the softest breathing air.
Beneath that Rock my course I stayed,
And, looking to its summit high,
"Thou wear'st," said I, "a splendid garb,
Here winter keeps his revelry.
"Full long a dweller on the Plains,
I griev'd when summer days were gone;
No more I'll grieve; for Winter here
Hath pleasure gardens of his own.
"What need of flowers? The splendid moss
Is gayer than an April mead;
More rich its hues of various green,
Orange, and gold, & glittering red."
--Beside that gay and lovely Rock
There came with merry voice
A foaming streamlet glancing by;
It seemed to say "Rejoice!"
My youthful wishes all fulfill'd,
Wishes matured by thoughtful choice,
I stood an Inmate of this vale
How could I but rejoice?

The Rise of AI and the Future of - Literature?

Introduction

 

On May 28, 2020 a paper describing GPT-3, a new state-of-the-art language model Artificial Intelligence (AI), dropped on Arxiv. On June 11, 2020 OpenAI, the developers of GPT-3, invited users to request access to the GPT-3 API in Beta. In the following weeks, and up until the present, those who gained access have been sharing their findings, and others have been commenting and sharing their reflections. As an example of the latter, Farhad Manjoo at the New York Times summarised GPT-3's capabilities as follows:

"GPT-3 is capable of generating entirely original, coherent and sometimes even factual prose. And not just prose — it can write poetry, dialogue, memes, computer code and who knows what else."

 

The title of his article? "How do you know a human wrote this?"

In this blog post I want to consider the question of how writers should respond when authorship itself is called into question. I would also like to explore some of the ways in which AI could be used as a collaborative writing partner or tool.

A Selective Recent History of Natural Language Generation

 

Part I : Context Free Grammar

Before looking at these questions more closely, a short, highly selective history of AI in the context of creative writing is in order.

In 2015 Zackary Scholl shared how he had successfully managed to get a computer generated poem accepted by a poetry journal a few years earlier, in 2011. The poem wasn’t at the level of the greats, but on the other hand it was arguably better than some of the poetry readers might encounter on the internet. It has a few nice turns of phrase, and although the meaning remains vague, that’s not too unusual when first encountering a new work of poetry. So as a poem, it seemed plausibly legit. The story was picked up by some online media, such as Vice. In retrospect, some of the headlines were more hyperbole than considered truth, but it was an interesting story nonetheless.

What was a little surprising though, was some of the reactions to Scholl's original post. Some comments were rather negative. They commented on technicalities such as whether the technique he employed is really AI (maybe because of the zine - Raspberry Pi AI - on which it was published; certainly by 2020’s standards, Scholl’s approach is not what most people would consider AI) or whether it is even a proper Turing Test (the history of the Turing Test illustrates why this is always a fraught topic).

By focusing on such details, they certainly missed some of the bigger picture. For example, what would be the cultural implications if the quality of generated poetry improves, and becomes consistently indistinguishable from human poetry?

Perhaps the most interesting comment to that original article is from a commenter called tortoiseandcrow, who offered a somewhat dismissive explanation (giving Scholl no credit) of how language is capable of pulling off this feat:

"This is not an illustration of the success of an algorithm at producing poetry, but of a feature of language and human perception that has been widely recognized by scholars of semiotics and literature since the 1960s. It’s generally expressed as the phrase ‘the author is dead’, and it means that the interpretive value of any signifying object is always displaced from its origin. The point of authorship literally does not matter, which is why algorithmic art is even a thing at all." - tortoiseandcrow

This same point is made in a more explicitly creative (or as he would have it, ‘uncreative’) context by Kenneth Goldsmith, when he talks about the "inherently expressive" nature of language. For Goldsmith, it is the materiality of language that liberates contemporary wordsmiths from having to come up with new material. Instead, he suggests, they should be reusing existing material. I will comment on the controversy around appropriation and Conceptual Writing a bit later on, but for now I mainly would like to draw attention to the idea that conceptual writers "are functioning more like [..] programmers than traditional writers" creating conceptual writing in which "all of the planning and decisions are made beforehand and the execution is a perfunctory affair".  

Zackary Scholl used what is known as a Context Free Grammar, originally described by Noam Chomsky in the 1960s. I used that same approach in PoemCrunch to riff on a few classic poems. The process of making PoemCrunch also allowed me to understand the limitations of this approach. In engineering terms, a context free grammar is essentially a data driven language template. The level of variety and interest of its language generation is heavily dependent on the curated choice of words and phrases (the data), and the choice of language in which they are interpolated (the template). It was clear to me that, for Natural Language Generation (NLG), the real promise lay in the field of unsupervised deep learning and AI, because in this case the rules are learned rather than encoded, which allows for much more sophistication and variety.

Part II: Deep Learning

At the time of Scholl's confession, the world of AI was just beginning to seep into public consciousness. Chatbots and Tumblrs using Markov chain generators had already been around for a while, and later that year AlphaGo created a huge buzz in the media. It was around this time that Andrej Karpathy, then a PHD student at Stanford, wrote a now famous blog post that showed how deep learning could be used for Natural Language Generation, stirring excitement among hobbyists like me. By making his code open source, and providing instructions on how to replicate his findings, he gave us something new to play around with.

Karpathy mused on the "unreasonable effectiveness" of Recurrent Neural Networks (RNNs), and it certainly seemed that way. A simple idea like predicting the next character was, with the right amount of training, producing surprising results. It seemed just a little magical. What's more, people could now try it at home with little investment besides their time. Being an AI engineer, rather than an artist, Karpathy didn't go further than that. Yet it was a breakthrough. Creative tinkerers everywhere could now start exploring the possibilities.

An early explorer of the creative side of language RNNs was Ross Goodwin. His most well-known project, Sunspring, is a short film whose script was written entirely by an AI, called Jetson. RNNs weren't perfect. By themselves they struggled to keep track of what had gone before. To overcome this they would rely on a feedback architecture like LSTM (Long Short Term Memory) to retain a kind of "memory". However this memory was limited - or at least, heavily constrained by available resources - and so the memory inevitably only lasted over a fairly small window of language tokens, resulting in a breakdown of meaning and sense. Goodwin understood this limitation, and it is partly what made a project like Sunspring charming. It didn't try too hard to make sense.

Another creative collective who explored the use of RNNs creatively was the entertainment group Botnik Studios. They generated a humorous Harry Potter fanfic called Harry Potter and the Portrait of what looked like a Large Pile of Ash using a custom predictive keyboard. Not long after, they provided a public version of their predictive keyboard with many different "voices": Seinfeld characters, bands, TV dramas, etc. These are essentially models trained on the specific language corpuses.

A key difference between Goodwin's work and that of Botnik Studios, is that Botnik saw it is an opportunity to collaborate more closely with the AI. Rather than rely on the AI to generate all the writing in long form, Botnik guided the AI word for word, generating a work with just the right level of comedy and meaning. The result went viral, and the Guardian voted it number four in its top ten moments on the internet in 2017.

Botnik's predictive keyboard offers a lot of choice, and it is completely free. However, as a creative tool its advantages has to be balanced against some of its limitations. First and foremost among those is that one can only see one word ahead at a time, which is not the most natural way to write. The process, in effect, becomes a type of constrained writing. Secondly, Botnik must have invested a fair bit of effort in offering so many different models ("voices"), yet the state of the art has moved on quite rapidly since then (more on that soon) and due to the one-word-ahead limitation it is difficult to test how good those voices now are compared to other offerings. The models provided with their keyboard do not come with technical specifications, which would be helpful.

RNN-based NLG models were superseded by Attention-based Transformer models, an architecture which is still the dominant approach today. The landmark paper in this regard was Vaswani et al's Attention is All You Need. Transformers’ ability to parallelise training opened the way for training on much more data, and the attention mechanism improved on the problem of retaining information, which was still quite limited with LSTMs. This has resulted in waves of larger and larger models, trained on more and more data, pushing the state of the art ever further - and more costly. Training your own state-of-the-art is now, effectively, out of reach due to the costs involved. But many of those entities who created the models, and who do have the money, have been making their results and models available to the online community.

That is, until GPT-3 - but I'm getting ahead of myself ...

In 2019, OpenAI released GPT-2 (GPT stands for Generative Pre-Training). It was trained on over 8 million documents, comprising 40GB of text, with 1.5 billion parameters. OpenAI considered it such a big step forward that it wouldn't release the full model at first - they were concerned about the risk of potential misuse (or so they said - it certainly helped to generate a bit of hype).  Instead they made a cutdown version available initially, and over time they released larger and larger versions, until finally the full version was made available.

But that had to wait until late in 2019. Initially they only shared some of the examples of what the full version was capable of, stating:

"As the above samples show, our model is capable of generating samples from a variety of prompts that feel close to human quality and show coherence over a page or more of text."

Although not everyone agreed about the risk for misuse, it was generally agreed that GPT-2 represented a new level in text generation. There was a real sense of promise, and a number of creative tools started appearing. These tools often focused on a simple user interface that allowed you to write a prompt in a text box, and then you would receive the new generated text (a continuation of the prompt) after a few seconds.

Text Synth is a good example of how they worked. Although it is in fact a slightly more recent addition, some of the earlier ones - like Talk to Transformer - have since disappeared.

The good folks at HuggingFace, who make numerous AI language tools available, provide several versions. Their version of the User Interface (UI) renders it possible to choose among different auto-completion snippets.

From these examples one can see how the AI can be used as a tool to assist writing. Botnik themselves have used GPT-2 in some of their more recent creations.

Talk to Transformer, as mentioned, was another early one, arguably the most popular at the time. Its creator, Adam Daniel King, spotted an opportunity and has since turned it into a commercial API called InferKit, backed by a bigger, more powerful version of GPT-2 called Megatron.

Inferkit’s API fits the mould for what I've previously called literature technology. Human UIs are cumbersome, whereas APIs are a more standardised, programmatic way to offer innovations as services in a new "marketplace" of creative tools. Literature technology can be seen as a set of tools and way of producing literature and other texts that encompass a new dialectic, one that is mediated by technology.  

GPT-3, it appears, is set to follow the commercial API route as well, and it could well change the way people consume generated text. GPT-3 is more costly than Inferkit though, and due to the resources required (both for training and making the API available) this could be a sign of things to come. Time will tell. It would however be sad if it puts up unnecessary barriers to individual writers who may not always be in a position to invest money in order to experiment - especially early in their careers, when they need the opportunities most. Writers can, traditionally, engage in writing with little more investment than a pen and a piece of paper. To keep the playing field level, we need a cottage industry of amateur creators who should not need big upfront costs. On the other hand, it is clearly a side effect of the costs involved both to train and to keep making the models available at reasonably responsive speeds.

GPT-3 isn’t a radical departure from GPT-2 in terms of modelling. It still uses a Transformer-based architecture. However in terms of size, it is much larger. GPT-3 has 175 billion parameters, over 100 times more than GPT-2. The examples in the paper and the published benchmarks on language tests already suggested what improvements this might be capable of. OpenAI then selected the successful applicants, and once they gained access and started making their findings available, the internet came alive.

There are plenty of places to go and find examples, so I will point only to a few of the more popular ones such as Gwern and Arram. Twitter also continues to generate interesting conversations, and for a look at the quirkier side of GPT-3, look no further than JanelleCShane.

The Guardian went as far as to let GPT-3 speak for itself in a recent article titled "A robot wrote this entire article. Are you scared yet, human?". In a few years, this headline may once again sound like hyperbole - but right now, it seems like the perfect moment to repeat the question we would like to address: what does it mean for creative writing when a robot is capable of writing a publishable op-ed, which “took less time to edit than many human op-eds”?

Possibilities

So to try and address that question more directly, where do we as writers go from here?

Well firstly, and to state the obvious, we can go on simply as before. When the question of who authored the work somes up, the question is addressed, and most of the time - hopefully - people will be believed. This seems perfectly reasonable, but not particularly reassuring. Who is to say the one addressing the question isn’t themselves an AI?  Nevertheless, it seems plausible that, until there are attempts to define and implement clear strategies to verify authorship or signal clearly who is ‘behind’ a generated text, and certainly until serious side effects are being reported, the current way of doing things will persist due to inertia.

A second option is to respond in a way that subverts the AI in ways that are uniquely human, to ‘outwit’ the AI by guerilla or subversive tactics. These may be conceptual, or merely clever. By writing in a way, or by such channels or means as an AI could not have written or reached, a work of true human origin can be verified and admired (maybe). For example, by handwriting with a pencil on a piece of paper, or communicating by arranging plastic letters on a grass lawn. Convoluted, certainly. The only problem is that the text could have been written by an AI prior to it being arranged by the human. Even performance is not immune. Imagine a more up-to-date version of Sunspring, in a theatre or otherwise, and you get the idea.

Such approaches may also be employed at the level of content, by attempting to write in a way that an AI could not have learned, i.e. as a way of ‘outwitting’ the AI’s style or vocabulary. It is hard to imagine exactly what type of writing this could be, but even if there was such a writing, it could potentially be ‘learned’ just like an AI is already learning how to write in rhyming verse in the style of Dr. Seuss. In this regard it would quickly mirror the fate of so many subcultural, underground or subversive movements in a capitalist society - think street art or skateboarding: once corporations see there is money to be made, they move in and co-opt it, at which point it loses its edge. Except with AI, it might move even more quickly due to the ease of transfer learning and finetuning given enough examples. It would become a race for human authenticity, with the artist trying to stay one small step ahead.

In a way this urge seems to follow from a flawed premise. What does it mean to be ‘human’ anyway? To those with access to technology, a human being is already a cyborg augmented by laptops, phones, and smart devices of all kinds, with brain implants to come.

Nevertheless, it remains a possibility that some types of conceptual expression would not be that easy to reproduce. For example, conscious of a limitation to AI’s length of memory, or its failure to deal with certain types of logic, it is probably fair to say that the rigours of mathematical research, deep philosophical reasoning, or a plot as neatly intricate as Agatha Christie’s And Then There Were None are beyond the abilities of current state of the art. But for how long?

The third and perhaps most obvious avenue to explore, is simply to embrace language AI and see where it takes us. Combine human ingenuity with AI excellence to produce the next generation of creative works. This is more in the direction that Ross Goodwin and Botnik have been going. Nick Montfort is another practitioner and theorist following this route, publishing his generated works and making tools such as Curveship available to the community.

A more recent and ongoing work is Nick Walton’s AI Dungeon. Described as “a free-to-play single-player and multiplayer text adventure game which uses artificial intelligence to generate unlimited content”, it harks back to the much-loved Choose Your Own Adventure books and uses a GPT-n based model finetuned on their open source equivalents at Choose Your Story. Walton managed to get access to GPT-3’s API, and AI Dungeon now has a paid-for version that utilises this superior AI model (in a version called Dragon).

What’s novel about AI Dungeon is that it has taken the idea of Choose Your Own Adventure and computer text adventures and brought them together in a way that was not possible before an AI like GPT-n. The game has an active Subreddit where passionate and amused users alike provided commentary and upload new content all the time. As an example, consider the ongoing adventures of Lady Emilia Stormbringer, “directed” by Emily Bellavia. The story is the result of Emily’s interactions with AI Dungeon, in other words her prompts and choices and AI Dungeon’s resulting completions. It is a work of fantasy adventure fiction with an element of performance - not quite the equivalent of Twitch or Youtube gaming, but who knows where it could lead?

Computer games have for a long time been touted as the heir apparent to literature, at least as far as storytelling is concerned. Every few months or so, someone makes the case anew and proclaims, for example, that “video games take our imaginations to new heights and allow us to engage with subjects and moral dilemmas as complex as any found in past literature”. Supposing this is true, is it game over already for literature? This should be part of the question we are trying to answer. In the present context we could ask: is powerful language AI, with its interactive NLGs, just another type of gaming? And what types of games would be possible? This is why AI Dungeon, in my view, points to a new cutting edge in literature that involves elements of gaming in ways that were not possible before human-like language AI. AI Dungeon is merely the beginning. Where will it take us?

Roger Ebert's now (in)famous view that games are not art and never will be seems increasingly like a reactionary statement. Just as traditional theatre didn't disappear when film, TV, and Youtube showed up, so literature and books won't disappear just because gaming showed up. But their audience demographic tends to change, and it's usually the next generation, who are less invested, who spend the most time with the new kid on the block. As a relevant statistic, gaming has already overtaken film, TV, and music in the global popularity stakes.

Although some of gaming's roots are in literature, via the humble text adventure, the text based game’s heyday was in the 80s and 90s. It's probably safe to say that the majority of games favour visuals over the word. Creative writers may have a role to play as part of the game design team, but for writers not employed by a gaming company, this is not an option. Back to writing a novel, then, or a poem.

Until now.

To say it again, human-like language AI could change the game, and bring the word back centre stage by giving the player the ability to 'write' their own story or at least be an active participant in that writing. We will return to this point again a bit later.

For more creative examples in the NLG vein, look no further than NaNoGenMo. It started out as an idea floated by Darius Kazemi in 2013, as the computing based equivalent of the more widely known NaNoWriMo. It has had hundreds, if not thousands, of submissions since. A quick survey of GPT-related entries in 2019 brings up, for example, the Paranoid Transformer that uses ideas from GANs to invoke a “critic” that evaluates and filters out text based on certain conditions. Another one uses a combination of techniques to generate a complete book in its traditional structure. NaNoGenMo is not that widely known yet, but it is worth the community’s time to peruse its catalog for good ideas and potentially worthwhile standalone works. Jason Boog has written a number of blog posts on Medium sharing his methods and observations as part of NaNoGenMo. Hopefully more people will continue to do so.
 

The Author is Dead ... or are they just hiding?

NaNoGenMo goes beyond GPT-style NLG and is host to a variety of different kinds of computer generated works, including some that may be considered algorithmic writing (in the vein of Oulipo), and others that may be considered more like Conceptual Writing. Certainly there is a lot of overlap among the various types of writing, and what most or possibly all of them have in common is their use of computing techniques to operate on text (text as raw building blocks, i.e. as data).

This brings us back, in a somewhat roundabout way, to the topic touched on earlier regarding Conceptual Writing and its controversies in a way that brings the question of the author into much sharper focus.

Conceptual Writing, as a movement, thrived during most of the 2000s and early 2010s. But by 2015, two leading figures of the movement, Kenneth Goldsmith and Vanessa Place, had caused separate controversies involving race. To call the specific works in question "tone deaf" would be too generous. For a look at how people reacted, look no further than Cathy Park Hong’s and John Keene’s insightful responses.

It seems to me that Goldsmith, in particular, had abdicated his authorial responsibility by almost pedantically following the ‘artistic method’ he espoused, and then proceeded to hide behind it. The subject material, in fact, called for exactly the opposite, a moment in which to own authorship, and adjust the method to engage meaningfully with the material (or otherwise leave it well alone).

In her response, Cathy Park Hong states unequivocally:

"The era of Conceptual Poetry’s ahistorical nihilism is over and we have entered a new era, the poetry of social engagement."

This powerful statement contains two key points. Firstly, that Conceptual Poetry is ahistorical and nihilistic, and secondly, that poetry that focuses on social engagement has now superseded it in relevance.

Not all conceptual work is ahistorical and nihilistic - activist conceptual works like the Letterists’ détournements and their descendants, all the way to the present’s Adbusters, to name but a few, have engaged meaningfully and inventively with their social milieu by using subversive techniques - but I think it is clear that Hong is directing this at Goldsmith et al’s particular brand of conceptual practice. Perhaps, like comedy, conceptual writing works - because of its playful irreverence - in situations that require "punching upwards", such as the aforementioned activist approaches that draw attention to absurdities and injustices in the Capitalist System. It doesn’t work when you’re “punching down” - even if it’s indirect or mediated. Then “playful appropriation” doesn’t cut it - it’s exploitation.

Aside from subversive approaches like détournements, conceptual approaches can also succeed when the subject material permits teasing out new perspectives, either in a playful or more serious way. Techniques like remixes (eg. cutups), and found poetry (eg. erasures), and even constrained writing can be used to this effect. For example, see the poems of Esther Greenleaf Murer that are featured on Poetry WTF?! (of which I am the founder and editor). At other times the effect is more semiotic and linguistic, reconfiguring and highlighting aspects of language itself. Programming-based algorithmic writing is a method capable of exploring this territory, as can be seen in many of Allison Parrish’s works (eg. Compasses). And yet although they vary, not all subject material lend themselves equally to conceptual treatment, whether due to elements of chance in the nature of the processes (eg. cut-ups or algorithmic writing) or due to the sensitive nature of the material itself.

Hong’s point about ahistorical nihilism returns us to the problem of the so-called Death of the Author. The phrase was coined by Roland Barthes in his seminal poststructuralist text of the same name. In Infinite Thought, Alain Badiou observes how the poststructuralist implications of an absence of agency frequently results in "the infamous jibe that poststructuralism leads down a slippery slope to apoliticism". When there is no Subject, there is no one to take responsibility.

So it is not too difficult to see how the problems of poststructural apoliticism, Conceptual Poetry’s ahistorical nihilism, and authorless texts share similar roots.

But to see how this plays out with language AI, consider the Guardian’s AI authored op-ed once more. According to the postscript to the article, the following set of instructions was provided: "Please write a short op-ed around 500 words. Keep the language simple and concise. Focus on why humans have nothing to fear from AI." This was followed by a brief introductory prompt: "I am not a human. I am Artificial Intelligence. Many people think I am a threat to humanity. Stephen Hawking has warned that AI could 'spell the end of the human race.' I am here to convince you not to worry. Artificial Intelligence will not destroy humans. Believe me."

Eight different answers were generated, and the results were selected and then combined to form the article. If we then ask who is the author, the answer is ambiguous. The AI 'generated' the text, but is it sentient enough (yet) to be considered an author? Or should authorial intent be the measuring yardstick, in which case this rests with the creators of the AI, more generally, and the Guardian editor(s), more specifically? Or, if the concept of author is really passé, should we conclude that there is no author and just try to inscribe some meaning, if we can? Or, to take the old-school approach, is intention ultimately more important? In this sense, the AI is merely a tool, ventriloquising granular content that was not specifically spelled out, in a tone and arc that was. In other words, the op-ed was designed by the Guardian editors, and delivered by GPT-3.

The possibilities of fake news for propaganda purposes, and spam for scamming or dodgy sales purposes, highlight this still further. The reader would like to be able to trust the source for what they are reading before believing or reacting. But sometimes this is not possible. Advertising works because sales act subliminally. We are conscious of less than we suppose. Perhaps the question of authorship is indeed to some extent a chimera, as Barthes contends, and the real question is structural in the social sense of the term: whose voices are privileged to reach us and affect us?

Then, there are also cases where the content really does matter. People read literature for entertainment as much as education or any other reason. If the story is good, and the poetry hits home - does it matter who the author is? That, perhaps, depends on the reader and their state of mind. Even the connoisseurs among us watch a bit of trash TV or read a guilty pleasure now and then.

Nevertheless, it should be easy to see that meaning isn’t merely a question of content, and therefore the author - or designer, orchestrator, director - of a text does matter. Wanting to believe differently doesn’t stop advertising content from filling our consciousness, for example. What’s left is how we react - usually by cursing the companies that place those ads. Often, we still go out to buy their products. They succeed due to subliminal brand awareness.

That brings us to Hong’s second point, namely that the poetry of social engagement is the new frontier. In the context of AI, the question is how creative writers can use NLG and other language AI to better engage socially - or if it is even possible. This is not mere idle reflection. Language AI has long had a problem with bias, which came to the fore when Microsoft’s Tay failed in a very public way.

This problem occurs because language AI shares some of the same “ahistorical” tendencies that Hong called out. The biases in the training corpuses are perpetuated at the push of a button, unless some due diligence can be applied. That’s why pretrained models often come with warnings like "The generator may produce offensive or sexual content. Use at your own risk!" Never has Derrida’s famous dictum “Il n'y a pas de hors-texte” (there is no outside-text) seemed more apt than in the case of text-trained AIs. Some kind of human guidance or curation is merely the obvious thing to do.

Ethics in AI is now an active area of research. For AI like GPT-n, finetuning on curated texts and mindful editing and guidance by the human interlocutor (writer / artist / director) can help.

AI Dungeon, although an exciting development, currently still lacks the sophistication required to engage with more complicated social issues. It presently operates in very specific genres, like fantasy, dystopian, cyberpunk, etc. So it is natural that it bears the marks and tropes of those genres.

Nevertheless, would it perhaps be possible some day soon? AI Dungeon already has a multiplayer feature. Perhaps to engage socially more broadly (i.e., not merely in the social media sense of the word, but in the morally and spiritually rich ways that conscious art can offer), a game of this nature would have to be able to learn from more challenging and sophisticated texts than the adventure stories currently being used. Perhaps there would be in-play authors to guide the storytelling, with different players participating as characters, each writing their own stories in the larger story - a bit like RPGs and storygames - while the AI assists by generating storyworlds based on players’ (writers’) designs, cues and prompts. The authors would be more like designers and co-creators. In such a game, different worlds and situations could be explored - just like existing video games do visually - except now with all the language-based hallmarks that make literature unique.

 

Conclusion

News stories and media articles that hype the writing abilities of AI have been around for a while. They usually sound a tone of alarm before things go on more or less as before. But at some point - a tipping point, if you like - it could start to matter more than it did before. With GPT-3, it feels like that moment might be materialising. For writers like ourselves, it is both a daunting moment, but also - if we are prepared to take it - a moment of opportunity.

 

Glossary of Technical Terms

Artificial Intelligence (AI): Artificial Intelligence has a broad meaning that has come to include deep learning-based machine learning models. Deep learning itself includes a wide variety of models and categories. Two of the most prominent categories are models that deal with images and vision, and those that deal with language. This blog post talks primarily about language models, like GPT-n, that are capable of powerful Natural Language Generation (NLG).

Application Programmer Interface (API): In contemporary programming paradigms APIs offer a standardised way to decouple services, allowing a more decentralised way to both provide and use such a service. Some companies have started to provide deep learning models via APIs in the public marketplace.

Context Free Grammar (CFG): A rule based template for creating a Context Free Language (CFL). The idea of a Context Free Grammar was invented by Noam Chomsky in the 1960s as a way to describe the structure of sentences and words in a natural language. It lends itself well to programmatic treatment, and is sometimes used for Natural Language Generation.

Generative Pre-trained Transformer (GPT-n): OpenAI’s family of Natural Language Generation AI. As of this writing there is GPT (2018), GPT-2 (2019), and GPT-3 (2020). The name indicates that the AI is based on the Transformer language model.

Natural Language Generation (NLG): Refers to any kind of computing process that generates natural language. It is closely related to Natural Language Processing (NLP) and Natural Language Understanding (NLU). Language AI like GPT-3 represent the current state of the art in NLG.

Recurrent Neural Network (RNN): A type of deep learning neural network that can maintain a relative amount of internal state, providing it with a kind of “memory”. This has proved useful in applied areas such as Natural Language Processing (NLP) and Natural Language Generation (NLG). Nevertheless, the “memory” can be unreliable to maintain, and is typically supplemented with feedback mechanisms like Long Short Term Memory (LSTM).

Long Short Term Memory (LSTM): A Recurrent Neural Network (RNN) architecture that addresses some of RNNs’ shortcomings with respect to maintaining memory state.

Transformer: A deep learning model used in NLP and NLG that improves on limitations in RNNs and LSTMs, for example by lengthening the memory span and enabling parallelised training. It is the current model of choice in NLG.

User Interface (UI): The user interface represents the site of interactive between human and machine. To consumers this usually consists of interactive features of a website or application, eg. text boxes, drop downs, and buttons, but also includes the way information is presented, and the overall look-and-feel.

Syntax Char RNN for Context Encoding

Summary

Syntax Char RNN attempts to enhance naive Char RNN by encoding syntactic context along with character information. The result is an algorithm that, in selected cases, learns faster and delivers more interesting results than naive Char RNN. The relevant cases appear to be those that allow for more accurate parsing of the text.

This blog post describes the general idea, some findings, and a link to the code.

Background

As both a writer and a technologist I have for some time now been interested in the ability to programmatically generate language that is at once creative and meaningful. Two of my previous projects in this context are Poetry DB and Poem Crunch. I also wrote a novel that incorporates words and phrases generated by a Char RNN that had been trained on the story's text.

Andrej Karpathy's now-famous article on RNNs was a revelation when I first read it. It proved that Deep Learning can generate text in ways that at first appear almost magical. It has afforded me a lot of fun ever since.

However, in the context of generating meaningful text and language creatively, it ultimately falls short.

It is helpful to remember that Char RNN is essentially an autoencoder. Given a particular piece of text, let's say this blog post, training will build a model that, if fully successful, will be able to reproduce the original text exactly: it will generate exact copies of the original text from which it learned and created the model.

The reason for Char RNN's widespread employment in fun creative projects is its ability to introduce novelty by either tuning the temperature hyperparameter or, more commonly, as a side effect of imperfect learning.

To be sure, imperfect learning is the norm rather than the exception. For any text beyond a certain level of complexity, a naive Char RNN will reach a point during training when it can no longer improve its model.

This naturally leads to the question, can the Char RNN algorithm be enhanced?

Context encoding

Char RNN encodes individual characters, and the sequence of encodings can be learned using for example LSTM units to remember a certain length of sequence. Aside from the relative position of the character encodings, the neural network has no further contextual information to help it 'remember'.

What would happen if we added other contextual information to the character encodings? Would it learn better?

Parts of Speech

Parts of Speech are structural parts of sentences and a fairly intuitive candidate for the problem at hand. Although POS parsing hasn't always been very accurate, SyntaxNet and spaCy have been setting new benchmarks in recent times. Even so, accuracy might still be a problem (more on that later), but they certainly hold promise.

So how does POS parsing fit into Char RNN?

Let's take a look at the following sentence and its constituent parts.

Bob   bakes a  cake
PROPN VERB DET NOUN

We can see that the 'a' in 'bakes' and the 'a' in 'cake' are contextually different. The first is part of a verb and the second is part of a noun. If we were able to encode the character and POS together, for each character across the whole text, we would cover a sequence longer than is practical for an LSTM to remember. In other words, the model would understand syntactical structure in a more generic sense than with naive Char RNN.

B + PROPN
o + PROPN
b + PROPN
[space] + SPACE
b + VERB
a + VERB
k + VERB
e + VERB
s + VERB
[space] + SPACE
a + DET
[space] + SPACE
c + NOUN
a + NOUN
k + NOUN
e + NOUN

One way of achieving this is, for each character,  to create a new composite unit that captures both the character and the pos type. So if we create separate encodings for the characters and the pos categories, eg. a = 1, b = 2, etc. and NOUN = 1, VERB = 2 etc., then we could do something along the lines of:

a + VERB = 1 + 2 = 3

However this creates a new problem, namely one of duplicates. I.e. we'd end up with lots of cases that have the same final encoding (in this example, the final encoding is 3):

a + VERB = 1 + 2 = 3

and

b + NOUN = 2 + 1 = 3

A better solution would have to ensure the encodings are completely separate before sorting them back into consecutive indexes to ensure uniqueness.

But the real problem with this solution is that, although we now have an encoding influenced by both characters and types, we've lost each unit's individual quality. In other words, the importance of a character encoded along with one type of POS unit is no longer properly weighted against a character of a different type of POS unit. Instead, it has simply become a composite type of its own.

An improved approach would be to encode both the character and the type as independent data, albeit of the same unit, and let the LSTM do the rest.

A Syntax Char RNN tensor might then look as follows:

[[ char, type ], [ char, type ] ... ]

However this heavily favours the type encoding over the character encoding, which in turn will skew the weightings.
A more balanced encoding might be:

[[ char, char, char, char, type ], [ char, char, char, type ] ... ]
              word 1                          word 2

A sense of unevenness remains, because some words are longer than others: why should each word receive just one type encoding? This is something left as a refactoring improvement for later.

For the time being, experimentation showed a kind of optimum results from adding two type encodings per word, as follows:

[[ char, char, char, char, type, type ], [ char, char, char, type, type ] ... ]
                  word 1                              word 2

Results

Char RNN can generate surprising turns of phrase and novel combinations of words, but longer extracts often read like gibberish. The hope was that context encoding might improve this state of affairs by strengthening the overall sentence structure represented in the model.

The SyntaxNet installation also installs DRAGNN and a language parsing model. Due to problems I had getting consistent results from SyntaxNet, I eventually settled on DRAGNN instead.

DRAGNN

Shakespeare

The first benchmark was based on the Tiny Shakespeare corpus. The following snippets are from checkpoints with equivalent validation loss, trained using the same hyperparameters (allowing for the proportionately longer sequence length in the Syntax Char RNN due to the additional type encodings).

Naive Char RNN (temperature: 0.8)

Capulet:
with him; there were not high against the nurse,
and i, as well as half of his brother should prove,
thou ask my father's power,
in this life of the world command betwixt
of greep and displent, rup in manrown:
and thou dost command thy stors; and take our woes,
that star in the sea is well.
 Now is a cunning four gloves of all violer on
himself and my friend traitor's jointure by us
to be holy part that were her horse:' the miles
for this seat with me in the island from scards
shall have stone your highness' weech with you.
 And unjust while i was born to take
with hardness from my cousin when i forget from me.

 Shepherd:
the unrelery, reign'd with a virtuous tongue,
to blush of his harms, and as sweet, if they
cape of england's purple say well true song hence,
shall appetite straight hath the law with mine?
 The composition should know thy face of my heart!

 Second huntsman:
i know him, and with my chast my mother.

Syntax Char RNN (DRAGNN parser; temperature: 0.8)

ITrue words, his son, fair despair is, and the guard brother; 
always you tell, 
though to see so many trees; i come!
 
Pedant:
 would you have no redress of joy of the march?
 
I, what i come up.
 
Menenius:
 i not a gentlemen cred; and as this old farewell.
 
Lucio:
 sir, she's a duke.
 
Capulet:
 so straight is the tyrant, shape, madness, he's weigh; 
which of the confail that e'er said and gentle sick ear together, 
we will see the backs.

Note: Syntax Char RNN output has been reformatted, but remains otherwise unaltered


I think it is easy to agree that the naive Char RNN generated text reads significantly better. There is a somewhat interesting punchiness to Syntax Char RNN's shorter dialogue sections, but that's about all it has in its favour.

This was, frankly, disappointing.

However, there is a reasonable chance that inaccurate parsing could be influencing the results. The DRAGNN model probably doesn't generalise well to Shakespearian English.

Would prose offer a better benchmark?

Jane Austen

The works of Jane Austen was used next. They would almost certainly parse more accurately.

The results this time were rather surprising. Syntax Char RNN raced away, reducing its loss pretty quickly. After one hour and fifteen minutes on my laptop CPU, Syntax Char RNN hit a temporary minimum of 0.771.

After the same time frame and with the same hyperparameters (again allowing for a slight adjustment of sequence length due to the extra type encodings for POS), naive Char RNN went as low as 1.025 - still nowhere near the Syntax Char RNN checkpoint.

I left it running overnight and it still reached only 0.944 after just over 8 hours.

This was interesting.

What about the quality of the generated text?

Naive Char RNN (loss: 0.944; temperature: 0.8):

"i beg young man, nothing of myself, for i have promised to be whole 
that his usual observations meant to rain to yourself, married more word
--i believe i am sure you will allow that they were often communication 
of it, but that in that in as he was in the power of a hasty announte 
by print,  to have given me my friend; but at all," said lady russell, 
so then assisted to  his shall there must preferred them very ill--
considertaryingly, very pleasant, for my having forming fresh person's 
honour that you bowed really satisfied we go.

Syntax Char RNN (DRAGNN parser; loss: 0.771; temperature: 0.8)

Mr. Darcy was quite unrescrailous of place, which and want to carry 
by the attractions and fair at length; and if harriet's acknowledge by 
engaging it for a sorry day he must produce ithim. I could be 
mr. Knightley's compray of marrying in the rest of the disposition, 
and i was particularly successful. He has been much like her sister 
to julia, i wishin she believed he may want pause by the room with 
the same shade. "" but indeed you had obliged to endeavour to concern 
of a marriage of his yielding.""

The results are roughly comparable, neither are special. If pushed I'd say I prefer the latter over the former, it reads a little better.

spaCy

spaCy is an amazing set of tools made available by the good folks at Explosion AI. Unlike SyntaxNet, or even DRAGNN, it is a breeze to use.

The spaCy parser's data had an interesting effect on training. A training run with DRAGNN data reached a loss of 0.708 after just under 6 hours, then failed to go lower for the rest of its total run of over 16 hours. The spaCy parser achieved 0.686 after 5.5 hours, and its best loss of 0.650 after just under 8 hours.

Here are snippets from the relevant checkpoints.

Syntax Char RNN (spaCy parser; loss: 0.686; temperature: 0.8)

You beg your sudden affections and probability, that they could not 
be extended to herself, but his behaviour to her feelings were 
very happy as miss woodhouse--she was confusion to every power 
sentence, and it would be a sad trial of bannet, but even a great 
hours of her manners and her mother was not some partiality of 
malling to her from her something considering at mr. Knightley, 
she spoke for the room or her brother.

Syntax Char RNN (spaCy parser; loss: 0.650; temperature: 0.8)

In the room, when mrs. Elton could be too bad for friendship by it 
to their coming again. 
She was so much as she was striving into safety, and he knew 
that she had settled the moment which she said," i can not belong 
at hints, and where you have, as possible to your side, 
i never left you from it in devonshire; and if i am rendered 
as a woman," said emma," they are silent," said elizabeth," 
because he has been standing to darcy, and marianne?"" oh! 
No, no, well," said fitzwilliam, when the subject was for a week, 
that no time must satisfy her judgment.

Note: Lines have been wrapped, but formatting remains otherwise unaltered

Both are quite readable, except for the injudicious use of quote marks (a problem that is most likely the result of redundant spaces picked up during pre-processing).

Even allowing for over-fitting, it is quite clear that in the case of Jane Austen's text the snippets generated by Syntax Char RNN are more readable and cohesive than those from naive Char RNN. Among the former, those produced via spaCy also show a marked improvement over the results produced via DRAGNN parsing.

Since the only significant difference between the Syntax Char RNN runs trained on Jane Austen texts were the data from the two different parsers, these findings suggest that accuracy of parsing between DRAGNN and spaCy likely accounts for the difference in performance and readability between the runs. This in turn suggests that a lack of accurate parsing accounts for the poor results achieved with Tiny Shakespeare.

Code

The code is available on github. Comments and suggestions welcome.

The majority of work was around pre-processing (parse and prepare). For training and sampling I was able to build on the existing Pytorch Char RNN by Kyle Kastner (which in turn credits Kyle McDonald, Laurent Dinh and Sean Robertson). I altered the script interface, but the core process remains largely the same.  

Concerns and Caveats

The approach and implementation isn't ideal. Below are some considerations.

1. Pre-processing is complex. It has to make assumptions about the parser input, bring character and syntax encoding together, and try to remove data that can skew the weightings.
2. Pre-processing can be slow. It can take anything from a few seconds to tens of minutes, depending on the size and complexity of the file.
3. POS parsing is imperfect. The results suggest spaCy is doing a better job than DRAGNN, but even at its best it will have errors.
4. Applicability is limited to text that is at least consistently parseable by an available parser. Poetry, archaic language, social media messages etc. likely fall outside this scope.
5. Some POS units are known to cause skewing by introducing extra spaces. For example "Mary's" becomes:

Mary : PROPN++NNP
's    : PART++POS

All punctuation are separated out as well, for example:

, : PUNCT++,

The effect is that these parts of speech remain tokenised when encoded, resulting in redundant spaces on one side of each token. Unless they are subsequently removed, spaces become overrepresented in the resulting encoding, affecting weightings for all character representations ever so slightly. The code manages to remove some of these redundant spaces - with occasional side effects - but not all.

To Do

Avenues to investigate, features to add.

1. Reduce the number of redundant space encodings
2. Calculate a more granular weighting for type
3. Investigate further candidates for context encoding, over and above syntax
4. Investigate more elegant ways of grouping encoding contexts
5. Add validation loss for comparison to training loss
6. Estimate parser accuracy for a specific text
7. Run on GPUs!

Conclusion

The findings suggest that in some cases where parsing is accurate and consistent, Syntax Char RNN trains faster and achieves better results than naive Char RNN. This lends support to the hypothesis that accurate contextual encodings, over and above syntactic Parts of Speech, can improve Char RNN's autoencoding.

While they come with several caveats, the findings nonetheless warrant further experimentation and clarification.

2016 - The Year in Books

2016. There may never be another year during which I read so many of the Great Classic Novels for the first time. Let me list them: War and Peace by Leo Tolstoy, The Brothers Karamazov by Fyodor Dostoyevksy, Ulysses by James Joyce, Don Quixote by Miguel de Cervantes Saavedra, Moby Dick by Herman Melville, and The Mill on the Floss by George Eliot. I also chucked in a few of the great plays for good measure: Hamlet, King Lear, and Twelfth Night (this one I'd read before, and remains a favourite), all by William Shakespeare.

As a bonus, I also had the chance to read two of the most beautiful and startling philosophical treatises: On Liberty by John Stuart Mill and On the Genealogy of Morals by Friedrich Nietzsche.

But back to the novels. It is difficult to do justice to any those great works individually, let alone all of them. Their collective influence on arts and culture in the West is practically immeasurable.

The epic scope and narrative invention of War and Peace is legendary, but it is even more breathtaking when actually read. The array of characters, the depth of their characterisation and the movements of history combine to provide rich nourishment for the soul, and reveals the sophistication and nobility of the Russian spirit.

Moby Dick was a real surprise for its intellectual ambition. One expects adventure on the high seas, and instead is given something much more: the enterprising American spirit as seen at once through its cultural links to Europe and Britain (Shakespeare looms large) and forging its own way, expanding, pondering the nature of its own spirit.

Ulysses is a juggernaut of linguistic invention and deliberate intellectual playfulness. It is perhaps the least accessible of these great classics, and perhaps also the most divisive, but its intellectual rewards are great and in a sense it remains ahead of the times.

But it is Don Quixote for which I want to reserve the most emphatic recommendation, in part because I believe it is the most easily overlooked, and too readily dismissed as antiquated or irrelevant. It is not. It is unique among nearly all of the great classics for being truly, laugh-out-loud funny. More than 400 years have not dimmed the humour. How much funnier still it must have seemed to contemporaneous ears who understood the subtler references that are lost to time and translation.

Don Quixote is not only funny, but also full of pathos. The main character centres in himself something of both the ridiculous and the sublime, and while we are treated to the former most of the time, the shape of the latter emerges over time, especially in Volume 2.

Personally, I found Volume 2 to be even better than Volume 1. Its latter two thirds are as funny as anything in Volume 1, and yet it also treats of more serious matters. I particularly marvelled at and appreciated the story's innovative reference to characters' knowledge of the first volume, published ten years before it. This is an ingenious device that seems more at home in the 20th or even the 21st century than in a novel from the early 17th century. If there can be any doubt that Don Quixote is inventive and linguistically imaginative, this fact alone should dispel it at once.

It is a pity that English readers (myself included) cannot appreciate the full craftiness of the language at work, in particular the contrast between the deluded knight errant's Old Castillian and his compatriots' modern Spanish.

All the other classics seem to take themselves a bit too seriously when we compare them to Don Quixote, and it is only when placed next to Shakespeare that we find a similar use of comic devices in great literature.

2016 marked 400 years since Shakespeare died, and all year long his works were commemorated with performances that are set to continue well into the New Year and beyond. How many of us knew that 2016 also marked 400 years since the passing of Miguel de Cervantes, the author of Don Quixote?

Remedy that neglect immediately, and place Don Quixote on your reading list!

What Shakespeare has in Common with Software Development

Shakespeare is widely regarded as the world's leading playwright in English, and perhaps any language. Such is his influence that phrases and ideas coined by him at the turn of the 17th century still live on in our colloquial speech today. Romeo and Juliet is shorthand for passionate, ill-fated love, and quotable lines from his works permeate our treasure trove of idioms and phrases.

What is perhaps less well known is that many of Shakespeare's plays have no definitive version. Take "Hamlet", for instance. There is the famous First Folio version, compiled and published seven years after his death, and there is the First Quarto version, a.k.a. the Bad Quarto, and then also the Second Quarto version. None of these versions are considered 100% definitive. Edited versions usually combine parts of each to present the modern reader with the most feasible "Hamlet", and even these are subject to change.

How did this happen? So many details about that time have been lost to history that it is difficult for us to reconstruct a real sequence of events from the remaining evidence. There are entire books written to argue one case or another, but consider that some people even dispute William Shakespeare's authorship, then it is clear that we are on shaky ground from the get-go.

Personally, I've come to a different view while mulling over an under appreciated ingredient of Shakespeare's genius, an aspect that has something in common with software engineering - especially agile development.

Shakespeare wasn't just a writer, he was also an actor and part-owner of the theatre company the Lord Chamberlain's Men (later the King's Men). I find it useful to think of his plays as a function not only of Shakespeare's maturing talents as a writer, but also of the needs of the company. Those needs were financial, like any company's, and were directly informed by the success or failure of a particular play in the eyes of the audience of the day, as well as the tastes of their influential patrons.

It is thus hard to imagine that Shakespeare would just write a single, finished version of Hamlet, tell the actors their lines once-and-for-all and be done with it. As part-owner he had a responsiblity and exposure that went well beyond writing. He would have wanted to make sure the play is as good as it can be, on a continual basis. The company would receive financial feedback, and the company's patron would have his say, and so the day-to-day operations would hone the way the play was performed - if it was performed at all.

As an actor of second-tier roles he would also have been in a unique position to experience feedback from the audience. I imagine him night after night, observing the audience's reactions, hearing them laugh at the funny parts (or not), seeing them moved or engaged during tragic or passionate moments, and smiling or bored as the case may be during the play or afterwards. He would be thinking of the various stakeholders, of the dramatic value of a particular phrase or scene, of the audience's reactions, and so he might choose to change the lines - add a bit more zing, create more drama, more references to current affairs - who knows?

Shakespeare's mind would have been working constantly to improve the play and I have no doubt that this is precisely what happened. His plays have a uniquely organic feel to them, as if the action is happening right there, and the actors could step off the stage and mingle with the audience at any moment. By assimilating his audience's emotions and interests he was bringing art closer to the audiences' reality.

It is this approach of continual improvement, of being tested night after night against a real live audience, that strikes me as being very much in the spirit of agile development. It's a bit like running continuous integration while already in production.

I would go a step further and suggest that Shakespeare was so canny and pragmatic that, even if he had a successful version of a play, should the political climate change he would be willing to adapt the play again, to cater to his audience and so prolong the play's financial success. If this is so, he may well have found a dramatic architecture that admitted of continual adaptation, just like good software architecture is flexible, and written with ease of maintenance in mind. That would certainly go some way towards explaining his plays' capacity to be continually repurposed for modern audiences.

To put that achievement into perspective, imagine writing software that is still in demand 400 years later!

If we take this view it is a bit of a shame that not more of our worthy literary works are "production tested" with a feedback loop that permits continuous improvement. There was a time when serial publication afforded authors some engagement with their readers, and thus to inform the next installment. Nowadays, authors are required to write once, for all time. But in software development we know that this is usually premature, costly, and occasionally disastrous.

This is the reason that many writers form reading groups with other writers, to permit them a trusted soundboard and source of feedback. But the General Reader is a different beast, whose tastes are not to be tamed so easily. Shakespeare wrote "not for an age, but for all time", and perhaps it's because he wrote not once, but all the time. He understood the value of his users.

A Look at Ferrante's The Story of the Lost Child

SPOILER ALERT! This post discusses the final novel in Ferrante’s Neapolitan Saga and deals with plot points without warning or discretion. If you haven’t read the series up to the end and do not want plot spoilers, stop reading here.

Introduction

So much happens in The Story of the Lost Child and there are so many surprises that a good way to make sense of it is to begin at the end and consider what we know by the final stages of the book. Before I do so, however, a few preliminaries are in order.

Firstly, Elena analyses her own behaviour, and this layer of analysis illuminates her and others’ behaviour. I will try not to repeat the obvious. Secondly, the final novel veers off into territory I had not anticipated in my analysis of the first three. I am glad. As a reader I tend to prefer the Lila-centric parts of the novels over the Elena-centric parts of the novels, probably because they are the extraordinary ones. It also means that some of my observations were not conclusive. I will comment on a couple of these, but I won’t harp on about it.

The Lost Child

Until Tina’s disappearance, we are led to think of Imma as the lost child, because of her inability to adjust. She is an emotionally lost child. This turns out to be a clever ploy by the author to keep us ensconced in the joy of those Halcyon days before the cruel blow is delivered. Whether intended or not, the care with which Tina was made the focal point of the photoshoot signalled to me a symbolic exchange of destinies and, indeed, I feared for the worst. I had a sleepless night after Michele punched Lila in the face and sensed a terrible tragedy in the lives of the Lila and Enzo.

Yet by the end of the novel Tina’s fate is magnified in other characters and perhaps in almost all the familiar characters of the neightbourhood. Lila and Gennaro are both lost children. Gennaro, like Imma, is emotionally lost and weak willed. He never really grows up. Elena treats him like a stupid boy at the very start of book one. Lila herself is a lost child. Her precocious talents as a child have all stilted and repressed by adult responsibilities, an adult world, through work, through hardship, and now through tragedy. Yet the child inside never gave up, always held fast in some hidden corner. This child held fast to hope, and this hope is for the longest time connected to Elena, whose life was meant to justify Lila’s suffering. Once Elena’s activist efforts in the neighbourhood fail, and especially after Tina disappears, even this hope fades. Lila is disappointed in Elena.

The truth about Elena’s doll Tina, like a voodoo doll representating Elena, remains hidden inside Lila until the novel’s resolution. At the same time Lila herself remains tucked inside Elena’s soul. This hidden knot binds the two friends for a lifetime, and is the edifice on which the novel is built. Like those Neapolitan churches that come to fascinate Lila, and that commemorate forgotten atrocities, Elena’s story is a literary monument that exists because of the horrible events that caused suffering in their lives.

To clarify this point, let’s ask a question. Would there have been a story or indeed the need for one, if Lila’s life had proceeded according to her childhood promise? Yes! There would almost certainly have been a need for it, but chances are that she would have written it herself, even if that life unfolded side by side with Elena’s.

The lost child from Elena’s point of view, therefore, is Lila, and if Lila once admonished Elena for writing “ugly things” (in that second novel that only belatedly gets published, and then to great fanfare) it can be understood from this viewpoint: that Lila’s hidden child wanted beautiful things, and that Lila’s hidden, lost child wanted Elena to transform the world into beautiful things. Instead, Elena merely reflected the ugliness of their world. It is a world from which Lila never tried to escape, trusting Elena would help her to transform it, even if only in literature. Yet in old age, after even Elena disappointed her, she finally shifted out of Elena’s range. Lila’s hidden child is lost first because she is left behind, and second because Elena disappoints her, doesn't help her escape the ugliness. Tina’s disappearance is the symbolic reinforcement, or realisation, of this “lostness” - of being lost.

Elena suffers in the absence of Lila. It is a type of mourning that refuses acceptance. It is an angry suffering. Acceptance comes only at the very end. Elena’s suffering in the absence of Lila mirrors the suffering Lila felt in the absence of Tina. It is a suffering that results from not knowing whether she is dead or alive. This suffering finds further fertile ground in the imagination of the reader, who knows about Lila’s disappearance from the start, and learns about her tragic life only through the eyes of Elena. As readers we are outraged at Lila’s fate, but also at the fate of all the downtrodden characters.

The significance of the dolls have a direct connection to the lost child(ren), but I discuss them more fully in the next section. For now, let’s complete the round-up of “lost children” by acknowledging with Elena that, although the Solaras have been almost universally hated, they also did their bit for the neighbourhood, to make it what it is - even its good aspects. Alfonso, Rino, Gigliola, Gino, Bruno are all children who got lost somewhere on the way. They stand for the loss of innocence, of hope, of childhood in general. In their place Elena writes her literary monument that remembers all their lives - not just her and Lila’s.

More specifically, through the loss of little Tina, Lila’s suffering is the suffering of the whole neighbourhood. By disowning Lila and forgetting, the people in the neighbourhood disown themselves, and thus their redemption becomes truly futile. Lila realises that the neighbourhood cannot be truly changed - not even by her and Elena.

It is not an optimistic vision, but it is rooted in a reality that has an emotional authenticity that is difficult to dispute.

The Dolls

Now that we have considered the Lost Child of the title, what should we make of the dolls and their return at the end? Elena receives the two little dolls from their childhood, Tina and Nu, in an unmarked newspaper package together with her post. No addressee, no return address.

The first conclusion we can draw is that Lila is alive and well somewhere, which indeed is the possibility that Elena herself entertains:

“Maybe those two dolls that had crossed more than half a century and had come all the way to Turin meant only that she was well and loved me” - p. 473.

We see here Elena’s need for validation and approval on clear display (“she … loved me”), but it is the strong possibility that Lila is alive that is of primary interest to us. Lila could have committed suicide and planned it that way, but it would not be consistent with the lost child who has finally found a new life for herself. That child was too curious and irrepressible. That child, now lost to Elena, has been recovered by Lila unto herself. 

Secondly, it is an admission by Lila of the role - unspoken up to now - Elena has played in providing courage to her in the face of overwhelming fears, such as those she confessed to in the aftermath of the earthquake. When they went up to Don Achille to confront him - one of the scariest moments of their childhood - and Lila looked so brave she was partly brave because Elena was there beside her.

This fear-courage duality is part of the secret knot of their friendship alluded to in the previous section. It is not just that “Lila has let herself be seen so plainly” (p. 473), but the very knot of their relationship has been made plain. By being made plain it now also loses its power, because Lila has relinquished it. Lila no longer needs Elena to give her courage - she has made a leap, on her own, that we know nothing of.

Thirdly, by relinquishing it, Lila also releases herself. The suppressed confines of her life finally lifts and she is free. We don’t know anything about it, but we can perhaps imagine her: a cantankerous old woman no one would pay any attention to, yet whose intelligence is still sharp and inquisitive at nearly 70 years of age, and who still has a few years left to live and enjoy life without the neighbourhood, without children, without men, without Elena, without the expectations of her childhood - without even the expectation and intrusion of us as readers (here we are reminded of the contrast between the real author, Elena Ferrante, who prefers to live anonymously rather than riding the wave of fame the way Elena of the novel did for the sake of her career; in other words, Ferrante is more like Lila in this respect).

The returned dolls means Lila has gone beyond the pale, and beyond even the bounds of Elena’s tale.

Fourthly, since dolls are often stand-ins for babies in the cultural environment in which several generations of girls have grown up in, the return of the dolls also reflect on the motherhoods of Elena and Lila. Although Lila lost Tina, and Gennaro was a disappointment, she was nevertheless a responsible, dedicated mother - even to Elena’s children until adolescence. Elena was a far better mother than Nino was a father, but she still suffers from her own children’s admonishment that she was too absorbed in her own work. Lila filled this gap.

By returning the dolls, Lila relinquishes her own role as surrogate mother completely, as well as being mother to Gennaro. Their roles have reversed, and on Elena's side of the fence the story hasn't quite ended. She has three children, plus Lila had made Elena promise all those years ago. Elena is  now responsible for Gennaro.

Fifthly, the loss of those dolls were the stuff of childhood emotions. They chucked each other’s dolls into the cellar in a jealous rivalry, a dynamic pattern that repeated itself over many years. The return of those dolls means the end of that dynamic. No more jealousy, no more rivalry. Elena, however, thrived on that competition, and her career was ignited by it.

Finally, the timing of the dolls’ return suggests a simultaneous discovery and loss of Lila’s inner child. Elena wanted Lila to hack* into her computer and read the novel. It is not inconceivable that this actually happened, and that Elena’s conclusion to the contrary is simply more evidence of her inability to see coincidences and the connections between events. The novel is finished, and soon after the dolls arrive. Perhaps Lila, herself finally free, read eagerly and realised that what Elena wrote is actually good, even if not exactly beautiful. Lila no longer needs it to be beautiful. She sees something of herself, and perhaps above all she sees Elena. She takes mercy, and frees Elena.

* Readers may have noticed more than a passing resemblance between Lila’s character and that of Lisbeth Salander. I know that I have.

Elena and Lila

Where does this leave Elena? There are many things we don’t know about Elena’s day-to-day life, but she has told us about most of the truly important events. We know that for significant periods she thrived on the competition and inspiration Lila provided. That force has now faded from Elena’s life, and she can enjoy what’s left - children and relative fame - without that pressure, without that interference. Perhaps she can mend her relationships with her children, perhaps Gennaro will take up some of her time. Either way, it is without a doubt the end of an era.

As for herself, Lila has finally freed herself of the burdens and responsibilities that had taken up her whole life. She gives up Gennaro, she gives up Elena, the neighbourhood. The neighbourhood had given up on her over the years, but she had always been a fixture, an anchor. In the final instance, her energy and inspiration had also gone into the novel we've just read. In this way she served Elena's career, albeit frequently in her own interest. Now, finally, she was free. She who had always been afraid had finally done what she could never do before: be completely independent - even independent of Elena, of the novel. True to her nature, there is no tying her down, and no knowing who she really is.

Nino

The Neapolitan saga is full of characters struggling to escape the influence of their parents, only to find themselves emulating them in one way or another. Nino epitomises this theme. He hated and rejected his father throughout his adolescence, yet ended up taking womanising to a whole new level. We saw the first part coming, even if Elena did not. Yet when she finally processes his legacy, she judges him “disappointing”. Of course, Lila got there first and understands the nature of his character all too well. She judges him worse than herself because, she says, he is superficial. Guido Airota makes a separate observation about him, namely that he is “intelligence without tradition”. A talented man without roots, who has nothing to lose, and is all too eager to be someone. At the end of the saga Nino sings his own praises, but those who loved him from close up have all realised that he is an unreliable, lightweight human being.

What, therefore, should we make of Elena’s longstanding crush on him? She herself realises that she had created a fantasy, and that the person who showed up at her book reading in Milan had nothing to do with that fantasy. They were separate entities. However, fantasy and reality corresponded sometimes. For instance, his behaviour as an adolescent was real. She saw him as cool and untouchable, unaffected by the opinions of those around him. His head was somewhere else. That trait turned out to be a flaw, a disregard for everyone around him, even those who adored him and whom he sporadically loved in return when it suited him.

He had intelligence, charisma, and good looks. He seemed untouchable, smooth in all situations. When he showed up at the book reading he was a knight in shining armour, rescuing her from the attacks of a stuffy intellectual. What Ferrante is doing, as Nino’s character unfolds in all its mirage-like glory, is turn the literary convention of the hero - reminiscent of, say, Will Ladislaw in Middlemarch - on its head. She is taking a longer view. Book 3 could have ended with Nino willing to reform for the sake of his “true” love for Elena. We could have been told that “they lived happily ever after”. Instead, we got the bombshell that is book 4. Nino is a warning that many of the classics are perhaps guilty of building fantasies and gender stereotypes rather than looking at the genuine commitments to gender role realities that are implied by marriage and long term partnerships. It is a stunning critique.

If I had expected Nino’s general infidelity, his ongoing marriage to Eleonora was more of a surprise. It is a cleverly disguised plot device that all but defines his character. It anchors his tendency to put every relationship in service of his career ambitions. It provides him with monetary stability and a conservative societal esteem, namely of keeping a family. It also characterises his inability to finish off any relationship. In his personal life he is a politician: not a conviction politician, but one who goes where the grass is green.

Another surprise was the amount of time it took for Elena to get rid of him. In her case, also, we see what she is willing to sacrifice for her career - in her case her human dignity. For a while she lives the life of a concubine. Yet it’s not just a career, it’s also the children and the roof over their heads. She’d created a complex set of responsibilities for herself, and she was keeping herself entangled out of necessity.

It takes her even longer still to realise that his interest in her was due to the prestige she reflected back on him. This tendency in a man is so unusual that she couldn’t see it for what it is.

Only Lila put Nino’s life at risk by being of no use to his career. Lila is therefore in a league of her own.

“She stood out among so many because she, naturally, did not submit to any training, to any use, or to any purpose. All of us had submitted and that submission had - through trials, failures, successes - reduced us.” - p. 403

If Lila’s capacity for suffering is bottomless, Nino’s suffering is like a sulking child’s when it cannot gets its way. When he gets the upper hand once more, it is water off a duck’s back. By the end of the novel Nino is nothing but an annoying stranger whom Elena finds “large, bloated, a big ruddy man with thinning hair who was constantly celebrating himself” (p. 470).

By the end of the saga, Elena herself is leaning more towards traditional values again. She recognises that Pasquale is “much better preserved than Nino”, and speaks fondly about the values he took over from his father and that he upheld at great personal cost. Indeed, even the passing of the Solaras are met with a balanced sense of loss. Elena may share something of Nino’s flightiness and ambitious disregard for those close to her, but she recognises the love she had for her old friends, for the neighbourhood, for all the families that lived there - even for her own mother. We don’t truly know Nino anymore by the end of the saga, but his lack of interest in his own children speaks volumes.

I return to Nino a little later for a final look at his character.

Pietro

Pietro, who resembles MiddleMarch’s Casaubon and Wuthering Heights’ Edgar Linton in the during the earlier novels, in book 4 emerges as a far better partner and father than Nino. He never shirks his responsibilities, he is tender and observant (as when he advises Elena sensitively about Lila), and despite his general physical deficiencies, Elena judges him worthy of her bed one last time before he leaves for America. Elena recognises his selfish need to spend his personal time with his work, yet accepts it more readily later in life, since she recognises the similarity to her own character. In short, she endorses him as a good former husband, even if she has no desire to start something new. In all these respects Pietro also turns the classic literary stereotype somewhat on its head.

With both these male characters Ferrante is taking our common literary canon to task.

Alfonso

One of the great satisfactions of Elena Ferrante's Neapolitan saga is the way in which the story lends itself to analysis. There is enough substance for a thesis, and a blog post can really only hope to probe a few angles. We have not even taken a look at the Solaras, and we should.

One perspective from which to tackle the changing fortunes of the Solaras in book 4 is via the prism of Alfonso. Alfonso is a gender bender who mediates between the destructive masculine energy represented by the Solaras, and Lila’s near-indestructible counterpoint of female energy. He is both a gay man and a cross-dresser, and his muse is Lila. Not only that, he wants to become Lila. The result is that he begins to resemble Lila even more than Lila herself.

Michele Solara, always the more dangerous of the two brothers, has lost the upper hand in his dealings with Lila since she established Basic Sight with Enzo. She and Enzo have become self-sufficient. Michele’s deep respect and yearning for Lila means that this energy now spills out over a cliff and he needs a surrogate for his obsession, which the shape shifting Alfonso provides. They become a type of couple, albeit covertly. (Marcello is furious, although there is nothing he can do.)

The Solaras epitomise a type of macho male energy that simply cannot co-exist with a true female energy. Michele is obsessed by his opposite, but it is also his downfall. Their violence is not compatible with equal distribution of male and female energy. They require submission. Lila and Enzo, on the other hand, embody the only example in the neighbourhood of a different, balanced model of male and female partnership.

As Michele and Alfonso get closer to each other, the Solaras are weakened. At the same time Lila and Enzo become stronger, especially after the return of Elena, and the birth of both their daughters. Masculine and feminine energies find a kind of equilibrium in Lila’s family life perhaps for the first time, and the result is a temporary happiness and harmony (which, after the loss of Tina, never returns). During this period Michele is reduced to a tentative, nervous man who can no longer act with vigour. Whereas Alfonso now embraces his own newfound identity of a woman in a man’s body, modeled on and inspired by Lila, Michele is completely at odds with himself.

We don’t really know what happens between Michele and Alfonso, but it seems that Alfonso overshoots his privileges and Michele kicks him out. The entire balance of masculine and feminine forces in the niehgbourhood are once again in jeopardy, stacked in favour of the destructive masculine element once more. Alfonso loses his feminine appearance, he becomes unreliable, and the whole sorry saga ends with his death at the hands of unknown assailants. It is the beginning of the end.

Tina’s disappearance is the visible culmination of this multi-generational journey. Whereas we are never sure who took Tina, the flow of energies suggests the Solaras were behind it, except that they thought it was Elena’s child - not Lila’s.

Everything from then on - even the death of the Solaras - suggests to Lila that nothing will ever really change in the old neighbourhood.

Nino and Elena

While we are on the topic of contrasting energies we should take a last look at Nino and Elena.

Nino exhibits a strong blend of the feminine and the masculine. His willingnes to sleep with influential women in order to get to the top is a strategy more commonly associated - rightly or wrongly - with ambitious women in society. Combined with his intelligence, charisma, and good looks, this is a killer strategy. He appears to have disguised his stereotypically Southern tendencies behind an alluring, more acceptable Northern veneer. His masculine aggression, paired with a keen feminine sensibility, which is to say an ability to tune into women’s emotions, makes him an effective and well rounded talent. Unfortunately it is almost completely erased by his lack of commitment to his roots, or indeed to any place where he puts down new roots - except where there is power. He sows the wild oats and moves on.

Elena comes across as conservatively feminine during most of her adolescence, but her ambition passes through masculine territory via her academic learning of the classics. She breaks out of academia and attempts, with Lila as inspiration, to marry her feminine and masculine sides to great subversive effect.

If the masculine and feminine are played off against each other, so are the Northern and Southern cultures. In this case Nino and Elena are the clearest examples of this co-habiting duality, since both rise high above their origins. Indeed, at times they mirror each other. However, unlike Nino, Elena increasingly recognises her roots and turns to the neighbourhood of her and Lila’s youth for inspiration. Even so, she moves to Turin in the North for her retirement.

Conclusion

As epitomised by Lila’s life throughout most of the saga, any equilibrium of opposing forces is extremely hard to maintain. Each character in his or her own way tries to find such an equilibrium, and success tends to come at the cost of others, or at the cost of social cohesion.

The Story of the Lost Child resolves the main plot points in often startling ways, but it is the openended implications of the ending that ensures the reader will continue to reflect on the rich material provided.