Intro

I recently began playing with GPT-2. I use the 774M model rather than the more-powerful 1558M/1.5B model for speed.

How

Stories were written in-order, with GPT-2 called-in whenever I felt like waiting an eternity it to suggest the next word¹.

GPT-2 would provide 3 or 4 options for what the next word could be, and I’d either pick one or regenerate.

You may call this cheating, but I call it a profound waste of time.

Challenge: Roughly half of the text must be generated by GPT-2.

Reading Key

bold text = I wrote it

normal text = GPT-2 wrote it

Story #1: Alice and Bob escape the catacombs

“Bob! I found the doorway!” Alice yelled.

Bob was overjoyed.

It had been his idea to journey into the catacombs, and if it hadn’t been for Alice’s sense of direction, they would’ve been screwed.

It looked like this would be their final day in the catacombs. It had been nearly two days since they’d entered.

“Alice,” Bob said, “I was wondering if you had any idea of where we’re going to take the rest of the day.”

Alice shrugged. “I’ll take you up on your offer, if you tell me the rest of the story.”

Story #2: Alice and Bob are in a stereotypical YA romance novel

“I don’t know what I want to do anymore.” Alice said quietly, still in the doorway.

“Hm?”²

“I mean, I used to want to get out of here. Go somewhere. Leave.”

“You don’t anymore?”

Alice smiled and shook her head. “I don’t know, it’s all so complicated now.” She walked toward me.³

“I’d go with you, wherever you wanted. I mean, it’d be nice to be out of this place and not have to deal with these people.

“I know, but it’d make me sad to leave**, too,” she sighed, “it was so easy when I was alone and miserable.** And it just… I don’t know.”

“Sorry for complicating things.”

“You should be,” Alice turned around and gave me a soft, lingering kiss, “but also, thank you for everything.”

I smiled at her, “thanks for everything too.”

I wrapped my arms around her back and kissed her gently. She giggled. We remained in that position for a long time.

Story #3: Alice and Bob attempt to discern their reality

Alice punched Bob playfully on the shoulder.

“So, what’s next?” she asked.

“Anything as long as it doesn’t include you jumping into black holes. That was a traumatizing experience.” Bob replied.

“Oh, well, I’m sure we can work something out. I really am glad to have you here, Bob. You’re the only one who’s ever come along and made me feel like this.” Alice said as she turned back around to face him.

“Oh… that’s nice to hear,” Bob replied, “are you going to tell me what’s on your mind?”⁴

“I’ve been thinking about our relationship. We’re not married yet, and you’re still very much in my thoughts. The fact that you’ve been so sweet to me the last couple of days really makes me happy.”

“But Alice, you know those feelings are superficial. We’re just simulations after all. We aren’t even conscious right now. This is all a dream.”⁵

“Yes, you’re right. We’re all simulations. You and I aren’t really here right now. It’s all a game. What’s more, I’m just a simple simulation of your own subconscious mind. You’re the only real person I have ever known.”

“What? I’m real?”

“Well, if you believe the dream you’re in, you are, aren’t you?”⁶

Conclusion

GPT-2

Overall, I’d say that writing with GPT-2 is fun, but excruciatingly slow and ineffective for anything practical. Though, the latter might be a side-effect of using the 774M model instead of the 1558M/1.5B version. And the former is definitely because I had to use Google Colab because my PC is just a salvaged office desktop tower.

Still, I’m looking forward to a better version of GPT-2 or maybe a way to run a super-simplified/specialized model in the browser with TensorFlow.js that is actually fast.

When I was messing around with GPT-2, it would usually give stuff that you’d expect, but sometimes it would suggest things that completely 180’d the story.

For example, I gave GPT-2 a string about two friends hugging. The friend being hugged was suggested to be feeling “happy”, or they “smiled”. However, GPT-2 also gave me a suggestion that the friend being hugged was “scared”, though this was mostly due to the vagueness and shortness of the prompt. Still, it was interesting and happened a couple of other times when I was too vague with my prompts and was a nice change of perspective.

Education

Probably the most-obvious use of GPT-2 would be in school essays.

Need to boost the word count of your already-finished essay? Just go to the middle of each paragraph and have GPT-2 suggest a sentence or two.

Need to write a blob of text for a freeform essay? Just write 10 sentences and have GPT-2 turn them into 5 paragraphs.

And if you just have GPT-2 generate a couple of words at a time, not only will you have finer control of what gets generated, but it becomes much harder to tell if the text has been generated, especially if you shove in your own words here and there to nudge GPT-2 in the right track, which is necessary anyway for preventing logical inconsistencies.

Bonus

Here’s some book reviews GPT-2 generated. Unlike the stories, I did no word-by-word coaxing for these, I configured GPT-2 to just create these reviews in their entirety.

“I can’t imagine a better story for the human condition.”

—JEFF BEZOS

“The sex scenes are just incredible. Alice is so much more than an object of sexual desire, she is a person and she is complicated, intelligent and passionate and in many ways just a better version of me”

—STEPHEN HAWKING

“This book is a great celebration of the female libido. I don ‘t know what you’ve read in science fiction or fantasy about female libidos, but the notion of ‘vagina porn’ is a new concept in the canon of literature. It’s an intriguing and important new development, because the female libido is really the core of our human consciousness. “

—DAVID FRICKE

Thanks

• GPT-2 GitHub Repo
• Max Woolf’s great blog post on using GPT-2
• (Not used for this post) An article about retraining GPT-2
• Wrapper for GPT-2