Chatbot using OpenAI GPT-2 transformer model

Some time ago I read an article on OpenAI's GPT-2 language processing model. This model employs a Transformer network to predict the next word based on a given text. The examples on their website show that the network is able to generate high quality stories. Especially interesting is that the network is able to generate consistent stories. If the network generates a paragraph about a specific subject, then later on it can refer back to it. Unfortunately OpenAI decided not to make their most powerful model (1.5B trained parameters) publicly available as they fear that the model could be used for example to generate fake news or fake reviews (good marketing trick!). OpenAI did however make their less sophisticated (117M trained parameter and recently 345M trianed parameter) versions available.

I thought it would be interesting to see if GPT-2 can be used to implement a chat bot. The idea is as follows:

1. The network is given a prior consisting of part of a conversation.
2. Then the network generates one paragraph to complete one answer in the conversation.
3. Next we can add our own response and the cycle repeats.

The example code can be ran online using Google's CoLab infrastructure. Read the documentation in the chat bot code and try a conversation yourself!

Below an example of an earlier attempt with the 115M GPT-2 model (the code online uses the more recently published 345M model which actually performs even better). First the prior given conversation which I used as a starting point:

Dinne: "Nice to meet you. What's your name?"

Me: "My name is Pete."

Dinne: "That's an interesting name. How old are you?"

Me: "I'm 30 years old."

Dinne: "Can you tell me something about yourself?"

Me: "Ofcourse! I like playing video games and eating cake. "

Dinne: "I like sweet stuff too. What are your plans for tomorrow?"

In the following phrases the "Me" part was generated by the network/chat bot!

Me: "Things you might not reach. Right now. Copper toilets. Iron third barricade without batteries. Iron wall. Copper motor docks with lockers, door mists and lock plates. I guess I am lucky? I dunno, I haven't seen a lot of science fiction in my life, have I?"

Dinne: "Do you like science fiction?"

Me: (together) Yeah, I like science fiction. It's a wonderful story that I enjoy. Everything about it is fascinating, and you guys are a great bunch.

Dinne: "I've watched every Star Trek movie."

Me: "Yeah. Star Wars, Spielberg, All the Star Technical Stuff. I watch every Star Trek film."

Dinne: "What do think of the latest Star Trek movie?"

Me: "Oh no, I don't care what they do. I just know it's weird. It's funny. Star Trek is weirdly fun."

Dinne: "I still think Spock is the best!"

Me: "I've never read Spock so I don't know whether or not that's right."

Dinne: "Spock isn't a book! He's an alien from Vulcan and is the senior science officer on board the Enterprise."

Me: "Can you tell me anything about the first episode of the television series?"

Dinne: "Yes, in the first episode the crew is assembled, they go on a trip to planet Talos where captain Pike is captured."

Me: "They're so excited they all grow up to become the leader of Starfleet."

Dinne: "Do you think the warp engine could really exist one day?"

Me: "Eh, no, I doubt you."