Creativity is a quality that we, as humans, think is ours alone. Prof. Georgios N. Yannakakis is creating computers that might have already taken this away from us. Computational creativity is here. His games are helping children be more creative, others to overcome dyslexia, and even combat bullying. Words by Dr Edward Duca.
I am going to reveal a secret: I wrote this article like a computer. Or at least, that was how I felt after a chat with computational creativity and adaptive machine learning expert Prof. Georgios N. Yannakakis from the Institute of Digital Games (University of Malta). ‘We [humans] do follow machine processes and algorithmic steps and are very creative’. A bold statement by Yannakakis. He goes even further, reasoning that philosophical thoughts that see humans as the only creative things are outdated. ‘We should allow machines to be creative in their own right, and then judge on the outcomes and
processes that they follow’.
Yannakakis’ research revolves around two pillars. The first sees him and his team develop ‘algorithms (computer programs that perform specific tasks) to understand how players behave, feel, and understand particular processes.’ The users’ data is used to form a mathematical model that can predict their behaviour and tailor the game for them. The second pillar takes this a step further with the game generating new content specifically for that person. A game could create new extremely challenging levels focused on anger. Forget mass production, Yannakakis’ work is the ultimate individualist ideology as a force for good.
His tailor-made games are not being used to increase sales, even though he has researched commercial titles. Yannakakis has used his algorithms to create a tailor-made Tomb Raider: Underworld and Super Mario experience. Data from thousands of players helped identify different player types to allow game adjustments or generate new personalised levels. That research propelled him into the media limelight with coverage by the magazine New Scientist and blogs worldwide; the papers are still highly cited by researchers. His other research has helped soldiers recover from war trauma, resolve childhood bullying, overcome dyslexia, enhance creativity, and even rehabilitate physical injury.
Games for Health
Wii fit, Nike+, fitness bands that monitor your every move, health has been digitised. Yannakakis has gone a step further with adaptive games for physical rehabilitation. His team worked with therapists in Denmark (before Malta, Yannakakis was an Associate Professor at the IT University of Copenhagen) to create rules for a ski slalom type game. The game ‘understands where an injury might be, […] how much weight you put on your left or right leg [measured using pressure mats] resulting in a patient’s profile for their injury by tracking their knee’.
So let us take things to the next stage; how about a game that can cure the mind? Yannakakis developed a game that can be used by psychiatrists to help soldiers recover from post-traumatic stress disorder (PTSD). Enter Startlemart, which was designed to place PTSD patients in a stressful supermarket experience with the player trying to match their shopping list while dodging ‘couples fighting, weird faces looking at them, […] even a person who suddenly maximally stressed when you provide this particular event at this moment. Do you (the psychiatrist) want to do that? Yes or no?’ The psychiatrist can use the game to tailor treatment, affect the exposure therapy by maximising stress or minimising frustration, and then talk the patient through the experience.
“Dodging ‘couples fighting, weird faces looking at them, […] even a person who suddenly transforms into an Afghan soldier”
The game was developed for one of the biggest clinics in Denmark. It was tested on patients to be used as a tool. The game was not designed to help patients deal with PTSD on their own. Unfortunately, due to the nature of research funds, the game has not been promoted to be used in other clinics; neither has it been monitored after it was released to see how well it is working or how it can be improved.
This problem with research funding troubles Yannakakis. ‘Startlemart was one of the most exciting projects. I interviewed these veterans, and the life they have had is horrible beyond any imagination. You would never have thought that these people live amongst us. Only 20 years old [and] their life is destroyed in several ways. They come back [from a war-zone] as different men. Their partners cannot tolerate them easily any more. They often get divorced. Their children are scared of them [and] cannot cry because their parent might get a panic attack. Most veterans are even afraid of driving. […] If they overcome the problem, it’s only after several years of good and effective therapy. […] War shouldn’t be there in the first place’ but Yannankakis has found a way to help these veterans. Yet, Startlemart was not the only game about conflict.
Games for Education
Across Europe, around a third of children think bullying in schools is a problem. Bullying is one type of conflict that can scar children for life if it is not tackled properly. Yannakakis worked with other researchers at the University of Malta (Prof. Rilla Khaled and Dr Antonios Liapis) and with institutions around Europe to build a game that could detect conflict in a multi-player game for children called Village Voices.
The game places children in different roles within a village. One child could be a carpenter and another a builder. They have to collaborate on certain jobs, while have the ability to steal and spoil the plans of others. In the background, the game profiles the children and their interactions to enhance the conflict the students experience. The idea is that offline, the children can discuss with a teacher the conflicts they experienced in the game. In this safe environment the children would have learnt how to deal with conflict—a very clever, if unorthodox, approach.
The experts consulted did not know how to modify in-game conflict. The researchers had to use machine learning to create a conflict model. Any model needs data, so they studied several children playing the game and asked them to report how high they thought the conflict was in different situations. Then, ‘statistical processes learnt the relationship […] and in which sequence [to] predict [future] conflict. So you end up with a model you can trust.’ The predictions of conflict reach accuracies of 80%. ‘It’s not great but sufficient.’ The learning process of the machine was so successful that the game won awards.
Another game was developed, using a similar technique, to help dyslexic children between nine and eleven years of age. ‘In iLearnRW, (the project behind the game Words Matter), we are not modelling the player, we are modelling the teacher instead.’ A teacher normally guides a dyslexic child through a set of activities and modifies them to maximise the child’s improvement in reading and writing. Yannakakis’ team (especially Dr Hector P. Martinez) trained ‘an artificial teacher on actual teachers’ [pedagogy]’ by turning nine well-known activities used by learning difficulty specialists into mini-games. The teachers then went through the game with the children they work with, picking out the best activities at specific difficulty levels according to each child’s needs. Repeat this with dozens of teachers working with various children, throw in some artificial intelligence, and you have the best possible teacher model.
Words Matter is the first game built in-house at the University of Malta and designed to be a teacher on a tablet. Not every dyslexic child has access to the best specialists. With Words Matter the teacher is always there. The children have someone recommending what is best for them. The artificial intelligence behind this program creates a profile of the student that the real world teacher can see at any point to monitor the child’s progress. The game can work on its own or with a teacher, and the system is highly accurate: ‘this artificial tutor would be able to predict what [a real world tutor] would say in the eight or even nine out of ten cases, on average’.
“Across Europe, around a third of children think bullying in schools is a problem”
In all these games Yannankakis develops the game module needed to learn how the user plays in order to adapt the game to that player. How is such a high accuracy reached? In Words Matter, set activities were carried out with dyslexic children, however the way they are incorporated is not standardised. ‘Teachers tend to deviate from the book’ to provide the best education to children who might get stuck. The teacher model was developed on the average best deviation the teacher would take in each case. Teachers do disagree on the best approach, but machine learning tools learn to predict a teachers’ choice and find overlaps in teaching practices to predict the best possible approach. In this way, Yannakakis managed to build a highly accurate model. He keeps ‘improving [his] algorithms as the years go by, by testing them on different domains’.
The game has impressed the Maltese Government. It is being incorporated into the One Tablet per Child project, along with other games produced by the Institute. The best part of this serious game is that it is fun. ‘It’s a really cool game because it carefully integrates the key learning objectives within a fun and [Khaled’s] well-designed activity. It is a game that anyone can play whether they have dyslexia or not [….] which is the biggest success of any educational game: to stealthily engage people in learning activities.’ I managed to play it at the Science in the City and Notte Bianca festivals and loved it. The game is inspired by the Mexican Day of the Dead celebration—an eerie and highly artistic vision. Apart from creativity in the games he creates, Yannakakis is trying to help others be creative through games.
Games for Creativity
In 2008, a credit crunch turned into a recession, which transformed into a depression of the World’s economy. Certain countries have bounced back but many Western economies are still struggling. Some think that a lack of creativity has led to sluggish economies that cannot adapt to new realities. Traditional education has no classes in creativity, with children needing to jump hoops to make it through the system until they reach University. Once they graduate, they suddenly have to be creative and ‘think outside the box’ to create a job that they love, is innovative, and helps the country’s economy. Rather than this unrealistic approach, Yannakakis is trying to ‘foster creativity in children with the C2Learn games’.
Within the games developed in the C2Learn project Yannakakis created models of players and measured their creativity. Yannakakis and his team drew inspiration from the philosopher ‘Edward de Bono’s lateral thinking [the UoM has an Institute dedicated to this great thinker] and they have split it into two main dimensions: the semantic (words) and diagrammatic dimensions (pictures).’ They’ve designed games that tackle both aspects. ‘We [consulted] pedagogues, creativity experts and teachers.’ For picture-based creativity they came up with 10–15 basic shapes featured in the game Iconoscope. With squares, circles, triangles, and a few more shapes, hearts, sticks, chains, and other diagrams that convey ideas can be created.
The children have to draw shapes depending on the given concept be it love, freedom, or others. So suppose a classroom is given the word ‘love’ to draw by the AI of the game and everyone draws a heart. The AI would find this uncreative: the class has failed. Students need to be able to convey the message of love [through a diagram] to half of the class and confuse the other half, and that requires creativity. Each child picks an assistant (the Mad Scientist, Typical Tom, Progressive Petra, Wise Oracle, or Chaotic Kate) and each one of these helpers has a particular AI behind it. For instance, the Mad Scientist comes up with very novel graphics. What is novel? [Novel is a diagram] divergent from the current stream of thought’. The other AI assistants suggest pictures ranging from typical love diagrams (Typical Tom) to unorthodox ones (Progressive Petra) or just purely random (Chaotic Kate). The game’s goal is for the children to draw various diagrams and link them to love; for example, an empty chair could represent love lost. What they want to achieve are ‘entirely new associations that have not been historically made by other students or [the player]’
In C2Learn ‘the expert [model] is the actual player’. The AI machine would be creating a profile of every single player and pushing those who are sticking to clichéd diagrams to try out new ideas. ‘AI searches for [diagrams] and sees what the features for love [for example] are, how many colours, shapes, and so on.’ Then if a new student joins the class ‘[the computer] would say “let me give [them] something entirely novel, [different] from whatever has been generated over the last year, something that no other student has thought about under love.” That suggestion could be super amazing or completely uninteresting, but we keep suggesting new diagrams constantly and let the student decide.’ Yet Yannakakis still does not know ‘how well this computational creativity intervention will be for human creativity’. The game is currently being piloted in several countries across Europe.
Apart from developing the creativity of humans, Yannakakis wants computers to be creative. Run a series of algorithms and out pops a new piece of artwork or, in the case of their latest project AutoGameDesign, a new game. ‘That is a project that deals with automatic generation of games that are novel, unexpected, yet playable and fun.’ Yannakakis envisages that one could set a constraint like: “I want a novel first person perspective game about fear which is playable” and the machine then creates such a game.’ Novel here can mean ‘completely different, or somehow atypical from what is already out there, or what [the computer created] in the past’. Through machine learning these computers will be able to invent new games.
AutoGameDesign is a first step towards having computers be creative in their own right. Yannakakis wants machines to ‘surpass [themselves] from what they have designed so far. So you can let [the computer] design, [then it can assess how well it did and how creative it was], and how I, [the computer], can learn from that so that I can develop my artificial brain to create something more complex and unique from what I have been doing.’ Yannakakis then continued to explain the concept of social computational creators. To create a game you could set up machines with different roles. One computer could evaluate and criticise the others, another could be the music designer or level designer, or the perfect game tester. In this way you would have trained the computers ‘to be creative and experts in their own field’. Yannakakis envisages a society of computers creating, then perfecting, that creation.
Terminator, Blade Runner, and The Matrix tend to enter our imagination when we hear these ideas, but Yannakakis is somewhat bored of these discussions. ‘Artificial Intelligence is an overstated term for a very human-centric term for intelligence. [Artifical Intelligence] is not biological intelligence.’ He sees a division of labour between humans and machines. After all, AI machines have already surpassed us in many tasks. No one can beat them at chess, or checkers any more. ‘We shouldn’t necessarily compare ourselves to machines […] Humans and machines have different but particular capacities. There are harder problems for machines to solve than artificial intelligence. A harder problem, for instance, is artificial stupidity as intelligence naturally implies a logical process behind it. Stupidity can break any logical process.’
I, and Yannakakis too, do not think stupidity is the only realm that will be left for humans. But these new creative computers are challenging what creativity is. Also, what does it mean for a human being to be creative? We already use machines to be creative. I am writing this article on a laptop. So is the designer to create the magazine. ‘Creative machines will be a tool to use. It is really up to us to which degree and in which context.’ Some already exist and ‘software tools already assist design.’ Yannakakis thinks that we should exploit machines to do what they are good at, when humans sometimes are limited.
“We already use machines to be creative, I’m writing this article on a laptop as is the designer to create the magazine.”
Yannakakis also reverses the concept of human and artificial intelligence being very different by seeing their similarity. ‘Computational creativity formalises aspects of creativity […] that could give us answers about human creativity. The particular way that machines operate and create could be relevant to how we operate and create.’ We behave like machines. ‘We sometimes follow machine processes and algorithmic steps and are very creative.’ So I write like a computer, or the computer writes like me.
Could a computer become the editor of Think magazine? Yannakakis thinks that one day it might, but it depends on a lot of factors. Human intelligence and artificial intelligence could be considered separately and the question I just posed is maybe unimportant to ask. What we should be doing is building computers that are better than us at completing certain creative tasks so we can focus on becoming even more creative on other levels. ‘The machine could handle the level of detail you don’t want to bother with and machines can be wonderful for concept generation or “what if” scenarios. Because they can rapidly search in huge databases and come up with really unexpected combinations of things and can—if you pose the right question—make you more creative.’ Now all I need to do is convince Yannakakis to create an Edward-adapted writing module and perhaps turn me into a mathematical function in the name of creativity. •
Further Reading:
- Drachen, A., et al. ‘Player Modeling using self-organization in Tomb Raider: Underworld. Computational Intelligence and Games, 2009. CIG 2009. IEEE Symposium on, 2009. IEEE, 1–8.
- Liapis, A., Yannakakis, G. N. & Togelius, J. Computational game creativity. Proceedings of the Fifth International Conference on Computational Creativity, 2014. 285–292.
- Lopes, P., Liapis, A. & Yannakakis, G. N. ‘The C 2 create authoring tool: Fostering creativity via game asset creation.’ Computational Intelligence and Games (CIG), 2014 IEEE Conference on, 2014. IEEE, 1–2.
- Togelius, J., et al. (2011). ‘Search-Based Procedural Content Generation: A Taxonomy and Survey’. IEEE Trans. Comput. Intell. AI Games, 3(3), pp.172–186.
- Yannakakis, G. and Togelius, J. (2011). ‘Experience-Driven Procedural Content Generation’. IEEE Trans. Affective Comput., 2(3), pp.147–161.
Comments are closed for this article!