The Painting Fool - Machine Intelligence Competition

The Machine Intelligence Competition 2008

Video Footage of Our Demonstration - Transcript

Simon Colton:

Thanks very much Max. And thanks to Richard and John and Sue for organising the competition and for inviting us along here.

So, we're going to be looking today at painting. So, I'm Simon Colton and this is Michel Valstar. Our third member of the team [Maja Pantic] couldn't make it, as unfortunately she is ill today.

Painting is an intelligent task, and we are interested in simulating the painting process, rather than [just] the results of the painting process - which is what you get from Photoshop, and we're not trying to reproduce that.

One aspect of the painting process is appreciation. And there is appreciation both of the subject matter (i.e., what you are being asked to paint), and there is appreciation of how your art materials and you colour palette and your painting styles can affect the mood, the atmosphere and the emotion being expressed in the painting.

So today we are bringing together a combination of two systems from Imperial. Firstly, there is the system that Michel and colleagues in Maja Pantic's machine vision group, that is going to be given a video sequence taken live from the video camera here. And in that video sequence, we are going to ask someone to express an emotion. Then their software is going to take the video and it's going to do two things. First of all, it's going to detect that emotion. And it's also going to detect where the features are, i.e., where the eyes, and the mouth of the subject are.

Then they are going to pass that to my software, which is called The Painting Fool. And The Painting Fool is going to use this information about where the eyes and the mouth are, and what emotion is being expressed, and it's going to choose its art materials accordingly, it's going to choose its colour palette accordingly, it's going to exaggerate some of the features, and it's going to alter the painting style to fit the emotion.

And we're hoping today to show that there has been an advance in machine intelligence because software is now able to appreciate what it is painting and how it is painting it in computer graphics.

OK, so we're going to do two demos. Firstly, Michel is going to show us how to do it. Then we have Paulo - who volunteered earlier to do this. And hopefully, we'll get two portraits which match the emotion that has been expressed in the video.

So... relax... go.

(Michel shows disgust into the camera at this stage).

OK. Great.

Michel Valstar:

Yes. So that expresses my current emotion! So, at this point, our program finds a number of facial features such as the mouth, eyes, the eyebrows, the nose and the chin. These are very important in the next stage - in the painting. Based on the motion of these points, we then decide what was the emotion that was shown. And we will then select the image that showed that emotion the most strongly, and pass it on to The Painting Fool, together with the location of the facial features, along with the emotion that we thought was there.

This will create a picture which depends on the emotion being shown.

Simon Colton:

This is always the scary bit - whether it has got the emotion correct. OK. So, The Painting Fool has taken note of that and I'll read out [what it says].

The Painting Fool says that the person is disgusted, which is good. So, he's going to stretch out the face, like Edvard Munch did in The Scream. He's going to use low-saturation colours in his palette. So, we're talking about greys, greens and sort of mottly browns, to emphasise the disgust being shown on the face. He's going to simulate acrylic paints. He can also simulate pastels and chalks and pencils according to how he wants to paint at the moment. He's going to use quite a messy painting style, and this one [painting] may look like the sitter, i.e., it will be a good representation of him, or a half good representation - but we're never quite sure.

So, it should start any second now...

honestly...

OK. So, I tend not to talk over the top of The Painting Fool. I just let it do its thing really. It will take about two minutes.

So, you can see that it has indeed chosen sort of mottley greys and browns, and colours you might expect a [rotten] piece of fruit to look like. So, we're looking at low saturation [colours] here. And it is simulating acrylics with quite fluid painting strokes.

Depending on the lighting conditions, it might choose to paint the whole thing grey! If it does, I'll put it down to artistic license and move on...

So, you might be able to see the outlines of the face. You can see in the top left hand corner the zoomed out shot and you can see details here.

So, there is a bit of the Rolf Harris about this, I'm afraid. It's a matter of wait and see. Sometimes I get it to download images from the internet, so I don't know what the underlying picture [photo] is, so it's even more exciting to see what it's doing.

There's an eye socket there, I think. Oh, here we go. It's painting the eyes now. You'll notice how it's found the eyes thanks to the vision software, and it's painting them in, in fairly acute detail, to try and capture the likeness. Sometimes it captures a better likeness, other times it doesn't.

It's managed to miss out an entire section of the face, but again I'll put that down to artistic license. Only about 10 or 20 seconds more to get the finished product.

It helps if you stand back, so people at the back might, er...

Remember it's a stretched out face, like The Scream was, so we're looking for elongated features.

The question I guess you should be asking is: is this more disgusting than the original photo?

So, it's finishing off the mouth.

OK. It's nearly there. To my mind, you can clearly see disgust being expressed. It does look a bit like Michel, but then I know his face better than most, as I've been looking at his face for the last couple of weeks, and nothing else - preparing for this demo.

And we're done.

So, without any further ado, we'll get on with the next demo.

Michel Valstar:

So, if you'll allow me a few seconds to get the camera ready.

So, when I say yes, go.

Yes. Thanks.

OK. Well, let's see with the [camera] flashes. It works well with all sorts of light conditions, except for time-varying light conditions. But then again, we'll have to see.

So, again at this point, we're finding the points. And they will be tracked through time. See [the light flashes]. Ah, well it copes very well. So, you can see that the green box is trying to track the points. If they are really big circles, this means that it's not exactly certain where it is, and it is looking around. So, at this point, you can see where the flashes start - you get big elongations of the circles. But, it is working quite well, so thank you [Paulo] for the robustness test!

Simon Colton:

Aaaah. So, it's thought that this person is angry. Ordinarily, it would use very happy colours, but [instead] it's going to use green and reds for this, and because we've only got a few minutes to paint this, it's going to do it very sketchily, and it's going to make the eyes quite distinctive. And it's going to use a very abstract overall image, with very distinctive eyes.

This isn't one of the best of The Painting Fool's repertoire of expressions, but at least it's quite different to the one you have previously seen.

So, right now, it's going around the outlines of the face. Basically, think of The Painting Fool as an expert system that has got some skill. So, it's an expert system which is able to take information and turn it into a painting style. So, in that sense, an expert system - obviously slapped onto the end of a non-photo-realistic rendering system to render the artwork.

It seems to have given him big owl eyes there!

I'm going to very quickly go over a couple of slides to show you the repertoire [that] The Painting Fool has available. And, we'll come back to this [painting] in a second, and see what the results are.

(Slide 1)
So, very quickly, This is me yesterday - looking pretty sad. So, this is if sadness was the emotion that was expressed. You can see there it's pastels and muted colours to highlight the sadness there.

(Slide 2)
This is myself, Maja and Michel - after the system didn't work - looking disgusted.

(Slide 3)
Surprise. We really liked this one. You can just about see the surprise in his eyes there. It's a very distinctive style that The Painting Fool has used on the back of the vision software.

(Slide 4)
This was fear. Maybe not so fearful, but we've got a ghostly effect using chalks.

(Slide 5)
Maja was very keen on the anger [paintings]. These are like the one you are going to see in a second. But hopefully it will look a bit more like Paulo than Maja.

(Slide 6)
And this is hte one we were hoping would come out. This is Michel looking very happy. Very slapdash, wacky colours. It doesn't look exactly like the sitter, but it's certainly vibrant.

(Slide 7)
Just to show you that we can get more realistic images - that's Michel from yesterday. It took about 15 minutes to produce this, because it's [been] carefully hatching pencil marks here. So, we couldn't do this in the demo. But, we are able to produce nice images.

(Slide 8)
Often, we can produce more flurried images. That's Michel painted again, with a smile on his face.

(Slide 9)
That's me at 3am this morning - doing these slides. We've used watercolours there.

(Slide 10)
If you want to know more about The Painting Fool side of things - and we're going to put details of the vision software on there hopefully - then please go to thepaintingfool.com.

(Slide 11)
To give you an indication of the range of styles, I'm going to quickly whip through some of these.

(Slide 12)
OK, so it goes from fairly representational like the ones here. These are of Audrey Tatou. If you go to this website, there are 222 paintings of the same woman, which is a little bit obsessive I guess, but it helped me train the software in producing emotionally aware paintings.

(Slide 13)
There is a whole range of techniques it can use, going all the way to the very abstract.

(Slide 14)
The next thing for The Painting Fool is to produce its own scenes and paint them - to make it appear more imaginative.

(Slide 15)
So, I'm going to thank you for your time and attention and show you what the resulting image was.

OK. It's fairly angry, but not as much as we are with the software for not getting it right.

So, we'll leave it there. Thank you very much.

Max Bramer:

Not something you see everyday, really.

If you would like to talk to Simon Colton about this, please see here >>

The Painting Fool. © Machine Creations Ltd.
A computer program that aspires to be an artist.
I sell prints and take commissions.