Is it possible to train neural network to draw picture in certain style? (So it takes an image and redraws it in a style it was trained for.)

Is there any approved technology for such kind of a thing? I know about DeepArt algorithm. It is good to fill main image with certain pattern (for example, vangoghify image), but I am looking for something different - i.e., for example, making cartoon in a certain style from the input portrait.

  • 3
    $\begingroup$ One obstacle to training a photograph-to-cartoon neural net may be finding a training dataset. It seems like the dataset would have to contain photographs, and cartoons that humans have drawn based on those photographs. I'm not aware of any such datasets. $\endgroup$ May 31 '16 at 23:25
  • $\begingroup$ @TannerSwett How do you think how much images is necessary for such kind of a training? $\endgroup$
    – zavg
    Jun 2 '16 at 1:23
  • $\begingroup$ I'm no expert, so I can only make a wild guess. I think you would need at least a thousand images. You might need a lot more than that. By the way, I suggest looking at this tool: github.com/hardmaru/sketch-rnn That tool has been used to generate imitations of Chinese characters; maybe a similar tool could generate imitations of cartoons. $\endgroup$ Jun 3 '16 at 2:07
  • $\begingroup$ I may be a little out of date as my NN training was some time ago but if you are thinking of just training a network with a few thousand images and expecting it to be able to render pictures in a style you may be reaching too far - if you are thinking that this is a good starter project then don't. To achieve what you describe would require a 'lot' of manual fiddling. Think about the knowledge required to interpret an image and not just pattern match. $\endgroup$ Jun 3 '16 at 14:09
  • $\begingroup$ Also worth looking at vox.com/2016/6/1/11787262/blade-runner-neural-network-encoding $\endgroup$ Jun 6 '16 at 14:51

There is a relevant paper: LA Gatus, AS Ecker, M Bethge, 2015, A Neural Algorithm of Artistic Style. Quoting from the abstract,

Here we introduce an artificial system based on a Deep Neural Network that creates artistic images of high perceptual quality. The system uses neural representations to separate and recombine content and style of arbitrary images, providing a neural algorithm for the creation of artistic images.

Here is Figure 2 from this paper:

enter image description here

There is also a very popular open-source implementation based on torch here which is quite easy to use. See the link for more examples.

Keep in mind, that the computations are heavy and therefore the processing of single images is the scope of this work.

Edit: after checking your mentioned DeepArt project, it seems it is using the same techniques. I'm not sure why this is not what you want, because the concept of style-transfer is as general as it gets.

  • 4
    $\begingroup$ Cartoons exaggerate and simplify the features of whatever it is they're depicting, producing shapes that are very much unlike the shapes in a photograph. I don't believe that existing style transfer neural nets have ever done that. $\endgroup$ May 31 '16 at 23:21
  • $\begingroup$ @TannerSwett Look at the examples here: imgur.com/a/ue6ap. Some of them are quite cartoon-ish. $\endgroup$
    – amoeba
    Jun 3 '16 at 1:03
  • $\begingroup$ @amoeba They do look cartoonish, yes, but none of them have the distorted proportions that most real cartoons have. $\endgroup$ Jun 3 '16 at 2:03
  • $\begingroup$ We are talking about style-transfer here. All the examples use some a-priori chosen style, where non of these have been cartoonish (why should the output look cartoonish, when it learned on Van Gogh). Maybe you could just get the open-source project to run (i had no problems in the past) and try it with cartoon-input. $\endgroup$
    – sascha
    Jun 3 '16 at 9:57

This is a pretty difficult problem to solve. You can see some examples here on how a cartoon style, e.g. from the Simpson's has been applied to an image.

A cartoon image generally doesn't have the structure that gives this artsy effect. The easiest way to try to apply this in some way would be to have a face-tracker, and then try to align two faces, e.g. a cartoon face and a human face, and then apply this. That might get you somewhere, but it might also look weird. You might then annotate landmarks in the images to help further and do a non-rigid registration before this. This is still somewhat a shitmix solution, but the closest I can think of that could work for faces.


The comment by @TannerSwett adds something to this, it is potential to go onto some artists webpages and try to find their illustrations and try to learn "their" style. I still do not think that will satisfactory or yield enough data, but that would be an interesting thing to test. There is no generally available solution right now, but I think that are definitely some people working on this, and we will see better results soon.

I think that maybe the way to go is not the artistic neural network approach. Maybe it is better to have a network that can classify objects in an image and then learn the correspondences between the objects and their cartoon counterparts, then blend the results in some meaningful way.

  • 1
    $\begingroup$ That Simpson example looks pretty awesome to me @Gumeo $\endgroup$
    – FabricioG
    Oct 26 '18 at 5:40

It shouldn't be too complicated to do. Haven't read the article mentioned, here's my recipe:

Variational Auto Encoders

Online demo with morphing faces: http://vdumoulin.github.io/morphing_faces/online_demo.html

and https://jmetzen.github.io/2015-11-27/vae.html for teh codez.

Basically, this gives you a way to parametrize the 'style' in your case, for example let's say how wide or fuzzy should the brush stroke be. Stuff that depends on the particular style you are trying to emulate.

In the example above different 'morphed' or 'imagined' faces are a function of the parameters in the latent space. In the image below that would be what you get by changing stuff at the 'code' level.

Here's the basic idea: original image left, stylised version of the same image on the right:

enter image description here

Now, in theory, if you would train such a model on a normal image and a stylised image as a target and add convolutions, you should be able to learn the kernel filters that correspond to the type of "brush strokes" that the artist uses.

Of course, that means that you need to have a few examples of images in both original and stylized versions. Such a dataset would be nice to donate to the community - if you end up doing this I'd be very keen to see this sort of work.

Good luck!

The wiki article on auto encoders would be a good starting point: https://en.wikipedia.org/wiki/Autoencoder


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.