I am new to machine learning! I need a way to generate vector image out of hand drawn sketch. I dont need to trace bitmap like it is usally done because it gives you exactly what you drawn. I need to generate "simplified" drawing. eg. When human hand draws something with strait lines, the lines are not true. When a human writes a text I want it to be recognised as such and converted to some existing font that best matches the text.

I would like somone to just guide mo to where and about what should I start research!

Thanks very much !

enter image description here

  • 1
    $\begingroup$ Optical character recognition $\endgroup$
    – Henry
    Commented Feb 13, 2019 at 16:54
  • $\begingroup$ arxiv.org/abs/1705.07962 $\endgroup$
    – shimao
    Commented Feb 13, 2019 at 19:07

2 Answers 2


It depends a little on the exact problem.

If you are interested only in text, then two fields come to mind:

Again in this case one would simply detect the text, classify it, and then replace it with a vector version (discarding the rest of the image, or e.g. blending it).

Things become tougher when you have general vocabularies of discrete objects. For instance, I noticed the box in your example becomes a nice straight box. How should this be done? Should it detect there is a box, and then figure out what size it should be? Or should it detect four lines, and separately compute their lengths? This is a non-trivial problem, but there are numerous ways to approach it (some rather effective):

Overall, due to the requirement for differentiability in deep learning, handling discreteness is challenging. One can use techniques from reinforcement learning to circumvent this, since the likelihood ratio (i.e., the REINFORCE estimator) can compute gradient estimations in very general scenarios. In other words, you can set up your problem as a deep RL problem, where the agent gets a reward for reproducing your target image using choices from a vector vocabulary. Papers like Tucker et al, REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models (as well as those papers cited by/citing it) might be a good place to start learning about that area.

Hopefully that's a useful starting point :)


In addition to user3658307's comprehensive answer I'd like to mention Generative Adversarial Networks (image to image translation GANs specifically) - they should come in handy if you can give them enough training data. The advantage over OCR-based methods would be simplicity - you don't need to build any pipeline, the method learns the transformation end-to-end.

Image to Image translation aims at recovering transform that maps images from one domain to second one - this looks exactly like your problem - in fact, I think it's not a difficult one (other examples cover generating images of cats/items from sketches, which seems harder). I encourage you to visit the method authors website. It contains many examples as well as links to implementations.

  • $\begingroup$ +1 good point, im2im like cyclegan could work. Some upsides being paired images are not needed for training, no need for detection as you said, etc... Only downside is these techniques in their standard form will still be raster output (without significant post-processing to get vector output). $\endgroup$ Commented Apr 28, 2019 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.