1
$\begingroup$

The size of training data is critical for deep learning. For image classification, I saw people making arbitrary training images to improve the accuracy.

Is it possible to generate arbitrary training data for document classification problems? For images, people can rotate, shift the images. So how can I manipulate the existing labeled training data, to get more data?

$\endgroup$
  • $\begingroup$ This is called "data augmentation." It's definitely not "arbitrary fake training data," which would mean it could be whatever you felt like, e.g. taking a pure-black image and labeling it an ostrich. $\endgroup$ – Dougal Apr 8 '17 at 1:13
3
$\begingroup$

Generating synthetic data for neural-network has indeed become a common practice lately, especially in the field of computer vision. However, it's important to differentiate between the practice (creating synthetic images) and the purpose, which can differ tremendously.

Let's consider the task of face recognition using a deep neural network.

Then two example purposes ("use-cases", if you wish) for this case are:

  1. robustness: For example, you'd want the network to recognize Brad Pit if he's staring straight to the camera, but also if the image is flipped, or rotated 180 degrees. In order to "help" the network learn that a face is a face even though it's rotated, a common practice is to manually introduce rotated faces to the training set. This is what is known as "data augmentation", and it's very common since it's essentially free - all you need to do is apply a simple pre-processing stage.
  2. accuracy: In some scenarios, getting labeled data is incredibly hard or time consuming. For example, in order to train a network to perform face recognition well enough, an order of at least half a million labeled faces are required (Google and Facebook, which have by far the best products, use order of millions of images). Without this order of data, any classifier will pay in accuracy. Note that generating new labeled faces is a task far more difficult than just flipping your training images. However, there have been some very nice results in this field using CG tools. For example, in "Frankenstein: Learning Deep Face Representations using Small Data" the authors suggest a very nice pipeline for generating new face images, thus reaching STOA performance with as few as 10,000 training images.

Now, let's move away from vision to the field you've asked about, NLP.

Generally in text, the practice of generating synthetic data is much less evolved. One reason I can point to relates to my comment in the beginning regarding the "why" factor. For example, in document classification, most methods common today are unsupervised, thus they don't require labeled data; un-labled data is so abundant, that trying to synthetically generate data really doesn't make sense.

Of course, I'm not saying this can't at all be useful for NLP. There are supervised algorithms which could potentially benefit from a stage of "pre-training" on some artificially large corpus before "fine-tuning" them on a smaller amount of real data. But again, this will really depend on your application and what you're trying to achieve. Simply "getting more data" isn't a goal in itself, not even in deep-learning :)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.