24
$\begingroup$

Can a generative adversarial network (GAN) be used for data augmentation (i.e. to generate synthetic examples that are added to a dataset)? Would it have any impact on the performance of a model trained on the augmented dataset?

$\endgroup$
3
  • 3
    $\begingroup$ Just from a theoretical perspective alone, this cannot be possible. If a GAN is trained on a given dataset, it can only learn the information represented in that dataset. If you then use this GAN to generate new data, it will generate data from the same space that the original data is in. By training a GAN, you're not adding any new information to the dataset, so naturally the GAN cannot produce data from a larger space than the space of the original dataset. It is thus pointless to try to generate new training data with a GAN, because this synthetic data will not contain any new information. $\endgroup$
    – Alex
    Aug 30, 2018 at 21:33
  • 5
    $\begingroup$ I don't understand why this question was closed. The question is very interesting indeed. I'd like to add a link to a study we performed regarding this issue, if anyone is interested. The relevant code can be found here. $\endgroup$
    – Djib2011
    Sep 16, 2019 at 9:59
  • $\begingroup$ I agree. thanks for re-opening. Could you describe your study as an answer? $\endgroup$ Sep 16, 2019 at 14:13

7 Answers 7

17
$\begingroup$

Yes, GAN can be used to "hallucinate" additional data as a form of data augmentation.

See these papers which do pretty much what you are asking:

If your GAN is sufficiently well trained, there's no reason why this shouldn't help improve model performance. If your GAN is bad, you'll get garbage.

$\endgroup$
4
$\begingroup$

If you train your GAN on dataset A and use it to augment data on B, I think the answer is yes, since it absorb some knowledge from A. If you train your data on B and try to augment on B, I think GAN is useless here because there is no gain in information.

$\endgroup$
3
$\begingroup$

As already mentioned, it depends on what data you train your GAN. But it also depends on what you expect as an outcome of the GAN. Most methods focus on complete new synthetic data, but that's not the only option. This approach of landing AI seems to be more promising than just generating new artificial data, they augment existing data using a GAN to enrich it with rare events. This can help to generate data for classes that are underrepresented in the training data or could help to mitigate bias in some cases.

$\endgroup$
3
$\begingroup$

(My answer will mostly focus on tabular data as it has proven the hardest to synthesize owning to its heterogeneity and general arbitrariness)

Yes, we can get a generative adversarial network (GAN) to generate synthetic data. An exceptional resource on this is: the Synthetic Data Vault initiative where a number of different approaches regarding synthetic data generation are included. As you correctly assess, GANs can be used for synthetic data generation, a number of approaches are implemented in the accompanying sdv package. I will note here that actually variational auto-encoders (VAEs) seem to be a very competitive alternative to GANs for this task. The last couple of years there have been quite a good papers on the subject, some obvious picks would be:

From personal experience: Synthetic data generation can be exceptionally hard to get "arbitrarily correct" but it can be possible to get reasonable mileage out it. For example: Making a synthetic dataset where a classifier cannot distinguish between original and synthetic data is very hard. Making a synthetic dataset to pass to collaborators for them to train some classifier such that it performs adequately in unseen real data is just hard. Making a synthetic dataset to pass to a collaborator to make some EDA and presentation material is usually doable. As such we need to consider why we need the synthetic data to begin with:

  • is it for a privacy preserving task?
  • is it a data-scarcity issue?
  • is it for model-testing requirements? (common in financial applications - eg. stress testing in banking)

(Other, more exotic use cases, also exist: complicated data-sharing agreements, adversarial learning applications, agent-based modelling, etc. but they are rarer beasts.)

These points are not particular to tabular data but rather extend to all types of data. For example, one healthcare trust might want/have to avoid sharing real patient MRI scans (imaging data / privacy preserving scenario), similarly a tech start-up might need to train its voice recognition software on multiple speakers (speech data / data-scarcity scenario). Notably even GANs models with strong theoretical foundations (Wassenstein GAN (2017) Arjovsky et al.) might not be fully appropriate, for example financial time-series data have long-term correlations, volatility clustering and asymmetries (e.g. many small positive moves but few large negative ones) that we require us to use specialised metrics to assess their representativeness.

To recap: using deep generative models (GANs, VAEs, EBMs) for synthetic data generation is an extremely fruitful area of ML research. They are already proofs of concepts from research groups as well as some early industrial products; their usefulness will vary wildly among different applications both in terms of data types (images, speech, tabular, etc.) as well as application fields (healthcare, retail, etc.). Therefore we have no established way to generally quantify their impact on the performance of a model. That said, we will have a clearer view in a couple of years; GDPR/CCPA are powerful motivators for certain multi-billion dollar companies to convince legislators about the benignity of their data usage - synthetic data generation is a piece of that puzzle.

$\endgroup$
8
  • $\begingroup$ Did you utilize sdv for augmentation? If yes, had it any performance gain? $\endgroup$
    – keramat
    Feb 1, 2022 at 6:49
  • 1
    $\begingroup$ Yes, I have used it. No, I didn't see any performance gains per se, I used it for model validation tasks. Generally speaking, I found VAEs to be "better" than GANs in a variety of situations (EDA, model validation, pre-training, naive similarity metrics) but maybe that was application dependent. $\endgroup$
    – usεr11852
    Feb 1, 2022 at 9:57
  • $\begingroup$ Thanks for your helpful reply. Did you find performance gain with syntactical sample augmentation with VAEs? $\endgroup$
    – keramat
    Feb 1, 2022 at 10:01
  • $\begingroup$ Yes, sdv is quite easy to use. $\endgroup$
    – usεr11852
    Feb 1, 2022 at 10:21
  • 1
    $\begingroup$ sdv has GAN- as well as VAE-based models. But yeah, you can always code your own VAE and go forward with that. $\endgroup$
    – usεr11852
    Feb 1, 2022 at 10:28
2
$\begingroup$

After long time, I would conclude the answer is no, based on some quite solid theoretical basis https://en.wikipedia.org/wiki/Data_processing_inequality

$\endgroup$
6
  • 1
    $\begingroup$ Could you edit your answer to clarify the connection between the data processing inequality and the application of GANs that OP writes about? $\endgroup$
    – Sycorax
    Dec 23, 2021 at 0:19
  • $\begingroup$ the OP is me :) $\endgroup$ Dec 23, 2021 at 9:31
  • $\begingroup$ All the same, how does this answer the question? $\endgroup$
    – Sycorax
    Dec 23, 2021 at 10:15
  • 1
    $\begingroup$ To follow-up a bit on @Sycorax 's ask for clarifications; I think this is not a correct rationale. We are not post-processing anything. GANs are generative models, they do not transform the existing data, they generate new data based on existing data. $\endgroup$
    – usεr11852
    Dec 23, 2021 at 16:45
  • $\begingroup$ @usεr11852 - the post-processing occurs when training the GAN. The GAN's weights (and, therefore, any images generated by the GAN) can only contain information about the training data because that's the only data it's ever seen. $\endgroup$
    – airalcorn2
    Aug 31, 2022 at 9:44
2
$\begingroup$

For visual tasks, data augmentation can often be accomplished by rotation, scaling, or rearranging patches. These transformations do not necessarily add information, but can be useful for models to learn to generalize better.

The generator in a GAN learns a complex distribution from its training data from which you can sample, and you can view these new samples as another type of transformation of the original data, akin to rotation, scaling, etc.

$\endgroup$
1
  • 2
    $\begingroup$ The GAN's weights (and, therefore, any images generated by the GAN) can only contain information about the training data because that's the only data it's ever seen. In contrast, rotation/scaling/etc. are human-defined transformations, i.e., they add new information to the dataset by injecting human knowledge acquired from images not included in the training set. $\endgroup$
    – airalcorn2
    Aug 31, 2022 at 9:50
1
$\begingroup$

2023: debates goes on https://arxiv.org/abs/2305.17493v2

We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.