I had an interesting idea of using artificial data gathered from screen shots of a high-resolution video game as a cheap substitute for labeled real data, which can be quite expensive or difficult to obtain. I've seen variants of this idea in the literature, for example this paper used screenshots from Grant Theft Auto V to rapidly generate semantic label maps for images. I'm specifically interested in using the video-game frames for use in object detection using modern approaches such as YOLO or Detectron.

For people with more computer vision experience and intuition than I have, should we expect that models trained on video game data (like GTA V) should be able to perform well when evaluated on a dataset of real images? Is there an obvious reason why this shouldn't be expected to work? This seems like an easy way to quickly generate a lot of data of reasonable quality.

  • $\begingroup$ what are the classes you are trying to predict ? is it cars bikes or people ? $\endgroup$ Aug 21 '19 at 4:21
  • $\begingroup$ If you want to generate images, look into generative adversarial networks. NVIDIA has a cool video of fake faces: m.youtube.com/watch?v=2edOMMREazo. $\endgroup$
    – Dave
    May 30 '20 at 21:07

There's quite a bit of research in this area. Keywords to use include domain adaptation and domain transfer. Also related: domain translation and domain randomization.

Another keyword is sim2real (using simulations to do learning for real world tasks), which is how it is more commonly called in robotics, rather than computer vision. There are some references here.

Basically, if you train on one dataset ($P_D$, a probability distribution on the space of images) and test on another ($P_T$), then the usefulness of doing so depends in part on the overlap (or divergence) between $P_D$ and $P_T$. (e.g., see here). Basically, training on images of bananas won't help you differentiate apples and oranges. In this case, the overlap is considerable (since videogames are, at least sometimes, indeed designed to mimic real images). The issue is that your network may learn to rely on spurious features/correlations that are present only in the simulated data (this failing when they are missing in the real data) or be thrown off by the differences in the real data (e.g., by reflections or lighting changes that don't occur in games).

Personally, I suspect it would (or at least can) be useful here, especially if you have at least some real data with which to fine-tune the network (meaning, first train on the game images, then fine-tune to convergence with only the real images - this is a seemingly common approach, which is a form of transfer learning). Notice that this is similar to how people often start with a network pretrained on ImageNet, even if their actual task is different and their dataset is quite different from ImageNet. The hope is that the features learned are sufficiently generic that they are useful across multiple tasks and datasets.

The idea is for why this works is indeed similar to transfer learning. It is also why data augmentation is often useful, or methods like mixup. Essentially, if there are transformations that you know the network should learn invariance to, then adding transformed versions of the data to your dataset is essentially getting useful data for free (and at least in theory, more novel data means better performance). It helps in generalization because it expands the volume of the dataspace that the network has access to in training, thus smoothing our network's behaviour over it. We just have to be careful not to overfit, especially to areas of the data-space that are far from the test set (as the game images will be compared to real training images).

TL;DR: it will probably not work very well if you don't train with any real images, due to domain mismatch. However, if you use the simulated images as a form of data augmentation, treat this as a transfer learning approach, or at least finetune on real data, then I would expect it to be useful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.