# Training an Object Detection Model Using with Artificial Data from Video Games

I had an interesting idea of using artificial data gathered from screen shots of a high-resolution video game as a cheap substitute for labeled real data, which can be quite expensive or difficult to obtain. I've seen variants of this idea in the literature, for example this paper used screenshots from Grant Theft Auto V to rapidly generate semantic label maps for images. I'm specifically interested in using the video-game frames for use in object detection using modern approaches such as YOLO or Detectron.

For people with more computer vision experience and intuition than I have, should we expect that models trained on video game data (like GTA V) should be able to perform well when evaluated on a dataset of real images? Is there an obvious reason why this shouldn't be expected to work? This seems like an easy way to quickly generate a lot of data of reasonable quality.

• what are the classes you are trying to predict ? is it cars bikes or people ? Aug 21 '19 at 4:21
• If you want to generate images, look into generative adversarial networks. NVIDIA has a cool video of fake faces: m.youtube.com/watch?v=2edOMMREazo.
– Dave
May 30 '20 at 21:07

Basically, if you train on one dataset ($$P_D$$, a probability distribution on the space of images) and test on another ($$P_T$$), then the usefulness of doing so depends in part on the overlap (or divergence) between $$P_D$$ and $$P_T$$. (e.g., see here). Basically, training on images of bananas won't help you differentiate apples and oranges. In this case, the overlap is considerable (since videogames are, at least sometimes, indeed designed to mimic real images). The issue is that your network may learn to rely on spurious features/correlations that are present only in the simulated data (this failing when they are missing in the real data) or be thrown off by the differences in the real data (e.g., by reflections or lighting changes that don't occur in games).