Definition of "Natural Images" in the context of machine learning Whilst reading up on the Deep learning literature, I noticed that a few variations on the standard network structure that were created specifically to better model "Natural/Real Images". For example, this paper says that:

Deep belief nets have been successful in modeling handwritten
  characters, but it has proved more difﬁcult to apply them to real
  images. ... [Our network] can better model the covariance structure
  of natural images.

The paper seems to imply that only real or natural images have a rich local covariance structure. If that's the case, then would a screenshot of a videogame count as a natural image? A digital painting? and in any case in there an algorithmic way to test for this "Naturalness"?
This may sound open-ended, so let eplicitly operationalize it: when given a training set of 2D pixel arrays, how to do determine if you should use a standard network structure or a structure intended for natural/real images?
 A: I suspect it depends on the context.
In the example you gave, they are contrasting handwritten characters with photographs. Both obviously have covariance structure. However, there are a finite number of letters (26, 52, etc) and the covariance structure of a letter is, by convention, pretty tightly constrained. The number of possible photographs is much larger than 26, and there are considerably weaker constraints on the structure of the covariance matrix. In this case, I think the authors are suggesting that belief nets work well on a tightly constrained problem (map pixels onto [a-zA-Z]), but their method is superior when there is less a priori knowledge about the images.
In other cases, "natural images" is essentially used as shorthand for "images which have a rich local covariance structure." This is particularly true in visual neuroscience, where the contrast is often between simple parametric stimuli (e.g., sine waves or white noise), which have somewhat boring covariance structure, and movies/photographs, which don't. Under this definition, I think it'd be completely reasonable to call paintings, video game screenshots, and the like "natural." I have noticed that people often hedge and call them "naturalistic" instead of "natural", but I think the point stands.
People have compared the characteristics of these naturalistic scenes to arbitrary visual stimuli. There's a good review on natural scene statistics by Geisler (2008). For example, the power spectra of natural(istic) scenes has a stereotypical 1/f shape. You could use that criteria to distinguish between arbitrary input and "naturalistic" input. There has been a lot of interest in whether/how the visual system has evolved to represent naturalistic input; I'd be happy to provide you with more pointers if you're interested.

Geisler, W. S. (2008). Visual perception and the statistical properties of natural scenes. Annu. Rev. Psychol., 59, 167-192.
