3
$\begingroup$

Whilst reading up on the Deep learning literature, I noticed that a few variations on the standard network structure that were created specifically to better model "Natural/Real Images". For example, this paper says that:

Deep belief nets have been successful in modeling handwritten characters, but it has proved more difficult to apply them to real images. ... [Our network] can better model the covariance structure of natural images.

The paper seems to imply that only real or natural images have a rich local covariance structure. If that's the case, then would a screenshot of a videogame count as a natural image? A digital painting? and in any case in there an algorithmic way to test for this "Naturalness"?

This may sound open-ended, so let eplicitly operationalize it: when given a training set of 2D pixel arrays, how to do determine if you should use a standard network structure or a structure intended for natural/real images?

$\endgroup$
6
$\begingroup$

I suspect it depends on the context.

In the example you gave, they are contrasting handwritten characters with photographs. Both obviously have covariance structure. However, there are a finite number of letters (26, 52, etc) and the covariance structure of a letter is, by convention, pretty tightly constrained. The number of possible photographs is much larger than 26, and there are considerably weaker constraints on the structure of the covariance matrix. In this case, I think the authors are suggesting that belief nets work well on a tightly constrained problem (map pixels onto [a-zA-Z]), but their method is superior when there is less a priori knowledge about the images.

In other cases, "natural images" is essentially used as shorthand for "images which have a rich local covariance structure." This is particularly true in visual neuroscience, where the contrast is often between simple parametric stimuli (e.g., sine waves) and white noise, which have somewhat boring covariance structure, and movies/photographs, which don't. Under this definition, I think it'd be completely reasonable to call paintings, screenshots, and the like "natural." I have noticed that people often hedge and call them "naturalistic" instead of "natural", but I think the point stands.

People have compared the characteristics of these naturalistic scenes to arbitrary visual stimuli. There's a good review on natural scene statistics by Geisler (2008). For example, the power spectra of natural(istic) scenes has a stereotypical 1/f shape. You could use that criteria to distinguish between arbitrary input and "naturalistic" input. There has been a lot of interest in whether/how the visual system has evolved to represent naturalistic input; I'd be happy to provide you with more pointers if you're interested.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.