29
$\begingroup$

I'm aware that there's been lots of advances with regards to image recognition, image classification, etc with deep, convolutional neural nets.

But if I train a net on, say, PNG images, will it only work for images so encoded? What other image properties affect this? (alpha channel, interlacing, resolution, etc?)

$\endgroup$
2
  • 3
    $\begingroup$ I don't have enough experience with the statistics of images to give a confident answer, but I would expect that different models use completely different schemes to code images into features, some of which will be affected by lossy JPEG compression, alpha channels, etc., and some of which won't, with the details being model-specific. I know that there exist both models that treat each pixel as a feature and models that carve images into regions and use region properties as features. $\endgroup$ Jun 18, 2017 at 0:27
  • $\begingroup$ what happens when you trained a model using PNG images and tested it for jpeg images. PNG has 4 color channels right? while jpeg only has 3. will it have a big effect with regards to its accuracy in classification. $\endgroup$
    – Aaron Ruiz
    Mar 4, 2021 at 12:27

4 Answers 4

24
$\begingroup$

Short answer is NO.

The format in which the image is encoded has to do with its quality. Neural networks are essentially mathematical models that perform lots and lots of operations (matrix multiplications, element-wise additions and mapping functions). A neural network sees a Tensor as its input (i.e. a multi-dimensional array). It's shape usually is 4-D (number of images per batch, image height, image width, number of channels).

Different image formats (especially lossy ones) may produce different input arrays but strictly speaking neural nets see arrays in their input, and NOT images.

$\endgroup$
10
$\begingroup$

While Djib2011 answer is correct, I understand your question as more focused on how the image quality/properties affect neural network learning in general. There is only little research in this topic (afaik), but there might be more research on it in the future. I only found this article on it. The problem at the moment is, that this is more a problem appearing in practical applications and less in an academic research field. I remember one current podcast where researchers observed that even the camera that was used to take a picture could have a big effect.

$\endgroup$
2
  • $\begingroup$ Could you point me to the podcast please? $\endgroup$ Sep 27, 2017 at 16:13
  • $\begingroup$ It's a one hour episode of "talking machines", where they have a discussion with a researcher practicing machine learning in Africa (ca. middle to end) iTunes link: itunes.apple.com/de/podcast/talking-machines/… $\endgroup$
    – Bobipuegi
    Sep 27, 2017 at 16:51
3
$\begingroup$

This is a riff on the first answer from Djib2011. The short answer has to be no. Longer - Firstly photos are always encoded as a tensor as follows. An image is a number of pixels. If the photo is considered to have m rows and n columns, each pixel is specified by it's row and column location, that is by the pair (m,n). In particular there are m*n pixels which is very large even for 'small' photos. Each pixel of the photo is encoded by a number between zero and one (blackness intensity) if the photo is black and white. It is encoded by three numbers (RGB intensities) if the photo is color. So one winds up with a tensor that is either a 1xmxn or a 3xmxn. Image recognition is done through CNN's which, taking advantage of the fact that photos don't change that much from pixel to pixel, compress the data via filters and pooling. So the point is that CNN's work by compressing the incredibly large numbers of data points (or features) of a photo into a smaller number of values. So whatever format you start with, CNN's start off by further compressing the data of the photo. Hence the per se independence from the size of the representation of the photo.
However, a CNN will demand that all images being run through it are all of the same size. So there is that dependency that will change depending on how the image is saved. In addition, to the extent that different file formats of the same size produce different values for their tensors, one cannot use the same CNN model to identify photos stored by different methods.

$\endgroup$
0
$\begingroup$

While changes in camera or image compression after training can be severe, if it is the same, the problem is much less. Of course with more noisy images the performance is less, but I never heard that standard JPEG compression would make a big difference. But it will depend on the application.

If you change things after training, it very much depends. E.g. for some networks changing the resolution doesn't work at all. For others its possible. Its very network specific. In general any change (even lens, lighting, background, etc) needs to be evaluated and needs to be included in the training from a theoretical perspective.

In general its not a good idea to have training data that is qualitatively different. If you want to classify PNG and JPG, then it would be best to also train on both. The same for other image properties.

A CNN cannot extrapolate, it usually just works within the training set space. Other models can do that, like rule based models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.