I have an image dataset in which there is a response $Y$ (dependent variable) for each image. There are about 1000 images, and accordingly 1000 response variable Y per image. My question is how should I define the independent variables since each image contains several pixels but there is only one response value per image. Should I take the average of the color band values (red, green, blue) of all the pixels in one image and consider that the independent variable for that image? This will produce three independent variables and one dependent variable per image. However, if I just take an average over all pixels, I may lose or leave out some information, such as the variance. Is there a machine learning algorithm better than simply taking the average for this type of problem?
1 Answer
If this is an image categorization problem, meaning that the response variable $Y$ is categorical, something like 'dog' or 'cat,' then you should look at a convolutional neural network (CNN). You are correct that merely taking the average will lose quite a lot of information - Think about how many different combinations of the number of pixel values in your RGB channels could lead to the same 'average' value in each channel. Let's say you have $256 \times 256$ pixel images. You thus have $2^{16}$ different pixel values in each of your three channels. Depending on the color spectrum of your $10^3$ samples, that could be quite a few images that are categorically different yet with close average values in each channel. Besides that, small differences in the average value would be interpreted as high similarity in the images, further muddying any inference the model produces.
If this is a true regression problem, meaning that $Y$ is a continuous variable, then you may still have to worry about losing positional information of the pixels by vectorizing the images. I would need more information about the details of this specific problem in order to give any suggestions.
edit
Taking into account your comment, you could be able to just perform a linear regression, possibly with some modifications. Assign a value of 0 on all channels to all your pixels that get masked as 'background.' Your predictor variables will just be each pixel - you will need to standardize your images so that, before any masking has occurred, they all have the same pixel count. If some images are smaller than others before masking, just perform zero-padding to expand the smaller images to match the larger ones as this would just result in them being set to 0 - the value they already are assigned - in the masking step. Some issues that might be of concern here are: the fact that spatial information is still retained in a sense, as each pixel is distinguished from the others. Thus, if you were to perform a spatial translation on your partially masked image, it could produce a completely different set of relevant 'variables' (the pixels) but actually still be, for all intents and purposes, the same image. Another issue is that of correlation among the 'variables' - let's say you have lots of images of green leaves. Then the probability of a green pixel having a neighbor that is also green may be higher than a pixel which is not green having a green neighbor. Both of these issues may be addressable by performing a down-sampling by selecting a representative pixel for each 'neighborhood' of pixels. Another issue that could arise is the fact that the color black in RGB is also 0 in all channels, which would only be an issue if the extracted images after masking may also contain black - these pixels would get dropped by the method and produce an erroneous extracted image. One way to handle this would be to assign an unused value, like 256 or 500, to all masked pixels. Alternatively, you could assign a weighting to pixels that reflects the intrinsic geometry, for instance assigning a higher weight to pixels near the edges and center of the extracted image. This would bypass any worries about translation invariance or losing black pixels, as it is performed only after masking and image extraction.
-
$\begingroup$ Thanks for your suggestions. Y is a continuous variable (regression problem). I agree that vectorizing may not be an option here given the fact that the number of pixels per each image is different. Also, there is no meaningful spatial information since some pixels are masked out from each image as background from various location in each image. Let's say image are 256x256x3, and different number of pixels representing background are disregarded from each image, so the number of segmented pixels for each image is different. I have only one Y per all pixels segmented from each image. $\endgroup$– AliSep 3, 2018 at 22:36
-
$\begingroup$ I am not sure if I agree with your statement saying "Your predictor variables will just be each pixel". I don't have a response/target value Y for each pixel, rather it is for a whole image. So I am looking for a technique to combine the pixels within an image and then assign the measured Y to that. Thanks! $\endgroup$– AliSep 4, 2018 at 15:50