Image classification where images are of different dimensions, resolutions, etc

Question

I am working on an image classification problem where I have black and white images of all different dimensions and resolutions. The images belong to 1 of 5 groups including an unknown category. These images are stored sparsely, by which I mean as a list of coordinates.

For example the first three observations for an image might look like this:

X Y
0 0
0 1
2 3
: :

where there are points at (0,0), (0,1), and (2,3). Since the images are black and white, these coordinates correspond to the pixels in the image that are black.

My goal is to classify these images into 1 of 5 groups. I have some metadata that I also plan to use, such as date the image was created, but ultimately I need to find meaningful features from the images themselves. My plan for this was to run PCA and feed the features into SVM or random forests model.

I am somewhat familiar with the classic handwriting digits example, however I am not familiar with cases where images are not of standard dimension or resolution.

One idea that I have had is to divide each image into a fixed number of cells, say a 100x100 grid. This way each image would be described by 1000 values which I could then run PCA on. My concern with this idea is that it reduces the information available for classifying the images.

The Question

Much of the work on image processing that I have seen uses examples where the images are of comparable size, resolution, etc. but how do we develop models for image classification where the images are not standardized in this way?

Answer 1

I would try LBP (or any other descriptor, based on what your images (and task, i.e. classes) actually are). To deal with different sizes, you can use Bag of Words encoding.

Answer 2

I agree. In most of the works on image pools, images are all of the same size. So, you can consider this as one of the major challenges of your work.

Segmentation

Segmentation of images (as you had thought) is always a good practice. I would suggest the same. You can consider a grid of size g*g for all images (regardless of their size) and calculate your parameters on those cells one by one. This would give you a g*g image parameter matrix, for each image. Regarding the parameters, you could start with descriptive statistics (e.g.,mean, sd, skewness, ...) or texture-related parameters (e.g., gradient, orientation, etc.)

Sampling

But before that, perhaps you need to make a general data analysis to see what the best size is to rescale all images to. That size is not necessarily the size of the smallest image in your dataset. You can do up-sampling images (i.e., enlargement) as well as down-sampling. For instance, if only a very small fraction of your images are below d*d, then you can rescale them to d*d using any interpolation method you prefer. Or perhaps, your images are of such a nature that they won't be affected much by interpolation (e.g., they don't have much of details).

This gets more complicated if your images are not all squared. If this is the case, then you would also need to answer this question that whether or not the subjects of your images will be significantly affected if stretched horizontally or vertically.

But does this make your results will be less accurate since you are (over/under) sampling and interpolating? Yes, but your model is as good as your data. If this is the best data you could access for your work, then there is not much one can do for missing data. Here, the missing data is the images that were supposed to be all of the same size, n*n, but apparently many of them are smaller.

Other Dimensionality reduction techniques?

For PCA, you definitely need to have images of similar sizes, since you will be dealing with vector space. I know that you would also want to use SVM which deals with vectors as well, and they must be of the same size. But if PCA was the issue, you could have used other dimensionality-reduction techniques. For instance F-test (Wikipedia, F-test). Since your data is labeled, you only need to have an equal size set of features. F-test would sort your features and you can find the best ones.

---

A minor point that perhaps you already know; the concept of resolution is defined only in the context of visualization (on screen or print). It is all about how many pixels will represent a certain unit of the image. Hence the units of resolution, ppi (Pixel per inch) or dpi (dot per inch). So, you only need to worry about the image dimension, i.e., width and height.

Answer 3

Grauman and Darrel's Pyramid Match Kernel deals with how to compare histograms of images of different sizes (and thus different number of features). But since your features are simply the number of black pixels, there isn't a proper histogram to start with.

I'd normalize the pixels count by the total number of pixels in the image. That way you could compare small images against large ones.