What is the name of this technique? (some type of classification?)

Question

Say you have a bunch of simple images that are basically just 25 pixel blocks. Each of these images has a corresponding set of class labels (for the whole block) but you really just want to know what class each individual pixel is. For instance, using the two blocks below:

Perhaps you know up-front the left block is 32% gravel (8/25), 32% bare earth, and 36% vegetation (9/25). You also know that the right block is 60% gravel (15/25) and 40% bare earth (10/25). Therefore, like an algebra problem, you could deduce that the dark gray is gravel, the light gray is vegetation, and the white is bare earth.

I need to do this for real images (RGB + near infrared) that are a bit more complex; but luckily I have a lot of blocks, so I think the classes (n=5) should be separable. However, I have no clue what method/technique this is - does anyone have an idea?

Edit: the motivation for this work is that (in real life) the blocks are my ground truth points (vegetation sampling quadrats) for a larger image that I want to classify. The original approach I thought of was resampling the images to the size of my blocks and then training an artificial neural network off the mixed pixels (after Foody (1996)). But I would like to avoid this if it's at all possible to derive more information by classifying at the level of the original pixels first.

Answer 1

The problem itself is essentially semantic segmentation or pixelwise classification. In this case, you're taking coarser labels and trying to extend them into a segmented pixelwise labeling. As such, the segmentation side is more important, I think. (And the more I think about it, the more I would recommend graph cuts (last paragraph).)

Since you're attempting to create a neural network with these, there may be some oddities from biases whatever model you create has. Still, a neural network should be able to learn to be roughly as good and more generalizable, so it makes sense.

It is a good idea to look into probabilistic graphical models such as conditional random fields. Perhaps a Gaussian mixture model classification of some sort is all you need, where the inputs are binned RGB(I) histograms. That will take a little experimentation on your part.

However, simpler models may be good enough. I would even say a majority vote or nearest neighbor approach with superpixels would be suitable. Graph cuts is another method that may work, where the energy function would a function be how close it is to the center of the labeled block in color and physical space.