Say you have a bunch of simple images that are basically just 25 pixel blocks. Each of these images has a corresponding set of class labels (for the whole block) but you really just want to know what class each individual pixel is. For instance, using the two blocks below:

enter image description here

Perhaps you know up-front the left block is 32% gravel (8/25), 32% bare earth, and 36% vegetation (9/25). You also know that the right block is 60% gravel (15/25) and 40% bare earth (10/25). Therefore, like an algebra problem, you could deduce that the dark gray is gravel, the light gray is vegetation, and the white is bare earth.

I need to do this for real images (RGB + near infrared) that are a bit more complex; but luckily I have a lot of blocks, so I think the classes (n=5) should be separable. However, I have no clue what method/technique this is - does anyone have an idea?

Edit: the motivation for this work is that (in real life) the blocks are my ground truth points (vegetation sampling quadrats) for a larger image that I want to classify. The original approach I thought of was resampling the images to the size of my blocks and then training an artificial neural network off the mixed pixels (after Foody (1996)). But I would like to avoid this if it's at all possible to derive more information by classifying at the level of the original pixels first.

  • $\begingroup$ So this is an image labeling problem? Have you looked into UNets? If the images are complex and vary quite a bit in scale, rotation, translation, etc..., then a conv net like this may be needed, or at least helpful. $\endgroup$ Commented Jan 16, 2018 at 20:05
  • $\begingroup$ I edited to take out the "quite" on a bit more complex - my problem space is actually very similar to what I've described :) The only additional complexity is that, of course, in real images not every pixel of vegetation (for instance) looks the same. And not every image was taken on the same day, although there is good coverage of the different days and lighting conditions were similar. I am hoping to find the simplest solution possible though so if I can avoid a neural network that would be great, haha $\endgroup$
    – HFBrowning
    Commented Jan 16, 2018 at 20:12
  • $\begingroup$ And yes, an image labeling problem for sure $\endgroup$
    – HFBrowning
    Commented Jan 16, 2018 at 20:12
  • $\begingroup$ If the pixels are quite separable on RGB channels for the 5 classes then you could treat each pixel's values as a sample, instead of the entire image. Then use standard classification algorithms to separate them into classes (i.e. SVM or trees). This way at least you could account for some of the variability and uncertainty around decision boundaries. At least, this might be a way to start. If you just plot the 3 RGB values in 3 dimensions, do the classes look separable? $\endgroup$ Commented Jan 16, 2018 at 20:24

1 Answer 1


The problem itself is essentially semantic segmentation or pixelwise classification. In this case, you're taking coarser labels and trying to extend them into a segmented pixelwise labeling. As such, the segmentation side is more important, I think. (And the more I think about it, the more I would recommend graph cuts (last paragraph).)

Since you're attempting to create a neural network with these, there may be some oddities from biases whatever model you create has. Still, a neural network should be able to learn to be roughly as good and more generalizable, so it makes sense.

It is a good idea to look into probabilistic graphical models such as conditional random fields. Perhaps a Gaussian mixture model classification of some sort is all you need, where the inputs are binned RGB(I) histograms. That will take a little experimentation on your part.

However, simpler models may be good enough. I would even say a majority vote or nearest neighbor approach with superpixels would be suitable. Graph cuts is another method that may work, where the energy function would a function be how close it is to the center of the labeled block in color and physical space.

  • $\begingroup$ Interesting. I didn't realize skimage already has an implementation for this clustering. Superpixels seem like a good non-net beginning approach to this. Generate superpixels -> feature extraction -> classification. Might work well enough if the data isn't too complex. $\endgroup$ Commented Jan 16, 2018 at 20:34
  • $\begingroup$ I should probably experiment, as both you and neuroguy suggest (since I haven't yet confirmed for sure that the pixels are easily separable). The only reason I wanted to avoid a neural net is that actually these are my ground truth plots for a larger classification, which is going to be a neural net. I was looking to avoid using mixed pixels/fuzzy classification by hard classifying these pixels in advance. I can't tell which of these would be more valid (fuzzy classification off the whole block or classifying based upon the output of another classifier)? $\endgroup$
    – HFBrowning
    Commented Jan 16, 2018 at 20:39
  • $\begingroup$ @Poik I think you understand me correctly the first time? Not sure. The goal is understanding class on a pixel-by-pixel basis. It's almost like I want to downscale the information from coarser to finer resolution $\endgroup$
    – HFBrowning
    Commented Jan 16, 2018 at 20:42
  • $\begingroup$ OH! I did misunderstand a bit. Okay. It's still a segmentation problem, with the semantic part being sort of filled in. I would use superpixels still, and use the percentage of pixels in each superpixel that falls under each coarse label as a fuzzy label or vote for what that superpixel is. So where superpixels fall on coarse label lines, it would be both classes at some percent, or the majority class. There are other ways to do this, but that should be a good place to start? I think graph cuts were an older method that worked well (but was somewhat computationally expensive?). $\endgroup$
    – Poik
    Commented Jan 16, 2018 at 20:47
  • $\begingroup$ @Poik want to edit your answer to suggest that? I will wait to see what other people suggest (since a checkmark deters others from answering) but I like your idea. Thank you. $\endgroup$
    – HFBrowning
    Commented Jan 16, 2018 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.