Let's assume I have dataset of image-like 2D samples where values can be divided into few discrete levels (for example 1, 2, 3 and 4) like in the image below, where each color maps different value, from 1 to 4. Number of how many times given color occurs on the picture varies from sample to sample though.

enter image description here

I would like to classify these images into different classes but based on the spatial relations of these values between each other (not the values themselves). By spatial relations I mean basically (left, right, up, down), for example:

  • If blue is above and to the right of the red
  • Another blue is above and to the left of the same red
  • Yellow is to the right of one blue (same height)
  • One green is below red
  • ...

My question is, what machine learning or deep learning algorithm I should use for that task? I would appreciate even just some keywords or clues of what might help here.

[EDIT] These data are not proper real images. Just more-less 50x50 arrays with one integer value per cell (range of these values is limited to just few, like 1, 2, 3, 4).

  • $\begingroup$ Are the four colors really separate objects, or are you just referring to the color of a particular pixel in an image? If the four colors are separate objects, what happens if two colors occupy the same position? $\endgroup$ Apr 22, 2021 at 17:57
  • $\begingroup$ @FransRodenburg yes, these are separate objects. One data sample is more less 50x50 array with some cells filled with these values. Colors are only used to help visualize the question, these are not regular images. Two colors cannot be in the same cell. Single cell from such array can only have one discrete integer value. $\endgroup$
    – GKozinski
    Apr 22, 2021 at 18:34
  • $\begingroup$ Is there a prespecified number of classes, or do you want to somehow group 'similar' arrangements? $\endgroup$ Apr 22, 2021 at 19:04
  • $\begingroup$ @FransRodenburg that is very good and hard question that would be too long to explain, but in simple view I need to do both, first regular classification with 2 classes and then try to do clusterisation of this data, by the use of the model in question. $\endgroup$
    – GKozinski
    Apr 22, 2021 at 19:53
  • $\begingroup$ actually what you said: "to group similar arrangements" is very close to what I need it for, after some thinking $\endgroup$
    – GKozinski
    Apr 22, 2021 at 19:54

1 Answer 1


I'm still not quite sure I understand whether you already have two classes to perform classification on, or whether you are trying to detect clusters in an unsupervised manner. My answer assumes the latter, although the first part of my answer would be largely the same for a classification task.

You should be able to capture the approximate position of an object can be captured by a convolutional neural network.

You would then split your data into 4 channels:

  • The first channel is equal to $1$ if the position is occupied by class 1, and $0$ otherwise;
  • The second channel is equal to $1$ if the position is occupied by class 2, and $0$ otherwise;
  • The third channel is equal to $1$ if the position is occupied by class 3, and $0$ otherwise;
  • The fourth channel is equal to $1$ if the position is occupied by class 4, and $0$ otherwise.

Each channel is then be used in a 2D convolutional layer, with the first activation set to linear, since the input is just ones and zeroes. Provided you have enough of these 'images', your network should be able to learn combinations of positions of objects 1–4.

The next question is what to use use as output. Since you don't have a label to train these arrangements from (as far as I understood the question), what you could try is to have the network simply map the arrangements based on similarity.

One thing that comes to mind is a variational autoencoder. This maps the input to a latent normal distribution. Variational autoencoders penalize both for the reconstruction error and the variance of the latent distribution. This forces similar input to be mapped more closely to each other.

Take for example the example of a convolutional variational autoencoder from the tensorflow tutorial page:


The VAE loss forces images of similar digits to be closer to one another and vice versa.

That should at least get you to the point where you can "group similar arrangements". Of course a simple $t$-SNE should be able to do the same, but after training a network like this, you end up with an actual model that you can use.

  • $\begingroup$ Thank you for your anwser, using VAE looks promising but I wonder about what you wrote on the begining about CNNs, are you sure that convolutions will somehow learn the positions of the ones in particular channels? I was thinking about it before and it seemed to me that convolutions are better for determining shape of things on the picture but they don't care much for the position. But here there are no shapes. $\endgroup$
    – GKozinski
    Apr 23, 2021 at 9:11
  • $\begingroup$ By using convolution and max pooling, your network can more efficiently learn that something was "in the top right corner" instead of at a precise location. $\endgroup$ Apr 23, 2021 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.