What algorithm to use to classify image-like data by spatial relations? Let's assume I have dataset of image-like 2D samples where values can be divided into few discrete levels (for example 1, 2, 3 and 4) like in the image below, where each color maps different value, from 1 to 4. Number of how many times given color occurs on the picture varies from sample to sample though.

I would like to classify these images into different classes but based on the spatial relations of these values between each other (not the values themselves). By spatial relations I mean basically (left, right, up, down), for example:

*

*If blue is above and to the right of the red

*Another blue is above and to the left of the same red

*Yellow is to the right of one blue (same height)

*One green is below red

*...

My question is, what machine learning or deep learning algorithm I should use for that task? I would appreciate even just some keywords or clues of what might help here.
[EDIT] These data are not proper real images. Just more-less 50x50 arrays with one integer value per cell (range of these values is limited to just few, like 1, 2, 3, 4).
 A: I'm still not quite sure I understand whether you already have two classes to perform classification on, or whether you are trying to detect clusters in an unsupervised manner. My answer assumes the latter, although the first part of my answer would be largely the same for a classification task.

You should be able to capture the approximate position of an object can be captured by a convolutional neural network.
You would then split your data into 4 channels:

*

*The first channel is equal to $1$ if the position is occupied by class 1, and $0$ otherwise;

*The second channel is equal to $1$ if the position is occupied by class 2, and $0$ otherwise;

*The third channel is equal to $1$ if the position is occupied by class 3, and $0$ otherwise;

*The fourth channel is equal to $1$ if the position is occupied by class 4, and $0$ otherwise.

Each channel is then be used in a 2D convolutional layer, with the first activation set to linear, since the input is just ones and zeroes. Provided you have enough of these 'images', your network should be able to learn combinations of positions of objects 1–4.
The next question is what to use use as output. Since you don't have a label to train these arrangements from (as far as I understood the question), what you could try is to have the network simply map the arrangements based on similarity.
One thing that comes to mind is a variational autoencoder. This maps the input to a latent normal distribution. Variational autoencoders penalize both for the reconstruction error and the variance of the latent distribution. This forces similar input to be mapped more closely to each other.
Take for example the example of a convolutional variational autoencoder from the tensorflow tutorial page:

The VAE loss forces images of similar digits to be closer to one another and vice versa.
That should at least get you to the point where you can "group similar arrangements". Of course a simple $t$-SNE should be able to do the same, but after training a network like this, you end up with an actual model that you can use.
