# Neural Networks, what is the best way to deal with sequence when order does not matter?

Right now I am working on a problem where I have sets of 5 profile pictures of people. I am trying to classify those pictures using a CNN which feeds into an RNN. As a whole the set of 5 images should fall into one class although there may be some outliers within the 5 (like someone who posts a picture of their dog rather than themselves.) Ideally the network should be able to learn to ignore these. Right now this CNN->RNN approach works decently well. However, in an RNN the order in which the images are presented seems to matter. If I shuffle the profile images and feed them to the network the prediction probability tends to vary by about 5%. I was wondering if there is an architecture other than RNNs where the whole sequence can be considered yet where the ordering of the sequence does not matter?

• what if the order did matter? for instance, people are more likely to put good pictures first, some may not have 5 good pictures so they stick garbage after 2-3 etc. – Aksakal May 29 at 13:22

In case of RNN order matters. hence your architecture will not work well when you shuffle the images and then tries to predict a sample. For this problem, RNN is not required at all and they are not even a better choice for such kind of problems. The best architecture for your problem is the use of CNNs only. It will give you an exceptional accuracy if you implement it correctly. In order to get an overview how it works, you can take a look at the example of CIFAR10 small images classification: Convolutional Neural Network (CNN) with realtime data augmentation given here

• Yes but a CNN can only handle one image. – chasep255 Sep 25 '16 at 14:46
• Have you looked at the CIFAR10 dataset? Each of the 10 class is trained on a set of images for each class. – Nain Sep 25 '16 at 15:16
• Yes but you are still classifying each of the images independently of each other. 1 image -> 1 class. What I am talking about is taking 6 images -> 1 class where just one image alone may not represent that class. That is why I feed the images through an RNN after the CNN. – chasep255 Sep 27 '16 at 8:51
• For CNNs the order would matter as well. – Tim May 28 at 21:27

Going by this comment:

Yes but you are still classifying each of the images independently of each other. 1 image -> 1 class. What I am talking about is taking 6 images -> 1 class where just one image alone may not represent that class. That is why I feed the images through an RNN after the CNN.

If order does not matter, then you could provide samples as channels, with images constantly being shuffled within each set.

Alternatively, you could build a Siamese network from all samples and sum or aggregate their (vector or scalar) outputs. This would be order-agnostic as well.

## To the point

If you're not modelling a sequence (a set of 5 or 6 images with no defined precedence) then I find recursive architectures hard to justify. Just stick to a convolutional (with possibly Siamese flavor) architecture.

You can process each image with the same CNN, outputting a $$H$$ dimensional vector $$h_i$$ for each image (in Keras this is a TimeDistributed application of your CNN). So if you have $$N$$ images, your output will be $$(N,H)$$ dimensional.

Then you can use max and/or mean pooling across the $$N$$ dimension, giving you a fixed output of dimension $$2H$$, which you can pass onto a classification layer.

• +1, this is basically my second answer explained more clearly Alternatively, you could build a Siamese network from all samples and sum or aggregate their (vector or scalar) outputs. This would be order-agnostic as well. – Firebug May 29 at 13:06

As far as I understand your question, you have five images, that should, but don't have to show the same thing, and given the pictures you want to predict something. For processing each of the pictures you probably should stick with stat of the art for the pictures, that is, some kind of convolutional neural network. Since each of the pictures is assumed to show the same thing, as said by others, you should use the Siamese architecture in here, so the same convolutional network should be used for processing each of the images. As I understand it, the problem is with combining the information from each of the pictures. Obviously, recurrent network would not work in here, because it explicitly assumes that the order does matter. Notice however that if you used convolutional network, then order would matter as well (pictures appearing next to each other would be processed by a single convolutional kernel). Same with dense network, since it would have separate parameters for each of the positions of an image, so it would make a difference if it appeared on the first position. If you really want to ignore the position, then if I were you, I would try an architecture that aggregates the information from all the pictures weighting them equally. For this purpose you can use single aggregating function (e.g. sum, product), or multiple such functions in parallel.

The architecture, that I am thinking of is something like

\begin{align} \boldsymbol{z}_i &= \operatorname{CNN}(\mathbf{X_i}) \\ \boldsymbol{a} &= g(\boldsymbol{z}_1, \boldsymbol{z}_2, \dots, \boldsymbol{z}_k) \\ y &= f(\boldsymbol{a}) \end{align}

where $$g$$ is something like

$$\boldsymbol{a}_i = g(\boldsymbol{z}_{1i}, \boldsymbol{z}_{2i}, \dots, \boldsymbol{z}_{ki}) = \sum_{j=1}^k \boldsymbol{z}_{ji}$$

So each of the $$\mathbf{X_i}$$ for $$i \in 1,2,\dots,k$$ pictures you use the same CNN network to output some latent representation $$\boldsymbol{z}_i$$. Next, you aggregate it using aggregation function $$g$$, and pass this through an "output" network $$f$$ (e.g. dense layer on top of it followed by an adequate activation function).