Problem description
I have a dataset consisting of 200.000 images divided between 20 classes. I'm making a classifier that predicts the class of an image. Each image belongs to one class only.
The dataset has an interesting property. Multiple images belong together and share the same class label. These subsets of the dataset differ in size from 1 to 50.
In other words; the dataset $X$ consists of $m$ disjunct sets of varying size, such that:
$$ X = \{\{x_{11}, x_{12}, \dots, x_{1a}\}, \{x_{21}, x_{22}, \dots, x_{2b}\}, \dots, \{x_{n1}, x_{n2}, \dots, x_{nm}\}\}, $$
where $x_{nm}$ is sample $m$ in set $n$. Note that the subsets are not of the same length. That is, the condition $a = b = m$ does not have to be true. Also, two subsets can belong to the same class.
Each of the samples, $x_{n1}, x_{n2}, \cdots, x_{nm}$, in one subset has the same class label.
The question is: How can we utilize this information?
Note that the structure of the dataset is the same for the test data as well.
What I have tried
I have approached the problem without using the extra information at training time. I could not find a suiting classifier that accepts variable sized inputs.
I have used the pre-trained inception-v3 network (on the ImageNet dataset), retraining it to fit my own dataset.
I would like to combine the predictions from the images in the same subset. Since I know they belong to the same class, I can, for instance, average the predictions to make a prediction for the whole subset.
This is similar to ensemble methods, but instead of testing the same input on different classifiers, I would like to test different inputs on the same classifier before combining the results.
I have failed to find any relevant literature. Does anyone know about papers that describe this exact problem? Would it be correct to approach the problem in the same way as ensemble methods?
I know it would probably be better to use this information on train time as well, but I have no idea how I would approach this problem.