Combining predictions on different inputs

Question

Problem description

I have a dataset consisting of 200.000 images divided between 20 classes. I'm making a classifier that predicts the class of an image. Each image belongs to one class only.

The dataset has an interesting property. Multiple images belong together and share the same class label. These subsets of the dataset differ in size from 1 to 50.

In other words; the dataset $X$ consists of $m$ disjunct sets of varying size, such that:

$$ X = \{\{x_{11}, x_{12}, \dots, x_{1a}\}, \{x_{21}, x_{22}, \dots, x_{2b}\}, \dots, \{x_{n1}, x_{n2}, \dots, x_{nm}\}\}, $$

where $x_{nm}$ is sample $m$ in set $n$. Note that the subsets are not of the same length. That is, the condition $a = b = m$ does not have to be true. Also, two subsets can belong to the same class.

Each of the samples, $x_{n1}, x_{n2}, \cdots, x_{nm}$, in one subset has the same class label.

The question is: How can we utilize this information?

Note that the structure of the dataset is the same for the test data as well.

What I have tried

I have approached the problem without using the extra information at training time. I could not find a suiting classifier that accepts variable sized inputs.

I have used the pre-trained inception-v3 network (on the ImageNet dataset), retraining it to fit my own dataset.

I would like to combine the predictions from the images in the same subset. Since I know they belong to the same class, I can, for instance, average the predictions to make a prediction for the whole subset.

This is similar to ensemble methods, but instead of testing the same input on different classifiers, I would like to test different inputs on the same classifier before combining the results.

I have failed to find any relevant literature. Does anyone know about papers that describe this exact problem? Would it be correct to approach the problem in the same way as ensemble methods?

I know it would probably be better to use this information on train time as well, but I have no idea how I would approach this problem.

Answer 1

I found the term I was looking for and that helped the literature search a lot. The correct term is Multiple Instance Learning (MIL).

There are multiple ways to perform MIL. Basically, either define your own loss function that calculates loss across "bags of instances", or simply combine the predictions that belong to a bag. The latter can be done by averaging or weighting confident predictions. There are other more intricate solutions as well, but I have not looked too deep into it.

Some papers on the subject:

• Viola, Paul, John C. Platt, and Cha Zhang. "Multiple instance boosting for object detection." NIPS. Vol. 2. 2005.
• Xu, Xin, and Eibe Frank. "Logistic regression and boosting for labeled bags of instances." Pacific-Asia conference on knowledge discovery and data mining. Springer Berlin Heidelberg, 2004.
• Singh, Diveesh, and Pedro Garzon. "Using Convolutional Neural Networks and Transfer Learning to Perform Yelp Restaurant Photo Classification."
• Babenko, Boris, Ming-Hsuan Yang, and Serge Belongie. "Visual tracking with online multiple instance learning." Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 2009.

Hopefully this answer will help furture people with the same problem as I.