Problem description

I have a dataset consisting of 200.000 images divided between 20 classes. I'm making a classifier that predicts the class of an image. Each image belongs to one class only.

The dataset has an interesting property. Multiple images belong together and share the same class label. These subsets of the dataset differ in size from 1 to 50.

In other words; the dataset $X$ consists of $m$ disjunct sets of varying size, such that:

$$ X = \{\{x_{11}, x_{12}, \dots, x_{1a}\}, \{x_{21}, x_{22}, \dots, x_{2b}\}, \dots, \{x_{n1}, x_{n2}, \dots, x_{nm}\}\}, $$

where $x_{nm}$ is sample $m$ in set $n$. Note that the subsets are not of the same length. That is, the condition $a = b = m$ does not have to be true. Also, two subsets can belong to the same class.

Each of the samples, $x_{n1}, x_{n2}, \cdots, x_{nm}$, in one subset has the same class label.

The question is: How can we utilize this information?

Note that the structure of the dataset is the same for the test data as well.

What I have tried

I have approached the problem without using the extra information at training time. I could not find a suiting classifier that accepts variable sized inputs.

I have used the pre-trained inception-v3 network (on the ImageNet dataset), retraining it to fit my own dataset.

I would like to combine the predictions from the images in the same subset. Since I know they belong to the same class, I can, for instance, average the predictions to make a prediction for the whole subset.

This is similar to ensemble methods, but instead of testing the same input on different classifiers, I would like to test different inputs on the same classifier before combining the results.

I have failed to find any relevant literature. Does anyone know about papers that describe this exact problem? Would it be correct to approach the problem in the same way as ensemble methods?

I know it would probably be better to use this information on train time as well, but I have no idea how I would approach this problem.

  • $\begingroup$ Since the Large Scale Visual Recognition Challenge (ILSVRC) is a competition and not a particular solution, look over that link and tell us which application is in play here, please. $\endgroup$
    – Carl
    Jun 3, 2017 at 18:31
  • $\begingroup$ @Carl: I have updated the question. Also, I think Multiple-Instance-Learning should be a tag on this question, but it doesn't seem to exist. $\endgroup$ Jun 4, 2017 at 18:16
  • $\begingroup$ The language in the question is still a bit vague. If you would be so kind as to improve it, I will vote to reopen the question. Can you do that? $\endgroup$
    – Carl
    Jun 4, 2017 at 19:55
  • $\begingroup$ @Carl: Thank you for your time. I have edited it now. It should be more clear. The reason the question was vague was that I did not know what I was looking for. Note that I have provided an answer for the question as well :) $\endgroup$ Jun 4, 2017 at 20:35
  • $\begingroup$ OK, clearer now. $\endgroup$
    – Carl
    Jun 5, 2017 at 3:40

1 Answer 1


I found the term I was looking for and that helped the literature search a lot. The correct term is Multiple Instance Learning (MIL).

There are multiple ways to perform MIL. Basically, either define your own loss function that calculates loss across "bags of instances", or simply combine the predictions that belong to a bag. The latter can be done by averaging or weighting confident predictions. There are other more intricate solutions as well, but I have not looked too deep into it.

Some papers on the subject:

  • Viola, Paul, John C. Platt, and Cha Zhang. "Multiple instance boosting for object detection." NIPS. Vol. 2. 2005.
  • Xu, Xin, and Eibe Frank. "Logistic regression and boosting for labeled bags of instances." Pacific-Asia conference on knowledge discovery and data mining. Springer Berlin Heidelberg, 2004.
  • Singh, Diveesh, and Pedro Garzon. "Using Convolutional Neural Networks and Transfer Learning to Perform Yelp Restaurant Photo Classification."
  • Babenko, Boris, Ming-Hsuan Yang, and Serge Belongie. "Visual tracking with online multiple instance learning." Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 2009.

Hopefully this answer will help furture people with the same problem as I.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.