# Neural Networks: utilizing weakly-labeled data to improve fully-supervised network?

The problem:

I have built a fully-supervised CNN that localizes an object in different scenes. As you can imagine, it quite time-consuming to label data: I have to manually localize the object in an image and draw a bounding-box around it. Although the labeling itself is "as comfortable as possible" (I created a labeling tool that only requires spanning a rectangle over the object in an image, automatically saving the label and loading the subsequent images) it is still a really slow process, because every image has to labeled manually - one at a time.

My current labeled data sets count >= 1,000 images and the CNN localizes the object with a high accuracy in the test/validation data. However, changing lighting, adding occlusion or testing backgrounds unknown to the network affects the results. Thus, I would like to make the localization more robust.

In addition to the labeled data sets I have over 50,000 weakly-labeled images. They are basically unlabeled, but they are split into two different sets: "object" and "no-object". The first set just contains images where the object that gets localized is present, while the latter are just background scenes without the object (for positive and inhibitory feedback).

The fully-supervised architecture looks like this:

              input
|
convolutional-layers L1...Ln
|
|
---flattened output---
|                    |
|                    |
fully-connected      fully-connected
x                    y


...where x and y are the two different fully-connected layers (+outputs) for the x and y location of the object. The network then gets fed images to the input and is trained via backpropagation, by minimizing the combined loss of x & y.

My first idea was copying the trained weights of the convolutional layers into a slightly different network:

              input
|
convolutional-layers L1...Ln
|
|
flattened output
|
fully-connected
|
single-output-neuron  # 1.0 -> object is present
# 0.0 -> object is most likey not present


I thought by copying the weights of the trained convolutional layers this new network should have a sufficient starting point in order to classify the weakly-labeled data correctly into "object" and "no-object". Besides some false-positives the network indeed is classifying most images correctly (outputs above .95 have over 99% accuracy in test/validation data sets, where the object is present in the images).

My plan was if the network is accurately classifying the weakly-labeled images into "object" & "no-object", I possibly could analyze the activations of the convolutional layers. A high output should indicate that the object was present in the input image and the convolutional layers should show some activation at regions where the object was in the original image. Thus, I thought by analyzing and extracting the activations of the convolutional layers, I should be able to find the object's location and therefore "automatically" label the weakly-labeled dataset, turning it into a fully-labeled data set.

Unfortunately this does not work: the activations in the convolutional layers always "overshoot". The regions of the image with a high activation never match the edges of the object, they are always much bigger. Additionally, the activations are always of different shapes, the edges blunt and they are not centered on the object. Thus, the activations are more or less useless, because they just supply me some sub-region of the image where the object can be anywhere inside this region.

The question:

How could I utilize the weakly-labeled data to imporove the robustness & generalization of the fully-supervised network?

Is it possible to drastically increase the "sharpness" of the edges and implicit object localization of the second, weakly-supervised network? In a way that I can use it to label the weakly-labeled data sets?

Or should I drop this idea and go, for example, for some kind of ladder network in order to make use of the weakly-labeled data?

Typically large amounts of weakly-labeled or unlabeled data exists, while (fully) labeled data is normally pretty rare. However, there is a large potential in weakly-labeled / unlabeled data. The problem is using this potential without having to manually label every sample by hand. Do efficient approaches for such scenarios exist?

I hope the question is not too broad...it is hard to get this important / big topic into smaller chunks...

Bounty summary:

• How can a feed-forward network (especially CNNs) possibly utilize weakly-labeled (or unlabeled) data?
• How can we generally utilize weakly-labeled data? (Like the example: data sets separated into "containing a specific object" and "not containing the object")
• Do any promising architectures exist, that enable fully-supervised architectures to use weakly-labeled data or unlabeled data? (to improve the network)
• How would you (practically) tackle such problems? (where lots of weakly-labeled data or unlabeled data is available, but labeled data is not)