How can I do a one vs all classification (binary classifier) with a neural network

Question

I have a set of images that belong to a particular class. Then, I have another set of images that do not contain any image of the above particular class.

So, effectively I have two sets of images viz. one that belongs to a class and another that does not belong to the class.

I need to design a neural network that can model the positive class. What I am doing is the following using Theano in python:

1. A multiple layers neural network with convolutional layers at the beginning.
2. Use a mean squared error as loss function.
3. Use a single sigmoid neuron at the output.

My question is whether using a single sigmoid neuron is the right approach or not. Otherwise, should I use two separate sigmoid neurons at the output for this scenario.

Any recommendations is appreciated.

Thanks

Answer 1

Easy question: Yes you are correct. A single sigmoid neuron output is exactly what you want for this. Could also use any other 0,1 bounded neuron: eg tanh.

Like your choice of sigmoid vs tanh, your loss (ie error, ie cost) function doens't really matter. I'ld use cross-entropy, but mean squared world work fine -- any differences are going to vanish over a few epochs of training.

I like to initialise with small normally distributed values, generaly mean zero, varience 0.01 (Its often with testing dropping or raising that varience by an order of magnitude). But once again, small uniformly distributed also works fine.

A futhur recommendation is that you blanance the number of positive and negative examples. Otherwise your network will learn the prior (that things are for example 2x more likely to have as not to have.).

You should take a good look through Yann Lecun's "Effient Back-propergation". It's written by the creator of the convoltional neural network.

Answer 2

You have a single class in the problem as you've posed it. This is not a one-vs.-all situation where you have multiple possible class outputs. If you were trying to classify pictures of cars into classes representing their body style, for example, you would need k output nodes (where k is the number of labeled classes of car bodies, e.g., sedan, pickup, hatchback, SUV, etc.). But all the training data must be labeled, and the output node that receives the largest output value determines the decision class, regardless of how close the others are. If you have more output nodes than you have labels the meaning of the NN is in question because you cannot train for the "none-of-the-above" label.

Supposing you were to have labels for "sedan," "hatchback," and "pickup" and then you wanted to add a catch-all category for none-of-the above. You would have to label all input images with one of the categories, even if the catch-all category has cars, kittens, kitchens, and koalas in it. The net will learn some feature set for the catch-all set, and what it actually learns is very difficult to determine. You will never really be able to fully train the catch-all set with a basis set of all possible other inputs. So the accuracy of its classification in this category is dubious. Worse, this additional category will dilute the other categories and increase the probability of false negatives in all the other classes.

So you are right to have a single output node with a single class. With NNs you have only known cases, there is no "else" catch-all block equivalent.