2
$\begingroup$

I have a set of images that belong to a particular class. Then, I have another set of images that do not contain any image of the above particular class.

So, effectively I have two sets of images viz. one that belongs to a class and another that does not belong to the class.

I need to design a neural network that can model the positive class. What I am doing is the following using Theano in python:

  1. A multiple layers neural network with convolutional layers at the beginning.
  2. Use a mean squared error as loss function.
  3. Use a single sigmoid neuron at the output.

My question is whether using a single sigmoid neuron is the right approach or not. Otherwise, should I use two separate sigmoid neurons at the output for this scenario.

Any recommendations is appreciated.

Thanks

Improve this question
$\endgroup$
2
  • $\begingroup$ Input neurons: assuming 100x100 pixels, it should be 10k neurons // Hidden: atleast 60 or more // Outputs: 2 (for 2 classes) // Activation function: sigmoid (-1 and 1) --> Make sure you normalize the data between 0 to1 $\endgroup$
    – pbu
    Jun 22, 2015 at 12:27
  • $\begingroup$ beware that (depends on your data), if you use square loss, the outliers will cause dramatic changes on your decision boundary (sensitive to outliers). But if you use cross entropy, it will be robust to outliers. $\endgroup$
    – Charles Chow
    Sep 12, 2019 at 3:37

2 Answers 2

Reset to default
2
$\begingroup$

Easy question: Yes you are correct. A single sigmoid neuron output is exactly what you want for this. Could also use any other 0,1 bounded neuron: eg tanh.

Like your choice of sigmoid vs tanh, your loss (ie error, ie cost) function doens't really matter. I'ld use cross-entropy, but mean squared world work fine -- any differences are going to vanish over a few epochs of training.

I like to initialise with small normally distributed values, generaly mean zero, varience 0.01 (Its often with testing dropping or raising that varience by an order of magnitude). But once again, small uniformly distributed also works fine.

A futhur recommendation is that you blanance the number of positive and negative examples. Otherwise your network will learn the prior (that things are for example 2x more likely to have as not to have.).

You should take a good look through Yann Lecun's "Effient Back-propergation". It's written by the creator of the convoltional neural network.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Can you please recommend what error function should I use? $\endgroup$
    – London guy
    Jun 22, 2015 at 11:00
  • $\begingroup$ Thanks so much. I used this configuration but while the recall is very high (90%), the precision is quite low (50%). it is like classifying every neagtive sample as a positive sample. So the precision is shown as 50%. Any strategy to increase the precision now? Oh, my output threshold is 0.5. Is it important to change that? $\endgroup$
    – London guy
    Jun 22, 2015 at 11:09
  • $\begingroup$ Any idea around weights initialization? I have initialized using a uniform distribution. $\endgroup$
    – London guy
    Jun 22, 2015 at 11:09
  • $\begingroup$ Your output threshhold (ie bias) is normally learn via backprop. Is this positive and negitive training cases balances to the same ratio as the test set cases? Are you using many epochs? $\endgroup$
    – Frames Catherine White
    Jun 22, 2015 at 11:16
  • 1
    $\begingroup$ @AbhishekShivkumar Ah you are using threshold as in the classification threshold. Often threshhold is used as a synomyn for bias. I misunderstood. But yes, the threshold for is it 1 or is it 0 should be 0.5 However you often can just output the value of the sigmoid neuron, as that is just the models confidance of it being in the 1 class. $\endgroup$
    – Frames Catherine White
    Jun 23, 2015 at 0:44
1
$\begingroup$

You have a single class in the problem as you've posed it. This is not a one-vs.-all situation where you have multiple possible class outputs. If you were trying to classify pictures of cars into classes representing their body style, for example, you would need k output nodes (where k is the number of labeled classes of car bodies, e.g., sedan, pickup, hatchback, SUV, etc.). But all the training data must be labeled, and the output node that receives the largest output value determines the decision class, regardless of how close the others are. If you have more output nodes than you have labels the meaning of the NN is in question because you cannot train for the "none-of-the-above" label.

Supposing you were to have labels for "sedan," "hatchback," and "pickup" and then you wanted to add a catch-all category for none-of-the above. You would have to label all input images with one of the categories, even if the catch-all category has cars, kittens, kitchens, and koalas in it. The net will learn some feature set for the catch-all set, and what it actually learns is very difficult to determine. You will never really be able to fully train the catch-all set with a basis set of all possible other inputs. So the accuracy of its classification in this category is dubious. Worse, this additional category will dilute the other categories and increase the probability of false negatives in all the other classes.

So you are right to have a single output node with a single class. With NNs you have only known cases, there is no "else" catch-all block equivalent.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.