3
$\begingroup$

I use a neural network with a topology of 17-30-1 (sigmoid, atan error function, mse as cost function, 5-fold cv) for text classification. (It's closely related to a previous question of mine.)

The input data is quite noisy thus I could live with not a "perfect" classification score, but the results I get are probably too bad (or even just random) and thus I ask for your opinion.

  1. The training error is around 0.06-0.09 (MSE), i.e. in average each classification differs approx. 0.25-0.3 from the predict label; in this binary case with a class threshold of 0.5 this might be acceptable. What do you think?

  2. The test error (MSE) is unfortunately around 0.20 sometimes even 0.25; i.e. the effective error for a test sample is around 0.5, which to me means that the network a) suffers from high variance and b) is just as good as random guessing.

I don't need a perfect classification, but the network should however represent the patterns of the input data. But with this results I think the neural network is more or less useless or rather the input features are crap.

Improve this question
$\endgroup$
1
  • $\begingroup$ Do I understand you correct: You have 17 input neurons, 30 hidden neurons and only one output neuron? $\endgroup$
    – Martin Thoma
    Commented Jun 18, 2014 at 15:51

1 Answer 1

Reset to default
3
$\begingroup$

First, I would advise you to not use squared error but the cross entropy error. Squared error results from the assumption that your labels are subject to Gaussian noise, which will probably not be the case.

First, the output of your network should be a softmax:

$z_k = \frac{\exp{y_k}}{\sum_i\exp{y_i}}$

This is basically a logistic regression layer on top of the neural network, and gives you a proper probability. You can train that with the cross entropy error function (see here for an explanation)--the derivatives stay the same as for squared error and linear outputs.

Regarding the interpretation of the results: this is data set specific. If it is a hard task, that looks good. However, you should look at the acutal number of correct classifications in the end and see if that is good enough for your application. Anyway I think you will get better results if you use cross entropy.

Improve this answer
$\endgroup$
8
  • $\begingroup$ You mean I should use Cross Entropy instead of ATAN as error function? I use mean squared error to measure the performance of the classifier (heatonresearch.com/wiki/Error_function). And in the binary classification scenario, is the output the probability of the sample that it belongs to class 0 or 1, i.e. a output value of 0.76 and the threshold at 0.5 means the sample belongs to class 1 whereas 0.31 means the sample belongs to class 0? Just to make sure, with SoftMax I need two output nodes instead of one. $\endgroup$
    – Andreas
    Commented Mar 3, 2012 at 16:25
  • $\begingroup$ I meant Cross entropy instead of MSE. And for softmax, you only need one output as well. (The probability is 1-p where p is the other probability.) And where you set the threshold depends on your application (e.g. use decision theory to find one) but 0.5 is typically right. $\endgroup$
    – bayerj
    Commented Mar 3, 2012 at 21:21
  • $\begingroup$ Ok, thanks. I've asked for the output node amount because Encog, the nn library I use, always gives me 1 as the final result when I use one output node. When I try two then both nodes sum up to 1. Thus I tried with two output nodes. But if one output node is right then there's probably a bug somewhere. $\endgroup$
    – Andreas
    Commented Mar 4, 2012 at 9:14
  • $\begingroup$ According to the author of Encog one output node always returns result 1, thus for Encog two classes are need when SoftMax is used. $\endgroup$
    – Andreas
    Commented Mar 7, 2012 at 6:25
  • $\begingroup$ I would consider using something else than Encog, since it seems to be rather limited. $\endgroup$
    – bayerj
    Commented Mar 7, 2012 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.