1
$\begingroup$

I am new to Neural Networks and but I have built a multi-classifier using the FANN neural network package.

My multi-classifier, regardless of the network hyperparameters, consistently gives an error around 10% (it changes based on the network configuration only 2-3 d.p. after the 10%, and these changes are completely undeterministic) on my test set. I am using k-fold cross validation with about 5000 events in total and about 5 folds.

Furthermore, I have built functionality for creating ROC curves, one for each class, assuming if that class' NN output value > some threshold T then it was predicted to be an example of that class and no otherwise. For some reason, I am almost consistently getting 0.5 AUCROC for each class' ROC curve. Sometimes, I get a slightly bigger value for some class with some set of network hyperparameters (never above 0.6 though) but again its undeterministic because if I run the network again with the same hyperparameters I get different AUCROC values for the different classes.

What exactly does this mean?

In obtaining my data, I am applying a pre-selection for obtaining samples (making my total samples go from around 50,000 to 5000) (e.g. dimension x > 5). Could it be that the classifier is having trouble classifying samples that pass the preselection?

Thanks.

$\endgroup$
  • $\begingroup$ Are your classes unbalanced? If 90% of the samples are in the same class, then an 10% error is trivial to achieve. How do you calculate the AUC? Once per run of the cross validation or pool the predictions of the test folds and then calculate the AUC? It's also not clear whether the ROC curves are correct, see answer by @user777. $\endgroup$ – Erik Mar 2 '16 at 15:52
  • $\begingroup$ @Erik I calculate AUC by area under the ROC, and my ROC is calculated on the test set results. As for how I calculate the ROC curves themselves, my output is a vector length C, C being the number of classes. Each element is a probability spit out by the network of how likely the example falls into the class. I vary the threshold value T, and then when analysing the ROC for a given class, I see whether or not the probability value at the index of the class is greater than T; if so, call it an example of that class. From there I calculate the standard confusion matrix. $\endgroup$ – annikam Mar 2 '16 at 15:53
  • $\begingroup$ @Erik Yes my classes are unbalanced. $\endgroup$ – annikam Mar 2 '16 at 15:56
  • $\begingroup$ @user1934426 Accuracy is a notoriously poor performance metric for unbalanced class compositions. $\endgroup$ – Sycorax Mar 2 '16 at 15:57
  • $\begingroup$ @user777 What measure would you suggest using for unbalanced class compositions? $\endgroup$ – annikam Mar 2 '16 at 15:59
2
$\begingroup$

This is almost certainly because of how you've mis-defined AUROC. Fixing a threshold and then doing performance assessment on the basis of the threshold is an improper scoring method because it only uses partial information about the data. I don't know about multi-class AUROC, but that's the literature you'll want to read to make progress. But methods like cross-entropy and Brier generalize more readily to multi-class cases, so you have options there as well. There are variations of multi-class ROC that are insensitive to class imbalance.

Proper vs improper scoring rules are discussed all over our archives, so you can find those discussions using the [search] feature. Frank Harrell's posts are particularly good. You should always use full information about model outputs, i.e. do not threshold the model outputs. Thresholds create vexing scenarios where the decision rule $\hat{y}>0.5$ treats $\hat{y_1}=0.99$ the same as $\hat{y_2}=0.51$ even though the model is very obviously much more confident about the first instance.

$\endgroup$
  • $\begingroup$ Thanks. Can you perhaps tell me why my scoring method is wrong and how I should do it otherwise? $\endgroup$ – annikam Mar 2 '16 at 15:52
  • $\begingroup$ Please see my revision. $\endgroup$ – Sycorax Mar 2 '16 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.