# Deep Neural Networks: AUCROC Values Consistently = 0.5 even though RMS Error on Test Set ~10%

I am new to Neural Networks and but I have built a multi-classifier using the FANN neural network package.

My multi-classifier, regardless of the network hyperparameters, consistently gives an error around 10% (it changes based on the network configuration only 2-3 d.p. after the 10%, and these changes are completely undeterministic) on my test set. I am using k-fold cross validation with about 5000 events in total and about 5 folds.

Furthermore, I have built functionality for creating ROC curves, one for each class, assuming if that class' NN output value > some threshold T then it was predicted to be an example of that class and no otherwise. For some reason, I am almost consistently getting 0.5 AUCROC for each class' ROC curve. Sometimes, I get a slightly bigger value for some class with some set of network hyperparameters (never above 0.6 though) but again its undeterministic because if I run the network again with the same hyperparameters I get different AUCROC values for the different classes.

What exactly does this mean?

In obtaining my data, I am applying a pre-selection for obtaining samples (making my total samples go from around 50,000 to 5000) (e.g. dimension x > 5). Could it be that the classifier is having trouble classifying samples that pass the preselection?

Thanks.

• Are your classes unbalanced? If 90% of the samples are in the same class, then an 10% error is trivial to achieve. How do you calculate the AUC? Once per run of the cross validation or pool the predictions of the test folds and then calculate the AUC? It's also not clear whether the ROC curves are correct, see answer by @user777. – Erik Mar 2 '16 at 15:52
• @Erik I calculate AUC by area under the ROC, and my ROC is calculated on the test set results. As for how I calculate the ROC curves themselves, my output is a vector length C, C being the number of classes. Each element is a probability spit out by the network of how likely the example falls into the class. I vary the threshold value T, and then when analysing the ROC for a given class, I see whether or not the probability value at the index of the class is greater than T; if so, call it an example of that class. From there I calculate the standard confusion matrix. – annikam Mar 2 '16 at 15:53
• @Erik Yes my classes are unbalanced. – annikam Mar 2 '16 at 15:56
• @user1934426 Accuracy is a notoriously poor performance metric for unbalanced class compositions. – Sycorax Mar 2 '16 at 15:57
• @user777 What measure would you suggest using for unbalanced class compositions? – annikam Mar 2 '16 at 15:59

Proper vs improper scoring rules are discussed all over our archives, so you can find those discussions using the [search] feature. Frank Harrell's posts are particularly good. You should always use full information about model outputs, i.e. do not threshold the model outputs. Thresholds create vexing scenarios where the decision rule $\hat{y}>0.5$ treats $\hat{y_1}=0.99$ the same as $\hat{y_2}=0.51$ even though the model is very obviously much more confident about the first instance.