I am new to Neural Networks and but I have built a multi-classifier using the FANN neural network package.
My multi-classifier, regardless of the network hyperparameters, consistently gives an error around 10% (it changes based on the network configuration only 2-3 d.p. after the 10%, and these changes are completely undeterministic) on my test set. I am using k-fold cross validation with about 5000 events in total and about 5 folds.
Furthermore, I have built functionality for creating ROC curves, one for each class, assuming if that class' NN output value > some threshold T then it was predicted to be an example of that class and no otherwise. For some reason, I am almost consistently getting 0.5 AUCROC for each class' ROC curve. Sometimes, I get a slightly bigger value for some class with some set of network hyperparameters (never above 0.6 though) but again its undeterministic because if I run the network again with the same hyperparameters I get different AUCROC values for the different classes.
What exactly does this mean?
In obtaining my data, I am applying a pre-selection for obtaining samples (making my total samples go from around 50,000 to 5000) (e.g. dimension x > 5). Could it be that the classifier is having trouble classifying samples that pass the preselection?