I have a dataset of N normalized features, and outcomes of the form 1.0 and 0.0 (win and loss), split 50/50 into training and test data (about 50000 samples each).
I train the artificial neural network on the training data with M hidden nodes, where the normalized features are the input and the target is the 1.0/0.0 outcome value of the outcome. Hence the ANN has layer nodes of (N, M, 1).
Then, I test the performance of the ANN by activating across the test data, using the output layer directly as a probability estimate. The accuracy of the probabilities is evaluated using the AUROC function against the actual outcomes.
The results are much better than expected, verified multiple times, and I suspect overfitting. However I have not optimized hyperparameters or selected from a large number of trained ANN's ie. this is really a first pass. The test data unseen during training, and the training data is not used for testing. And yet I'm almost sure it has overfitted.
My questions:
1) Does the excellent performance against unseen test data absolutely rule out the possibility of having overfitted?
2) Are there other ways to have overfit which could produce this result?
3) In this context, is it valid to use the output of the ANN as a probability estimate of win/loss? (I see other methods saying that it should be a classification output layer with a softmax layer to get probabilities.... is there any benefit over what I'm doing?)