Is verification with test data sufficient to rule out overfitting of neural network?

I have a dataset of N normalized features, and outcomes of the form 1.0 and 0.0 (win and loss), split 50/50 into training and test data (about 50000 samples each).

I train the artificial neural network on the training data with M hidden nodes, where the normalized features are the input and the target is the 1.0/0.0 outcome value of the outcome. Hence the ANN has layer nodes of (N, M, 1).

Then, I test the performance of the ANN by activating across the test data, using the output layer directly as a probability estimate. The accuracy of the probabilities is evaluated using the AUROC function against the actual outcomes.

The results are much better than expected, verified multiple times, and I suspect overfitting. However I have not optimized hyperparameters or selected from a large number of trained ANN's ie. this is really a first pass. The test data unseen during training, and the training data is not used for testing. And yet I'm almost sure it has overfitted.

My questions:

1) Does the excellent performance against unseen test data absolutely rule out the possibility of having overfitted?

2) Are there other ways to have overfit which could produce this result?

3) In this context, is it valid to use the output of the ANN as a probability estimate of win/loss? (I see other methods saying that it should be a classification output layer with a softmax layer to get probabilities.... is there any benefit over what I'm doing?)

• Can you share actual figures from you run that made you think there was overfitting? – Arun Jose Feb 7 '17 at 8:55
• A previous model produced an AUROC value of 0.72, and the baseline we are dealing with is 0.76 which is assumed to be a very strong model to start with (let's say commercial grade). The first pass at a ANN model produced 0.81 with only a subset of features, and some individual features on there own are can produce 0.77. It seems extremely unlikely that without considering the full feature set, and optimizing hyperparameters etc, that I would get 0.81, hence my suspicion. – Brendan Hill Feb 7 '17 at 10:14
• I ran into a similar problem once while using simple logistic regression. The problem was that one of my independent variables had a very strong relationship with the target variable. So the fit was extremely good even with the lack of other features. So if your classification inherently is that simple, it means you wouldn't need a complicated ANN to solve for it. You could probably test for this by running a logistic regression in addition to the ANN and studying the model coefficients. – Arun Jose Feb 7 '17 at 10:22
• Thanks Arun. On inspection it does seem like specific features can be modeled by logistic regression, but not their combinations. – Brendan Hill Feb 8 '17 at 10:44

3) From your description, it sounds like your current neural network outputs a single unbounded continuous value when you actually want a probability $p \in [0,1]$. You could use a sigmoid function to do this.