1
$\begingroup$

I have a dataset of N normalized features, and outcomes of the form 1.0 and 0.0 (win and loss), split 50/50 into training and test data (about 50000 samples each).

I train the artificial neural network on the training data with M hidden nodes, where the normalized features are the input and the target is the 1.0/0.0 outcome value of the outcome. Hence the ANN has layer nodes of (N, M, 1).

Then, I test the performance of the ANN by activating across the test data, using the output layer directly as a probability estimate. The accuracy of the probabilities is evaluated using the AUROC function against the actual outcomes.

The results are much better than expected, verified multiple times, and I suspect overfitting. However I have not optimized hyperparameters or selected from a large number of trained ANN's ie. this is really a first pass. The test data unseen during training, and the training data is not used for testing. And yet I'm almost sure it has overfitted.

My questions:

1) Does the excellent performance against unseen test data absolutely rule out the possibility of having overfitted?

2) Are there other ways to have overfit which could produce this result?

3) In this context, is it valid to use the output of the ANN as a probability estimate of win/loss? (I see other methods saying that it should be a classification output layer with a softmax layer to get probabilities.... is there any benefit over what I'm doing?)

|cite|improve this question
$\endgroup$
  • $\begingroup$ Can you share actual figures from you run that made you think there was overfitting? $\endgroup$ – Arun Jose Feb 7 '17 at 8:55
  • $\begingroup$ A previous model produced an AUROC value of 0.72, and the baseline we are dealing with is 0.76 which is assumed to be a very strong model to start with (let's say commercial grade). The first pass at a ANN model produced 0.81 with only a subset of features, and some individual features on there own are can produce 0.77. It seems extremely unlikely that without considering the full feature set, and optimizing hyperparameters etc, that I would get 0.81, hence my suspicion. $\endgroup$ – Brendan Hill Feb 7 '17 at 10:14
  • $\begingroup$ I ran into a similar problem once while using simple logistic regression. The problem was that one of my independent variables had a very strong relationship with the target variable. So the fit was extremely good even with the lack of other features. So if your classification inherently is that simple, it means you wouldn't need a complicated ANN to solve for it. You could probably test for this by running a logistic regression in addition to the ANN and studying the model coefficients. $\endgroup$ – Arun Jose Feb 7 '17 at 10:22
  • $\begingroup$ Thanks Arun. On inspection it does seem like specific features can be modeled by logistic regression, but not their combinations. $\endgroup$ – Brendan Hill Feb 8 '17 at 10:44
1
$\begingroup$

1) If your training loss is much lower than your test loss, then your model is likely overfitting. If training and test loss are similar, then your model is generalizing well. Accuracy on the held-out test set gives you an estimate of how well you can make predictions on unseen data.

2) Why do you believe that you're overfitting? Is it possible that this is just a very easy binary classification problem?

3) From your description, it sounds like your current neural network outputs a single unbounded continuous value when you actually want a probability $p \in [0,1]$. You could use a sigmoid function to do this.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks. The AUROC evaluation of the training data differs only negligibly from the test data. Given that, you would agree that there is therefore no reason, from the results along, to suspect overfitting? $\endgroup$ – Brendan Hill Feb 7 '17 at 10:24
  • $\begingroup$ Re. the output function - is there any problem with training with targets of {0.0, 1.0} then treating the ANN output (which will be in [0.0, 1.0] directly as a probability? It appears to work well in my example but I have found nothing about the theoretical aspects of this. $\endgroup$ – Brendan Hill Feb 7 '17 at 10:28
  • $\begingroup$ Your dataset is probably pretty large relative to the size of your neural network, which also makes overfitting less likely. As long as you didn't accidentally train on your test data you should be good! $\endgroup$ – Neal Jean Feb 7 '17 at 10:29
  • $\begingroup$ If you train with targets of {0, 1}, it's definitely possible to have predictions on test data that fall outside of this range (e.g., if a test point has features that are very different than those that appear in your training set). A sigmoid will restrict your predictions to valid probabilities between 0 and 1. $\endgroup$ – Neal Jean Feb 7 '17 at 10:32
  • $\begingroup$ Thanks for the comments. When I predict on test data I clamp to [0, 1] to avoid invalid probabilities. Seems to work - I just can't find any external validation for this technique! $\endgroup$ – Brendan Hill Feb 8 '17 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.