0
$\begingroup$

I am building a machine learning model to attempt to predict the winner of a sports match based on historical statistics of the two teams.

My model (a neural network) appears to get about 70% accuracy on test data which was better than I expected. However there are some weird things going on with the accuracy and loss over time charts.

Accuracy and Loss over time.

(Blue is training data, red is test data)

As you can see, the accuracy starts off flat, then at about 1000 training iterations it jumps straight to very close to the final values. It appears to get stuck predicting the same winners for each match in the first 1000 iterations despite the loss dropping significantly in that time.

The other thing I'm not sure about is how closely the loss functions for training and test data match. it looks like they are the same just offset.

What could be going wrong here? I'm not sure what direction to look.

More info:

My loss function is cross entropy, activation is ReLU, regularization is dropout. Weights are initialized with truncated normal distribution. The network itself is just a 5 layer feed forward ANN using Adam for training.

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ Why do you use 5 layers? Have you tried less and it didn't work? $\endgroup$ – Łukasz Grad Apr 4 '17 at 9:12
  • $\begingroup$ Fewer layers yield nearly the same result, just a few % less accuracy. I'm aware 5 is a bit of overkill :) $\endgroup$ – null0pointer Apr 4 '17 at 9:13
  • 2
    $\begingroup$ The loss graph looks pretty normal to me, sometimes even with momentum optimizers need some time to escape starting positions, though the higher accuracy on test set is suspicious. Also, try to change dropout for standard l2 regularization, drop outs, AFAIK, are used in very deep (e.g. convolutional) and large networks $\endgroup$ – Łukasz Grad Apr 4 '17 at 9:21
  • 1
    $\begingroup$ To continue on accuracy, what are the dataset sizes? The sudden increase may be due to homogeneity of your samples, i.e. large portion of them have the same characteristic, and once network learns to distinguish it, all samples are suddenly classified correctly $\endgroup$ – Łukasz Grad Apr 4 '17 at 9:27
  • $\begingroup$ The dataset only has about 3000 records (the last 16 years of match data). I do an 80/20 split on training/test. Thanks for the tips I'll try them out! $\endgroup$ – null0pointer Apr 4 '17 at 9:39
1
$\begingroup$

This is an interesting question. I guess that you have observed the effect of using a nonproper scoring rule. Accuracy is a noncontinuous (and nonproper) scoring rule, it will only change when the parameters under learning have changed sufficiently to actually change the decision. What you call Loss in the plot, cross-entropy, is akin to logistic regression, it is a continuous function of the parameters, and a proper scoring rule. So the difference you see in the plots is kind of expected, though the magnitude is interesting!

Some other posts with more information is Why isn't Logistic Regression called Logistic Classification? and Alternative notions to that of proper scoring rules, and using scoring rules to evaluate models

Here is another post which seems to run into the same problem: Probability Calibration messes Reliability

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ @null0pointer: To verify this, could you double-check the default threshold that accuracy is calculated by? Is it 0.5? Also, when you initialize your neural network, could you predict on a sample of your data, and verify that at least some of the predictions score above 0.5? Otherwise your initialization may not be optimal. $\endgroup$ – Alex R. Feb 20 '18 at 21:27
  • 1
    $\begingroup$ I strongly suggest dividing train and test along with time. Thus you you will know how the model behaves on examples coming from unseen future as you go in real life. $\endgroup$ – Alexey Burnakov Feb 20 '18 at 21:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.