0
$\begingroup$

So, While building a simpel neural net (MLP) for recognizing digits, I ordered my function to print me both the mean cost overall off the train dataset and %currect answers, also over the train dataset (even though the function doing this is called val()). the cost kept going down for an hour and a half, but the %currect answers stayed the same as the beginning. the wierd thing is that while the cosr is constantly going down, the train accurasy do not change AT ALL from the very beggining. enter image description here

Any ideas why this is happening?

Here is the nootbook, if needed.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

It's hard to say without more detail. But most likely, if you're selecting the correct class by picking the maximal class in your softmax, then the loss can keep going down because you're increasing the correct class probability, but if another class always dominates the correct class, then the answer remains the same. For example if you have two classes with probabilities (0.2,0.8), and if the first one is supposed to be correct, then (0.3,0.7) would reduce the loss, but would not change the percent correct.

Looking at your code some more, it looks like you have three layers with 10 hidden units each. You'll most likely need at least 100 hidden units within each layer, so try making them much bigger. Check out Yann Lecun's page on MNIST accuracy here:

http://yann.lecun.com/exdb/mnist

to compare your neural network with the baselines provided

Improve this answer
$\endgroup$
13
  • $\begingroup$ this is what i thought as well. any ideas on how to fix this? this is my first neural net so im kinda lost. $\endgroup$
    – Moran Reznik
    Dec 1, 2017 at 19:23
  • $\begingroup$ @מורןרזניק: I don't see anything obviously wrong with your code, so I suggest you take a look at Yann Lecun's MNIST accuracy page to get a sense of how good or bad you're doing: yann.lecun.com/exdb/mnist Yours looks to be a (non-convolutional) neural net, so I would see if increasing the width of your layers, or adding on more layers improves the accuracy. If you want more accuracy, you'll likely have to start using convolutional neural nets. $\endgroup$
    – Alex R.
    Dec 1, 2017 at 19:56
  • $\begingroup$ @מורןרזניק: See edit $\endgroup$
    – Alex R.
    Dec 1, 2017 at 20:00
  • $\begingroup$ I followd the stracture from here:youtube.com/watch?v=aircAruvnKk&t=480s. He got a pretty great accurasy with 2 hidden layers with 16 units each. $\endgroup$
    – Moran Reznik
    Dec 1, 2017 at 20:05
  • $\begingroup$ @מורןרזניק: Could you provide a reference for the accuracy (and whether the same MNIST dataset was used)? I didn't see a mention of it. $\endgroup$
    – Alex R.
    Dec 1, 2017 at 20:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.