Just for sanity checks, I am checking that, can my network overfit over some 50 examples of cifar-10. But every time I see that my training accuracy is not going above 40% but training error decreases with every iteration. I am using lasagne and vgg type network. Any help for choosing hyperparameters like learning rate, momentum, regularization parameter could be bonus for me. Thanks for your attention.
$\begingroup$
$\endgroup$
4
-
$\begingroup$ This may be a dumb question: but which accuracy are you referring to? You mean the accuracy on the training batch of size 50? Or the accuracy of your CNN on some validation/test set? $\endgroup$– Indie AICommented Mar 19, 2016 at 12:59
-
$\begingroup$ @IndieAI fine, seems dumb question but answer it and it is training accuracy. $\endgroup$– SiddharthCommented Mar 20, 2016 at 4:47
-
$\begingroup$ You'll need to be more specific about exactly what you mean by "training error" versus "training accuracy"; they typically mean one minus the other. $\endgroup$– DanicaCommented Mar 20, 2016 at 5:02
-
$\begingroup$ @Dougal training accuracy is the fraction of correctly predicted examples from training set and training error is cost function which is cross entropy. $\endgroup$– SiddharthCommented Mar 20, 2016 at 17:57
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
What you're calling "training error" would more typically be referred to as "training loss."
Call your training set $\{ (x_i, y_i) \}_{i=1}^n$, where $y_i$ is an integer from 1 to the number of classes $m = 10$; then your model predicts $\hat p_{ij}$ for each data point $i$ up to $n$ and each class $j$ up to $m$.
Your training accuracy can be written as $$\frac1n \sum_i \begin{cases}1 & y_i = \arg\max_j \hat p_{ij} \\ 0 & \text{otherwise}\end{cases}.$$ And your training loss is $$ -\sum_i \log\hat p_{i y_i} .$$ So, what your network is doing is probably driving up $\hat p_{i y_i}$ for the cases it's sure about, without changing the relative ordering for the "difficult" cases. You could diagnose this further by looking at the probability outputs over time for your training set, since there are only 50 of them.
So, why doesn't it get more of them right? There could be a lot of reasons for that, but my guess is probably that it's because you're just giving it the same input batch every time, so the gradient it steps along never changes, and it's stuck in a local minimum of sorts that it could probably get out of if you have it some different inputs. It could also definitely be something related to the regularization you're using, an adaptive step size scheduler getting confused, or other problems related to the exact specifications of the problem that you haven't told us and may be difficult to figure out even so.
$\begingroup$
$\endgroup$
Well to put it in simple way this happens when your model learns to recognize the right answer as well as the WRONG ANSWER.For example, if you classifying amongst 3 classes namely "apple", "mango", and "banana".Let say the output prediction for 3 classes in same order is [0.6,0.55,0.5].Lets also assume your threshold is 0.6 .Now assume the first label is apple, BAM!! u got correct answer i.e your model detected "apple" in the first iteration by yielding a prediction probability of 0.6 which is equal to the threshold.Lets say the same process gets repeated with same kind of output values for next few iterations. assuming everything going right including your predictions, you will see your accuracy boosting up.but loss almost same.Why? Because it gave u correct answer it is not very confident on wrong values as well as the correct values by giving value like 0.6, 0.5,0.55 which is not very far from threshold.At this point we say that model is not very good at determining wrong answers. had we got the values like 0.9,0.3,0.2 , now these kinda values suggests that model is becoming better at determining not only correct answer with full confidence but also what is wrong with full confidence. A prime reason for such kind of behaviour is BAD WEIGHT INITILIZATION. There are lot of papers related to it.Refer https://arxiv.org/abs/1704.08863
