6
$\begingroup$

Figure below shows the accuracy using different alpha values in L2 regularisation. As long as alpha is small in the range of $10^{-12}$ to $10^{-2}$ the accuracy remain the same. I do undarstand when alpha value is $10^{1}$ or greater it will increase the weights to a point where they do not fit the data optimal and then, resulting in under-fitting. What is the reason why the accuracy remain the same by smaller alpha values? The formula is: $$w^{2}=w^{1}-\alpha*w^{1}$$

where $w^{1}$ and $w^{2}$ are respectively the recent and the regularised weight parameters, $\alpha$ is the regularisation parameter specifying the amount of regularisation.

Applying different alpha values in L2 regularisation

$\endgroup$
2
  • $\begingroup$ What is alpha? Does it represent the significance level of a particular statistical test? $\endgroup$ Jan 16, 2017 at 18:32
  • $\begingroup$ @MichaelChernick please check the update $\endgroup$
    – AdiT
    Jan 16, 2017 at 18:54

2 Answers 2

16
$\begingroup$

The various loss functions reflect different answers to the question What makes a model "good"? Choosing one loss function over another is implicitly choosing one interpretation of "model goodness" over another.

In the case of inaccuracy, the loss function just checks whether the largest predicted value matches the target class, and the loss is the fraction incorrectly predicted.

Cross-entropy loss awards lower loss to predictions which are closer to the class label.

The difference between cross-entropy loss and inaccuracy is the difference between taking a course pass/fail or for a grade. A pass/fail rubric just tells you whether minimum requirements were satisfied. A letter grade tells you how well a student performed. Thus, the inaccuracy score is concealing the effect of the $\alpha$ parameter: the predicted values for the observations are almost certainly changing at different values $\alpha$, but the relative ranking of the scores isn't, so (in)accuracy is flat.

For illustration, consider the following results

observation $\alpha_1$ $\alpha_2$ label
1 0.01 0.49 0
2 0.99 0.51 1

These two models will obviously have the same (in)accuracy for this data, but the cross-entropy loss will be completely different.

The inaccuracy is obviously $0.0$ for both $\alpha_1$ and $\alpha_2$, using the rule that argmax of the predictions is the predicted class. Moreover, since the inaccuracy is bounded below by $0.0$, it is impossible to improve upon either model when the models are compared on the basis of inaccuracy loss.

However, the cross-entropy loss in the case of $\alpha_1$ is $-\log(0.99)-\log(1-0.01)=-2\log(0.99)\approx 0.02.$

For $\alpha_2$, the cross-entropy loss is $-\log(0.51)-\log(1-0.49)=-2\log(0.51)\approx 1.35$. This shows that a metric which is sensitive to the degree of precision of the predictions is more informative than one which is not, in the sense that more (less) confidence about the correct class is reflected in the loss.

Stated another way, the (in)accuracy of $\alpha_1$ and $\alpha_2$ is the same, so inaccuracy loss on its own is not sufficient to distinguish between the two models.

By contrast, the cross entropy loss is completely different: $\alpha_1$ is the better model according to cross-entropy loss, because its loss is lower than the loss for $\alpha_2$.

$\endgroup$
3
  • $\begingroup$ Hmm I do not understand your answer ? Can you explain it according the formula I wrote above? $\endgroup$
    – AdiT
    Jan 16, 2017 at 19:12
  • 1
    $\begingroup$ I don't know how I could make this more clear. Your formula doesn't directly bear on either the cross-entropy loss or accuracy. $\endgroup$
    – Sycorax
    Jan 16, 2017 at 19:31
  • 3
    $\begingroup$ When your alpha gets below about $10^-2$, your accuracy doesn't improve by making it smaller. @Sycorax 's point is that this is because your accuracy measure is very coarse. Moving your alpha around doesn't change the 0-1 classifications, so your accuracy doesn't change. It would change the probabilities, which would be captured by a loss such as cross-entropy, but basically small alphas are all the same from the point of view of 0-1 loss. $\endgroup$
    – jbowman
    Jan 16, 2017 at 21:34
0
$\begingroup$

There is no direct relationship between alpha and accuracy. The alpha influences the weights of the model. And the weights influence the loss of the model. Finally the loss influences the accuracy of the model. The change of alpha will certainly lead to the change of the value of the weight and the loss, but different weights and loss don't mean different accuracy. The reason is in the answer of @Sycorax.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.