Tag Info

5

If you have only two $\beta_j$ parameters, just plot it in a 3D plot with $\beta_1$ on $x$-axis, $\beta_2$ on $z$-axis, and the loss on $y$-axis. If there is more parameters, there is no easy way to plot them. What you can do, it to use a dimensionality reduction algorithm to reduce the dimensionality of inputs, as authors of the loss landscape paper did, ...

2

Gradient descent is unlikely to ever update such that it sets some weights to exactly 0. This is because at any given step, it's simply unlikely to estimate a gradient update for a parameter that is exactly enough to force the parameter to zero. Additionally, even if the update does force the value to exactly zero at some step, then this may be undone at a ...

1

Changing the mean of the dependent variable makes no practical difference, as all that happens is that $\beta_0$ changes to match that change and you are not penalising $\beta_0$ Rescaling also makes little difference beyond rescaling other parts of the calculation, since trying to minimise \sum^I_{i=1}\left(y_i - \left(\beta_0 + \sum^J_{j=1}\beta_jx_{ij} +...

1

If you use the SGDRegressor in scikit-learn with the epsilon_insensitive loss function specified and the epsilon value set to zero, you will get a model equivalent to LAD with L2 regularization.

Only top voted, non community-wiki answers of a minimum length are eligible