I am a little new to neural network. I have two questions: 1. Can we use neural network in the small $n$ large $p$ situation? 2. Any regularization methods designed for parameter elimination? (Shrink some $\omega$ to 0). For example to add $L_1$ regularizer as Lasso? I know some methods for avoiding over-fitting, (weight decay, early stopping, ect.) but none of them shrink parameters to 0.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
you could iteratively reweight the parameters estimated by the $L_1$ regularization. Also what I do, is to apply the $L_1$ Lasso with a relatively large weighting $\lambda$ and then to remove all the parameters, which are at zero. Then, ordinary ridge regression with a very small $\lambda$ can be applied efficiently, so you dont have drifts between the estimated and the "target" values due to regularization.
See
- Generalized cross validation for obtaining $\lambda$,
- Reweighting of lasso
- Sparse Redundant Representations, somewhere in the chapter for shrinking algorithms. M. Elad calls the refitting with LSQ "projection".
-
$\begingroup$ Yes. But your answer seems to apply for linear regression. My question is for artificial neural network. $\endgroup$ Commented Feb 3, 2014 at 15:16
-
1$\begingroup$ Well, these are very related problems. Consider the linear regression of type $y_j=\sum_i^N \varphi(x_i, x_j)\alpha_i$ where $\alpha_i$ are your parameters (weights in terms of ANN). In the case of ANN, you have the problem above multiple times, which leads to "deep learning". $\endgroup$– mojovskiCommented Feb 3, 2014 at 15:36