2
$\begingroup$

I am a little new to neural network. I have two questions: 1. Can we use neural network in the small $n$ large $p$ situation? 2. Any regularization methods designed for parameter elimination? (Shrink some $\omega$ to 0). For example to add $L_1$ regularizer as Lasso? I know some methods for avoiding over-fitting, (weight decay, early stopping, ect.) but none of them shrink parameters to 0.

$\endgroup$

1 Answer 1

2
$\begingroup$

you could iteratively reweight the parameters estimated by the $L_1$ regularization. Also what I do, is to apply the $L_1$ Lasso with a relatively large weighting $\lambda$ and then to remove all the parameters, which are at zero. Then, ordinary ridge regression with a very small $\lambda$ can be applied efficiently, so you dont have drifts between the estimated and the "target" values due to regularization.

See

$\endgroup$
2
  • $\begingroup$ Yes. But your answer seems to apply for linear regression. My question is for artificial neural network. $\endgroup$
    – yliueagle
    Feb 3, 2014 at 15:16
  • 1
    $\begingroup$ Well, these are very related problems. Consider the linear regression of type $y_j=\sum_i^N \varphi(x_i, x_j)\alpha_i$ where $\alpha_i$ are your parameters (weights in terms of ANN). In the case of ANN, you have the problem above multiple times, which leads to "deep learning". $\endgroup$
    – mojovski
    Feb 3, 2014 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.