I'm working on estimation of pressure based on a noisy sensor for an R&D project. I've achieved good results with GBM \ Random Forest using the excellent xgboost software library. I'm trying to compare this model to a Tensorflow neural network regression. I know this may not be the optimal approach, as highlighted here and the simplistic models recommended here.

Is there a good neural network model (not necessarily deep) that can be used for regression? I am aware that neural nets usually have "smooth" activation functions which can be used for regression, however I don't know which model is useful and\or is competitive with more classic "statistical" regression models (such as Lasso, ElasticNet etc.).

  • $\begingroup$ All neural networks are in a sense regression models, so all neural network models can be used for regression. Whether they are successful or the right choice is dependent on the problem domain. For the general question under what circumstances do neural networks out perform other models, I can't really help you. Maybe someone knows some research that gives guidelines on this, but it seems a very broad question $\endgroup$
    – dimpol
    Nov 10, 2016 at 8:52
  • $\begingroup$ Thanks for the comment @dimpol, I am aware that this is a broad question which is domain dependent. I'm looking for a model that has proven properties, such as dealing well with the bias\variance tradeoff or other useful properties which have shown promise over other regression methods. $\endgroup$
    – Leeor
    Nov 10, 2016 at 8:57

1 Answer 1


These are different weight (coefficient) regularization methods. Weight regularization modifies the objective function that we minimize by adding additional terms that penalize large weights. The suitability of the specific weight regularization you want to apply depends on your data and use case.

Lasso (L1) regularization is widely used in domains with massive datasets where efficient and fast algorithms are essential. The lasso is not robust to high correlations among predictors and will arbitrarily choose one and ignore the others and break down when all predictors are identical. The lasso penalty expects many coefficients to be close to zero, and only a small subset to be larger (and nonzero).

Ridge (L2) regularization is ideal if there are many predictors all with non-zero coefficients and drawn from a normal distribution. It shrinks the coefficients of correlated predictors equally towards zero, and is therefore suitable for cases with many predictors, each having small effects on the outcome. L2 regularization prevents coefficients of linear regression models with many correlated variables from being poorly determined and exhibiting high variance.

Elastic net is an extension of the lasso that is robust to extreme correlations among the predictors. It uses a mixture of the L1 (lasso) and L2 (ridge) penalties, and was first proposed for analyzing high dimensional data.

For further details on these three, see [1].

A final, (deep) neural network specific weight regularization with multiple hidden layers is the Max norm constraint.

Max Norm regularization has a similar goal of attempting to restrict the weights from becoming too large. Max norm constraints enforce an absolute upper bound on the magnitude of the incoming weight vector for every neuron and use projected gradient descent to enforce the constraint. In other words, anytime a gradient descent step moved the incoming weight vector such that L2||w||>c, we project the vector back onto the sphere (centered at the origin) with radius c.

Alternative ways to prevent over-fitting

Note that you can regularize a neural net with other methods as well, such as:

i. modifying the number of hidden units, where higher number of hidden units refer to a more complex model, and you can carry out model comparison by looking at the goodness of fit or MLE and a complexity penalty as in AIC or BIC.

ii. earlystopping - where you stop training your model once you observe that generalization is deteriorating, i.e. the test error starts growing.

iii. apply drop-out to prevent co-adaptation of neurons. This has a lengthier explanation, I am giving you the link to the original Srivastava & Hinton, et al paper instead.

  • $\begingroup$ Thanks for the answer! If I understand correctly, you suggest adapting the cost function and keeping the model simple (in a nutshell)? $\endgroup$
    – Leeor
    Nov 10, 2016 at 9:01
  • $\begingroup$ @Leeor, yes and make sure you are not overtraining (the early stop point) $\endgroup$
    – Zhubarb
    Nov 10, 2016 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.