I am trying to list the differences in the ways a regularized general linear model would, in theory, be different from a neural network and linear discriminant analysis.

I have a case where the number of cases is more than the number of features, and there is slight skewness in the sample cases. There is no co-linearity between the features. In most cases, the sample size of the training sample is large (> 100).

Given that a neural network is a non-linear classifier, I would expect it to out-perform the LDA (at least). THe LDA is parameteric, and I assume the regularized GLM is also parametric? I am not sure what effect the lasso or elastic net fitted to the generalized linear model would have on its performance ability as a classifier. I have read Friedman et al.'s paper, but still cannot postulate how such a penalty would improve overall classification.

Can anyone start me off, or direct me to literature that deals with the computational differences between these classifiers, specially the regularized GLM.

Anything posted will be appreciated

  • 1
    $\begingroup$ Your theory needs to take into account the available sample size and the uncertainty on your data as well as the properties/ characteristics of data and problem. Otherwise you are subject to a no free lunch situation: your problem is so general that it cannot be solved. Not even theoretically. $\endgroup$ Dec 3, 2013 at 9:13
  • $\begingroup$ Thanks, made some edits. I think I should have just asked for the differences in the way they classify data. I am particularly concerned with the regularized GLM. $\endgroup$ Dec 3, 2013 at 16:12

1 Answer 1


To get you started,

  • the Elements of Statistical Learning have a nice discussion about regularization, and also sound discussions of different models
  • To judge whether a particular regularization is a good idea, you need to take into account you data as well. E.g. for the LASSO, does it make sense for your data to assume that you have noise-only variates that should be excluded and information-carrying variates that should be included (as opposed to: do you expect that the separating information will be spread over all or nearly all variates)
  • Also, Ripley's Pattern Recognition and Neural Networks discusses the artifical neural nets in R package nnet. This package has a function multinom which uses an ANN without hidden layser to fit (multinomial) logistic regression models. I think the NN book possibly together with the MASS book should get you started on the connection between ANN and GLM in this case.
  • There is a close relationship between LDA and logistic regression: the posterior probabilities of a binary LDA is actually a logistic function.

  • Whether non-linear is better than linear obviously depends on the structure of your data: if the classes are linearily separable, then a non-linear classifier won't help (but may be worse due to overfitting).

  • Whether e.g. LDA is better than, say, logistic regression will depend on how much the assumptions of the LDA are violated.
    However, the Elements of Statistical Learning suggest that in practice, there is rarely much of a difference. And, of course, regularization or not is independent of whether you use LDA or GLM/logistic regression.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.