Theoretical performance of different classifiers

Question

I am trying to list the differences in the ways a regularized general linear model would, in theory, be different from a neural network and linear discriminant analysis.

I have a case where the number of cases is more than the number of features, and there is slight skewness in the sample cases. There is no co-linearity between the features. In most cases, the sample size of the training sample is large (> 100).

Given that a neural network is a non-linear classifier, I would expect it to out-perform the LDA (at least). THe LDA is parameteric, and I assume the regularized GLM is also parametric? I am not sure what effect the lasso or elastic net fitted to the generalized linear model would have on its performance ability as a classifier. I have read Friedman et al.'s paper, but still cannot postulate how such a penalty would improve overall classification.

Can anyone start me off, or direct me to literature that deals with the computational differences between these classifiers, specially the regularized GLM.

Anything posted will be appreciated

Answer 1

To get you started,

• the Elements of Statistical Learning have a nice discussion about regularization, and also sound discussions of different models
• To judge whether a particular regularization is a good idea, you need to take into account you data as well. E.g. for the LASSO, does it make sense for your data to assume that you have noise-only variates that should be excluded and information-carrying variates that should be included (as opposed to: do you expect that the separating information will be spread over all or nearly all variates)
• Also, Ripley's Pattern Recognition and Neural Networks discusses the artifical neural nets in R package nnet. This package has a function multinom which uses an ANN without hidden layser to fit (multinomial) logistic regression models. I think the NN book possibly together with the MASS book should get you started on the connection between ANN and GLM in this case.

• There is a close relationship between LDA and logistic regression: the posterior probabilities of a binary LDA is actually a logistic function.

• Whether non-linear is better than linear obviously depends on the structure of your data: if the classes are linearily separable, then a non-linear classifier won't help (but may be worse due to overfitting).

• Whether e.g. LDA is better than, say, logistic regression will depend on how much the assumptions of the LDA are violated.
  However, the Elements of Statistical Learning suggest that in practice, there is rarely much of a difference. And, of course, regularization or not is independent of whether you use LDA or GLM/logistic regression.