1
$\begingroup$

For datasets of higher dimensions, how do I decide if a Linear model is sufficient to fit the data or if I have to use non linear models like regression trees to fit the data ?

NOTE:I did try both linear and non linear models to fit the data and observed that the mean squared error is substantively reduced by replacing linear regression model with a non-linear regression model (like M5 Regression Trees). But I do not understand how to visualize linear relationships in higher dimensions. So if I fit my 5 dimensional dataset using a linear model, what would be a threshold that would suggest that there is a necessity to adopt non linear models like Regression Trees ?

I am beginning to learn statistics only recently, so apologize if this question is too elementary in nature.

$\endgroup$
1
$\begingroup$

Welcome to our site. Of course in regression problems like this (along with interpretability) our main goal is accuracy. Why are we afraid of going from linear models to more complex models, because by adding more parameters we may be over-fitting to our data.

Arguable the best way of dealing with this is hold-out-testing, and cross-validation. If you add parameters, and accuracy improves, but on a held-out-set the accuracy is poor, then you've over-fit to your data.

There are also information-theoretic methods for comparing models of varying complexity, such as the Likelihood-Ratio-Test and AIC. One special case of model comparison is nested models where all of the resulting regressors of one model are also possible in the other, for instance comparing Linear Regression with some higher order Polynomial Regression. In these cases, the training set accuracy will always be as good or better for the more general model.

$\endgroup$
  • $\begingroup$ Thank you very much for your answer. I see sharp increase in training accuracy as well as cross validation accuracy by moving to a regression tree (non Linear Model). I understand much clearly now $\endgroup$ – nsr Oct 1 '15 at 22:52
  • $\begingroup$ I did up-vote the answer. I guess I have to gain some reputation before it appears here. $\endgroup$ – nsr Oct 3 '15 at 22:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.