# Linear vs polynomial regression

If you have multiple dimensions to your data, where it is not possible to visualize them together, how to decide if your model should be linear or polynomial?

• why would a cost function be linear? That would imply erring hugely one way would incur a negative cost... Jul 18, 2013 at 5:41
• @Glen_b I think the OP means to ask about whether the cost function should be linear or polynomial in the explanatory variables. E.g. is the model for $Y|X_1,X_2$, or $Y|X_1,X_1X_2,X_2$, or $Y|X_1^2,X_1,X_1X_2,X_2,X_2^2$, etc. Jul 18, 2013 at 8:50
• hi Pranay how many dimensions do you have? Jul 18, 2013 at 8:51
• @TooTone surely the model is not the same thing as the cost function. That would be bizarre. OP should clarify. Jul 18, 2013 at 8:52
• @TooTone $h$ and $J$ are quite different things. $h$ is the model and is linear there; $J$ is the cost function and is not linear. You can't start confusing the two, or you end up asking nonsensical questions about linear cost functions... which takes us right back to where I first commented. Jul 18, 2013 at 16:15

If you did the same course as me, Andrew Ng's machine learning course on coursera, you will remember that it was suggested there to split the data into 3 parts.

1. Training set
2. Cross-validation set
3. Test set

Briefly, you fit each possible model to the training set. However you can't easily use the training set to decide which model is best, because additional terms in a model can only lead to a model fitting the training set better, with the possibility of overfitting (this is when the regression coefficients are "tuned" to noise in the training data). Rather you choose which model to use based on the error on the cross-validation set. Finally, you test how well the chosen model predicts using the test set.

If your model is highly dimensional, then it might be best to consider other approaches.

• Thanks for the answer. So finally it would be like a trial and error method, which in case of very high dimensions becomes infeasible Jul 19, 2013 at 4:21
• @PranayWarke Although there are ways to automate the process, yes it becomes infeasible. You also have to question whether a regression model (linear or polynomial) is suitable. On the whole you might prefer a parsimonious model with fewer parameters, especially if you don't have a large number of examples in your training set. So you could for example investigate whether some of your explanatory variables could be dropped, or choose a different kind of model altogether. You could try searching for "dimensionality reduction" or similar. Jul 19, 2013 at 9:03

@Pranay Just to add, one way to decide weather to fit a linear or a non-linear function is to do residual analysis. Starting from a linear function you need to find the residual values for all predicted values as :

Yactual - Yobserverd

When you plot these values you can decide weather to fit a linear or a polynomial based how the residual values are scattered from the mean as: If the values are randomly scattered then it may be safe to fit a linear function.

On the contrary if you can find a pattern in the residuals, then fitting a linear function is not going to work: The above are the few among the many things you can do with residuals; they can be used to analyze large number of patterns in data.

Hope this helps.

• the above diagrams were taken from link. It also details more ways of using residuals. May 4, 2017 at 0:47
• do you have permission to use the diagrams? ... if you do (such as some license -- which you should indicate for each diagram) you should give proper credit to the person that originally made them in your post, since stackexchange requires proper attribution May 4, 2017 at 2:57
• @Glen_b check my my comment where I have added the link to the above diagrams and stated that the above diagrams were taken from that source as stackexchange does not let users under a certain reputation to add more than 2 links. Unless that is still not sufficient. May 8, 2017 at 18:45