# What type of regression will improve the prediction for these data?

I'm quite new to regression problems in general. I have a simple data with 1 feature. I am trying to fit a regression model so that I can predict on new data set. So far, I tried to fit a linear regression and result are as below (input x plotted on x-axis and output y plotted on y-axis):

I wanted to ask what other regression models I can try to get better fit?

Update: I coded up polynomial regression and as per suggested I tried 3rd order polynomial. Also higher order polynomials were clearly overfitted. I also tried to predict in (-2.5 - 2.5) range. Here's what I got:

• I am getting the impression there is an S-shapes curve in your data. Have you compared your simple linear fit with a higher order polynomial? Jan 19, 2016 at 19:27
• So should I use features such x^2, x^3? Jan 19, 2016 at 19:40
• You could go as high as a cubic power and see what happens. Remember however that in case you want to include polynomial terms, you first have to center the predictors and include all lower order terms as well. Let me know how it goes. Jan 19, 2016 at 19:42
• You may want to try trigonometric functions (eg, sin & cos) instead of polynomials, as well. Jan 19, 2016 at 19:46
• @JohnK thank you for the suggestion. I tried cubic polynomial and I think it looks better, thank you! Jan 21, 2016 at 8:40

Your data seems slightly S-shaped, so you could try a cubic (higher order polynomial than cubic runs the risk of overfitting).

• I did try your suggestion of using cubic polynomial. I have updated my post to include the same. I think it looks better than linear. Thanks! Jan 21, 2016 at 8:38

What does the science behind the data suggest?

Yes, you can fit many different models including polynomials, splines, trigonometric functions, non-parametric smoothers, etc. But what is most important is what make sense based on the science behind the data (having fancy computers does not excuse us from understanding where our data comes from).

What do you expect to happen to your y variable as x increases beyond 2? decreases beyond -2? This can tell a lot about what types of models would be meaningful. A cubic polynomial will head quickly towards infinity as x increases and quickly towards negative infinity as x decreases. A natural spline would also head towards infinity and negative infinity, just not as quickly. But maybe based on the science you believe that there are upper and lower bounds that y will approach but not continue past, this would be a very different type of model.

Polynomials, splines, and smoothers can approximate other functions (remember Taylor Series from calculus?), but if you have an idea of what the true relationship will be then fitting that model directly can enlighten more than an approximation.

• I agree that the science behind the data would give much more insights to what model would be best, but unfortunately I do not have knowledge of the data as this was merely an exercise about linear regression and I was curious to know what would be better! Jan 21, 2016 at 8:33

I'd recommend a generalized additive model rather than a polynomial. This would allow a non-parametric smoother which is more flexible to follow your s shape.

However, there are numerous other options. Much depends on the purpose of fitting a model at all. Is it for prediction, or for explanatory purposes? What sort of inferences do need to make? etc.

• This is a good idea of course but I do not think the OP would like to go that far. Jan 19, 2016 at 20:27
• maybe... OP's call. If they want to use R, fitting a gam is very nearly as simple as fitting the original regression, and is just a one-liner. I would think that adding polynomials or trigonometric functions by hand (some of the other suggestions) would be in fact more complex. Jan 19, 2016 at 20:32
• It's not only about the programming though. It is about understanding as well. And it seemed to me that the OP has just learned about the simple linear model! Jan 19, 2016 at 20:35

I would say it depends in the type oft response. Is it deterministic or noisy?

If it is noisy data i would prefer Gaussian Processes so that you can not only extract a predictor value but also the likeliness.

A general introduction to this type of Model is given here: scikit-lean GP

• This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? We can also turn it into a comment. Jan 19, 2016 at 20:49