I'm quite new to regression problems in general. I have a simple data with 1 feature. I am trying to fit a regression model so that I can predict on new data set. So far, I tried to fit a linear regression and result are as below (input x plotted on x-axis and output y plotted on y-axis): enter image description here

I wanted to ask what other regression models I can try to get better fit?

Update: I coded up polynomial regression and as per suggested I tried 3rd order polynomial. Also higher order polynomials were clearly overfitted. I also tried to predict in (-2.5 - 2.5) range. Here's what I got: enter image description here

  • 5
    $\begingroup$ I am getting the impression there is an S-shapes curve in your data. Have you compared your simple linear fit with a higher order polynomial? $\endgroup$ – JohnK Jan 19 '16 at 19:27
  • $\begingroup$ So should I use features such x^2, x^3? $\endgroup$ – Pranav Jan 19 '16 at 19:40
  • 2
    $\begingroup$ You could go as high as a cubic power and see what happens. Remember however that in case you want to include polynomial terms, you first have to center the predictors and include all lower order terms as well. Let me know how it goes. $\endgroup$ – JohnK Jan 19 '16 at 19:42
  • $\begingroup$ You may want to try trigonometric functions (eg, sin & cos) instead of polynomials, as well. $\endgroup$ – gung Jan 19 '16 at 19:46
  • $\begingroup$ @JohnK thank you for the suggestion. I tried cubic polynomial and I think it looks better, thank you! $\endgroup$ – Pranav Jan 21 '16 at 8:40

Your data seems slightly S-shaped, so you could try a cubic (higher order polynomial than cubic runs the risk of overfitting).

  • 1
    $\begingroup$ I did try your suggestion of using cubic polynomial. I have updated my post to include the same. I think it looks better than linear. Thanks! $\endgroup$ – Pranav Jan 21 '16 at 8:38

What does the science behind the data suggest?

Yes, you can fit many different models including polynomials, splines, trigonometric functions, non-parametric smoothers, etc. But what is most important is what make sense based on the science behind the data (having fancy computers does not excuse us from understanding where our data comes from).

What do you expect to happen to your y variable as x increases beyond 2? decreases beyond -2? This can tell a lot about what types of models would be meaningful. A cubic polynomial will head quickly towards infinity as x increases and quickly towards negative infinity as x decreases. A natural spline would also head towards infinity and negative infinity, just not as quickly. But maybe based on the science you believe that there are upper and lower bounds that y will approach but not continue past, this would be a very different type of model.

Polynomials, splines, and smoothers can approximate other functions (remember Taylor Series from calculus?), but if you have an idea of what the true relationship will be then fitting that model directly can enlighten more than an approximation.

  • $\begingroup$ I agree that the science behind the data would give much more insights to what model would be best, but unfortunately I do not have knowledge of the data as this was merely an exercise about linear regression and I was curious to know what would be better! $\endgroup$ – Pranav Jan 21 '16 at 8:33

I'd recommend a generalized additive model rather than a polynomial. This would allow a non-parametric smoother which is more flexible to follow your s shape.

However, there are numerous other options. Much depends on the purpose of fitting a model at all. Is it for prediction, or for explanatory purposes? What sort of inferences do need to make? etc.

  • 2
    $\begingroup$ This is a good idea of course but I do not think the OP would like to go that far. $\endgroup$ – JohnK Jan 19 '16 at 20:27
  • $\begingroup$ maybe... OP's call. If they want to use R, fitting a gam is very nearly as simple as fitting the original regression, and is just a one-liner. I would think that adding polynomials or trigonometric functions by hand (some of the other suggestions) would be in fact more complex. $\endgroup$ – Peter Ellis Jan 19 '16 at 20:32
  • 2
    $\begingroup$ It's not only about the programming though. It is about understanding as well. And it seemed to me that the OP has just learned about the simple linear model! $\endgroup$ – JohnK Jan 19 '16 at 20:35

I would say it depends in the type oft response. Is it deterministic or noisy?

If it is noisy data i would prefer Gaussian Processes so that you can not only extract a predictor value but also the likeliness.

A general introduction to this type of Model is given here: scikit-lean GP

  • 2
    $\begingroup$ This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? We can also turn it into a comment. $\endgroup$ – gung Jan 19 '16 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.