Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm quite new to regression problems in general. I have a simple data with 1 feature. I am trying to fit a regression model so that I can predict on new data set. So far, I tried to fit a linear regression and result are as below (input x plotted on x-axis and output y plotted on y-axis): enter image description here

I wanted to ask what other regression models I can try to get better fit?

Update: I coded up polynomial regression and as per suggested I tried 3rd order polynomial. Also higher order polynomials were clearly overfitted. I also tried to predict in (-2.5 - 2.5) range. Here's what I got: enter image description here

share|improve this question
I am getting the impression there is an S-shapes curve in your data. Have you compared your simple linear fit with a higher order polynomial? – JohnK Jan 19 at 19:27
So should I use features such x^2, x^3? – Pranav Jan 19 at 19:40
You could go as high as a cubic power and see what happens. Remember however that in case you want to include polynomial terms, you first have to center the predictors and include all lower order terms as well. Let me know how it goes. – JohnK Jan 19 at 19:42
You may want to try trigonometric functions (eg, sin & cos) instead of polynomials, as well. – gung Jan 19 at 19:46
@JohnK thank you for the suggestion. I tried cubic polynomial and I think it looks better, thank you! – Pranav Jan 21 at 8:40
up vote 0 down vote accepted

Your data seems slightly S-shaped, so you could try a cubic (higher order polynomial than cubic runs the risk of overfitting).

share|improve this answer
I did try your suggestion of using cubic polynomial. I have updated my post to include the same. I think it looks better than linear. Thanks! – Pranav Jan 21 at 8:38

What does the science behind the data suggest?

Yes, you can fit many different models including polynomials, splines, trigonometric functions, non-parametric smoothers, etc. But what is most important is what make sense based on the science behind the data (having fancy computers does not excuse us from understanding where our data comes from).

What do you expect to happen to your y variable as x increases beyond 2? decreases beyond -2? This can tell a lot about what types of models would be meaningful. A cubic polynomial will head quickly towards infinity as x increases and quickly towards negative infinity as x decreases. A natural spline would also head towards infinity and negative infinity, just not as quickly. But maybe based on the science you believe that there are upper and lower bounds that y will approach but not continue past, this would be a very different type of model.

Polynomials, splines, and smoothers can approximate other functions (remember Taylor Series from calculus?), but if you have an idea of what the true relationship will be then fitting that model directly can enlighten more than an approximation.

share|improve this answer
+1. It is astonishing how often this needs to be said. – Nick Cox Jan 20 at 10:40
I agree that the science behind the data would give much more insights to what model would be best, but unfortunately I do not have knowledge of the data as this was merely an exercise about linear regression and I was curious to know what would be better! – Pranav Jan 21 at 8:33

I'd recommend a generalized additive model rather than a polynomial. This would allow a non-parametric smoother which is more flexible to follow your s shape.

However, there are numerous other options. Much depends on the purpose of fitting a model at all. Is it for prediction, or for explanatory purposes? What sort of inferences do need to make? etc.

share|improve this answer
This is a good idea of course but I do not think the OP would like to go that far. – JohnK Jan 19 at 20:27
maybe... OP's call. If they want to use R, fitting a gam is very nearly as simple as fitting the original regression, and is just a one-liner. I would think that adding polynomials or trigonometric functions by hand (some of the other suggestions) would be in fact more complex. – Peter Ellis Jan 19 at 20:32
It's not only about the programming though. It is about understanding as well. And it seemed to me that the OP has just learned about the simple linear model! – JohnK Jan 19 at 20:35

I would say it depends in the type oft response. Is it deterministic or noisy?

If it is noisy data i would prefer Gaussian Processes so that you can not only extract a predictor value but also the likeliness.

A general introduction to this type of Model is given here: scikit-lean GP

share|improve this answer
This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? We can also turn it into a comment. – gung Jan 19 at 20:49
This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post - you can always comment on your own posts, and once you have sufficient reputation you will be able to comment on any post. - From Review – General Abrial Jan 19 at 21:10

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.