5
$\begingroup$

How can we know which degree polynomial is the best fir for a data set composed of one predictor and one variable? And how can we evaluate them?

I have developed the linear regression and then went up to the third polynomial degree, but I just need to make how to assess the goodness of fit?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ You assess goodness of fit with cross validation. You pick a degree most generally by not picking a degree, instead you should use cubic splines, or if you must use polynomials, pick a highest degree you are willing to tolerate, and then use regularization to pick the correct complexity. $\endgroup$
    – Matthew Drury
    Commented Feb 13, 2017 at 0:55

2 Answers 2

Reset to default
8
$\begingroup$

This question can be generalized for selecting any machine learning algorithm hyper-parameters. For example, number of clusters in K-means, number of Hidden unit in neural networks, etc.

At very high level, there are two ways (not mutually exclusive, in fact combining two ways would be ideal.): Data driven and knowledge driven.

  • Data driven means using data to figure out which one is the best. We usually have training set and testing set. There are some other variations, such as adding one additional validation data set, run repeated cross validation etc. But the overall idea is and pick the best one in testing set, and we can make sure testing set is very close to production data.

  • Knowledge driven means using "domain knowledge" to make the decision on parameter tuning. For example, we are fitting some data from some trajectory data and we know our data from physics would generally follow a parabola trend, not a 5th order polynomial curve. Then we would like pick the 2nd order polynomial to fit. In addition, if we know our data is periodic, we may choose Fourier expansion on the data instead of polynomials. See this post What's wrong to fit periodic data with polynomials?

In sum, if we have a lot of data, and can make sure we have a fair representation to production data in testing set. Then data driven would be good. On the other hand, if we have lots of domain knowledge about the relationship between input and output, then knowledge driven is good. The ideal case would be combining two: know the relationship in data and testing it carefully using a good testing set.

Improve this answer
$\endgroup$
0
$\begingroup$

In a polynomial regression process(gradient descent) try to find the global minima to optimize the cost function. We choose the degree of polynomial for which the variance as computed by

$\frac{Sr(m)}{n-m-1}$

is a minimum or when there is no significant decrease in its value as the degree of polynomial is increased. In the above formula,

  • Sr(m) = sum of the square of the residuals for the mth order polynomial
  • n= number of data points
  • m=order of polynomial (so m+1 is the number of constants of the model)

Refer : https://autarkaw.org/2008/07/05/finding-the-optimum-polynomial-order-to-use-for-regression/

Improve this answer
$\endgroup$
3
  • $\begingroup$ Don't know what you mean by "where the cost is minimal, so the residual must be tends to zero". $\endgroup$
    – Michael R. Chernick
    Commented May 7, 2018 at 5:28
  • $\begingroup$ i have rephrased, hope it will work :) $\endgroup$
    – Reeves
    Commented May 7, 2018 at 8:15
  • $\begingroup$ The link brings to a "broken" web page. Do you have an up-to-date reference to the cost function? Is there a name for such function? $\endgroup$
    – w4bo
    Commented Jan 2, 2023 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.