0
$\begingroup$

My input data

x = 
[[ 1.]
[ 2.]
[ 3.]
[ 4.]
[ 5.]
[ 6.]
[ 7.]
[ 8.]
[ 9.]
[10.]]

y = 
[[ 2.45151081]
 [ 3.85549532]
 [ 6.98435572]
 [ 7.11091693]
 [ 5.91860089]
 [ 7.33033817]
 [ 8.77851672]
 [ 9.77186197]
 [ 9.87780907]
 [11.52364423]]

... is actually approximating the function y = -1/10 * X**2 + 2*x

Using:

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

I am trying to understand how SKLearn takes the following to approximate the function above...

polynomial_features = PolynomialFeatures(degree=odr_polynomial)
x_poly = polynomial_features.fit_transform(x)    
model = LinearRegression()
model.fit(x_poly, y)

if I print(x_poly) I get

[[   1.    1.    1.    1.]
 [   1.    2.    4.    8.]
 [   1.    3.    9.   27.]
 [   1.    4.   16.   64.]
 [   1.    5.   25.  125.]
 [   1.    6.   36.  216.]
 [   1.    7.   49.  343.]
 [   1.    8.   64.  512.]
 [   1.    9.   81.  729.]
 [   1.   10.  100. 1000.]]

it looks like this function has iterated x and generated

i**0 , i**1 , i**2 , i**3 (based on me asking for 3rd order polynomial)

I don't know the proper name for this but I'd describe it as "the base bits of the polynomial without the coefficients". I'm an amature coder. To the best of my ability to read the code of sklearn.linear_model.LinearRegression it uses scipy.linalg.lstsq under the hood and that function calls this the "design matrix"

Reading scipy.linalg.lstsq code is where I get lost. I understand how LEast Squares works but I can't understand how least squares is being applied to the 'design matrix" to derive the coefficients.

If I run the code with odr_polynomial = 3 I get:

coefficients: [[ 0.  3.05733751   -0.44144351   0.02535316]]
y-intercept: [-0.12880603]

If I run the code with odr_polynomial = 2 (because the underlying function we're trying to approximate is 2nd order) I get:

coefficients: [[ 0.  1.12796227   -0.02311642]]
y-intercept: [2.04649483]

Although I have asked this question in the context of using SKLearn and Scipy, I think this is also something that could be done with paper & pen. I really want to understand how the coefficients are being derived from the "design matrix".

I'd be grateful for any advice or insight you can share.

$\endgroup$
  • $\begingroup$ Please consult any of our threads that describe multiple regression. A good key phrase is "normal equations:" stats.stackexchange.com/… and stats.stackexchange.com/… $\endgroup$ – whuber Jul 25 '19 at 12:49
  • $\begingroup$ The coefficients are the "least squares" solution to the problem of y = coeff1 * x^3 + coeff2 * x^2 + coeff3 * x + coeff4. $\endgroup$ – theGD Jul 25 '19 at 12:49
  • $\begingroup$ I’m new to stats.SE.com. It's like SO used to be before they admitted they had a problem stackoverflow.blog/2018/04/26/… It comes across as though there is a lot of people here out to prove how great they are vs helping learners. I asked a super-specific question & the response is basically “google it” like I haven’t watched/read tonnes of videos & blogs b4 resorting to this site knowing the typical tone of the responses here.You could have a truly positive impact on other people’s lives. It’s a missed opportunity $\endgroup$ – Axle Max Jul 25 '19 at 22:30
  • $\begingroup$ If the linked thread isn't what you want / you still have a question after reading it, edit your question to state what you learned & what you still need to know. Then we can provide the information you need without just duplicating material elsewhere that already didn't help you. $\endgroup$ – gung - Reinstate Monica Jul 26 '19 at 3:41