Polynomial regression using scikit-learn

Question

I am trying to use scikit-learn for polynomial regression. From what I read polynomial regression is a special case of linear regression. I was hopping that maybe one of scikit's generalized linear models can be parameterised to fit higher order polynomials but I see no option for doing that.

I did manage to use a Support Vector Regressor with a poly kernel. That worked well with a subset of my data, but it takes much to long to fit larger data sets so I still need to find something faster (even if trading some precision).

Am I missing something obvious here?

Answer 1

Given data $\mathbf{x}$, a column vector, and $\mathbf{y}$, the target vector, you can perform polynomial regression by appending polynomials of $\mathbf{x}$. For example, consider if

$$ \mathbf{x} = \begin{bmatrix} 2 \\[0.3em] -1 \\[0.3em] \frac{1}{3} \end{bmatrix}$$

Using just this vector in linear regression implies the model:

$$ y = \alpha_1 x $$

We can add columns that are powers of the vector above, which represent adding polynomials to the regression. Below we show this for polynomials up to power 3: $$ X = \begin{bmatrix} 2 & 4 & 8 \\[0.3em] -1 & 1 & -1 \\[0.3em] \frac{1}{3} & \frac{1}{3^2} & \frac{1}{3^3} \end{bmatrix}$$

This is our new data matrix that we use in sklearn's linear regression, and it represents the model:

$$ y = \alpha_1 x + \alpha_2x^2 + \alpha_3x^3$$

---

Note that I did not add a constant vector of $1$'s, as sklearn will automatically include this.

Answer 2

Theory

Polynomial regression is a special case of linear regression. With the main idea of how do you select your features. Looking at the multivariate regression with 2 variables: x1 and x2. Linear regression will look like this: y = a1 * x1 + a2 * x2.

Now you want to have a polynomial regression (let's make 2 degree polynomial). We will create a few additional features: x1*x2, x1^2 and x2^2. So we will get your 'linear regression':

y = a1 * x1 + a2 * x2 + a3 * x1*x2 + a4 * x1^2 + a5 * x2^2

This nicely shows an important concept curse of dimensionality, because the number of new features grows much faster than linearly with the growth of degree of polynomial. You can take a look about this concept here.

Practice with scikit-learn

You do not need to do all this in scikit. Polynomial regression is already available there (in 0.15 version. Check how to update it here).

from sklearn.preprocessing import PolynomialFeatures
from sklearn import linear_model

X = [[0.44, 0.68], [0.99, 0.23]]
vector = [109.85, 155.72]
predict= [[0.49, 0.18]]
#Edit: added second square bracket above to fix the ValueError problem

poly = PolynomialFeatures(degree=2)
X_ = poly.fit_transform(X)
predict_ = poly.fit_transform(predict)

clf = linear_model.LinearRegression()
clf.fit(X_, vector)
print clf.predict(predict_)

Answer 3

In case you are using a multivariate regression and not just a univariate regression, do not forget the cross terms. For instance if you have two variables $x_1$ and $x_2$, and you want polynomials up to power 2, you should use $y = a_1x_1 + a_2x_2 + a_3x_1^2 + a_4x_2^2 + a_5x_1x_2$ where the last term ($a_5x_1x_2$) is the one I am talking about.