# Implementing ridge regression in python

I was trying to implement ridge regression in python. I implemented the following code:

import matplotlib.pyplot as plt
import numpy as np
from sklearn import linear_model, preprocessing

alpha = 1e-5

x = np.linspace(0, 2*np.pi, 1000).reshape(-1, 1)
y = np.sin(x)+np.random.normal(0, 0.1, (1000,1))

regressor = linear_model.Ridge(alpha=alpha)
regressor.fit(x, y)

a = np.linspace(0, 2*np.pi, 100).reshape(-1, 1)
b = regressor.predict(a)

plt.scatter(x, y, c='b', s=1, alpha = 0.6, label='Training data')
plt.plot(a, b,'.-r', label='prediction')
plt.plot(x, np.sin(x), 'k', label='True function')
plt.legend()
plt.show()


But the result I get is not correct

I understand that later I should do cross-validation and choose best alpha value, but for now, whatever alpha value that I use, the result is underfitting. I was wondering what the problem is. Thanks!

You're trying to fit a linear model to distinctly nonlinear data. While linear regression does not limit you to lines, planes, and hyperplanes, it does limit you to lines, planes, and hyperplanes once you've decided on what you're going to feed into your regression.

Try doing your regression on $$x$$, $$x^2$$, and $$x^3$$, maybe going up to $$x^{11}$$ or so to really work the regularization. (It should squash down the even-power terms. Consider the series expansion of sine.) "This would be nonlinear regression," you protest? NO! It's still linear in the parameters. Linear regression doesn't care how nonlinear your input features are. You can do linear regression on polynomials, interactions (e.g. $$x_1x_2$$ or $$w^2x$$), or most anything else you desire.

If you go up to $$x^{11}$$, you will wind up with the following regression equation:

$$\hat{y_i} = \beta_{intercept} + \sum_{j=1}^{11}\beta_j x_i^j.$$

You will then regularize the $$\beta_j$$ parameters (not the intercept, but that should be built into the software).

As you're working now, you're always going to be trying to fit a straight line to a sine curve, which will not give a good fit for any kind of regularization.

(This idea of being linear in the parameters after you design your input features is tricky to wrap the brain around, so please do ask for clarification if you're confused.)

import matplotlib.pyplot as plt
import numpy as np
from sklearn import linear_model, preprocessing
from sklearn.preprocessing import PolynomialFeatures

alpha = 1e-5

x = np.linspace(0, 2*np.pi, 1000).reshape(-1, 1)
y = np.sin(x)+np.random.normal(0, 0.1, (1000,1))

regressor = linear_model.Ridge(alpha=alpha)

polynomial_features = PolynomialFeatures(degree = 7)
x_trans = polynomial_features.fit_transform(x.reshape(-1, 1))

regressor.fit(x_trans, y)

a = np.linspace(0, 2*np.pi, 100).reshape(-1, 1)
a_trans = polynomial_features.fit_transform(a.reshape(-1, 1))

b = regressor.predict(a_trans)

plt.scatter(x, y, c='b', s=1, alpha = 0.6, label='Training data')
plt.plot(a, b,'.-r', label='prediction')
plt.plot(x, np.sin(x), 'k', label='True function')
plt.legend()
plt.show()


• Output looks good, but please provide some context as to what you added/changed. – Cam.Davidson.Pilon Aug 14 '19 at 1:55