Polynomial regression using gradient descent [closed]

Question

How can I fit a polynomial to my data using gradient descent in python?

Answer 1

Agreed with Appletree that it should be done directly. But for the sake of practicing gradient descent and/or gaining some intuition on a problem for which we already know the answer it could be useful. With the math complete the Python can be found here. So,

Consider $\textbf{y}=\textbf{X}\mathbf{\beta}+\mathbf{\epsilon}$. With $\textbf{y} \in \mathbb{R}^n$, $\textbf{X} \in \mathbb{R}^{n\times d}$, $\mathbf{\beta} \in \mathbb{R}^d$, $\mathbf{\epsilon} \sim N(0,\sigma^2)$.

The goal of linear regression is to find the $\beta$ which minimizes the squared loss, i.e. $\text{argmin}_{\beta} \| \mathbf{y - X\beta}\|_2^2 $. To do this by gradient descent we must first find the gradient of the loss function with respect to $\mathbf{\beta}$:

$\frac{\partial}{\partial \mathbf{\beta}} \| \mathbf{y - X\beta}\|_2^2 = 2 \mathbf{X}^T(\mathbf{y-X\beta})$

Now, we follow the algorithm for gradient descent.

1. Select an initial guess $\mathbf{\beta_0}$
2. Set $\beta_{k+1} = \beta_k - \alpha_k \mathbf{X}^T(\mathbf{y-X\beta}_k)$

Where $\alpha_k$ can be a constant or adaptive stepsize. You may notice that the 2 from the gradient disappeared. That can just be absorbed into the $\alpha_k$.

Answer 2

Why would you even want that? There's an analytic solution based on linear regression. You simply expand your predictor matrix. For instance, this

$$ X = \begin{pmatrix}1 & x_1 & x_1^2 \\ 1 &x_2&x_2^2\\ \vdots& & \vdots\end{pmatrix} $$

specifies the predictor matrix for a quadratic polynomial. You can add higher order polynomials as additional columns. If you have multiple features/variables, you have to add a quadratic, cubic etc. term for each of them.

If y is the vector containing your responses, your regression model then becomes

$$ y = X\ \begin{pmatrix}\beta_0\\\beta_1\\\beta_2\end{pmatrix} + \epsilon $$

and $\beta_0$ is the intercept term, and $\beta_1$ and $\beta_2$ are the linear and quadratic terms. The solution is given as

$$ \hat{\mathbf{\beta}} = X^\dagger y $$

where $X^\dagger$ is the pseudo-inverse of X. No iterative methods required. Python pseudo-inverse code is here.