Polynomial regression using gradient descent How can I fit a polynomial to my data using gradient descent in python?
 A: Agreed with Appletree that it should be done directly. But for the sake of practicing gradient descent and/or gaining some intuition on a problem for which we already know the answer it could be useful. With the math complete the Python can be found here. So,
Consider $\textbf{y}=\textbf{X}\mathbf{\beta}+\mathbf{\epsilon}$. With $\textbf{y} \in \mathbb{R}^n$, $\textbf{X} \in \mathbb{R}^{n\times d}$, $\mathbf{\beta} \in \mathbb{R}^d$, $\mathbf{\epsilon} \sim N(0,\sigma^2)$. 
The goal of linear regression is to find the $\beta$ which minimizes the squared loss, i.e. $\text{argmin}_{\beta} \| \mathbf{y - X\beta}\|_2^2 $. To do this by gradient descent we must first find the gradient of the loss function with respect to $\mathbf{\beta}$:
$\frac{\partial}{\partial \mathbf{\beta}}  \| \mathbf{y - X\beta}\|_2^2 = 2 \mathbf{X}^T(\mathbf{y-X\beta})$
Now, we follow the algorithm for gradient descent.


*

*Select an initial guess $\mathbf{\beta_0}$

*Set $\beta_{k+1} = \beta_k - \alpha_k \mathbf{X}^T(\mathbf{y-X\beta}_k)$


Where $\alpha_k$ can be a constant or adaptive stepsize. You  may notice that the 2 from the gradient disappeared. That can just be absorbed into the $\alpha_k$. 
A: Why would you even want that? There's an analytic solution based on linear regression. You simply expand your predictor matrix. For instance, this
$$
X = \begin{pmatrix}1 & x_1 & x_1^2 \\ 1 &x_2&x_2^2\\ \vdots& & \vdots\end{pmatrix}
$$
specifies the predictor matrix for a quadratic polynomial. You can add higher order polynomials as additional columns. If you have multiple features/variables, you have to add a quadratic, cubic etc. term for each of them.
If y is the vector containing your responses, your regression model then becomes
$$
y = X\ \begin{pmatrix}\beta_0\\\beta_1\\\beta_2\end{pmatrix}
 + \epsilon $$
and $\beta_0$ is the intercept term, and $\beta_1$ and $\beta_2$ are the linear and quadratic terms. The solution is given as 
$$
\hat{\mathbf{\beta}} = X^\dagger y
$$
where $X^\dagger$ is the pseudo-inverse of X. No iterative methods required. Python pseudo-inverse code is here.
