How can we get the weights of ridge regression if there is bias term? For ridge regression I learned before, $\hat{w} = argmin_{\theta}||y-Xw||_2^2 + \lambda||w||_2^2$.
Thinking about if the bias is added, so the new $X$ become $[1,X]$, and we have a new weight $\theta$. In the meantime, we would not like to shrink the bias term in $\theta$. So, it becomes:
$$\hat{\theta} = argmin_{\theta}||y-X\theta||_2^2 + \lambda||w||_2^2$$.
Can this weight $\theta$ be solved in a closed-form as well?
 A: Yes, it can. 
$$||y - X\theta||^2 + \lambda||w||^2 = y^Ty - 2y^TX\theta + \theta^TX^TX\theta + \theta^T \Lambda \theta$$ where $\Lambda$ is diagonal with diagonal entries [0, $\lambda$, $\lambda$, ..., $\lambda$]. The $y^Ty$ does not affect the optimum, so I'll omit that. The objective becomes
$$- 2y^TX\theta + \theta^T(X^TX + J)\theta $$,
which is a nice, convex, differentiable, and utterly pleasing bowl-shaped quadratic as long as $(X^TX + J)$ is positive definite. (I think it is always positive definite but I don't quite have a proof.)
To optimize, set the derivative to 0. Its derivative w.r.t. $\theta$ equals 
$$- 2y^TX + 2\theta^T(X^TX + J)$$
so the optimum is at $(X^TX + J)^{-1}X^TY$.
A: A standard approach is to center $y$ and all of the input features. Typically, the input features would then be divided by their standard deviations to standardize them, so that each feature is penalized equally (independent of its original units/scale). Ridge regression can then be run as usual on the transformed data (e.g. solved in closed form). Because of the centering, no bias term is needed. The estimated weights can then be transformed back to the original units, and a bias term recovered, giving a model that applies to the original, untransformed data. No shrinkage is applied to the bias term when using this method.
How to do it
Suppose training data points are given on the rows of $n \times d$ matrix $X$, and corresponding target values are given in vector $y$. Let $\bar{x} = [\bar{x}_1, \dots, \bar{x}_d]$ be the mean feature values across data points, and $[\sigma_1, \dots, \sigma_j]$ be the corresponding standard deviations. Let $\bar{y}$ be the mean of $y$. Use these values to standardize each column of $X$ and center $y$, giving transformed dataset $\tilde{X}, \tilde{y}$. Perform ridge regression on the transformed data to obtain weights $\tilde{w}$.
The predicted output $\hat{y}$ given a new data point $x = [x_1, \dots, x_d]$ is then:
$$\hat{y}-\bar{y} =
\sum_{j=1}^d \tilde{w}_j \frac{x_j - \bar{x}_j}{\sigma_j}$$
Re-arrange this to obtain:
$$\hat{y} =
\sum_{j=1}^d \frac{\tilde{w}_j}{\sigma_j} x_j
- \sum_{j=1}^d \frac{\tilde{w}_j}{\sigma_j} \bar{x}_j + \bar{y}$$
We can see that this corresponds to a linear function (plus bias) on the original input space with weights $w$ and bias term $b$:
$$\hat{y} = w \cdot x + b$$
where:
$$w = \left[ \frac{\tilde{w}_1}{\sigma_1}, \dots, \frac{\tilde{w}_d}{\sigma_d} \right]^T$$
$$b = \bar{y} - w \cdot \bar{x}$$
