Variance of $\hat{\beta}$ in Ridge Regression If you are using ridge regression, what happens to the variance of your parameter estimates relative to regular regression?
My intuition is telling me that it would decrease because you are doing a constraint optimization problem so the possible values the function could take are fewer. 
 A: This question seeks information that is similar to an answer in another question here, though it is not a duplicate of that other question.  Most of the present answer is adapted from the answer to the linked question.
It is possible to obtain a general form for the variance matrix of the ridge-estimator, which holds for all penalty parameters $\lambda$.  To do this, we first write the ridge-estimator \in terms of the underlying regression coefficient and error vector as:
$$\begin{align}
\hat{\boldsymbol{\beta}}(\lambda)
&= (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} (\mathbf{x}^\text{T} \mathbf{Y}) \\[6pt]
&= (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T} (\mathbf{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}) \\[6pt]
&= (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} (\mathbf{x}^\text{T} \mathbf{x}) \boldsymbol{\beta} + (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T} \boldsymbol{\varepsilon}. \\[6pt]
\end{align}$$
Using the Woodbury matrix-inverse formula gives the exact form:
$$\begin{align}
(\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} 
&= \frac{1}{\lambda} \bigg( \frac{\mathbf{x}^\text{T} \mathbf{x}}{\lambda} + \mathbf{I} \bigg)^{-1} \\[6pt]
&= \frac{1}{\lambda} \bigg( \mathbf{I} - \frac{1}{\lambda}  \mathbf{x}^\text{T} \bigg( \mathbf{I} + \frac{\mathbf{x} \mathbf{x}^\text{T}}{\lambda} \bigg) \mathbf{x} \bigg) \\[6pt]
&= \frac{1}{\lambda^3} \bigg( \lambda^2 \mathbf{I} - \lambda (\mathbf{x}^\text{T} \mathbf{x}) - (\mathbf{x}^\text{T} \mathbf{x})^2 \bigg), \\[6pt]
\end{align}$$
and this allows us to express the ridge-estimator (or its moments) in terms of matrix operations that do not involve inversion.  (We do not need to give this formula here, and it is somewhat cumbersome at this stage.)  Consequently, the variance matrix for the ridge estimator is given by:
$$\begin{align}
V(\lambda, \mathbf{x}) 
&\equiv \mathbb{V}(\hat{\boldsymbol{\beta}}(\lambda) | \mathbf{x}) \\[6pt]
&= (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T} \mathbb{V}(\boldsymbol{\varepsilon}) ((\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T})^\text{T} \\[6pt]
&= \sigma^2 (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T} \mathbf{I} ((\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T})^\text{T} \\[6pt]
&= \sigma^2 (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} (\mathbf{x}^\text{T} \mathbf{x}) ((\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1})^\text{T} \\[6pt]
&= \sigma^2 (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} (\mathbf{x}^\text{T} \mathbf{x}) (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \\[6pt]
&= \frac{\sigma^2}{\lambda^6} \bigg( \lambda^2 \mathbf{I} - \lambda (\mathbf{x}^\text{T} \mathbf{x}) - (\mathbf{x}^\text{T} \mathbf{x})^2 \bigg) (\mathbf{x}^\text{T} \mathbf{x}) \bigg( \lambda^2 \mathbf{I} - \lambda (\mathbf{x}^\text{T} \mathbf{x}) - (\mathbf{x}^\text{T} \mathbf{x})^2 \bigg) \\[6pt]
&= \frac{\sigma^2}{\lambda^6} \begin{bmatrix}
\lambda^4 \mathbf{I} - \lambda^3 (\mathbf{x}^\text{T} \mathbf{x}) - \lambda^2 (\mathbf{x}^\text{T} \mathbf{x})^2 \\
- \lambda^3 (\mathbf{x}^\text{T} \mathbf{x}) + \lambda^2 (\mathbf{x}^\text{T} \mathbf{x})^2 + \lambda (\mathbf{x}^\text{T} \mathbf{x})^3 \\
- \lambda^2 (\mathbf{x}^\text{T} \mathbf{x})^2 + \lambda (\mathbf{x}^\text{T} \mathbf{x})^3 + (\mathbf{x}^\text{T} \mathbf{x})^4 \\
\end{bmatrix} (\mathbf{x}^\text{T} \mathbf{x}) \\[6pt]
&= \frac{\sigma^2}{\lambda^6} [ \lambda^4 (\mathbf{x}^\text{T} \mathbf{x}) - 2 \lambda^3 (\mathbf{x}^\text{T} \mathbf{x})^2 - \lambda^2 (\mathbf{x}^\text{T} \mathbf{x})^3  + 2 \lambda (\mathbf{x}^\text{T} \mathbf{x})^4 + (\mathbf{x}^\text{T} \mathbf{x})^5 ]. \\[6pt]
\end{align}$$
In a related question here we see that as $\lambda \rightarrow \infty$ the variance matrix reduces to the simple asymptotic form:
$$V(\lambda, \mathbf{x}) \sim \frac{\sigma^2}{\lambda^2} (\mathbf{x}^\text{T} \mathbf{x}).$$
