Ridge regression and distribution of estimate? When OLS overfits observed data,  does it give skewed distribution of estimates?
 A: It depends, Ridge regression can be expressed in two different ways
$${\text{minimize } f(\beta) \text{ subject to } g(\beta) \leq t}$$
or
$${\text{minimize } f(\beta)  + \lambda g(\beta)}$$
with
$$\begin{align} 
 f(\beta) &= \frac{1}{2n} \vert\vert y-X\beta \vert\vert_2^2 \\
 g(\beta) &= \vert\vert \beta \vert\vert_2^2 
\end{align}$$
You can perform ridge regression with fixed $t$ (situation 1 below) or with fixed $\lambda$ (situation 2 below). Also, you might figure out the optimal $t$ or $\lambda$ with some sort of cross validation.
Situation 1
The image below from an answer to another question here might help.
Why does regularization wreck orthogonality of predictions and residuals in linear regression?


*

*The OLS solution is an orthogonal projection of the observations into a subspace defined by the model.


*The ridge regression solution is also a sort of orthogonal projection, but now the subspace is restricted by the regularisation conditions.
With OLS you get that this orthogonal projection is on a flat surface and is like splitting the random noise (which is spherically symmetric in tha case of white noise) into two orthogonal parts.
With ridge regression the projection is on a curved surface (the same is true for non-linear least squares) and indeed, this will make the sample distribution of the estimate slightly skewed. With larger sample size, this skewness will reduce because the curvature of the surface becomes less.
I have no illustration of this, but you can imagine it as somewhat similar to the Delta method approaching a normal distribution.
Situation 2
For a given $\lambda$ we can express ridge regression as a linear estimator (a weighted sum of the observations) $$\hat\beta = (\textbf{X}^T\textbf{X} + \lambda \textbf{I})^{-1} \textbf{X}^T y$$ If  the estimate is a weighted sum of the components of $y$ (with finite variance) then it will approaches a normal distribution for large sample sizes (or it equals a normal distribution for any sample size if the components of the vector $y$ are normal distributed).
