Iterative generalized ridge regression I am looking for some references. Assume I have a series of observable input/output pairs $(y_t, X_t)$ for which I assume the following relations to hold:
$$\beta_t\text{ are i.i.d. }\sim N(\bar{\beta},\Sigma_{\beta})$$
and
$$y_t | \beta_t, X_t \sim N(X_t\beta_t, \Sigma_w)$$
where $\Sigma_{\beta}$ and $\Sigma_w$ are fixed unknown covariance matrices.
Starting from some initial guesses $\hat{\Sigma}^{(0)}_\beta$ and $\hat{\Sigma}^{(0)}_w$, do their exist closed-form iterative expressions for the estimates of $\hat{\beta}^{(i)}$, $\hat{\Sigma}^{(i)}_{\beta}$ and $\hat{\Sigma}^{(i)}_w$ after having observed  $(y_t, X_t)$, $t=1,...,i$? I was able to find references which treat the case $\Sigma_w=\sigma I$, with $\sigma$ known, but not for this more general setting.
 A: It follows from your model that the conditional distribution of $y_t$ given $X_t$ is
$$ y_t|X_t \sim N(X_t \bar \beta, \Sigma_w + X_t^2 \Sigma_\beta) $$
Now if you calculate the posterior probability of $\Sigma_w$,$\Sigma_\beta$ you can see that is doesn't reduce to any simple form because of the 'mixing' between $\Sigma_w$ and $\Sigma_\beta$, so a closed-form estimate based on an exact Bayesian updating of the posterior probably doesn't exist.
There could be however various approximations.
A rather simple one in this case is a least squares estimation: since the conditional variance of $y_t$ is  $\Sigma_w + X_t^2 \Sigma_\beta$, estimate the covariance matrices by minimizing :
$$ \min_{\Sigma_w,\Sigma_\beta} \sum_t \|z_tz_t^T -  \Sigma_w - X_t^2 \Sigma_\beta \|_F^2 $$
where $z_t = y_t - X_t\bar \beta$. (Note that I'm assuming that the $X_t$'s are scalars, otherwise the expression $X_t\beta_t$ doesn't make sense). This leads to a pair of linear equations :
$$ \sum_t z_tz_t^T = n \Sigma_w + (\sum_t X_t^2) \Sigma_\beta $$
$$ \sum_t X_t^2 z_tz_t^T = (\sum_t X_t^2) \Sigma_w + (\sum_t X_t^4) \Sigma_\beta $$
so the estimates are :
$$ \hat \Sigma_\beta^{(n)} = \frac{1}{n}\frac{\sum_t z_tz_t^T(X_t^2 - \langle X^2 \rangle)}{\langle X^4 \rangle - \langle X^2 \rangle^2}$$
$$ \hat \Sigma_w^{(n)} = \frac{1}{n}\frac{\sum_t z_tz_t^T(\langle X^4 \rangle - X_t^2\langle X^2 \rangle)}{\langle X^4 \rangle - \langle X^2 \rangle^2}$$
where $\langle X^2 \rangle = \frac{1}{n} \sum_t X_t^2 $ ,  $\langle X^4 \rangle = \frac{1}{n} \sum_t X_t^4 $ .
Note that these matrices are generally not guaranteed to be positive definite, although they should converge to the true values when $n \to \infty$. You could fix that by linearly combining them with the initial guesses, with a weight that increases with $n$ (similarly to the case of a standard Bayesian updating of a covariance matrix of a normal distribution).
