1
$\begingroup$

I came across the paper Fast Variational Bayesian Linear State-Space Model. They work with the following model: $$\begin{align} {\bf{x}}_n &= {\bf A} {\bf{x}}_{n-1} + {\text{noise}} \\ {\bf{y}}_n &= {\bf C} {\bf{x}}_n + {\text{noise}} \end{align}$$

where $M >> D$, the noise is Gaussian, $\bf A$ is the $D × D$ state dynamics matrix and $\bf C$ is the $M × D$ loading matrix.

They assign a prior to $\bf C$ to be $$p \left(\bf C \;\middle\vert\; \gamma \right) = \prod_{m=1}^M \prod_{d=1}^D \mathcal{N}\left( c_{md} \;\middle\vert\; 0, \gamma_d^{-1} \right),$$ where $c_{md}$ is the element in the $m$-th row and $d$-th column of the matrix.

I am studying a factor model of similar form where the first $D × D$ part of $\bf C$ is assumed to be lower triangular, e.g. if $D = 3$

$${\bf C} = \begin{bmatrix} 1 & 0 & 0 \\ c_{21} & 1 & 0 \\ c_{31} & c_{32} & 1 \\ c_{41} & c_{42} & c_{43} \\ \vdots & \vdots & \vdots \\ c_{M1} & c_{M2} & c_{M3} \end{bmatrix}$$

I don't need to worry about a hyper-prior and I believe the components should be standard Gaussian.

I have some non-random terms in this matrix, so how can I go about assigning a prior distribution?

I am wondering if I can just ignore the non-random parts and do something like $$p\left(\bf C \right) = \mathcal{N}\left( c_{21} \;\middle\vert\; 0, 1 \right) \mathcal{N}\left( c_{31} \;\middle\vert\; 0, 1 \right) \mathcal{N}\left( c_{32} \;\middle\vert\; 0, 1 \right)\prod_{m=4}^M \prod_{d=1}^3 \mathcal{N}\left( c_{md} \;\middle\vert\; 0, 1 \right)$$

I've never seen this type of thing handled in literature so I'm not sure if it's a proper way to handle this issue.

$\endgroup$
1
$\begingroup$

If $\mathbf{C}$ has no restrictions on its elements, the prior $$ p_1 \left(\bf C \;\middle\vert\; \gamma \right) = \prod_{m=1}^M \prod_{d=1}^D \mathcal{N}\left( c_{md} \;\middle\vert\; 0, \gamma_d^{-1} \right) $$ is proper, and so the posterior is proper, and so you have valid Bayesian inference. However, as you allude to, the likelihood is not identifiable, so your posterior might be exchangeable for certain combinations of elements of $\mathbf{C}$. This is a problem if you want to make inferences on individual elements of this matrix.

If you are defining $\mathbf{C}$ as $$ \begin{bmatrix} 1 & 0 & 0 \\ c_{21} & 1 & 0 \\ c_{31} & c_{32} & 1 \\ c_{41} & c_{42} & c_{43} \\ \vdots & \vdots & \vdots \\ c_{M1} & c_{M2} & c_{M3} \end{bmatrix} $$ then you only need to define a prior for the free element of this matrix, and so $$ p_2 \left(\bf C \;\middle\vert\; \gamma \right) = \prod_{m=2}^M \prod_{d=1}^{ \min(m-1,D) } \mathcal{N}\left( c_{md} \;\middle\vert\; 0, \gamma_d^{-1} \right) $$ is a proper prior that assumes all of the lower-diagonal elements are independent.

$\endgroup$
  • $\begingroup$ That makes sense. I'm not sure if this is worth posting another question for, but do you think that $p_2 \left(\bf C \;\middle\vert\; \gamma \right)$ is a bit of an abuse of notation? In the sense that $\bf C$ is not explicitly shown on the right-hand side of the equation. $\endgroup$ – grxxvytony May 23 at 0:59
  • $\begingroup$ For example, if I say $X$ is standard Gaussian, then $p(x) \propto e^{-\frac12 x^2}$. The right-hand side of $p(x)$ shows you exactly what happens to the input, $x$. $\endgroup$ – grxxvytony May 23 at 1:02
  • 1
    $\begingroup$ @grxxvytony I do prefer that notation usually, but my notation is pretty common, too, particularly in this Bayesian situation. For example, if you're deriving say a Gibbs sampler, and you have all these distributions everywhere, then sometimes it's shorter and easier to write the more concise notation $\endgroup$ – Taylor May 23 at 1:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.