How to assign a prior distribution to a loading matrix that has restrictions? I came across the paper Fast Variational Bayesian Linear State-Space Model. They work with the following model:
$$\begin{align}
{\bf{x}}_n &= {\bf A} {\bf{x}}_{n-1} + {\text{noise}} \\
{\bf{y}}_n &= {\bf C} {\bf{x}}_n + {\text{noise}}
\end{align}$$
where $M >> D$, the noise is Gaussian, $\bf A$ is the $D × D$ state dynamics matrix and $\bf C$ is the $M × D$ loading matrix.
They assign a prior to $\bf C$ to be 
$$p \left(\bf C \;\middle\vert\; \gamma \right) = \prod_{m=1}^M \prod_{d=1}^D \mathcal{N}\left( c_{md} \;\middle\vert\; 0, \gamma_d^{-1} \right),$$
where $c_{md}$ is the element in the $m$-th row and $d$-th column of the matrix.
I am studying a factor model of similar form where the first $D × D$ part of $\bf C$ is assumed to be lower triangular, e.g. if $D = 3$
$${\bf C} = \begin{bmatrix}
1 & 0 & 0 \\
c_{21} & 1 & 0 \\
c_{31} & c_{32} & 1 \\
c_{41} & c_{42} & c_{43} \\
\vdots & \vdots & \vdots \\
c_{M1} & c_{M2} & c_{M3}
\end{bmatrix}$$
I don't need to worry about a hyper-prior and I believe the components should be standard Gaussian.

I have some non-random terms in this matrix, so how can I go about assigning a prior distribution? 

I am wondering if I can just ignore the non-random parts and do something like 
$$p\left(\bf C \right) = \mathcal{N}\left( c_{21} \;\middle\vert\; 0, 1 \right) \mathcal{N}\left( c_{31} \;\middle\vert\; 0, 1 \right) \mathcal{N}\left( c_{32} \;\middle\vert\; 0, 1 \right)\prod_{m=4}^M \prod_{d=1}^3 \mathcal{N}\left( c_{md} \;\middle\vert\; 0, 1 \right)$$
I've never seen this type of thing handled in literature so I'm not sure if it's a proper way to handle this issue.
 A: If $\mathbf{C}$ has no restrictions on its elements, the prior
$$
p_1 \left(\bf C \;\middle\vert\; \gamma \right) = \prod_{m=1}^M \prod_{d=1}^D \mathcal{N}\left( c_{md} \;\middle\vert\; 0, \gamma_d^{-1} \right)
$$
is proper, and so the posterior is proper, and so you have valid Bayesian inference. However, as you allude to, the likelihood is not identifiable, so your posterior might be exchangeable for certain combinations of elements of $\mathbf{C}$. This is a problem if you want to make inferences on individual elements of this matrix.
If you are defining $\mathbf{C}$ as 
$$
\begin{bmatrix}
1 & 0 & 0 \\
c_{21} & 1 & 0 \\
c_{31} & c_{32} & 1 \\
c_{41} & c_{42} & c_{43} \\
\vdots & \vdots & \vdots \\
c_{M1} & c_{M2} & c_{M3}
\end{bmatrix}
$$
then you only need to define a prior for the free element of this matrix, and so 
$$
p_2 \left(\bf C \;\middle\vert\; \gamma \right) = \prod_{m=2}^M \prod_{d=1}^{ \min(m-1,D) } \mathcal{N}\left( c_{md} \;\middle\vert\; 0, \gamma_d^{-1} \right)
$$
is a proper prior that assumes all of the lower-diagonal elements are independent.
