How can a covariance matrix for a normal distribution not be quadratic?

Question

Currently Im reading this paper and in section 3.3., I came across the definition of a multi-dimensional standard normal distribution:

\begin{align} q(\pmb{\epsilon}) = \mathcal{N}(\textbf{0}, \textbf{I}_{Q \times K}). \end{align}

What does $\textbf{I}_{Q \times K}$ suppose to mean? I thought that covariance matrices must always be quadratic! How can one define the unit matrix of dimensions $Q \times K$ to be covariance function for a standard normal distribution?

Perhaps I missunderstood something here, but does anyone have a clue by chance what the meaning of this is?

Thanks

Answer 1

It's unfortunate notation, but my understanding is that they mean an identity matrix of size $N = Q \cdot K$, ie Q times K. This is needed since $\mathbf{W}_1$ is a $Q \times K$ matrix so the noise/error term $\mathbf{\epsilon_1}$ in that equation must be of size $Q \cdot K$ to match LHS in eq (14).

I think it's more clear to look first at the bias term: here $\mathbf{\epsilon}$ is of dimension $K$ because the bias term $\mathbf{b}$ is of dimension $K$. They do the same for the matrix equations, but now think of it as a flattened out $Q \times K$ matrix into a $Q \cdot K$ vector; in order to add a noise term $\epsilon_1$ to that (flattened) vector it has to come from a normal distribution of dimension $K \cdot Q$, which you get for an identity matrix of size $(Q \cdot K) \times (Q \cdot K)$.