Currently Im reading this paper and in section 3.3., I came across the definition of a multi-dimensional standard normal distribution:

\begin{align} q(\pmb{\epsilon}) = \mathcal{N}(\textbf{0}, \textbf{I}_{Q \times K}). \end{align}

What does $\textbf{I}_{Q \times K}$ suppose to mean? I thought that covariance matrices must always be quadratic! How can one define the unit matrix of dimensions $Q \times K$ to be covariance function for a standard normal distribution?

Perhaps I missunderstood something here, but does anyone have a clue by chance what the meaning of this is?


  • $\begingroup$ Agree, it is a confusing choice of notation. $\endgroup$
    – Tim
    Jun 9, 2020 at 10:55
  • $\begingroup$ Any intuition about what is meant? $\endgroup$
    – MJimitater
    Jun 9, 2020 at 10:59

1 Answer 1


It's unfortunate notation, but my understanding is that they mean an identity matrix of size $N = Q \cdot K$, ie Q times K. This is needed since $\mathbf{W}_1$ is a $Q \times K$ matrix so the noise/error term $\mathbf{\epsilon_1}$ in that equation must be of size $Q \cdot K$ to match LHS in eq (14).

I think it's more clear to look first at the bias term: here $\mathbf{\epsilon}$ is of dimension $K$ because the bias term $\mathbf{b}$ is of dimension $K$. They do the same for the matrix equations, but now think of it as a flattened out $Q \times K$ matrix into a $Q \cdot K$ vector; in order to add a noise term $\epsilon_1$ to that (flattened) vector it has to come from a normal distribution of dimension $K \cdot Q$, which you get for an identity matrix of size $(Q \cdot K) \times (Q \cdot K)$.

  • $\begingroup$ thanks for looking into it! So you suggest the identity matrix to be of dimension $Q*K \times Q*K$, I don't see yet how this results into matrix $W_1$ being dimension $Q \times K$ in eq. (14). I'd be grateful if you could elaborate on this $\endgroup$
    – MJimitater
    Jun 9, 2020 at 12:56
  • $\begingroup$ updated my answer with more details on formula / equations. let me know if that is more clear now. again, this is just my guess of what they mean based on freely switching between matrix and flattened vector notations. $\endgroup$ Jun 9, 2020 at 13:30
  • $\begingroup$ Excellent, thank you friend, crystal clear now! Cheers $\endgroup$
    – MJimitater
    Jun 9, 2020 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.