I'll assume that the elements of $\mathbf{y}$ are i.i.d. and likewise for the elements of $\mathbf{X}$. This is important, though, so be forewarned!
The diagonal elements of the covariance matrix equal the sum of $m$ products of i.i.d. random variates, so the variance will equal $m \mathbb{V}(x_{ij}y_j)$, which variance you have above in your first row.
The off-diagonal elements all equal zero, as the rows of $\mathbf{X}$ are independent. To see this, without loss of generality assume $\mathbb{E}x_{ij} = \mathbb{E}y_i = 0 \space \forall\thinspace i,j$. Define $\mathbf{x}_i$ as the $i^{\text{th}}$ row of $\mathbf{X}$. Then:
$\text{Cov}(\mathbf{x_iy},\mathbf{x_jy}) = \mathbb{E}(\mathbf{x_iy})^\text{T}(\mathbf{x_jy}) = \mathbb{E}\mathbf{y}^{\text{T}}\mathbf{x}_i^\text{T}\mathbf{x}_j\mathbf{y}=\mathbb{E}_y\mathbb{E}_x \mathbf{y}^{\text{T}}\mathbf{x}_i^\text{T}\mathbf{x}_j\mathbf{y}$
When $i \ne j$, the expectation with respect to $x$ of $\mathbf{y}^{\text{T}}\mathbf{x}_i^\text{T}\mathbf{x}_j\mathbf{y}$ equals 0 for any $\mathbf{y}$, as it is just the expectation of a sum of random variables each with expected value 0 (due to the assumed independence between the elements of $\mathbf{X}$). Consequently, the entire expectation equals 0.