Mixture models and K-means clustering similarities: What is the covariance matrix? Simple question: Is the covariance matrix used in the algorithm the covariance between the groups (Ie. Covariance between group 1 and group 2, group 1 and group 3 and so forth)? 
Or is it between the predictors? (Covariance between x1 and x2, x1 and x3 etc.) Because it mentions in the photo, independence among the variables)
I understand that it mentions independence among the variables, but wouldn't be making the assumption that your predictor variables are independent an egregious assumption?
Lastly, within a mixture model, is the first and most important assumption for this unsupervised method is the assumed number of groups within our data? Are there any methods to assure we have the correct groups? Since cross-validation is sort--of out of the equation.
Thank you!

 A: Before answering the question, let's first make clear the mindset behind kmeans and mixture model:


*

*kmeans: you believe that your samples are generated by k hidden components, each sample belongs to only one of the k components. The characteristics of a component is completely described by it's centre. Say for each component, there's a centre $c_i,i=1...k$.

*mixture model: (take mixture of multivarite normal for example)you believe that the samples are generated by k hidden components, each sample belongs to only one of the k components. The characteristics of a component is completely described by it's centre and it's "concentrations" around the centre. Say for each component you use a multivariate normal distribution $MVN(\mu_i,\Sigma_i),i=1...k$ to represent the characteristics of it, where $\mu_i$ is the mean vector, the "centre"; and $\Sigma_i$ is the covariance matrix (between your predictors/features), the "concentration", of the component.
When you assume the predictors/features are independent of each other and with the same scale, this is equivalent to setting $\Sigma_i = I\sigma^2,i=1...k$ in the mixture model, where $I$ is the identity matrix, $\sigma^2$ is a constant. Apply EM algorithm on this model is the same as applying the kmeans iterations on it, the two models will reach the same result and $c_i=\mu_i$.
As for your second question, how to decide the number of groups/components.
There are two commonly used methods:


*

*Method1: Use Bayesian mixture model and calculate the BIC of your model with different instances of $k$, pick the one $k$ that can minimize the BIC.  

*Method2: Use Bayesian non-parametric methods such as Dirichlet process to select the most likely k automatically.
Hope it helps!
A: Just like the variance in the sample with one variable(1 dimension, or univariate), the covariance represents the spread of the data in each dimension. And it also represents the orientation. 
For the 1D data we calculate the variance by: 
 
where $\mu$ is the mean of the data(all $x_i$'s) and $N$ is the size of the data. 
Along the same line, we calculate that(we call it covariance) for the multivariate data:  

Each element(a scalar) can be obtained by the same method we work out that for the univariate data(for instance for $\sigma(x, y)$): 

$x$ and $y$ represent two features of the data and there can be more than 2 features. 
References:
1. A geometric interpretation of the covariance matrix.
2. The Multivariate Gaussian Distribution
