I undestand in general that it is important to take correlational structure into account while applying almost any statistical techniques.

First question - could you help with the examples why it is so important and what can go wrong? For example, I have vector of correlated random variables, identically distributed. Can I wrongly reject the $H_0 : \{sample \in distribution\}$ not by chance, but because of correlational structure? (i.e., reject $H_0$ much more often than 0.05 with fixed level of significance $\alpha=0.05$). Could you explain it, using the picture (samples are 2D-vectors):

enter image description here

Second question: for example, I have two normal distributions with different means $N(0, \sigma), N(-1, \sigma)$. I have a sample of rows of numbers, like

$$(0.1, 0.5, -0.3), (-1.2, -1.0, -0.7, -1.1), \ldots$$

$$(0.2, 0.4, -0.2, 0.1), (-1.1, -0.5, -1.0), \ldots$$


$$ \xi_1(i), \xi_2(i), \xi_3(i), \ldots $$

here I denotes "islands" generated by one distribution in brackets, $i$ indicates one of the distributions (so $i$ is choosen randomly fro each $\xi(i)$).

I want to determine the borders of the islands, but I know that there is a correlational structure inside each island (may be also a correlational structure between islands, but it does not matter).

How should I deal with this problem? Let us assume that it is not enough samples to calculate the whole matrix of covariances (it will overfit the data). The initial idea is to use something like linear discriminant analysis and regularisation for it, but the borders are unknown. How can PCA help in this case? Typically, the strongest correlations arise from random variables that are neighbors in the row, so one of the possible solutions is to use block covariance matrices, but the size of one block can be large enough to cause problems with overfitting.

Does HMM takes into account the correlational structure? If so, what if I do not have a set of states (for HMM), but a continuous set of parameters for distributions, which technique is better to use?


1 Answer 1


For your first question

Why is it important to take correlation into account and what can go wrong?

It depends on the model you use. Let say you want to do some clustering on a dataset that has some correlated features. If you use K-means with the Euclidian distance, the clusters will be spherical around their center.

K-means can be viewed as a special case of fitting Gaussian distributions to your data; the center of each cluster is the mean, and, more importantly, since the cluster is spherical, there is an assumption that the features are not correlated. If the features are correlated, K-means clustering loses meaning.

So if you fit a Gaussian with a diagonal covariance matrix to a your data, you might indeed accept or reject your hypothesis wrongly. In the case of the Outlier example you cite, the 95% interval of the Gaussian would not consider the circled data point as an outlier, while it "obviously is".

In such a setting, using PCA to decorrelate your data into principal components helps. The principal components are the dimensions that maximize variance, and are orthogonal, there should not be any correlation between dimensions, which should solve the correlation problem. There are a lot of visualizations on how PCA works, they might be helpful if you have trouble understanding what it does.

Second question

I want to determine the borders of the islands, but I know that there is a correlational structure inside each island (...) How should I deal with this problem?

It is unclear to me what your data is, as your "sample of rows of numbers" is not cristal clear, and I do not see a clear link between estimating Gaussian distributions and HMM. From your question, I understand that you assume your data comes from multiple Gaussian distributions, and you want to approximate them.

If your model accounts for correlation and tries to learn it from data instead of posing too strict assumptions, you will probably be fine (as long as there is sufficient data). For this I would go for a Gaussian Mixture Model and, indeed, learn the covariances matrices.

There is always the problem of over/underfitting, but no method will give you the true model behind no data. The more assumption you make, the less data you need, but you lead the risk of those assumptions being false. On the other end, making no assumptions and trying to learn everything is nice, but requires more data to learn something meaningful. If you can make educated/domain-specific assumptions, put those into the model but do not rely on picking a model without understanding the assumption it makes.

Response to comments:

  • How will PCA behave if I the covariance matrix is sparse? A sparse covariance matrix, mostly composed of 0, means most features are not correlated. In that case, the principal components will mostly be the original features vectors, and some of the original vectors will be decomposed.
  • How would you regularise the covariance matrix? If you want the covariance matrix to be sparse, I guess you could add the L1|L2 norm of the covariance matrix to the cost function that is minimized by the model you choose. In the case of GMM, it would be added to the log-likelihood. I could not find much background in a relatively short search, other than this paper.
  • $\begingroup$ Nice example, thank you. So should I represent each value with the vector of PCs? I do not understand how PCA will behave if I have such a block structure (the covariance matrix is sparse)...Regarding second question, how would you regularise the covariance matrix? Also the question that is not really connected - how to operate with large covariance matrix, for example, 100.000 x 100.000 of floats? Is there any special techniques for working with large matrices? $\endgroup$ Jan 18, 2016 at 15:34
  • $\begingroup$ A sparse covariance matrix, mostly composed of 0, means most features are not correlated. In that case, the principal components will mostly be the original features vectors, and some of the original vectors will be decomposed. I guess you could regularize the covariance matrix by adding an L1/2 norm to the cost function if you think they should be mostly 0, but I have no backing material. As for large matrix manipulation, there probably is some fancy GPU stuff you can take advantage of, but I have no experience with it. $\endgroup$
    – Winks
    Jan 18, 2016 at 15:50
  • $\begingroup$ not sure if I understand you correctly, but where should I use the L1 or L2 norm? I have just a covariance matrix of $n \times n$ features and $n^2$ is too much for the dataset size...As for the data, my data is a set of correlated random variables, and the true distribution of current random variable is hidden, that's why it is possible to use HMMs. $\endgroup$ Jan 18, 2016 at 16:05
  • 1
    $\begingroup$ I edited my answer to add the comment content, but you will probably not find it satisfactory. You seem to have a lot of questions, and as your situation is unclear to me, I'm struggling to help you. I would advise thinking about what your problem is, try to split it into smaller pieces, think about what you really need help on. It gets difficult to help you if you go in every direction at once. Try to stay focused :) $\endgroup$
    – Winks
    Jan 18, 2016 at 16:48
  • $\begingroup$ @Winks Welcome to the site. Nice first answer! $\endgroup$ Jan 18, 2016 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.