Correlated features in dataset

I undestand in general that it is important to take correlational structure into account while applying almost any statistical techniques.

First question - could you help with the examples why it is so important and what can go wrong? For example, I have vector of correlated random variables, identically distributed. Can I wrongly reject the $H_0 : \{sample \in distribution\}$ not by chance, but because of correlational structure? (i.e., reject $H_0$ much more often than 0.05 with fixed level of significance $\alpha=0.05$). Could you explain it, using the picture (samples are 2D-vectors): Second question: for example, I have two normal distributions with different means $N(0, \sigma), N(-1, \sigma)$. I have a sample of rows of numbers, like

$$(0.1, 0.5, -0.3), (-1.2, -1.0, -0.7, -1.1), \ldots$$

$$(0.2, 0.4, -0.2, 0.1), (-1.1, -0.5, -1.0), \ldots$$

$$\ldots$$

$$\xi_1(i), \xi_2(i), \xi_3(i), \ldots$$

here I denotes "islands" generated by one distribution in brackets, $i$ indicates one of the distributions (so $i$ is choosen randomly fro each $\xi(i)$).

I want to determine the borders of the islands, but I know that there is a correlational structure inside each island (may be also a correlational structure between islands, but it does not matter).

How should I deal with this problem? Let us assume that it is not enough samples to calculate the whole matrix of covariances (it will overfit the data). The initial idea is to use something like linear discriminant analysis and regularisation for it, but the borders are unknown. How can PCA help in this case? Typically, the strongest correlations arise from random variables that are neighbors in the row, so one of the possible solutions is to use block covariance matrices, but the size of one block can be large enough to cause problems with overfitting.

Does HMM takes into account the correlational structure? If so, what if I do not have a set of states (for HMM), but a continuous set of parameters for distributions, which technique is better to use?

Why is it important to take correlation into account and what can go wrong?

It depends on the model you use. Let say you want to do some clustering on a dataset that has some correlated features. If you use K-means with the Euclidian distance, the clusters will be spherical around their center.

K-means can be viewed as a special case of fitting Gaussian distributions to your data; the center of each cluster is the mean, and, more importantly, since the cluster is spherical, there is an assumption that the features are not correlated. If the features are correlated, K-means clustering loses meaning.

So if you fit a Gaussian with a diagonal covariance matrix to a your data, you might indeed accept or reject your hypothesis wrongly. In the case of the Outlier example you cite, the 95% interval of the Gaussian would not consider the circled data point as an outlier, while it "obviously is".

In such a setting, using PCA to decorrelate your data into principal components helps. The principal components are the dimensions that maximize variance, and are orthogonal, there should not be any correlation between dimensions, which should solve the correlation problem. There are a lot of visualizations on how PCA works, they might be helpful if you have trouble understanding what it does.

Second question

I want to determine the borders of the islands, but I know that there is a correlational structure inside each island (...) How should I deal with this problem?

It is unclear to me what your data is, as your "sample of rows of numbers" is not cristal clear, and I do not see a clear link between estimating Gaussian distributions and HMM. From your question, I understand that you assume your data comes from multiple Gaussian distributions, and you want to approximate them.

If your model accounts for correlation and tries to learn it from data instead of posing too strict assumptions, you will probably be fine (as long as there is sufficient data). For this I would go for a Gaussian Mixture Model and, indeed, learn the covariances matrices.

There is always the problem of over/underfitting, but no method will give you the true model behind no data. The more assumption you make, the less data you need, but you lead the risk of those assumptions being false. On the other end, making no assumptions and trying to learn everything is nice, but requires more data to learn something meaningful. If you can make educated/domain-specific assumptions, put those into the model but do not rely on picking a model without understanding the assumption it makes.

• not sure if I understand you correctly, but where should I use the L1 or L2 norm? I have just a covariance matrix of $n \times n$ features and $n^2$ is too much for the dataset size...As for the data, my data is a set of correlated random variables, and the true distribution of current random variable is hidden, that's why it is possible to use HMMs. Jan 18 '16 at 16:05