2
$\begingroup$

E.g. Anscombe's quartet teaches that "visualizing data is important, as the summary statistics can be the same while the data distributions can be very different". But if you have millions of samples it would be difficult to look on all of them. So:

Question 1: are there any automatic tests which would help to check that obtained high correlation (say 0.6, p-value = 1e-16) is really meaningful ? I.e. fits to idea that there is a kind of reasonable dependence between variables

In particular consider the case on the left figure. High correlation 0.6, very low p-value. But visual inspection shows that actually it is not the situation where we have real dependence between data. Because: data consists of two parts - mostly we have ball like point cloud - which means almost zero correlation (that SUBpart is shown on the right figure separately) and some part of points with "y=0" ("dropout") - it is technical unwanted noise, but put together we get high correlation 0.6.

Question 2: Is there any way to detect the problem with correlation in such situation except directly excluding samples with "y=0" ? (Excluding "y=0", may work in that case, but I am afraid there can be other unexpected situations, so I am looking for a more general recipe).

enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

I tried to illustrate a good approach (for me) to deal with this kind of issue in practice. First, I simulated some data to mimic what you observed in yours.

$y = 10 + 0.5 * N(1000, 10,2) + N(1000, 0, 5)$.

enter image description here

Fitting a linear regression, I obtained a small proportion of variance explained by the model (R2 = 0.03).

Then, I set all values less than 10 to zeros, the data look like this: enter image description here

If you have fixed your intercept to be zero (which is rarely a good idea in practice), you will biased your model towards the significance. Indeed, fitting a model removing the intercept leads to a R2 of 0.80 (red line for the model without intercept). However, increasing your intercept with an offset term to the original intercept value gives an equivalent R2 observed in the original data (purple line).

My advice here should be to add an arbitrary value corresponding to your y mean value to force your model to not pass through the origin, but to the expected value in the absence of technical noise. Indeed, we can assume that a large proportion of zeros will tend your intercept towards zero.

Here you need to have a strong knowledge of your data to set a reasonable intercept value corresponding to a true mechanism. Unfortunately, there is not "magic trick" in statistics to systematically do stuff.

Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.