2
$\begingroup$

Suppose that there is a property $\mathbf{p}$, that for each study case $i$ we want to estimate as: $$p_i = c_1 x_{i1}+c_2 x_{i2}$$

If $p_i$ and $x_{i2}$ are uncorrelated, then we simply would do

$$p_i = c^\prime _1 x_{i1}$$

In practice: How to be sure that this procedure is correct?

Wikipedia article on Uncorrelated random variables states that two random uncorrelated variables $X$ and $Y$ satisfy $$0= E(XY)-E(X) E(Y)$$ but I bet it won't be the case for a finite number of observations.

¿Can (in practice) it be used to say something about the correlation between $p_i$ and $x_{i2}$?

Alternatively, I think that the corresponding p-value from the bivariate regression can be enough to state that: it is very probably that there is not correlation and because of that the first model is not better than the second one. Is this correct? Are there other useful, convenient or necessary test that I can use for the purpose?

Edit: I found that the two answers are complementary and equally useful and interesting. So in few days I will choose the less voted answer to try to equally the recognition that they receive.

$\endgroup$
0

2 Answers 2

2
$\begingroup$

The following statement (paraphrased your question a bit) is not quite true:

Given that $p_i$ and $x_{i2}$ are uncorrelated, the model $p_i \sim x_{i1} + x_{i2}$ can be reduced to $p_i \sim x_{i1}$.

In practice: How to be sure that this procedure is correct?

You have to define "correctness." One definition of correctness could be -- "The reduced model have lower mean squared error than the original model", -- in which case it's easy to show a counter example in which the statement need not be true.

You can check a discussion and example here: https://stats.stackexchange.com/a/206663/54725, in which I constructively show an example where a variable (here $x_{i2}$) that is uncorrelated with the outcome (here $p_i$) can actually have a statistically-significant non-zero coefficient in the regression model and improve predictive accuracy of the model. Note that this is true even if we are only considering linear relationships between variables, and even if you ignore all artifacts due to finite data.

To answer your original question: How do I prove that two variables $X$ and $Y$ are uncorrelated?

  • If you have reasons to believe that $X$ and $Y$ are drawn from a multivariate normal distribution, then you can test whether the Pearson product-moment correlation is 0 at some significance level.

  • If you don't know the relationship between $X$ and $Y$, then this problem can be hard.

    • If $X$ and $Y$ are nominal, then you can use the Chi-squared test of independence: https://onlinecourses.science.psu.edu/stat500/node/56.

    • If $X$ and $Y$ are real valued, you could also try to use distance correlation (https://en.wikipedia.org/wiki/Distance_correlation), which can help establish a stronger property that two variables are independent of each other (and are hence uncorrelated). I don't know off-hand any statistical test that uses this correlation, but you could always bootstrap the distribution of the sample's distance correlation and work with it.

    • I don't know how to deal with cases that are a mix of categorical and continuous variables. Perhaps someone else can add a pointer!

Hope this helps.

$\endgroup$
5
  • $\begingroup$ Thank you very much. So, $x_2$ can be correlated or uncorrelated to $p$, but it does not imply something about the usefulness to add $x_2$ to the linear model. Right? I'm puzzled by the statement of the cited Wikipedia article "If two variables are uncorrelated, there is no linear relationship between them". Is it wrong? Is it only applicable to one variable model? Am I missing something? $\endgroup$ Commented May 3, 2016 at 16:10
  • $\begingroup$ Your first statement is correct assuming "usefulness" is defined precisely as I noted in my answer. The statement in Wikipedia is also correct in a loose sense, and your interpretation is right -- the statement is applicable only to a one-variable model $X \sim Y$. I say "loose sense" because some people use the term correlation to also mean non-linear forms of relationship. $\endgroup$
    – Vimal
    Commented May 3, 2016 at 16:22
  • $\begingroup$ An example where $x_2$ is correlated to $p$ but is not useful in reducing the MSE is when $x_2=x_1$ (multicollinearity), and is correlated with $p$. $\endgroup$
    – Vimal
    Commented May 3, 2016 at 16:26
  • $\begingroup$ I got it. And in practice I have to perform something like leave-one-out procedure. If not, I will always get lower mean squared error with the two model variables. Right? $\endgroup$ Commented May 3, 2016 at 16:27
  • $\begingroup$ Yes, in practice, with finite data, what you say is correct. (I assume by leave-one-out you meant leave-one-out cross-validation to avoid over-fitting.) $\endgroup$
    – Vimal
    Commented May 3, 2016 at 16:30
2
$\begingroup$

If the assumption that the regression errors are i.i.d and normal is correct then yes, the $p$ value is precisely what you want to look at. It tells you that, to keep your notation, $x_2$ is linearly uncorrelated to $p$ after controlling for $x_1$.

However a few issues might arise.

First, you might not believe that the errors are normal. In this case you should generate confidence intervals, and $p$ values, through re-sampling bootstrap to make sure.

Second, even if there is no linear correlation there might be some non-linear effect. You want to capture that effect to improve your prediction of $p$. The easiest way to spot a non-linear correlation while keeping safely within the linear regression confines is probably the Frisch-Waugh-Lovell theorem which gives you a nice visual clue about possible structure. I gave a simple R example here.

$\endgroup$
4
  • 1
    $\begingroup$ Thank you very much. If I understand, you're suggesting to do the following: regress $x_1$ on $_p$ and get residuals $e_1$, $x_2$ on $x_1$ and get residuals $e_2$. Plot $e_1$ vs $e_2$ to search for a tendency. It would be better than plot $x_2$ vs $p$ because what I want to know if it is useful to add $x_2$ to the model. Is there any difference between this and: Regress $x_1$ on $_p$ and get residuals $e_1$. Plot $e_1$ vs $x_2$, or regress $x_2$ on $e_1$ and inspect parameters like $R^2$ ? $\endgroup$ Commented May 3, 2016 at 16:00
  • $\begingroup$ yeah, it would be different. The regression would only look for linear relationships so you'd have no more information than what you started with. Plotting $e_1$ on $x_2$ is also not effective as you need to remove all the parts of $x_2$ linearly related to $x_1$ in order for the plot to be informative (or you catch linear structure between regressors rather than between $x_2$ and $p$). If you are a graphical thinker (or grasp intuitively orthogonal projections) the slides here are of great help econ.uiuc.edu/~wsosa/econ507/OLSGeometry.pdf $\endgroup$
    – CarrKnight
    Commented May 3, 2016 at 16:09
  • $\begingroup$ I did not realize that the non orthogonality between $x_1$ and $x_2$ would turn the plot uninformative. But, if I only want to compare the two linear models (of one or two variables), and I do not care (for some convenience) if it is possible to get a better non linear model, I just would use the p-value right? In such case, you said: "If the assumption that the regression errors are i.i.d and normal is correct ". Which regression? The bi-variate one? What i.i.d. stand for? $\endgroup$ Commented May 3, 2016 at 16:18
  • $\begingroup$ i.i.d. stands for independent and identically distributed. It's basically assuming that each case study is an independent realization and follows the same mechanisms as all the other case studies. If you want to simply select the "better" model only between $\sim x_1$ and $\sim x_1+x_2$ then yes just check the $p$ value. You should however at least test for normality (see people.duke.edu/~rnau/testing.htm for example) $\endgroup$
    – CarrKnight
    Commented May 3, 2016 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.