183 votes

What happens if the explanatory and response variables are sorted independently before regression?

I'm not sure what your boss thinks "more predictive" means. Many people incorrectly believe that lower $p$-values mean a better / more predictive model. That is not necessarily true (this being a ...
gung - Reinstate Monica's user avatar
135 votes

What happens if the explanatory and response variables are sorted independently before regression?

If you want to convince your boss, you can show what is happening with simulated, random, independent $x,y$ data. With R: ...
RUser4512's user avatar
  • 9,766
109 votes

What happens if the explanatory and response variables are sorted independently before regression?

Your intuition is correct: the independently sorted data have no reliable meaning because the inputs and outputs are being randomly mapped to one another rather than what the observed relationship was....
d0rmLife's user avatar
  • 1,957
81 votes

Does no correlation imply no causality?

does an absence of correlation imply absence of causality? No. Any controlled system is a counterexample. Without causal relationships control is clearly impossible, but successful control means - ...
conjugateprior's user avatar
69 votes

Does correlation = 0.2 mean that there is an association "in only 1 in 5 people"?

The quoted passage is indeed incorrect. A correlation coefficient quantifies the degree of association throughout an entire population (or sample, in the case of the sample correlation coefficient). ...
Kodiologist's user avatar
  • 19.7k
63 votes

When to remove insignificant variables?

Let me first ask this: What is the goal of the model? If you are only interested in predicting if a customer will buy, then statistcal hypothesis tests really aren't your main concern. Instead, you ...
Demetri Pananos's user avatar
62 votes

Why zero correlation does not necessarily imply independence

Correlation measures linear association between two given variables and it has no obligation to detect any other form of association else. So those two variables might be associated in several other ...
Marcelo Ventura's user avatar
56 votes

Generate a random variable with a defined correlation to an existing variable(s)

I will describe the most general possible solution. Solving the problem in this generality allows us to achieve a remarkably compact software implementation: just two short lines of ...
whuber's user avatar
  • 316k
55 votes

Is there any relationship among cosine similarity, pearson correlation, and z-score?

The cosine similarity between two vectors $a$ and $b$ is just the angle between them $$\cos\theta = \frac{a\cdot b}{\lVert{a}\rVert \, \lVert{b}\rVert}$$ In many applications that use cosine ...
GeoMatt22's user avatar
  • 12.6k
52 votes

What happens if the explanatory and response variables are sorted independently before regression?

Actually, let's make this really obvious and simple. Suppose I conduct an experiment in which I measure out 1 liter of water in a standardized container, and I look at the amount of water remaining ...
heropup's user avatar
  • 5,386
50 votes

Generating correlated binomial random variables

Binomial variables are usually created by summing independent Bernoulli variables. Let's see whether we can start with a pair of correlated Bernoulli variables $(X,Y)$ and do the same thing. Suppose ...
whuber's user avatar
  • 316k
49 votes

Example where $X$ and $Z$ are correlated, $Y$ and $Z$ are correlated, but $X$ and $Y$ are independent

Intuitive example: $Z = X + Y$, where $X$ and $Y$ are any two independent random variables with finite nonzero variance.
fblundun's user avatar
  • 3,929
44 votes

Difference between Random Forest and Extremely Randomized Trees

The Extra-(Randomized)-Trees (ET) article contains a bias-variance analysis. In Fig. 6 (on page 16), you can see a comparison with multiple methods including RF on six tests (tree classification and ...
user91213's user avatar
  • 2,084
44 votes

When A and B are positively related variables, can they have opposite effect on their outcome variable C?

The other answers are truly marvelous - they give real life examples. I want to explain why this can happen despite our intuition to the contrary. See this geometrically! Correlation is the cosine of ...
sds's user avatar
  • 2,226
43 votes

Interview question: If correlation doesn't imply causation, how do you detect causation?

There are a few ways around this. You are right that A/B testing is one of these. The economics Nobel this year was awarded for the pioneering of field experiments in the study of policies against ...
Student's user avatar
  • 938
42 votes

Why does independence imply zero correlation?

By the definition of the correlation coefficient, if two variables are independent their correlation is zero. So, it couldn't happen to have any correlation by accident! $$\rho_{X,Y}=\frac{\...
OmG's user avatar
  • 1,077
42 votes

Can statistical units measured per thousand inhabitants be bigger than 1000?

This is not a rate per one thousand people, this is the absolute number of people, with one unit equating 1,000 people. So if you see something like 3,258.1, it simply means 3,258,100 people. This is ...
J-J-J's user avatar
  • 3,123
38 votes

Such thing as a weighted correlation?

Formula for weighted Pearson correlation can be easily found on the web, StackOverflow, and Wikipedia and is implemented in several R packages e.g. psych, or weights and in Python's statsmodels ...
Tim's user avatar
  • 135k
37 votes

Correlation does not imply causation; but what about when one of the variables is time?

I'll provide another answer, since I think the ones currently provided miss an important point of the statement the physicist made. The quoted statement is: "correlation does not imply causation, ...
Duncan's user avatar
  • 1,302
35 votes

How is it possible to obtain a good linear regression model when there is no substantial correlation between the output and the predictors?

A pair of variables may show high partial correlation (the correlation accounting for the impact of other variables) but low - or even zero - marginal correlation (pairwise correlation). Which means ...
Glen_b's user avatar
  • 277k
34 votes

Zero correlation of all functions of random variables implying independence

Using indicator functions of measurable sets like$$f(x)=\mathbb I_A(x)\quad g(x)=\mathbb I_B(x)$$leads to$$\text{cov}(f(X),g(Y))=\mathbb P(X\in A,Y\in B)-\mathbb P(X\in A)\mathbb P(Y\in B)$$therefore ...
Xi'an's user avatar
  • 103k
33 votes

Does no correlation imply no causality?

No. Mainly because by correlation you most likely mean linear correlation. Two variables can be correlated nonlinearly, and may show no linear correlation. It's easy to construct an example like that, ...
Aksakal's user avatar
  • 60k
32 votes

PCA on correlation or covariance?

A common answer is to suggest that covariance is used when variables are on the same scale, and correlation when their scales are different. However, this is only true when scale of the variables isn'...
Aksakal's user avatar
  • 60k
32 votes

Does causation imply correlation?

Things are definitely nuanced here. Causation does not imply correlation nor even statistical dependence, at least not in the simple way we usually think about them, or in the way some answers are ...
Carlos Cinelli's user avatar
32 votes

When A and B are positively related variables, can they have opposite effect on their outcome variable C?

Yes, two co-occuring conditions can have opposite effects. For example: Making outrageous statements (A) is positively related to being entertaining (B). Making outrageous statements (A) has a ...
Matthew Gunn's user avatar
31 votes

Does mean centering reduce covariance?

If $X$ and $Y$ are random variables and $a$ and $b$ are constants, then $$ \begin{aligned} \operatorname{Cov}(X + a, Y + b) &= E[(X + a - E[X + a])(Y + b - E[Y + b])] \\ &= E[(X + a - E[X] - E[...
Artem Mavrin's user avatar
  • 3,927
29 votes

Why zero correlation does not necessarily imply independence

There is a generalized lack of rigor in the use of the word "correlation" for the simple reason that it can have widely differing assumptions and meanings. The simplest, loosest and most common usage ...
Mike Hunter's user avatar
  • 10.2k
29 votes

Why are random walks intercorrelated?

Your independent processes are not correlated! If $X_t$ and $Y_t$ are independent random walks: A correlation coefficient unconditional on time does not exist. (Don't talk about $\operatorname{Corr}(...
Matthew Gunn's user avatar
28 votes

Simple examples of uncorrelated but not independent $X$ and $Y$

I think the essence of some of the simple counterexamples can be seen by starting with a continuous random variable $X$ centered on zero, i.e. $E[X]=0$. Suppose the pdf of $X$ is even and defined on ...
28 votes

Difference between Random Forest and Extremely Randomized Trees

ExtraTreesClassifier is like a brother of RandomForest but with 2 important differences. We are building multiple decision trees. For building multiple trees, we need multiple datasets. Best practice ...
Ashish Anand's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible