Questions tagged [simpsons-paradox]
Simpson's Paradox is an example of the Reversal Paradox, where an association appears in several different groups of data but disappears, or even reverses in sign, when these groups are combined.
65
questions
5
votes
1
answer
307
views
How do I know if I'm seeing Simpson's paradox?
I need some guidance on how to analyse the results of a subgroup breakdown in an A/B test.
I have the results of an (ongoing) A/B test and need to do an interim analysis on
The overall headline ...
- 115
2
votes
0
answers
18
views
How do we draw parallels between the BBG drug example and the businessman (affected by the election) anecdote in "The book of why"?
In "The book of why", Dr Pearl concludes that a BBG drug cannot exist after phrasing the sure-thing principle in 'a more correct way' (pg 214). He does this by first "insisting" ...
0
votes
0
answers
30
views
How to address Simpson's paradox when aggregating ratios?
I have a table of means and ns from several studies comparing x and y groups. I've also computed the ratio of the means for each ...
- 190
1
vote
1
answer
46
views
Algorithm for Creating 2x2 Tables to Demonstrate Simpson's Paradox
Suppose I have a 2x2 table: $T = \lbrace a,b,c,d \rbrace$, where $a=T(1,1), b=T(2,1), c=T(1,2), d=T(2,2)$, where all entries of $T$ are positive integers. Let us assume that $\frac{ad}{bc} > 1$.
...
- 323
0
votes
2
answers
117
views
Combining biased noisy measurements
I'm looking for a way to combine noisy biased measurements to find confidence intervals. As an example, we have two people Tom & Mary that are each taking free throws on separate days, we have the ...
- 113
0
votes
0
answers
40
views
How do i explain the difference of correlation between mean income and total income?
I am doing a report on the complaints made to the municipality by the people of an italian city. I'm doing the part of the analysis where I try to understand where people complain the most.
The ...
- 15
2
votes
1
answer
207
views
Negative correlation in two groups (separately) but overall data shows positive correlation
Regression on the overall data gives positive correlation coefficient. However, if I divide the data by gender (male and female) and run the same regression model separately on each group - I get ...
- 33
1
vote
0
answers
33
views
Why does my predictor in two-way fixed effects models have opposite signs in the full panel data vs. period time subsets?
I have a heavily unbalanced panel data set in which certain entities are missing on the DV in certain years. On this data set, I run the two-way fixed effects model: ...
- 11
0
votes
0
answers
21
views
Linear Mixed Model in lme4 package- estimate is correct magnitude but wrong direction
I am using lmer in lme4 to do linear mixed modeling. I have simplified my model to just one fixed effect and one random effect to isolate the problem. I have a continuous DV (score), a dichotomous ...
- 1
2
votes
3
answers
211
views
Is a 2-way ANOVA immune to Simpson's Paradox?
Sometimes continuous data is of the type that would result in Simpson's Paradox findings if groups were simply aggregated. (See, for example, "Example #2: Baseball" on this page https://www....
- 3,276
0
votes
1
answer
91
views
How to decompose total slope into between-group and within-group contributions?
Consider the following data (I put the data as a table in the bottom of this question.)
These data are in two groups, blue and orange. In each group there is a positive relationship, while pooling ...
- 283
0
votes
1
answer
104
views
Simpson paradox with lmer
I am measuring 2 responses in patients from different age cohorts. Each response is negatively correlated with age. This gives me a positive correlation of ResponseX and ResponseY in a total ...
- 718
4
votes
2
answers
242
views
Making a ML model robust to Simpson's paradox
Suppose a model which predicts which location/landmark a walking tourist is going to visit next, based on two geographical input features:
the last neighborhood this person has walked through
the ...
- 571
3
votes
1
answer
743
views
Simpson's Paradox
I need some help figuring out whether the phenomenon of Simpson's paradox has occurred.
Here is a plot of the first dataset (correlation between probability of contracting a disease vs. hours slept ...
- 43
1
vote
1
answer
30
views
Is there a phenomenon like this?
It's like the mean of every part increases, but the overall mean decrease; or the mean of a group is larger in every part but is smaller in all samples; or something like this.
I remember there is a ...
- 423
2
votes
1
answer
460
views
What is the solution of simpson paradox?
In my nonexperimental data, when running regression, I faced the Simpson Paradox.
Simplistically speaking, Pearl,2014 said:
Simpson’s paradox refers to a phenomena whereby the association between a ...
- 319
17
votes
3
answers
3k
views
How to handle Simpson's paradox
Simpson's "paradox" is a well-known phenomenon that can be counter-intuitive for beginners: it is possible, say, for a medical trial to reveal that a certain treatment is beneficial to men ...
- 301
2
votes
1
answer
103
views
Alternative Aggregation Method to Avoid Simpson's Paradox
The table below shows an example of Simpson's Paradox arising from some fictitious data for the success of operations performed by two doctors from the Simpsons TV show.
Dr Hilbert
Dr Nick
Heart ...
- 275
8
votes
2
answers
359
views
Is it possible that marginally independent random variables are conditionally dependent?
Suppose that $X,Y$ and $Z$ are random variables. If $X$ is independent of $Z$ and $Y$ is independent of $Z$, is it possible that $X$ is dependent on $Z$ given $Y$ and $Y$ is dependent on $Z$ given $X$?...
- 4,455
3
votes
1
answer
276
views
Simpson's paradox in Judea Pearl's book?
I'm looking at the following question in Judea Pearl's primer on causality
In an attempt to estimate the effectiveness of a new drug, a randomized experiment is conducted. In all, 50% of the patients ...
- 551
1
vote
0
answers
34
views
Can we calculate the correlation between 2 variables just from the correlations between subsets of the data?
If I have two variables X, Y and I already have correlations for subsets of the data that that are mutually exclusive and exhaustive, can I compute the overall correlation directly from this?
It seems ...
- 111
3
votes
0
answers
158
views
When random effects fail to address Simpson's paradox
This site has a couple of nice questions and answers dealing with Simpson's paradox and random effects (e.g. Simpson's Paradox & Random Effects, Understanding Simpson's paradox with random effects)...
- 1,888
1
vote
0
answers
112
views
Stochastic independence, but functional dependence
Here is an extract from "Comment: A Fruitful Resolution to Simpson's Paradox via Multiresolution Inference" by Keli Liu and Xiao-Li Meng of Harvard University (The American Statistician, ...
- 71
4
votes
1
answer
489
views
How to see Simpson's paradox effect via glm in R
I am using the UCBAdmissions dataset and glm from R to check for Simpson's paradox. I am ...
- 237
6
votes
1
answer
1k
views
Understanding Simpson's paradox with random effects
Simpson's paradox is well known as a situation where the correlation between 2 variables in groups (ie within-group slope) is of opposite sign to the overall correlationed between the 2 variables, ...
- 57k
19
votes
4
answers
4k
views
Examples of Simpson's Paradox being resolved by choosing the aggregate data
Most of the advice around resolving Simpson's paradox is that you can't decide whether the aggregate data or grouped data is most meaningful without more context.
However, most of the examples I've ...
- 684
8
votes
1
answer
2k
views
Intuition behind Partial Residual Plots
I've been reading how univariate analysis in data with a lot of variables can be misleading due to "Simpson's paradox". I found the explanation of this phenomenon pretty fascinating but easy to ...
- 380
1
vote
1
answer
1k
views
Does Linear Regression solve Simpson's Paradox?
I'm facing a problem where I think I might be falling into Simpson's Paradox. I remember proving Linear Regression by using partial derivatives, so I'm thinking that even though Linear Regression is a ...
- 650
15
votes
4
answers
3k
views
Simpson's Paradox vs. Berkson's paradox
Can someone explain what is the difference between the two? They seem to me to be identical. In both paradoxes you start from a narrow distribution and find out the correlation switches when you move ...
- 3,271
4
votes
4
answers
468
views
Simpson Paradox Question
I am trying to understand if the following statement is an example for Simpson paradox:
"In the US elections a certain candidate got more votes than the other, but the other one was elected".
I ...
- 43
0
votes
0
answers
118
views
UC Berkeley gender bias study formulation in terms of probabilities
I'm trying to strengthen my foundations of statistics, and I thought it would be interesting to turn the problem below into some set of equations instead of a more qualitative treatment as is done in ...
- 31
0
votes
1
answer
237
views
Estimate the effect of dummies with unbalanced data
Imagine having a factory with jobs of different complexity, and a few machines that can execute these jobs; a job can succeed or fail. However, jobs are not assigned completely randomly, there is some ...
- 111
0
votes
0
answers
140
views
HELP: LM shows no relationship, but LMM does
My research question is assessing if a variable (let’s call it ‘x') can predict another variable (let’s call it ‘y’). The two variables x and y are in the same units, but they just come from ...
- 9
3
votes
0
answers
225
views
Looking to identify a book by a top statistician with a chapter on Simpson's Paradox
It was more than 20 years ago. I had just gotten acquainted with Simpson's paradox. I was browsing in a bookstore and saw a book by an eminent statistician -- eminent in the sense that I had come ...
- 71
0
votes
0
answers
35
views
Relationship between correlation and multiple hierarchial regression [duplicate]
I got a negative correlation value between two variables. Results of running Multiple hierarchical linear regression on the same variables indicated a positive beta coefficient between them. Explain ...
3
votes
1
answer
674
views
Distribution to match an example with collider bias?
I am doing exercises from "Causal Inference in Statistics: A Primer", by Pearl et al (2016). In chapter 1.2 there is a training challenge that goes like:
In an attempt to estimate the effectiveness ...
- 344
1
vote
1
answer
252
views
When should we use the segregated as opposed to the aggregated data?
In the book "Causal Inference In Statistics" by Pearl et al., there is the following problem (study question 1.2.2.)
A baseball batter Tim has a better batting average than his teammate Frank. ...
user82135
1
vote
0
answers
53
views
Recovering hidden confounder in Simpson's paradox trends
I just watched a video of an interested talk from PyData LA: "Using Simpson’s Paradox to Discover Interesting Patterns in..." - Nazanin Alipourfard, Peter Fennell (https://www.youtube.com/watch?v=...
- 121
6
votes
2
answers
337
views
When does Simpson's Paradox "end"?
Disclaimer: This is not a duplicate of How to resolve Simpson's paradox.
As given in this blog, the following is the data of people on the titanic:
This is the same data when divided on basis of ...
- 643
1
vote
1
answer
309
views
Detecting Simpson Paradox using lme
I am playing with a toy data where the Simpson's paradox exists for two variables NO2 and temperature:
A scatter plot clearly shows that the correlation between NO2 and temperature was reversed when ...
- 33
9
votes
2
answers
2k
views
Does Simpson's Paradox cover all instances of reversal from a hidden variable?
The following is a question about the many visualizations offered as 'proof by picture' of the existence of Simpson's paradox, and possibly a question about terminology.
Simpson's Paradox is a fairly ...
- 1,863
3
votes
2
answers
1k
views
Intuition needed when using weighted average to explain Simpson's paradox
In Freedman's Statistics (chapter 2), the author uses Berkeley's admission statistics (that 44% men and 35% women were admitted to graduate programs in general) to illustrate Simpson's paradox: the ...
- 200
1
vote
0
answers
29
views
Language data as as a ratio of the number of words?
I am analyzing language data (nouns, verbs, etc) of people with and without autism and all of them have different number of words.
Option 1: Would it be best to count each result as a ratio of the ...
- 11
1
vote
1
answer
3k
views
Effect of combining predictor variables in a regression model
Let's say I first run a linear regression model Sales = f(TV Spend, Digital Spend).
Now I add TV Spend and Digital Spend and run the second model. My second model is Sales = f(TV Spend+Digital Spend)...
- 41
3
votes
1
answer
112
views
Any examples of generalisation of simpson's paradox to other metrics
Simpson's paradox is introduced on wikipedia using 'metrics' of success rates and regression coefficients, for the first (success rate of kidney stone treatments): How to resolve Simpson's paradox?...
- 4,110
13
votes
2
answers
1k
views
Can you please explain Simpson's paradox with equations, instead of contingency tables?
I probably don't have a clear understanding of the Simpson's paradox. Informally I know that the average of response Y1, grouped over all possible levels of factor A, can be higher than the average of ...
- 17.4k
7
votes
3
answers
2k
views
Is Simpson's Paradox always an example of confounding?
Is Simpson's Paradox always an example of confounding? Or is it possible to have a Simpson's paradox effect without an extra variable lurking in the background?
- 597
5
votes
0
answers
122
views
Is stagewised feature engineering/ selection an invalid approach? What to do when all the features are not ready at one time?
Suppose we want to build a regression or classification model. However, the features (independent variables used) are not all ready at one time.
This is very realistic in business, because the data ...
- 36.1k
4
votes
3
answers
1k
views
A way to show Simpson's paradox is happening in the data
I'm not entirely sure what I am talking about is Simpson's paradox, because an opposite relationship does not appear when you combine two data sets, but merely a different one. Still, I think it is ...
- 123
7
votes
1
answer
2k
views
Is this Simpson's Paradox on the Titanic data set?
With the well known "Survival of passengers on the Titanic" data set I get a strange behaviour by plotting the fare vs. the age (see below). Without a constraint on Pclass the correlation is positive. ...
- 672