2
$\begingroup$

Suppose you constructing model whose training data is cumulative in nature; meaning each year you can add new observations with all prior observations being kept the same. (e.g. training set is non-decreasing)

As a close analogy, let's rephrase the objective of the true model in terms of studying college student's behavior across six campuses. One added bonus from the raw data is the detection of the lying rates across students pertaining to a specific question which the truth is known. (Think: Do you attend University A? Their answer can be verified)

After adding in new observations to the previous year's data, we discovered:

  • The rate of lying for all campuses, individually, increased from last year; but
  • The rate of lying, overall, decreased.

Seems like a classic case of Simpson's Paradox. However, I am having difficulty interpreting it as the times I saw Simpson's Paradox used was either the same cohort at the same time between 2x2 table of before/after (link), or same cohort across time (link).

Is this set-up different as new samples are added each year?

I tried to construct simulated data using only three "campuses" to show an example:

Original Group ("Last Year Training Set") \begin{array}{c|rr|r} \text{Campus} & \text{Lied} & \text{Total} & \%\\ \hline \text{A} & 3 & 154 & 1.9 \\ \text{B} & 155 & 783 & 19.8\\ \text{C} & 109 & 624 & 17.5 \\ \hline \text{Total} & 267 & 1561 & 17.1 \\ \end{array}

Final Group (Original + Newly Added = "This Year Training Set") \begin{array}{c|rr|r} \text{Campus} & \text{Lied} & \text{Total} & \%\\ \hline \text{A} & 38 & 1095 & 3.5 \\ \text{B} & 207 & 964 & 21.5\\ \text{C} & 212 & 1012 & 20.9 \\ \hline \text{Total} & 457 & 3071 & 14.9 \\ \end{array}

Notice each campus, individually had an increase of lying which was positive; but overall there was a decrease in lying.

How can one explain these results to one unfamiliar with Simpson's Paradox, and may think the entire analysis may be wrong?

$\endgroup$
  • $\begingroup$ Simpson's paradox can occur in any such comparison across groups where you consider them separately and then all combined, it doesn't have to relate to either cohorts or time. $\endgroup$ – Glen_b Jul 28 '14 at 5:50
  • $\begingroup$ If it was only two groups I think the BK plot would be a good illustration - one point for the original and one point for the final. It isn't obvious how to extend that type of plot to 3 groups though. $\endgroup$ – Andy W Jul 28 '14 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.