Learning statistical concepts through data analysis exercises

Question

I find that simple data analysis exercises can often help to illustrate and clarify statistical concepts. What data analysis exercises do you use to teach statistical concepts?

Answer 1

As I have to explain variable selection methods quite often, not in a teaching context, but for non-statisticians requesting aid with their research, I love this extremely simple example that illustrates why single variable selection is not necessarily a good idea.

If you have this dataset:

y      X1     x2
1       1      1
1       0      0
0       1      0
0       0      1

It doesn't take long to realize that both X1 and X2 individually are completely noninformative for y (when they are the same, y is 'certain' to be 1 - I'm ignoring sample size issues here, just assume these four observations to be the whole universe). However, the combination of the two variables is completely informative. As such, it is more easy for people to understand why it is not a good idea to (e.g.) only check the p-value for models with each individual variable as a regressor.

In my experience, this really gets the message through.

Answer 2

Multiple Regression Coefficients and the Expected Sign Fallacy

One of my favorite illustrations of a statistical concept through a data analysis exercise is the deconstruction of a multiple regression into multiple bivariate regressions.

Objectives

• To clarify the meaning of regression coefficients in the presence of multiple predictors.
• To illustrate why it is incorrect to “expect” a multiple regression coefficient to have a particular sign based on its bivariate relationship with Y when the predictors are correlated.

Concept

The regression coefficients in a multiple regression model represent the relationship between a) the part of a given predictor variable (x1) that is not related to all of the other predictor variables (x2...xN) in the model; and 2) the part of the response variable (Y) that is not related to all of the other predictor variables (x2...xN) in the model. When there is correlation among the predictors, the signs associated with the predictor coefficients represent the relationships among those residuals.

Exercise

1. Generate some random data for two predictors (x1, x2) and a response (y).
2. Regress y on x2 and store the residuals.
3. Regress x1 on x2 and store the residuals.
4. Regress the residuals of step 2 (r1) on the residuals of step 3 (r2).

The coefficient for step 4 for r2 will be the coefficient of x1 for the multiple regression model with x1 and x2. You could do the same for x2 by partialing out x1 for both y and x2.

Here's some R code for this exercise.

set.seed(3338)
x1 <- rnorm(100)
x2 <- rnorm(100)
y <- 0 + 2*x1 + 5*x2 + rnorm(100)
lm(y ~ x1 + x2)  # Multiple regression Model
ry1 <- residuals(  lm( y ~ x2)  )  # The part of y not related to x2
rx1 <- residuals(  lm(x1 ~ x2)  ) # The part of x1 not related to x2
lm( ry1  ~ rx1) 
ry2 <- residuals(  lm( y ~ x1)  ) # The part of y not related to x1
rx2 <- residuals(  lm(x2 ~ x1)  ) # The part of x2 not related to x1
lm( ry2 ~ rx2)

Here are the relevant outputs and results.

Call:
lm(formula = y ~ x1 + x2)

Coefficients:

(Intercept)           ***x1***           ***x2***  
   -0.02410      ***1.89527***      ***5.07549*** 

Call:
lm(formula = ry1 ~ rx1)

Coefficients:

(Intercept)          ***rx1***  
 -2.854e-17    ***1.895e+00*** 

Call:
lm(formula = ry2 ~ rx2)

Coefficients:

(Intercept)          ***rx2***  
  3.406e-17    ***5.075e+00***