How to assess whether a correlation between play and number of friends holds in different contexts?

Question

I am trying to ascertain if a relationship exists between hours of play and number of friends. I have two contexts for the study: "at school" and "at a place other than the school".

My data set relates to the "at school" context and the correlation r=0.8766 (significant).

Now the difficult part....

As I want to focus on the same group of students for the "at a place other than the school" context, I have asked a question that filters my respondents and puts them in the "at a place other than the school context".

The question is: Did you make any friends in the last three weeks? The answer can be either "yes" or "no". If it is yes, then I have asked them to indicate whether they made the friends "at the school", "at a place other than the school" or "at both places".

So I have four data sets as follows:

1. Those who did not make any friends
2. Those who made one or more friends at the school
3. Those who made one or more friends at a place other than the school
4. Those who made one or more friends at both places (i.e. school and a place other than the school)

I have ignored data set 1 because there is obviously no relationship between hours of play and number of friends because no new friends are made.

I have also ignored data set 2 because I already have this finding (in the correlation above, which was at the school)

I have also ignored data set 4 because it does not distinguish between "at school" and "at a place other than the school".

I have used data set 3 and worked out the correlation, which is 0.7687 (significant). My interpretation of this finding is the relationship between hours of play and number of friends exists at the school and also outside the school.

Does this make sense?

Answer 1

But in group 1, those people did go to school and did go somewhere else. So you are throwing away observations/information and potentially biasing your results. Did people in this group not make friends because they didn't play much? Or did they play a lot, but still didn't make friends? These two possibilities play out very differently for calculating your correlation.

Ideally, you would want, for each kid, the number of hours of play at school, number of hours of play somewhere else, number of friends made at school, number of friends made somewhere else. You don't have that; you only have total friends made.

You could do a regression of total friends made on hours of play at school and hours elsewhere. This gives you the impact of an hour of play on friend making for each location. You can compare the estimated impacts/coefficients to see if they are the same. Regression coefficients essentially measure correlation between your outcome (friend making) and your predictor (hours at a particular place) holding other stuff fixed (hours at the other place). That's the approach that I'd try.

Answer 2

Your results certainly suffer from sample selection bias. You do not have a random sample of kids who made friends out of school; you only have a random sample of children (at best; the original sample may have had its own biases). By selecting a subsample for analysis that is based on what the subjects have chosen to do (or were forced to by the parents choice), you confound your "significant correlations" with whatever factors were relevant for the original choice. E.g., in a family with both working parents, the child has to attend an after-school program for pretty much a fixed amount of time, and may be making friends there. Generally, restricting your sample based on the response variable is a very poor idea. Consider this:

    x <- runif(1000)
    y <- x + rnorm(1000,mean=0,sd=0.15)
    lm(y~x)
    lm(y[y>0.5]~x[y>0.5])
    lm(y[x>0.5]~x[x>0.5])

The regression coefficient is 0.65-ish instead of 1 in the second regression; there aren't any problems in the second regression, except for lower power, of course.

I would analyze the whole data set as a Poisson regression with hours of play as an offset variable, and settings as additional explanatory variables (i.e., they increase proportionaly the rate of "friend-making"). May be as a zero-inflated Poisson regression, if there are any characteristic of the child or the location that preclude them from making friends (e.g., walkable vs. car-only neighborhoods).