1
$\begingroup$

There are two independent samples of people, drawn from a population of a city at times $t_1$ and $t_2$, a decade apart*. The people were asked rate their preference regarding some question $Q$ on a scale of 1...10, as well as report their age and sex. The research question is, has average preference towards $Q$ changed in the city over time.

enter image description here

However, the sample composition varies somewhat between the two measurement time points in terms of both sex ratios and ages represented in the sample. The issue is therefore comparing means of $Q$ between the two temporal samples in a way that controls for sampling differences. E.g., in case say old men are thought to give a higher $Q$ rating, and there are slightly more older people and slightly more men in the $t_2$ sample, the the mean might look higher than in $t_1$ - but it might be really just an effect of having more old dudes in there. So it seems I can't just run a simple t-test.

Here is an attempt: concatenate data from the two samples and record time {$t_1, t_2$} as a variable, build two regression models predicting the $Q$ rating, with age and sex as control variables, one model has time as a categorical predictor, and the other doesn't.

model a: $Q \sim age ~+~sex $

model b: $Q \sim age ~+~sex ~+~time $

...and then compare the models to see if time adds descriptive power to the model (could also run cross-validation, compare prediction error, etc.). The idea is, if model b performs significantly better, then the differences between the two temporal samples must be significant too, after controlling for differences in sampling. So, for example, if mean $Q$ rating is a point higher in $t_2$ - but model b is not an improvement over model a - then I would conclude that the apparent "increase" is just a sampling effect.

Bottom line question: does this make sense? If not, why, and is there a better way to answer this sort of a research question?


* So technically, two samples from two different populations. There are only these two samples in the dataset, not any longer time series; and there are no overlaps in terms of participants, i.e. not that q; the sample sizes are almost the same, i.e. not that q. The differences between sex&age ratios are assumed to stem from sampling issues, not because of major changes in the city population in terms of sex ratio or age composition.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
+50
$\begingroup$

Don't jump the gun - clear research question first, statistical model after

Before you get to the point where you decide on your statistical test, you have to be careful to clearly frame the question you actually want to ask, and the sampling method that was used to get your data. One thing you are missing in your framing of the question is that the demographic composition of a city can change substantially over a decade, so it is entirely possible that the change in age/sex composition in your data reflects an actual change in the demographic composition of the city, rather than an aspect of the sampling. In order to determine which of these two things has led to the different demographic composition of the sample, you will need to clearly review the sampling method, and seek outside data on the demographic composition of the city over time (e.g., from census data).

You say that you want to detect a change in preference "in the city", which is fine. But remember that "the city" is not a fixed set of people --- it is a composition of people that changes over time. If you want to know whether the aggregate preference of that changing composition of people is changing then it is a bad idea to "control" for age and sex. That would be tantamount to filtering out effects due to the change in composition in the city, which is a real aspect of the new composition of the city. Alternatively, if you decide you want to filter out these effects via statistical methods, then you need to adjust your research question to reflect the fact that you are filtering out that part of the change in preference that is due to changing demographic composition of the city. Either of these research questions might potentially be interesting, depending on what you want to know.

Most importantly, it is wrong to assume that changes in the age/sex composition in your samples is purely an artefact of your sampling (as opposed to reflecting underlying changes in the demographic composition of the city). Consequently, it is wrong to conclude that controlling for age/sex is equivalent to controlling for sampling error in the composition.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I completely agree, it would be wrong to make the assumption of equivalence of the two populations, if we didn't have independent census data - fortunately, we do (I guess the footnote at the end of my question didn't spell it out clearly enough). So given there's data showing reasonably stable composition over time, we can be relatively confident that controlling for age/sex is equivalent to controlling for sampling error. Sorry for the confusion! $\endgroup$
    – user3554004
    Mar 29, 2019 at 12:54
  • $\begingroup$ If you have census data for both times then you should be able to weight your sample values so that they match the known (or almost known) demographic composition of the city at each time. (This technique is called post hoc weighting.) That will adjust your sample weights for the demographics to be equal to the population weights, but you will still have sampling error. $\endgroup$
    – Ben
    Mar 30, 2019 at 0:50
  • $\begingroup$ I hadn't thought of that. I looked up, but it would seem weighting has its own problems of possible bias and researchers degrees of freedom. I may need to read up on that more; right now I can't quite see the benefit of weighting over explicit controlling (that is, assuming relative demographic equivalence). $\endgroup$
    – user3554004
    Mar 30, 2019 at 15:36
  • $\begingroup$ There is no benefit to the assumption if it is false. $\endgroup$
    – Ben
    Mar 30, 2019 at 23:25
  • $\begingroup$ I don't understand the last comment; the assumption (of little to no change in demographic composition) is supported by data, and while possible, unlikely to be false. Or perhaps I misunderstood something. $\endgroup$
    – user3554004
    Mar 30, 2019 at 23:30
0
$\begingroup$

This does not appear to be the best way to test your hypothesis. In fact, there is a very simple way to test your hypothesis: You should just run model b, where time is a dummy variable indicating whether you are in the second time period. Then you can test whether the coefficient on time is significantly different from zero, just using a simple t test. Of course, the controls (age, sex, etc) you include matter, they should capture all the relevant differences in sampling.

More importantly, note that this procedure will test a slightly different hypothesis: Whether, conditional on your controls the average preference has changed.

For example, let's say for comparable persons, attitudes towards Q have not changed at all. Thus, running model B will result in an estimate of approximately zero. Probably this is the effect you are interested in. However, say old people like Q much more than young people. If average age goes up a lot, average attitudes will thus improve. If you want to include those indirect effect in your estimates (as your questions suggests), you can insert the average values at t1 (for age and sex) into model B (with estimated coefficients) and thus calculate the conditional expectation at time t1 (insert zero for time). Note that these should not be the averages in your sample but rather in the population (so look them up somewhere else). Compare that to inserting the averages for time t2 and inserting time=1. This is the best estimate of how the average attitude has changed over time.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for considering this! I simulated some data and thought more about this. It seems the suggested solution is in principle identical to mine though: in the end it's a test about the significance of the (addition of the) time variable (either via nested model comparison, or within a model as suggested). Therefore I'm not sure why you'd say that this does not appear to be the best way to test the hypothesis. What led you to see it that way if I may ask? $\endgroup$
    – user3554004
    Mar 27, 2019 at 17:18
  • 1
    $\begingroup$ Yes your core idea is ok, you want to consider whether time has an effect. However, you suggest complicated methodology to compare two models, while a t test does exactly what you want, is much simpler and probably has higher power. That is why it would be strange to see you run both models and compare them in some complicated way (although, depending on how exactly you implement the comparison, you might obtain a valid test with similar properties to the t test). $\endgroup$
    – user1587692
    Mar 28, 2019 at 22:36
  • $\begingroup$ Well, no. A (two-sample) t-test (or what it commonly refers to) compares means of two samples. A t-test and a linear regression model are basically the same thing; but multiple regression allows for additional varibles/controls to be specified. From what I understand, it makes little difference if the p-value is looked up within a single model (observe p-value of time in model_b) or via standard F-test model comparison (e.g. in R, anova(model_a, model_b)). No difference in complexity. $\endgroup$
    – user3554004
    Mar 29, 2019 at 13:28
  • $\begingroup$ Well, no? :D ok boss. But a test and a model are different things and I'm not sure why you want to look up p values either. If you have a single restriction a t test and F test are equivalent. But all I was saying is that it is uncommon to present two models when you're just testing the significance of one covariate. You could have described that testing problem much more concisely, which indicates you hadn't understood it fully. But I guess now you do. You also suggested to "run cross-validation, compare prediction error, etc.". I pointed out there is an easier solution. $\endgroup$
    – user1587692
    Mar 29, 2019 at 16:44
  • $\begingroup$ I wouldn't say model comparison is uncommon (particularly when using mixed effects models). Your suggestion did make me think about the problem more deeply - my first instinct was to go the model comparison route, but you pointed out the same can be achieved another way. What I'm still not sure of is whether this way to do comparison between time points is a good idea in general (regardless of the operationalization of the model, i.e. single vs nested multiple). $\endgroup$
    – user3554004
    Mar 30, 2019 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.