1
$\begingroup$

I would appreciate folks' thoughts on my proposed analysis.

  • There are two waves of survey data, the first administered in 2016 and then 2017. In the survey, there are 20 questions.

  • For both waves, respondents were randomly selected via probability sampling from two different public higher education institutions to take the survey.

  • Of the two schools surveyed, one school received an education-related intervention while the other school did not (Control group).

  • Respondents who took the survey in Wave 1 could have also taken the survey in Wave 2, and these respondents were not excluded.

  • The purpose of Wave 1 was to construct a baseline portrait of attitudes about education-related topics and the follow up wave was to be able to measure changes, if any, to possibly attribute to the intervention.

  • For instance, in Wave 1, across both schools, 50 of 454 respondents or 11% chose ‘Strongly Agree’ to a question, whereas in Wave 2, 31 of 388 or 8% responded ‘Strongly Agree’ to the same question.

  • How should I go about testing whether these proportions are significantly different from each other?

  • Would I use a t-test to compare the percentages or a chi-square test for the proportions?

Thank you in advance.

$\endgroup$
1
  • $\begingroup$ Welcome! What is the range of available responses? I assume it is a Likert-type scale. Do you only care about those answering at either anchor of the scale (i.e., "Strongly Agree/Disagree")? $\endgroup$ Commented Apr 22, 2020 at 15:18

1 Answer 1

0
$\begingroup$

As this study was at different points in time in a changing world, the assumption that even re-sampling the same people would produce statistically consistent answers is, perhaps, questionable.

However, as you do have some repeat respondents in the survey data, and I would suggest an examination of the consistency of repeat individuals scoring as a metric to evaluate reputed differences between years.

Some related work, for example, 'Repeated Measures, Interventions, and Time Series Analysis, to quote from the abstract:

Classical repeated measures designs assume treatments are given in a randomized order. When randomization is not performed and an experiment involves a sequence of observations on each subject collected over time, serial correlations may become important. An example of these types of data is an intervention experiment wherein subjects are observed before and after a treatment or other manipulation. This situation falls within the realm of time series analysis. The correlations between observations often depend on the time intervals between the observations; observations that are closely spaced in time usually are more highly correlated than those with a larger time separation. This report demonstrates a test for such serial correlation and discusses a method of adjusting for it in repeated measures experiments.

So, as you mentioned that a school was subject to an intervention, and as the above cites possible serial correlation issue occurring with time series data, a more advanced modeling perspective may be appropriate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.