Question about attributing change from a pre- to post- survey to the larger population if the respondents are different from time 1 to time 2?

Question

I administered a front office staff customer service satisfaction survey anonymously to consumers at different mental health clinics. Then the front office staff at these clinics received a course aimed at improving their front office customer service delivery. We plan to administer the same survey again (post- survey) to consumers anonymously at the same clinics to then be able to evaluate change in counts in several different variables (answers to likert scale questions) measuring aspects of consumer front office staff satisfaction following the front office staff customer service training.

EDITED TO REPHRASE THE QUESTION WITH THE FOLLOWING ADDITIONAL INFORMATION: The pre- and post- samples are similar in size; each is about 1/7 of the same total population of consumers who are actively receiving services at these clinics.

The main focus of my original question is the following: Given the above information, is there a difference between the following scenarios in how we can determine whether the training had an effect on change from pre- to post- and in attributing change to the larger population?

Scenario 1: The pre- and the post- samples consist of the same individuals, but they are not matched.

Scenario 2: The pre- and post- samples are completely different individuals, but they come from the same population.

Scenario 3: Let's assume there is 80% overlap of individuals in pre- and post- samples.

In summary, in my experiment, the pre- sample size is 4700, and the post- sample size is likely to be 4600-4800. The size of the whole population of consumers actively receiving services at these clinics is 35,000.

Question: How can we tell whether the training resulted in a significant change in satisfaction ratings in each of the three scenarios? Moreover, does having the same individuals in both samples give us an advantage in determining whether there was a significant change?

Thank you in advance.

Answer 1

It is not out of the question that you will be able to validly and reliably conduct the sort of test you describe. But the answers for the other thread you cite are statistical in nature; what you need is to bring to bear information from your research situation. To answer your question you need to use whatever you know or can learn or can surmise about comparability -- or lack thereof -- between the pre- and post-survey groups. E.g.,

• Will the procedures used to recruit respondents be exactly the same for both periods?
• Are processes occurring that will be expected to result in different characteristics of the post-survey group relative the pre-?
• Will the post-survey group consist disproportionately of people who have undergone treatment at the facilities?

To some extent you may be able to collect measurements on variables that account for group differences that are relevant to satisfaction. If these differences are relatively minor, you may find it workable to control for them in a procedure such as an ANOVA or ANCOVA. But most would advise you not to rely on such control if the differences are large. As Elazar Pedhazur has said, it would be bizarre to estimate how high a tomato plant would grow if it were a corn plant.

Instead, you might try this sort of strategy. If satisfaction is highly dependent on the number of times a person has visited a facility, conduct pre-post comparisons separately for those with, say, a low, medium, and high number of visits. This more basic type of control will make for a conceptually cleaner design, and one that should be easily interpretable.