3
$\begingroup$

I administered a front office staff customer service satisfaction survey anonymously to consumers at different mental health clinics. Then the front office staff at these clinics received a course aimed at improving their front office customer service delivery. We plan to administer the same survey again (post- survey) to consumers anonymously at the same clinics to then be able to evaluate change in counts in several different variables (answers to likert scale questions) measuring aspects of consumer front office staff satisfaction following the front office staff customer service training.

EDITED TO REPHRASE THE QUESTION WITH THE FOLLOWING ADDITIONAL INFORMATION: The pre- and post- samples are similar in size; each is about 1/7 of the same total population of consumers who are actively receiving services at these clinics.

The main focus of my original question is the following: Given the above information, is there a difference between the following scenarios in how we can determine whether the training had an effect on change from pre- to post- and in attributing change to the larger population?

Scenario 1: The pre- and the post- samples consist of the same individuals, but they are not matched.

Scenario 2: The pre- and post- samples are completely different individuals, but they come from the same population.

Scenario 3: Let's assume there is 80% overlap of individuals in pre- and post- samples.

In summary, in my experiment, the pre- sample size is 4700, and the post- sample size is likely to be 4600-4800. The size of the whole population of consumers actively receiving services at these clinics is 35,000.

Question: How can we tell whether the training resulted in a significant change in satisfaction ratings in each of the three scenarios? Moreover, does having the same individuals in both samples give us an advantage in determining whether there was a significant change?

Thank you in advance.

$\endgroup$
1
$\begingroup$

It is not out of the question that you will be able to validly and reliably conduct the sort of test you describe. But the answers for the other thread you cite are statistical in nature; what you need is to bring to bear information from your research situation. To answer your question you need to use whatever you know or can learn or can surmise about comparability -- or lack thereof -- between the pre- and post-survey groups. E.g.,

  • Will the procedures used to recruit respondents be exactly the same for both periods?
  • Are processes occurring that will be expected to result in different characteristics of the post-survey group relative the pre-?
  • Will the post-survey group consist disproportionately of people who have undergone treatment at the facilities?

To some extent you may be able to collect measurements on variables that account for group differences that are relevant to satisfaction. If these differences are relatively minor, you may find it workable to control for them in a procedure such as an ANOVA or ANCOVA. But most would advise you not to rely on such control if the differences are large. As Elazar Pedhazur has said, it would be bizarre to estimate how high a tomato plant would grow if it were a corn plant.

Instead, you might try this sort of strategy. If satisfaction is highly dependent on the number of times a person has visited a facility, conduct pre-post comparisons separately for those with, say, a low, medium, and high number of visits. This more basic type of control will make for a conceptually cleaner design, and one that should be easily interpretable.

$\endgroup$
7
  • $\begingroup$ Roland, thank you for highlighting factors/considerations that may confound attributing change in satisfaction to the training intervention, e.g., number of prior visits of a client and how busy the clinics are. Let's assume that recruitment procedures are the same pre- and post, and that we account for factors other than the training that change systematically from pre- to post-.To what extent will the fact that pre and post groups have different individuals interfere with attributing change to the intended intervention? $\endgroup$
    – LeeZee
    May 22 '18 at 20:15
  • $\begingroup$ In an idealized situation in which you can account for all factors that systematically affect the pre-post difference, then all that is left is the unsystematic: sampling error, i.e, chance. A test like ANOVA or ANCOVA will address this type of error and show you how likely your results would be if chance were the only force at work. $\endgroup$
    – rolando2
    May 22 '18 at 22:33
  • $\begingroup$ Please see above changes to the question $\endgroup$
    – LeeZee
    May 30 '18 at 6:05
  • $\begingroup$ Yes, agreed. Controlling for as many factors as possible that systematically affect the pre-post difference (e.g., busy-ness of the clinic; number of times visiting that particular clinic; front office staff turnover between pre- and post- collection times) is ideal. But let's assume we co-vary for those variables (ANCOVA) or control for them in other ways. To what extent does using different individuals in the pre- survey from the post- survey (with unknown degree of overlap) significantly interfere with attributing change to the intended intervention? $\endgroup$
    – LeeZee
    Jul 13 '18 at 17:45
  • $\begingroup$ I'm afraid that's like asking to what extent your audience for the research will accept the validity of the results. The answer will flow from their assessment of the totality of your results and methodology. And perhaps more: your reputation...their prior experience with similar analysis...their mood that day... :-) $\endgroup$
    – rolando2
    Jul 13 '18 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.