1
$\begingroup$

I am trying to test whether the difference between the two proportion I have found is significant or not. I am just unsure as to whether my samples are independent or dependent and as a result what test to use.

I have a sample of patients from one GP practice. Some have been sent recalls by text message and some have been sent recalls by letter. I have found the % of people who followed through with their recall for each group (those who were sent a text message and those who were sent a letter). It is possible that the same person has been sent a letter and a text message as the data is over a 2.5 year time period.

What test should I use to see if the % increase or decrease is significantly different? z-test (independent sample)? chi-squared test?

Improve this question
$\endgroup$
5
  • $\begingroup$ "It is possible that the same person has been sent a letter and a text message as the data is over a 2.5 year time period." -- are you able to identify these users? $\endgroup$
    – Jon
    Aug 22, 2016 at 21:49
  • $\begingroup$ not really, the process would be extremely long and complicated. Also due to confidentiality I am not sure I am able to $\endgroup$
    – Poppy
    Aug 22, 2016 at 21:54
  • $\begingroup$ Hmm....that's going to be tricky then. I mean, you can run your test as if no one received both alerts. Then you should at least, if possible, estimate the number of people who received both alerts and if this sample size is small, you can throw it as a caveat for statistical significance. One other strategy you can try is to only use people who you can be 95% certain received only 1 alert; but I'm sure that wouldn't be easily feasible either. $\endgroup$
    – Jon
    Aug 22, 2016 at 22:01
  • $\begingroup$ Do you know how big is potentially the overlap? How big is your sample? If your sample is very big and the overlap potentially very small it could be that you can simply ignore the fact. However it really depends on what is your data. Is it possible for you to ask some of the participants what message did they get? $\endgroup$
    – Tim ♦
    Aug 22, 2016 at 22:02
  • $\begingroup$ my samples range from 150 - over 5,000 people. And I am not sure about the potential cross over... however my instinct says it wouldn't be small enough to make negligible $\endgroup$
    – Poppy
    Aug 22, 2016 at 22:03

1 Answer 1

Reset to default
0
$\begingroup$

Given the context of your data, you have two factors:

  1. Did the person receive a letter or a text?
  2. Did the personal follow through with the recall? (Outcome: 1/0)

It seems you want to compare whether one factor was more effective than the other as well. Given that context, I would suggest the z-test.

Here is a reference: https://web.archive.org/web/20160320132012/https://onlinecourses.science.psu.edu/stat414/node/268

The Z-statistic is fairly straightforward to compute, and you can test whether group 1 has a statistically significant higher proportion of success than group 2.

Also, if you can resolve the other issues with your data, I would. That would ensure the integrity of your statistical results.

Hope this helps.

Improve this answer
$\endgroup$
3
  • $\begingroup$ That is kinda correct, the only difference would be that they are not compared over the same time period. Those sent letters were from before the practice started using the text software. Individuals could have been receiving recalls from that practice for over 10 years or individuals who received a text could have had a letter before or after the text depending on the practices protocol around recalls. Then those who were sent texts are the most recent individuals. $\endgroup$
    – Poppy
    Aug 22, 2016 at 22:13
  • $\begingroup$ So those who received letters were from time period 1 where no texts were ever sent out? And those who received texts potentially received letters as well? If this is the case, then I would make the factor: time period 1 vs time period 2; this should maintain the integrity of the analysis. Your analysis would however be a comparison of time periods not outreach methods (letters vs texts). $\endgroup$
    – Jon
    Aug 22, 2016 at 22:18
  • $\begingroup$ Even if those who received letters in the in the second time period are not considered in my analysis. The reason they are not is because I am not looking at how the texts affected the overall follow through, due the the number or texts being sent out are so low compared to letters. I am only looking at text, with the idea of showing that those receiving texts follow up more and therefore if practices used it more their overall rates of follow up would increase... does that make sense? $\endgroup$
    – Poppy
    Aug 22, 2016 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.