0
$\begingroup$

I am looking to evaluate if the change in proportion for a group changed pre-to-post intervention.

enter image description here

Essentially, is the ~7% increase from pre (~2%) to post (~9%) significant. I am running the McNemar test in SAS 9.3 as follow:

ODS HTML CLOSE; ODS HTML;
PROC FREQ DATA = AGE0_10_SUM ORDER = DATA; 
WEIGHT MEMCOUNT;
TABLES AGE0_10*SRC / AGREE; 
TITLE "AGE0_10"; 
RUN;
;QUIT;

This is resulting in a p-value less than 0.0001. I think the change is significant but not at this magnitude, also, when running other age groups the results for a 0.02% increase was also <0.0001. Something is wrong.

How should the table be set-up to run the McNemar test with AGE0_10 as being the repeated subject pre and post.

Thank you.

$\endgroup$
0
$\begingroup$

You have very large denominators, so you should not be surprised if quite small differences give low p-values.

One should be careful to not confuse p$\leq$0.05 with practical relevance or the magnitude of an effect. Differences that are completely irrelevant for any practical purposes can be "statistically significant" with enough data.

The language you use of "repeated subject" does not seem to make a lot of sense in connection to a variable that seems to have two levels and does not sound like it would indicate a particular individual person/experimental unit.

For McNemar's test, your data should be structured like this with each row giving the number of units (Number) with result of assessment 1 = Result1 and the result of assessment 2 = Result2):

Result1    Result2    Number
0          0          900
0          1          100
1          0          100
1          1          900

Other note: pre-/post-tests are typically irrelevant for (almost) any question a researcher truly wants to answer (particularly, if they are to see whether some intervention/change/something had an effect).

$\endgroup$
  • $\begingroup$ Thank you. You are right. The variable isn’t repeated. There is no way the data can be structured in the table you describe. What is a more appropriate way to understand if the change in distribution of that group is different in the post period? Should I take the difference and test if the difference is equal to 0? $\endgroup$ – Ronald Sanchez Jun 14 at 11:53
  • $\begingroup$ If you do not have the necessary information, then you cannot use McNemar's test, in fact you might have a bit of a problem, because if you assess the same subjects at two time points, you will have a correlation between the assessments, which you have trouble taking into account, if you do not know how the observations line up within subject. There might be some solution for that, but it's not immediately obvious to me. On the other hand, if these are independent records from totally separate subjects, this would be easier. $\endgroup$ – Björn Jun 14 at 11:53
  • $\begingroup$ Hmmm, ok. This is the “intervention”. We placed a social worker inside a doctors office. We are evaluating the change in volume of mental health and behavioral health services received at the clinic. First, we need to understand if the demographics (e.g. age groups at different bands) changed before and after the social worker was hired. So there will be patients In both time periods and then new patients in the post period. Thoughts in assessing the change in the post period for age groups? $\endgroup$ – Ronald Sanchez Jun 14 at 12:30
  • $\begingroup$ Another thought, could I compare the percent change in the total population (171% increase) to the target group (92%) to assess if the subgroup increased proportionally with the general population, the H-null would be that 171% is not different than 92%. Thoughts? $\endgroup$ – Ronald Sanchez Jun 14 at 13:24
  • $\begingroup$ a) There is uncertainty around 171% and 92%, and the two are presumably not independent (overlap in populations). And you do not really know the uncertainty around a change, because you do not know how much is existing patients changing vs. different people being in the population (at least it seems so from your comment that you cannot get the data into the format suggested). $\endgroup$ – Björn Jun 14 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.