4
$\begingroup$

I have an existing data set that comes from the same group of people before and after they received a treatment.

The data set comes from when participants tested their blood sugar values over a 30 day period before receiving an insulin pump and a 30 day period after receiving an insulin pump. This data was obtained from user logs (archival data) and was not controlled to ensure that they were tested at fixed intervals. Participants tested themselves when they needed to test throughout the day for a period of 30 days.

My goal is to determine whether the average blood sugar across 30 days was different for the before vs after group.

Normally this would be a paired samples t-test but unfortunately the before and after group have an unequal number of data points with the after group having significantly more. People are testing more often after receiving an insulin pump.

What is the correct way to handle this?

I can collapse the data to find a mean for each participant before treatment and after treatment (across all participants) so that the data matches up and then run a paired samples t-test on this data but I think this is not the ideal solution.

Would a one way within subjects ANOVA be the appropriate test to run in this case?

Improve this question
$\endgroup$
5
  • $\begingroup$ Clearly if you have unequal numbers, not all points can be paired. Do you have data where there are some in each group (before/after) that are unpaired with an observation in the other? The situation needs to be made more explicit. Can you for certain identify both measurements for those individuals that have two? (I find the "find a mean for each participant" part worrisome because it seems to imply that the answer there might be "no") $\endgroup$
    – Glen_b
    Commented Dec 9, 2014 at 23:36
  • 1
    $\begingroup$ Yes, I have data in the after group that does not have a corresponding data point in the before group. Would it be appropriate to crop/throw out extra data points? The data I am working with involves blood sugar levels. Some people tested their blood sugar levels much more often after receiving the experimental treatment in this case. I am also unsure what you mean by "identify both measurements for individuals that have two". For example one user might have 20 blood sugar tests before treatment but 40 blood sugar tests after treatment. $\endgroup$
    – Arctic
    Commented Dec 10, 2014 at 0:19
  • 3
    $\begingroup$ If you have multiple observations (more than one before and one after) for at least some participants, that's not paired - it sounds like repeated measures. Are these fixed occasions/intervals before and after or some random set of occasions (whenever people happened to get measured)? Please edit your question to describe the situation explicitly. $\endgroup$
    – Glen_b
    Commented Dec 10, 2014 at 1:18
  • 1
    $\begingroup$ I have updated the original question. The data set comes from when participants tested their blood sugar values over a 30 day period before receiving an insulin pump and a 30 day period after receiving an insulin pump. This data was obtained from user logs and was not controlled to ensure that they were tested at fixed intervals. Participants tested themselves when they needed to test throughout the day. One participant can have hundreds of data points across 30 days. $\endgroup$
    – Arctic
    Commented Dec 10, 2014 at 2:45
  • $\begingroup$ To add: the data set comes from an archival data source. $\endgroup$
    – Arctic
    Commented Dec 10, 2014 at 2:50

4 Answers 4

Reset to default
3
$\begingroup$

For the purposes of hypothesis testing, I often find that the simpler approach is the best.

In this case, I would do exactly what you considered yourself: average all the pre-treatment values and all the post-treatment values for each participant, obtaining two values per participant. Then you can run a paired t-test on the resulting averages.

There is nothing wrong with this approach. If you do that and you get $p$-value sufficiently low for your purposes, you can call it a day. I guess there are much more complicated mixed models that one can set up here, but I would be skeptical that they produce much lower $p$-values (and if not, then there is no gain). Whereas two big advantages of the simple t-test on averages are: (1) it takes five minutes to perform; (2) it takes two lines to explain in a paper.

PS. If I am not mistaken, then a simple repeated measures ANOVA (that you asked about) cannot be applied in your case.

PPS. Note that without a control group you will not be able to say if the difference between post and pre (in case you observe any) is due to treatment or due to some time passing.

Update. What I wrote above was under assumption that either you have no information about times of individual measurements, or you are happy to assume that the time is irrelevant. @psarka argued (+1) that time of the day is very relevant and, worse, it is unlikely that measurement pre- and post-treatment were distributed along the day in the same way. So if you have the information about measurement times, then you should better take it into account, and the exercise becomes more complicated then. If not, then well, not.

In addition, @robin argued that the day number is important as well, see discussion in the comments.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I don't think I'm entirely comfortable with the phrasing that suggests the point of the exercise is to produce a low p-value... Though of course if it talked about "sufficient power" that sounds much more respectable while meaning almost exactly the same thing! $\endgroup$
    – Silverfish
    Commented Dec 11, 2014 at 12:36
  • 1
    $\begingroup$ I like the PPS. That's important. A related word of warning: If an intervention is undertaken partly because "before" scores are high, then "after" scores in the intervention group may be lower due to "regression to the mean", even if there isn't a downwards trend over time or a genuine intervention effect. One of the perils of before vs after testing. $\endgroup$
    – Silverfish
    Commented Dec 11, 2014 at 12:40
  • $\begingroup$ As I put in my comment below, I think this approach assumes consistency across 30 days post treatment. From my limited knowledge of this field, that seems unreasonable to me. If I read that result, I would want to see something indicating both theoretically and statistically that all measures over 30 days are consistent measures of the general sugar level. Again, it may be commonly understood in health practice and I'm ignorant. But in my field I would approach it with a more appropriate method for repeated measures. $\endgroup$
    – robin.datadrivers
    Commented Dec 11, 2014 at 14:30
  • $\begingroup$ @robin: I would say that this approach assumes either consistency OR random sampling of measurements across 30 days for each participant. There might be some temporal evolution over 30 days, but if all participants measured their values across the whole time randomly, then sample average is a good proxy for average over these 30 days. As I wrote below, the OP stated in bold font that the goal is "to determine whether the average blood sugar across 30 days was different", so I assumed that "the average blood sugar" is a meaningful construct. $\endgroup$
    – amoeba
    Commented Dec 11, 2014 at 14:45
  • $\begingroup$ @Silverfish: agree with everything you said, good points (+1). Re p-values: well, one can state the same thing differently, saying that the point is to estimate the difference between post and pre with a confidence interval around it (it's commonplace to argue that this is more useful than just reporting a p-value), and of course one would want to have as narrow confidence interval as possible. That's why e.g. paired test is better than non-paired when the data are paired (more power <=> narrow confidence intervals). $\endgroup$
    – amoeba
    Commented Dec 11, 2014 at 20:16
1
$\begingroup$

I'm skeptical about possibility to say anything meaningful in this situation without taking daytime into account. The reason is the process that generates your data.

Blood sugar levels have a daily pattern (more erratic or less erratic, depending on the patient), partly related to carbohydrate consumption. In theory, after eating your sugar levels rise and then fall as insulin does its job.

Blood sugar measurements also have a daily pattern: patients usually measure their blood before eating, as they need to know if they have to adjust the amount of insulin. This should be especially true, if patient is used to his/her therapy (that is, before switching to pump).

If we agree that the true average blood sugar level is the area under curve which we could get if we measured blood glucose continuously, then

  • it seems likely that average of measurements before switching to pump generate biased estimate of true average
  • and unlikely that the average of more frequent measurements generate the estimate which is biased in the same way.

Even if these statements are not true in your case, you'll have to convince everyone (at least me) that they indeed aren't.

Now if you are willing to take daytime into account, then you can infer so called modal day before switch and after switch, or compare corresponding measurements, say before breakfast.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Sounds very reasonable, and seems to address well the biology of the question. But what is "modal day", and whatever it is -- how to "infer" it, and how to make statistical test between them? Also, say OP decides to follow your suggestion to compare corresponding measurements (e.g. "before breakfast"). How to do it? The same question about paired t-test with "repeated" measurements arises. $\endgroup$
    – amoeba
    Commented Dec 11, 2014 at 17:08
  • $\begingroup$ I'm not at all a fan of "modal day" and I'm not sure if you can do anything with it rigorously. Maybe I should not have mentioned it at all, but this notion pops up a lot when you look at what people try to do with discrete measurements. You can try googling it for numerous examples. The before breakfast part depends on what data OP has. Unless it is something non-standard, it should have labels, due to special status of pre-meal measurements. Then you have 30 measurements before switch and 30 after, all measuring the same thing. Easy. $\endgroup$
    – psarka
    Commented Dec 11, 2014 at 17:38
  • 1
    $\begingroup$ Oh, did not answer your question: modal day is some sort of smoothing of the overlaid time series (plot all days in the same 24 hour graph and do something with it). $\endgroup$
    – psarka
    Commented Dec 11, 2014 at 17:45
  • $\begingroup$ It does not sound so easy even if each subject has exactly 30 measurements pre- and exactly 30 post-treatment, and all measurements performed at the same time before breakfast. What test would you run then? +1, btw. $\endgroup$
    – amoeba
    Commented Dec 11, 2014 at 20:12
  • $\begingroup$ hmm, you are right. I assumed that this becomes a standard situation with a known test, but it seemingly doesn't. On the other hand, your answer seems to cover it pretty well, so +1 in return :) $\endgroup$
    – psarka
    Commented Dec 12, 2014 at 8:15
0
$\begingroup$

If I'm understanding you correctly, you should be running a repeated-measures model, sort of like an interrupted time series model except you only have 1 (in some cases, none) observations before the treatment. Longitudinal regression models can handle imbalance, if you treat it as a repeated-measure hierarchical or multilevel model. In this case, the level 2 model is the person, and the level 1 model is for the observation within the person.

Something like this:

Level 1: $y_{it}=\hat{\beta_{0i}}+\hat{\beta_1}treat+\hat{\beta_2}time_t + \epsilon_{it}$

Level 2: $\hat{\beta_{0i}}=\alpha+\Sigma\hat{\gamma_i}X_i + \mu_i$

Where indices $i$ and $t$ represent person and time, respectively. $\epsilon_{it}$ is the level 1 error term, for the observations within persons, and $\mu_i$ is the error term for the persons. The variable $treat$ is a binary variable, 1 if the person was treated at time $t$, 0 otherwise. I have here time as a linear term, but you could add polynomials as well. This specification should tell you if on average, independent of when the measurement was taken, if it was taken after the treatment was it higher than before. You can then also see if there is a time effect independent of treatment.

$\gamma_i$ is the coefficient for time-invariant person characteristics $X$, assuming you have that. You could also include time-varying person characteristics, if you tracked them. I've also only assumed you are interested in a random intercept - which means the intercept for each person is a normally distributed, random variable, centered around the mean.

This is a basic model - you can make it far more complicated. But it can get your started.

Regarding individuals with no pre-treatment observation, that can still be included in your model - they will not contribute any pre-treatment information, but they will provide additional comparison for those with pre-treatment observations. If you have additional covariates this would make it much stronger.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Can you comment on whether this model is likely to perform noticeably better than a simple paired t-test on the pre and post per-subject averages? Or maybe it will perform better in some particular circumstances? $\endgroup$
    – amoeba
    Commented Dec 11, 2014 at 9:35
  • $\begingroup$ You have more than two observations, so a paired t-test doesn't apply. I don't get why you would average multiple measures, unless you suspected that they are indicators of an underlying measure of overall sugar level. I'm no health expert, so that may be legit, but it seems to me individuals' blood sugar levels fluctuate throughout the day (and across days maybe?). You could run some reliability stats, like Cronbach's $\alpha$ to see if you have something consistent. The approach I laid out will account for that variability and still get at your average difference. $\endgroup$
    – robin.datadrivers
    Commented Dec 11, 2014 at 14:28
  • $\begingroup$ Thanks, @robin, good point about controlling for time of the day, but from how I read the OP these data might not be available at all ("Participants tested themselves when they needed to test throughout the day" -- does that mean they recorded the time of the day?). The OP stated in bold font that the goal is "to determine whether the average blood sugar across 30 days was different", so I assumed that "the average blood sugar" is a meaningful construct. $\endgroup$
    – amoeba
    Commented Dec 11, 2014 at 14:43
  • $\begingroup$ Ah - I interpreted it to mean average as in the average across persons, not within person. @Arctic can you clarify? $\endgroup$
    – robin.datadrivers
    Commented Dec 11, 2014 at 14:59
-1
$\begingroup$

Your stated goal is to compare the means from the two samples, each of which has different sample size (i.e. n1 does not equal n2). The classic way to handle this problem is to use Welch's t test ( http://en.wikipedia.org/wiki/Welch's_t_test).

This test is exactly appropriate in your case because it relaxes the requirement of the paired t-test, namely that n1 = n2.

This approach will of course require you to properly construct your hypothesis test, but that shouldn't be a problem since you were already thinking of applying a paired t-test. In short, the rest of the hypothesis test should proceed exactly as expected with the only difference being the application of Welch's test.

Good luck.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Welch's test is for two samples, but I am sampling from the same group of people before and after. $\endgroup$
    – Arctic
    Commented Dec 10, 2014 at 4:24
  • 1
    $\begingroup$ (-1) This does not address the original question, as @Arctic has already commented. $\endgroup$
    – amoeba
    Commented Dec 10, 2014 at 22:21
  • $\begingroup$ The experiment that you are describing actually has two populations: the before-insulin-pump population and the after-insulin-pump population. They are the same people, but from a statistics perspective they are different populations. I'll edit my response to make the condition more explicitly applicable. $\endgroup$
    – user2743
    Commented Dec 11, 2014 at 1:48
  • 1
    $\begingroup$ This approach isn't strictly speaking invalid - it's what you'd have to do if, eg, you'd lost your records of which measurement corresponded to which patient! $\endgroup$
    – Silverfish
    Commented Dec 11, 2014 at 12:42
  • 1
    $\begingroup$ The reason I think it is a poor choice is that I'd expect substantial variation from person to person, effectively in terms of what their "baseline" is. A paired t test is more powerful than an unpaired one because it takes this between-subjects variation into account (of course the problem for the OP is that the data aren't naturally paired). This approach effectively discards data about patient identity, which suggests it is going to lose a lot of power. $\endgroup$
    – Silverfish
    Commented Dec 11, 2014 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.