1
$\begingroup$

I initially will apologise for any lack of clarity with regards to this question. I'm posing this on behalf of my sibling who is in the biomedical sciences.

They have done an intervention on a patient. They are measuring the effect of the intervention by looking at test scores (6 questions), and repeating this test over four weeks.

The data looks like this

Week 1: 4/6 Week 2: 5/6 Week 3: 5/6 Week 4: 6/6

My sibling wants to see if the increase in the test scores is significant or not. I didn't really know how to go about this.

Any help would be greatly appreciated

$\endgroup$

3 Answers 3

1
$\begingroup$

Firstly, this is a very small sample size (4 observations), hopefully you have more, or at least have the same four observations for many different patients. If not, it will be difficult to find a model. Generally it is good to have a sample size greater than 100, or at a bare minimum, 20.

Secondly, the survey (6 questions) is also small. Does the patient receive the same questions each time? If so, they can be more prepared next time. But, this doesn't answer your question, but tell your sibling to seriously consider the set-up of the survey.

That being said, if this is the only data you have, there are a couple quick hacks. First, just regress the survey data on the index (you can make it just 1:4 as long as the surveys were taken at regular intervals):

> summary(lm(c(4,5,5,6)~c(1,2,3,4)))

Call:
lm(formula = c(4, 5, 5, 6) ~ c(1, 2, 3, 4))

Residuals:
   1    2    3    4 
-0.1  0.3 -0.3  0.1 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)  
(Intercept)     3.5000     0.3873   9.037   0.0120 *
c(1, 2, 3, 4)   0.6000     0.1414   4.243   0.0513 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.3162 on 2 degrees of freedom
Multiple R-squared:   0.9,  Adjusted R-squared:  0.85 
F-statistic:    18 on 1 and 2 DF,  p-value: 0.05132 

Here we see that the slope has a p-value of .0513, making it significant with 94.87% confidence. On average, at each step, the patient improves by .6 points. It seems obvious that there is a significant increase over time based on the sample size. However, the next question would be why does the increase occur?

$\endgroup$
2
  • $\begingroup$ how is that p-value being calculated though? What are the assumptions? I am not sure what exactly summary(lm()) is doing. $\endgroup$
    – Flask
    Nov 20, 2013 at 16:12
  • $\begingroup$ Sorry, should have given more context. lm() is an R function that does a linear regression. The p-value is from a t-stat table, using the t-values calculated from the coefficient and standard deviation : t = [x̄ - μ ] / [ s /√ n] $\endgroup$
    – Stu
    Nov 20, 2013 at 22:40
1
$\begingroup$

Caveat emptor: I am NOT a biostatistician...but I play one on TV ;-) In all seriousness, I have a strong statistics background, but it is not medically oriented. However, your question was simply stated, so my suggestions utilize a general approach, not one that is domain specific to biomedical research.

First, you have a very small sample size. I hope your sibling isn't looking for anything too exact! Also, I am assuming the test your sibling is using has been properly validated and calibrated, and that your sibling has access to that data. The validation/calibration data will add MUCH to this analysis. Without it, you have limited power. I am also surprised that a biomedical researcher doesn't have ready access to a trained biostatistician. However, I digress....

What you have presented is a short time series. What your sibling would like to know is if there is evidence for a positive trend. Without any assumptions about how these scores are supposed to behave, I can propose a simple resampling approach that may help shed light on the strength of the evidence:

If there were no trend, then the above scores can be seen as 4 samples from the same population, as there would be no difference between week 1 vs week 2 etc.

There are two approaches to quantifying the trend that I see:

  1. There is a net change of 2 units in 4 weeks?
  2. The scores increased twice, and never decreased.

The difference between the two is that (1) cares only about the overall change, which would allow a situation where there is no change for 3 weeks then rapid improvement in the last week while (2) cares more about the consistency of the behavior vs the magnitude of the effect. I'll outline both, but given the small range of possible scores, (2) may be more appropriate.

Resampling Approach

The null hypothesis is the same in both cases: $H_0$: the samples were drawn from the same distribution vs $H_a$: The samples were drawn from distributions with increasing medians.

Since we are assuming for $H_0$ that all samples are from the same distribution, you should draw a sample of size 4, with replacement, from the set of scores {4,5,5,6} and assign each score to a week: $W_1,W_2,W_3,W_4$. Then, calculate the relevant measure of trend (i.e., 1 or 2 above) and record its value. Repeat this a large number of times (say 1000) and see how often you get a trend value at least as big as the one for your sample. This will indicate how likely it would have been to see this type of improvement even if there were no real improvement. Obviously, you will need a computer stats package, like R or Minitab, that allows resampling, to do this effectively.

Specifically:

For 1: The measure of trend, T, is $T=W_4 - W_1$. You will want to know how many of your resampling trials result in $T=2$.

For 2: The measure of trend is T= (# of inter-week increases) - (# of inter-week decreases). For example if one of the resamples give us {6,4,5,6} we get T = 2 - 1 = 1.

I doubt this method will give very strong results either way, given the small sample size. If your sibling has some prior information about the distribution of test scores for healthy individuals vs sick individuals, then you could do A LOT more, since you will know the typical range and score for each. For example, if healthy patients score 5 30% of the time and 6 70% of the time while a sick patient has score distribution {1 (10%), 2(40%), 3(30%), 4(15%), 5(5%), 6(0%)} Then the above scores carry much more weight.

Context matters, so without any, you have very limited power. With more context, the tests could be quite powerful..that is the value of properly validated and calibrated tests.

Numerical Results

Distribution of differences between Week 4 and Week 1 Distribution of differences between Week 4 and Week 1 Distribution of net inter-week direction changes Distribution of net inter-week direction changes

Surprisingly to me, the observed test statistics is are quite unusual if the patient were showing now improvment and the test scores were all noise.

$\endgroup$
8
  • $\begingroup$ Generally a good approach IF the sample size is large. It is hard to make any generalizations about the distribution of test scores. The small sample size seems to be the biggest hurdle here. $\endgroup$
    – Stu
    Nov 20, 2013 at 15:09
  • $\begingroup$ @Stu Agreed. I tried to generously caveat and qualify my proposed answer. Without contextual data (i.e., validation and calibration results) or more samples, its going to be tough to say anything very definitive. $\endgroup$
    – user31668
    Nov 20, 2013 at 15:14
  • 1
    $\begingroup$ @Eupraxis1981 It may be possible to make a relatively strong claim that the treatment did not decrease the scores. This information could be useful in helping to decide whether to continue the intervention or not. $\endgroup$
    – Flask
    Nov 20, 2013 at 15:16
  • $\begingroup$ @Flask: I'm not sure about that. This test will have low power, so it will be hard to detect any deviations from the null, either way. What we really need to know is the scores from a typical healthy and sick person to know anything more. How would you propose testing for a decrease in scores? $\endgroup$
    – user31668
    Nov 20, 2013 at 15:19
  • $\begingroup$ @Eupraxis1981 I agree without prior information there is little justification for assuming any distribution. I don't think there is any reasonable test, there is just the fact that scores never decreased. $\endgroup$
    – Flask
    Nov 20, 2013 at 15:32
0
$\begingroup$

How about this: Take the null model that at each timepoint there is 1/3 chance the score will be higher, lower, or the same as the previous timepoint. The probability you would get no scores worse than the previous week 3 weeks in a row is (2/3)X(2/3)X(2/3) = 8/27 = 0.296

Edit:

Here is one way to look at it using an approach similar to that proposed by @Eupraxis1981. Lets say the intervention does nothing, the test measures nothing but noise, and the first score on the test=4. This histogram shows distribution of slopes could we expect:

enter image description here

Under these conditions the slope would be greater than or equal to the observed slope of 0.6 about 12% of the time. I still think there should be a way to rule out large negative effects of the treatment though.

R code:

x<-c(1,2,3,4)
y<-c(4,5,5,6)
actual.slope<-round(lm(y~x)$coefficients[2],2)

probs<-rep(1/6,6)
n=100000
sim.slopes<-matrix(nrow=n,ncol=1)
for(i in 1:n){
  tmp<-c(4,sample(1:6,3, replace=T, prob=probs))

  sim.slopes[i,]<-round(lm(tmp~x)$coefficients[2],2)
}

perc<-round(100*length(sim.slopes[which(sim.slopes>=actual.slope)])/length(sim.slopes),2)

hist(sim.slopes,breaks=seq(-2,2,by=.1), 
     xlab="Slope", col="Grey",
     main=paste(perc,"% of slopes >=", actual.slope ))
hist(sim.slopes[which(sim.slopes>=actual.slope)],
     breaks=seq(-2,2,by=.1), add=T, col="Red")
$\endgroup$
3
  • $\begingroup$ I don't think a random walk model applies here. What happens if a previous score is 6 or 1? $\endgroup$
    – user31668
    Nov 20, 2013 at 16:57
  • $\begingroup$ @Eupraxis1981 Yes, I see what you mean. Check the updated answer. $\endgroup$
    – Flask
    Nov 20, 2013 at 18:25
  • $\begingroup$ Cool output. I don't think anyone would say that there is evidence that the the treatment is harmful, so you are OK there. I'll post the results for my test and we can compare, just for interest. $\endgroup$
    – user31668
    Nov 20, 2013 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.