I've processed my data in two ways to determine significant difference, one method suggests a very high significant differences (p = 0.0000...), while the other returns non-significant results (p = 0.52). Immediately the second option seems more likely, as having such a small p number is suspicious to me, but if someone could tell me which of the following methods is correct (or if neither are correct please let me know) it would be really appreciated.

Data has been collected from four subjects for two tasks. First task involved monitoring heart rate for 1 min before an exercise task, and another 1 min is monitoring of the heart rate following the task.

Method 1) Take the one minute data set per subject for the 'before' task, and average them, so I have one data set representing the one minute. Do the same for the 'after' task. Data is sampled at 100Hz, so there are 6000 data points per 'before' and 'after' task. Use this data for the T-test and get p = 0.0000 (very small).

Method 2) Take the average of the one minute data set per subject so I have a single value, do the same for all subjects so I have four values each representing the average heart rate of that subject for the 'before' task. Do the same for the 'after' task so I have the average heart rate per subject following the exercise. Take these much smaller data sets (4 per set) and get p = 0.52.

Are either correct, or both wrong? Any suggestions would be great.

  • 1
    $\begingroup$ Method would depend on what exactly is your research question. Do you want to determine if the heart rate, averaged over 1 minute, increase after exercise task? Or do you want to determine the change in beat-to-beat variation after exercise task? You must have a clear question before choosing the data organization and statistical method. $\endgroup$ – rnso Apr 29 '15 at 3:59

Your first method gives a p-value that is (probably) too low, as the points are not independent.

Your second method gives a p-value that is (probably) too high, as you have discarded information about the sample.

I would use the first method, but include the fact that the measures are not independent, using a multilevel model or a sandwich estimator. What software do you use?

  • $\begingroup$ I have access to SPSS, MATLAB and Excel. Predominantly using MATLAB. $\endgroup$ – anon Apr 28 '15 at 22:56
  • $\begingroup$ I don't know about matlab, but you can do this with a mixed model in SPSS (or you could use complex samples). You can't do this with excel. $\endgroup$ – Jeremy Miles Apr 29 '15 at 2:18

Method 1 seems wrong to me as the data you are comparing are not independent samples, i.e., they come from a time-series obtained from a persons' heart measures and by taking 6000 data points you inflate N sample size. Method 2 seems correct even though it is probably not correct to use a t-test here as the dataset is very small.

  • $\begingroup$ Thank you. What method would you suggest to determine significance with such a small dataset? $\endgroup$ – anon Apr 28 '15 at 20:05
  • $\begingroup$ If it were possible to collect more data that would be best. If you insist I would say use a non-parametric test, e.g., Wilcox test. $\endgroup$ – Ruthger Righart Apr 28 '15 at 20:12
  • $\begingroup$ Why non-oarametric? $\endgroup$ – Jeremy Miles Apr 28 '15 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.