The task: I need to calculate a one-sample t-test to see if the mean of a vector of values differs significantly from zero.

Data: The dataset consist of about 25 such vectors, each with a length of 19 (counts collected at different points in time). So there's the small sample problem, but additionally, some data from these vectors (up to 10-20%) are missing. However, since the vectors are essentially transformed trendlines, it seems it would be possible to linearly interpolate the missing values prior to the transformation.

Edit to answer comments about data: each of the 25 vectors (or rows of a table, or samples, if you will) are conceived the following way: counts of certain occurrences of X and Y were observed on 19 consequtive occasions, and normalized so that each value of a vector reflects the propertion of X (vs Y) at that time point when the observation was made (so they are in essence trends of X). The values in these trend vectors were then further transformed (the function itself is irrelevant here, and a bit too involved). But this yields those vectors; the null hypothesis is that their mean is 0, the alternative is that it is not 0.

Proportions of X over time:
x = 0.05 0.09 0.11 0.13 0.14 0.16 0.22 0.23 0.23 0.24 0.34 0.26 0.29 0.37 0.40 0.34 0.48 0.53 0.46
Transformed into:
xtr = 0.36  0.14  0.11  0.03  0.08  0.24  0.03 -0.01  0.04  0.27 -0.17  0.07  0.21  0.05 -0.12  0.31  0.09 -0.13 (element at first position gets removed in the process)

One sample t-test of xtr yields p-value of 0.01993; if vector x is interpolated to length 50, the p-value of the t-test on the transformed vector is 0.00005278.

Problem: I am not sure how to intepret the p-values after that though: the more interpolation I add (used the approx function in R), the more significant results I get - starting with 3 significant (alpha=0.05) out of 25 at no interpolation and only interpolating missing; but going up to 14-17 out of 25 with p<0.05 results, if adding more interpolated datapoints; but interestingly with not much increases when getting up to 180-200 interpolated points. The plot below (going in increments of 10 interpolated points) illustrates this.


How to make sense of that? On the one hand, the data collection was done in a sparse manner across time, so it is reasonable to assume there could be more data in there (if the data were collected more frequently and by doing a broader sweep out there) - this makes intepolation tempting, particularly in cases where there seems to be a qualitatively pretty straightforward upward or downward trend (just with gaps in some places, or just being too small samples in general). Then again, if I add too much, almost each test becomes "significant".

Question: Is there some (justifiable) middleground here - would it be valid to use (some justifiable amount of) interpolation prior to a t-test, and if so, should I (somehow) reconsider the alpha threshold for significance?**

  • $\begingroup$ It doesn't make sense to ask whether the mean of a vector (or set of vectors) "is significantly different from zero": it either is zero or not. Statistical significance attaches to a hypothesis test about some model parameter. What is your model? For instance, many people would model counts with Poisson distributions. But then, observing a single nonzero count would be completely convincing evidence ($p=0$) that the Poisson parameter (its mean) is nonzero. $\endgroup$ – whuber Jun 29 '17 at 19:36
  • $\begingroup$ I think you're getting more significant results because interpolated values are inherently less noisy. One solution might be to not use a deterministic function for the missing data, and to instead add a random error term. With suitable chosen variance for the error term the number of significant results wouldn't change. You could also use multiple imputation, and draw on the relationship between all observed consecutive data points to impute the missing data. $\endgroup$ – PaulB Jun 29 '17 at 19:37
  • $\begingroup$ @whuber why would you ever make a distributional assumption to test a mean?? $\endgroup$ – PaulB Jun 29 '17 at 19:40
  • $\begingroup$ @PaulB Because "test a mean" has absolutely no meaning without some kind of distributional assumption. The word "mean" in this phrase refers to a property of a distribution; if no distribution is posited, there is nothing to test. $\endgroup$ – whuber Jun 29 '17 at 19:57
  • $\begingroup$ @whuber if I randomly sample enough individuals from a population, and I want to test the null hypothesis that the population mean is zero, no distributional assumptions need to be made. $\endgroup$ – PaulB Jun 29 '17 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.