1
$\begingroup$

I'm currently collecting time series data from an experiment. I'm collecting the position of an object in 1D, i.e. $x$, at a rate of 60 times per second. I'm then interested in summary statistics. The main summary statistic that I'm interested in is the fraction of measurements where the absolute value is larger than a limit, e.g. 0.1.

Every time I perform the experiment, I get a slightly different number. Sometimes the fraction is 0.2, sometimes 0.21, 0.17 etc. I expect that these fractions follow an unknown distribution. With enough experiments, the mean value that I get will be stable so that the standard deviation of the mean value will be smaller than some value.

I believe that I can estimate the standard deviation of the mean value by bootstrapping. However, I would like to estimate in advance how many experiments I have to perform for the standard deviation of the mean to go below some value.

Can I do that using the e.g. 10 first experiments?

I'm open to simulations, Bayesian approaches, or whatever is possible.

Improve this question
$\endgroup$
5
  • $\begingroup$ Is the number of experiments you refer to the number of position measurements (i.e. you want to fix total measurement time)? And does it make sense to assume that the probabiliy/fraction of measurements wtih x > 0.1 is constant? $\endgroup$
    – cbeleites
    Commented Nov 27, 2019 at 20:09
  • $\begingroup$ The provided information is very vague. Maybe you could tell us, why you believe that the central limit theorem is not applicable. If you really collect data with a frequency of 60Hz you should obtain a very good estimate of your distribution within 1min (=3600 data-point). $\endgroup$
    – Semoi
    Commented Nov 27, 2019 at 20:10
  • $\begingroup$ @Semoi Indeed, I believe that I can get an excellent estimate of the fraction for one individual, but each individual has its own fraction as described. They are not exactly the same, and therefore yield slightly different results. The individuals are, however, drawn from a limited pool of possible individuals. Much too large to sample all of them, but there are reasons to believe that enough experiments will characterize this pool of individuals. I'm sorry for being vague, I'm doing the best I can. Please ask more questions if needed. $\endgroup$
    – C. E.
    Commented Nov 27, 2019 at 20:53
  • $\begingroup$ @cbeleitessupportsMonica The experiments that I refer to are distinct time series. Recordings, if you will, of different individuals with slightly different characteristics. I believe that this pool of individuals that I'm sampling from has variation in it; the fractions corresponding to different individuals follow a distribution, I guess a normal distribution since they are independent. So I suppose one way of looking at it is how to estimate the $\mu$ och this normal distribution, and how to know when the standard deviation of this $\mu$ is smaller than some value. $\endgroup$
    – C. E.
    Commented Nov 27, 2019 at 20:58
  • 1
    $\begingroup$ @Semoi That expectation is unwarranted because it relies implicitly on two assumptions that are likely to be strongly violated: namely, that the observations are independent and that one is interested in their full distribution. In this case a strong serial correlation, if present, would make the effective number of observations much smaller; and the the focus on characterizing excursions above a threshold places us into extreme-value territory. The CLT is unlikely to apply for both reasons. $\endgroup$
    – whuber ♦
    Commented Nov 27, 2019 at 21:16

1 Answer 1

Reset to default
1
$\begingroup$

  • For the single individual, have a look at binomial confidence intervals iff you can assume each observation to be independent of all other observations. If you record at least a few seconds, the normal approximation should work well. The interesting feature of the binomial confidence intervals is that you can determine appropriate recording times beforehand, since p = 0.5 is the "worst case" = has highest variance.

  • If the observations of a single individual are not independent, look into time series to find out how long to record for each of them.

  • For the number of individuals, I'd then indeed bootstrap (by individual and including intra-individual variation if that's possible). Individuals being independent of each other doesn't make their fractions being normally distributed. However, CLT should work for the mean fraction (but may not be so very normal yet if only a few individual fractions are averaged).

Standard deviation of the mean is $\frac{s}{\sqrt n}$ (s = standard deviation of the population and, n = number of individuals). I.e., if the standard deviation of the mean for your preliminary data with $n_1$ individuals is too large by a factor $f$, in first approximation you'll need to measure $n \approx n_1 f^2$ to get the standard deviation down to your requirements. In practice, this is often sufficient with a somewhat generous rounding up to account for the uncertainty in the estimation of s based on only few individuals.

To get a conservative estimate of $n$ (i.e. one unlikely to underestimate the required sample size) you can also calculate required sample size with the upper limit of the confidence interval for the standard deviation (second approximation). For a normal distribution (this is where CLT is needed: if the distribution of the fractions isn't already normal, the $x'$ here is already the average of sufficient individuals to make $x'$ normally distributed), you can use the one-sided confidence interval $\sigma_{x'} \leq \sqrt{\frac{(n' - 1)s_x'^2}{\chi^2_{n'-1, 1 - \alpha}}}$ with confidence level $1 - \alpha$ - don't forget to multiply the resulting $f^2n'$ with the number of individuals that were averaged to get the actual number of individuals $n$. (Note that this is still approximate)

What @whuber comments about ending up in extreme-value theory does have consequences here as well: if you find out that you basically end up with a bimodal distribution (fractions being pretty much either 0 or 1 for each individual) you'd need to average far more individuals until you can work with a normal distribution for the mean than if you find already your distribution of individual fractions basically looks normally distributed around 0.2.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you, however, this doesn't address how I can predict how many experiments I would need to bring the standard deviation of the mean of all those individuals below a certain number. I can use bootstrapping to tell me what the standard deviation is with the number of experiments that I have. But it doesn't tell me how that number will change as I perform more experiments. Or does CLT somehow tell me this? The problem I'm facing is that I need to be able to estimate in advance how many experiments I need to make in order to get a good mean value over all individuals. $\endgroup$
    – C. E.
    Commented Nov 27, 2019 at 21:53
  • $\begingroup$ See update. For my experiments, the first approximation is usually sufficient. $\endgroup$
    – cbeleites
    Commented Nov 27, 2019 at 23:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.