3
$\begingroup$

I am measuring the read speed of a device that reads files from a microSD card through a custom device (I therefore cannot use benchmarks posted on the internet for this). I'm doing this because I am developing a time-critical application and I need a measure of how much this time varies. However, I've encountered a problem I can't wrap my head around:

I have collected about 1000 datapoints which measure the time it takes to read a 100 Megabyte file in milliseconds.

I'm a novice in statistics but have recently completed a course about inferential statistics and I thought to myself "This is kinda like drawing a sample out of a population with unknown variance and such" but then I realized that while there might not be a maximum time it takes to read a file (Imagine you're theoretically reading data from a microSD card connected with a 5 kilometer cable) there is a minimum value.

This minimum value is bound to how fast the hardware platform is. Theoretically, there is a speed in milliseconds where a file can't be read any faster, but how it currently stands is that I have trouble making normality and other assumptions because of this. The figure below shows the skewness. We can clearly see the measurements do not go lower than 5000 ms (Our "physical limit"), but there are enough (varying) datapoints to the right.

How should I deal with this? Is this still usuable for inferential statstics? Ideally I would like to extract a confidence interval for the mean read time of this file so we can handle this in our application accordingly.

enter image description here

$\endgroup$
  • 2
    $\begingroup$ You don't establish confidence intervals for the read time; you might establish one for say the mean read time or the median read time or some other parameter of the distribution. If you want some kind of interval for a single read time, or a collection of them you'll need to specify the kind of interval you have in mind (tolerance interval? prediction interval? something else?) $\endgroup$ – Glen_b -Reinstate Monica May 19 '17 at 9:40
  • 2
    $\begingroup$ There is nothing wrong with a distribution whose support starts above zero. If you can find a good model for your data you can construct the confidence intervals you are after. If R is the read time, it looks like (R-5000) might be approximately distributed as a gamma. $\endgroup$ – F. Tusell May 19 '17 at 9:51
  • 1
    $\begingroup$ @Glen_b My apologies, I indeed meant a confidence interval for the mean read time. Edited my question. $\endgroup$ – Zimano May 19 '17 at 10:58
  • $\begingroup$ Thank you for providing the pointers! I guess it doesn't matter whether there is a certain cap on the minimum values of a set of datapoints. @F.Tusell Do you mean I should transform my datapoints by substracting 5000? or did you use 5000 because it is the minimum value in my sampling distribution? $\endgroup$ – Zimano May 19 '17 at 11:01
3
$\begingroup$

The short answer is yes, you can certainly use these data for inferential statistics. You're interested in the mean, so this makes it pretty straight-forward.

The usual parametric confidence interval

The distribution of a mean estimate $\hat \mu$ will (per the central limit theorem) converge to a normal distribution with mean $\mu$ (the population mean) and standard deviation $\sigma/\sqrt{n}$. Your underlying data don't have to be normal for this to be true, and can actually look pretty crazy. But the speed of convergence depends on "how normal" the sample is.

Usually $\sigma$ isn't known and you have to estimate it from the data, which leads to some extra uncertainty and the estimate will follow a Student's $t$ distribution. However, at your sample size the normal approximation is probably fine as Student's $t$ is asymptotically normal. Using this information you just estimate the mean and use the parametric distribution to compute confidence intervals in the usual way.


About the "limits" of your data: So your data is limited below at 5000ms due to physical laws. I'm going to show how this might impact the central-limit-theorem-type confidence interval with some R code and some plots.

Below I sample from a Poisson distribution with $\lambda=.5$. This distribution has mean and variance both $=.5$. This distribution has a limit in the same sense as your data: its support is the non-negative integers, so it has no values below 0. Hence we can never estimate a negative mean from this, as you could never estimate a mean below 5000ms in your data.

The asymptotic normal approximation for the mean of an $n$-sized sample in this setting is $\hat \mu \sim N(0.5, \sqrt{\frac{0.5}{n}})$. A normal distribution has the entire real line as support, so there is some chance of negative numbers here. In some sense the normal approximation must be wrong no matter what! We'll see that this problem is more real for smaller sample sizes. The data look like this:

library(plyr)

set.seed(2017-06-08)

l=0.5

# this distribution has mean and variance both = l above
# importantly it never produces values < 0
plot(table(rpois(100, lambda=l)))

Now for estimating the mean. For each histogram below I have repeated 100000 times the following: draw a sample of $n$ observations from the Poisson with $\lambda=.5$ and estimate $\hat\mu$ in this sample. The histograms thus show 100000 draws from the actual distribution of $\hat \mu$. The red line shows the normal approximation.

At $n=5$ we're not doing so great. Your confidence interval would probably contain some negative values. At $n=10$ it's looking better, but there is still some issue with the truncation at 0. At $n=100$ and $n=1000$ it doesn't look as though we need to worry. This is of course because the distribution gets narrower with larger $n$.

asymptotic_v_actual <- function(sample_size) {
  means <- raply(100000, mean(rpois(sample_size, lambda=l)))
  hist(means, prob=T, col="grey", border = NA,
       main=paste0("sample size=", as.character(sample_size)))
  curve(dnorm(x, mean=l, sd=sqrt(l/sample_size)), add=T, lwd=2, col="red")
}

asymptotic_v_actual(sample_size=5)

asymptotic_v_actual(sample_size=10)

asymptotic_v_actual(sample_size=100)

asymptotic_v_actual(sample_size=1000)

Finally I will show the probability of having $\hat\mu < 0$ according to the normal approximation at different sample sizes. At $n=20$ this is practically zero in our toy example.

# probability of x < 0 for different sample sizes:
sizes <- 1:20
plot(sizes, pnorm(0, mean=l, sd=sqrt(l/sizes)), type="o", 
     main="p(x < 0) for different sample size")


Nonparametric alternative

If you don't want to make any parametric assumptions (if you worry that your data aren't normal enough), you can use the bootstrap to build confidence bands around the estimated mean. This is a computational method that goes roughly as follows (variants exist):

  1. Make $B$ bootstrap samples by resampling from your original sample with replacement.
  2. Estimate the bootstrapped mean $\hat \mu^*_b$ in each of your bootstrap samples. This gives you an empirical distribution of over $\hat \mu$.
  3. Use the quantiles of this distribution as confidence bands. Eg if you want a 95% confidence interval, use the 0.025 quantile as lower limit, the 0.975 quantile as upper limit.

The bootstrap is very general and the bootstrapped statistic doesn't have to be a mean. If you want to look at the median or something else that isn't as simple to work with as the mean there's always the bootstrap.

$\endgroup$
  • $\begingroup$ In your second paragraph, talking about the distribution of a mean, are you referring to the central limit theorem? Thank you for a great answer! I would have liked to see you focus more on the fact that there is a limit on this dataset! $\endgroup$ – Zimano Jun 7 '17 at 18:11
  • $\begingroup$ @Zimano yes that's what I'm thinking about. I specified this in the text, I also added some words and plots about how the lower limit of your data affects this parametric estimate $\endgroup$ – einar Jun 8 '17 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.