Performing a power analysis on finding the mean of a single sample, non-normal dataset

Question

I would like to perform a power analysis on my pilot data. My test statistic is a single-sample mean with only 14 observations. The data are non-normal (it's percent vegetation cover, which I think follows a beta distribution?) so I assume that I will be boostrapping, but that's about as far as I've made it.

My thoughts at the moment are that I'll boostrap the data to create a sampling distribution of means.

• Do I then perform a t-test on each bootstrapped mean to find if it is significantly different from the sample mean?
• Is this t-test appropriate because the sampling distribution is normally distributed even though the original sample isn't?
• Would $\mu$ simply be my original sample mean of 0.18?
• Where do I define how comfortable I am with the mean being off. I don't know the term for that; what I mean to say is. I'm comfortable with an error margin of +/- 0.05.
• I will, of course, be extending this to a sample size calculation (using R). Any clarity on the process is much appreciated.

edit to add: the end result I'm looking for is the sample size required to be 90% confident in my estimation of cover and to have that estimation of cover to be +/- 5% of the true cover. For extra fun, here are my 14 observations in my pilot data: (0, 0, 0.01, 0.01, 0.01, 0.03, 0.09, 0.20, 0.22, 0.23, 0.37, 0.42, 0.47, 0.57)

---

okay, Here's what I came to. I understand that 'power analysis' is the wrong term for what I was looking for. I am really asking, "How much can I trust these 14 observations to tell me the mean?" and secondly, "How many samples do I need to feel confident in my result?".

I first used bootstrap resampling to construct my confidence intervals of the original data with 14 observations. Then I resampled the data 1000 times while simulating capturing 10 through 20 observations, watched my confidence intervals get tighter and my margin of error shrink. With this I can estimate the number of samples I need to determine my mean within the margin of error I'm looking for.

Answer 1

You don't have a hypothesis anywhere here. Don't test anything unless you do have a suitable hypothesis; this must NOT be based on things you find in the sample.

What are you collecting data for? What were you trying to find out?

Is this t-test appropriate because the sampling distribution is normally distributed even though the original sample isn't?

(i) The sampling distribution of the mean is not normal; the average of 14 non-normally distributed quantities (even if some other typically assumed conditions were reasonable) is not normally distributed.

(ii) the derivation of the ratio in a t-statistic being t-distributed relies on more than the distribution of the numerator.

(iii) nevertheless, it's pretty likely that a suitable t-statistic would have a distribution close enough to the t-distribution that the significance level of the test would be very nearly what you selected $\alpha$ to be (quite possibly even at $n=14$). The power would be somewhat reduced compared to a better choice of distributional model, but in many situations probably not enough to worry you.

Would μ simply be my original sample mean of 0.18?

No. If you didn't have a $\mu$ before you saw any data, you didn't have a hypothesis to test

Where do I define how comfortable I am with the mean being off. I don't know the term for that; what I mean to say is. I'm comfortable with an error margin of +/- 0.05.

It's unclear what you mean. Are you here trying to specify a margin of error for the estimate of the mean, as if you were creating a confidence interval for the mean?

Do I then perform a t-test on each bootstrapped mean to find if it is significantly different from the sample mean?

No. You do not compare means of simple bootstrap resamples to the sample mean. That doesn't test anything.

If you were to have a suitable hypothesis, and you specifically wanted a non-parametric test of the mean (not simply that you didn't just want to use an ordinary one-sample t-test), I'd be suggesting a permutation test.

But you don't have a hypothesis. There's literally nothing to test.

I will, of course, be extending this to a sample size calculation (using R).

You're going to have to be a lot clearer about your aim. You don't have a test, so no sample size calculation on that basis.

You could specify a margin of error for a CI and get a sample size that way but you haven't really been very clear that this is actually what you want.

"(it's percent vegetation cover, which I think follows a beta distribution?)"

No, the beta distribution is one potentially plausible model for a continuous proportion like vegetation cover, but don't confuse the model for the real thing: -

Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.
$-$ George Box

If you believe the beta model would be a pretty good model (and it might, as long as none of your proportions are actually 0 or 1), why would you not directly use the beta model as the basis for a test, confidence interval, or whatever else?

...

Ideally you should have dealt with these issues in detail before gathering even the pilot sample, but at least you're doing it before you collect the main sample, which is important.