# Performing a power analysis on finding the mean of a single sample, non-normal dataset

I would like to perform a power analysis on my pilot data. My test statistic is a single-sample mean with only 14 observations. The data are non-normal (it's percent vegetation cover, which I think follows a beta distribution?) so I assume that I will be boostrapping, but that's about as far as I've made it.

My thoughts at the moment are that I'll boostrap the data to create a sampling distribution of means.

• Do I then perform a t-test on each bootstrapped mean to find if it is significantly different from the sample mean?
• Is this t-test appropriate because the sampling distribution is normally distributed even though the original sample isn't?
• Would $$\mu$$ simply be my original sample mean of 0.18?
• Where do I define how comfortable I am with the mean being off. I don't know the term for that; what I mean to say is. I'm comfortable with an error margin of +/- 0.05.
• I will, of course, be extending this to a sample size calculation (using R). Any clarity on the process is much appreciated.

edit to add: the end result I'm looking for is the sample size required to be 90% confident in my estimation of cover and to have that estimation of cover to be +/- 5% of the true cover. For extra fun, here are my 14 observations in my pilot data: (0, 0, 0.01, 0.01, 0.01, 0.03, 0.09, 0.20, 0.22, 0.23, 0.37, 0.42, 0.47, 0.57)

okay, Here's what I came to. I understand that 'power analysis' is the wrong term for what I was looking for. I am really asking, "How much can I trust these 14 observations to tell me the mean?" and secondly, "How many samples do I need to feel confident in my result?".

I first used bootstrap resampling to construct my confidence intervals of the original data with 14 observations. Then I resampled the data 1000 times while simulating capturing 10 through 20 observations, watched my confidence intervals get tighter and my margin of error shrink. With this I can estimate the number of samples I need to determine my mean within the margin of error I'm looking for.

You don't have a hypothesis anywhere here. Don't test anything unless you do have a suitable hypothesis; this must NOT be based on things you find in the sample.

What are you collecting data for? What were you trying to find out?

Is this t-test appropriate because the sampling distribution is normally distributed even though the original sample isn't?

(i) The sampling distribution of the mean is not normal; the average of 14 non-normally distributed quantities (even if some other typically assumed conditions were reasonable) is not normally distributed.

(ii) the derivation of the ratio in a t-statistic being t-distributed relies on more than the distribution of the numerator.

(iii) nevertheless, it's pretty likely that a suitable t-statistic would have a distribution close enough to the t-distribution that the significance level of the test would be very nearly what you selected $$\alpha$$ to be (quite possibly even at $$n=14$$). The power would be somewhat reduced compared to a better choice of distributional model, but in many situations probably not enough to worry you.

Would μ simply be my original sample mean of 0.18?

No. If you didn't have a $$\mu$$ before you saw any data, you didn't have a hypothesis to test

Where do I define how comfortable I am with the mean being off. I don't know the term for that; what I mean to say is. I'm comfortable with an error margin of +/- 0.05.

It's unclear what you mean. Are you here trying to specify a margin of error for the estimate of the mean, as if you were creating a confidence interval for the mean?

Do I then perform a t-test on each bootstrapped mean to find if it is significantly different from the sample mean?

No. You do not compare means of simple bootstrap resamples to the sample mean. That doesn't test anything.

If you were to have a suitable hypothesis, and you specifically wanted a non-parametric test of the mean (not simply that you didn't just want to use an ordinary one-sample t-test), I'd be suggesting a permutation test.

But you don't have a hypothesis. There's literally nothing to test.

I will, of course, be extending this to a sample size calculation (using R).

You're going to have to be a lot clearer about your aim. You don't have a test, so no sample size calculation on that basis.

You could specify a margin of error for a CI and get a sample size that way but you haven't really been very clear that this is actually what you want.

"(it's percent vegetation cover, which I think follows a beta distribution?)"

No, the beta distribution is one potentially plausible model for a continuous proportion like vegetation cover, but don't confuse the model for the real thing: -

Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.
$$-$$ George Box

If you believe the beta model would be a pretty good model (and it might, as long as none of your proportions are actually 0 or 1), why would you not directly use the beta model as the basis for a test, confidence interval, or whatever else?

...

Ideally you should have dealt with these issues in detail before gathering even the pilot sample, but at least you're doing it before you collect the main sample, which is important.

• Thank you for your comments, and for wading through my difficulty with terminology. I am in the process of learning, and am grateful for the guidance, though I must say, your tone comes across as condescending. Perhaps I should have simply started with my end goal. Yes, I have no hypothesis. I am simply trying to find the mean cover and feel confident that my sample size is adequate. Every sample size equation I can find assumes normal data and my data are decidedly not. This is not uncommon for ecological data, I am simply trying to determine where to go next. Commented Apr 29, 2023 at 6:02
• 1. I apologize for coming across as condescending; I'll see if I can find something to edit above; your post gave the impression of having more knowledge but with some (in that case) strange choices. It seems you were just somewhat led astray. 2. The resources you can find by searching the internet are often not of great quality, sadly, and often focus almost exclusively on hypothesis testing, which is not much use here -- there's a lot of people writing outside their area of expertise (many more than are writing within it). ... ctd Commented Apr 29, 2023 at 10:42
• 3. "I am simply trying to find the mean cover and feel confident that my sample size is adequate" -- as mentioned in my post, you could specify a margin of error for a CI (for the mean in this case) and get a sample size that way. That the distribution might be approximately beta is of no great consequence as long as it's not highly skewed, the usual (normal-assuming) CI should be adequate with a reaosnable-sized sample. Commented Apr 29, 2023 at 10:46
• this makes sense, thank you. The tricky thing is understanding what 'reasonable-sized' samples might be. I've found the sample size equations that assume normal data. I would prefer to know a better way to do it. For fun, here are the 14 points of data from the pilot data, (0, 0, 0.01, 0.01, 0.01, 0.03, 0.09, 0.20, 0.22, 0.23) Commented May 3, 2023 at 23:13
• You seem to be asking two new questions now -- "How do I figure out what a reasonable sample size might be?" and "How do I calculate a suitable sample size?" -- the first would involve investigating the properties under a variety of plausible assumptions for your variables, perhaps via simulation, for example, though algebraic calculation might be doable. The second would require (i) some assumptions - not just distributional ones, (ii) a specific test, CI or other procedure that you specify enough properties for, and (iii) criteria you need to satisfy. Again, simulation might be used. Commented May 3, 2023 at 23:45