# What can be concluded from small sample size?

I had just conducted a test where the customer wants to see a 20% improvement in perforation length from a baseline perforator. The baseline perforator has not been tested in the customer's specific conditions in which he will eventually perforate in.

The customer plans to shoot 40,000 perforations in the future. I'm not sure if that bit of information is important but I wanted to be sure that it is understood that the true population mean of perforation length does not exist yet.

Under the specific conditions tailored for the customer the baseline perforator was shot 3 times with the following results: [7.68, 8.72, 8.08] inches.

Three new perforator designs were constructed in hopes to out perform the baseline perforator and each design was shot 3 times with the following results:

• D1 [8.04, 8.93, 10.05] inches
• D2 [11.15, 6.91, 9.21] inches
• D3 [7.1, 7.6, 7.02] inches

As I am relatively new to statistics I was wondering if some of you professionals can help me out with what information can be gained from these tests results. If you can point me in the right direction as in what methods I should refer to in gaining these insights you suggest would be of much help.

I know that effort is encouraged on my part so I would like to include some of my initial attempts at inferential statistics and maybe you can correct me where I am wrong or include what more I should/can do.

Estimating Sample Size Requirements: Solving for the sample size requires the population standard deviation, $\sigma$. Most often we do not know it so we have to use an estimate or “planning value” in its place. Here are a few options:

1. Estimate σ from previous studies using the same population of interest
2. Conduct a pilot study to select a preliminary sample. Use the sample standard deviation from the pilot study
3. Use a judgment or “best guess” for $\sigma$. A common “guess” is the data range (high value – low value) divided by 4

The question I asked is: “What minimum sample size is necessary to produce 95% confidence (assuming a normal distribution) that the sample mean is $\pm$ 1 inch of the true population mean, $\mu$?”

Since we do not have an estimate of $\sigma$ from previous studies we will go with option 2 and consider our information as pilot studies.

Test charge Design 2 perforation length data is as follows: [11.15, 9.21, 6.91] inches. The sample mean is 9.09 inches and sample standard deviation is 2.12 inches. How large a sample should be shot to provide a 95% confidence interval with a margin of error (E) of 1 inch, assuming the population standard deviation is equal to the sample standard deviation?

$$n = \frac{(z_{\alpha /2})^2 (\sigma)^2}{E^2}$$

$$n = \frac{(1.96)^2 (2.12)^2}{(1)^2}$$

$$n = 17$$

Interpretation: To have 95% of our sample means contain the true population mean, $\mu$, we need a sample size of 17 shots. Or, 95 out of 100 tests made up of 17 perforation shots will contain within its average length $\pm$ 1 inch the perforation design’s true average perforation length.

It is not clear to me, however, how to move forward with this result. Does it mean I need to make an additional 17 shots as it's own sample or do I need to make 14 more shots to add to the original 3 shots? How do I handle the new information? I would suspect that the standard error of the mean would narrow with more measurements and would thus adjust the required sample size?

If we do not assume that the population standard deviation is equal to the sample standard deviation we can set the planning value for the population standard deviation as the Range of the perforation lengths we observed divided by 4 (option 3). From there we can estimate the minimum number of shots that should be taken with the margin of error of 1 inch to contain the true population mean 95% of the time.

Test charge Design 2 perforation length data is as follows: [11.15, 9.21, 6.91] inches.

$$\text{Planning Value for Std. Dev.} = \frac{\text{Range}}{4}$$

$$\text{PV} = \frac{(11.15-6.91)}{4} = 1.06$$

$$n = \frac{(z_{\alpha /2})^2 (\sigma)^2}{E^2}$$

$$n = \frac{(1.96)^2 (1.06)^2}{(1)^2}$$

$$n = 4.3$$

Interpretation: To have 95% of our sample means contain the true population mean, $\mu$, we need a sample size of 4 shots. Or, 95 out of 100 tests made up of 4 perforation shots will contain within its average length $\pm$ 1 inch the perforation design’s true average perforation length.

So which would be the more appropriate method (option 2 or 3)? And how should that information be used? In both cases more shots were required. But what does it tell me if the opposite occurs? e.g. for charge Design 3, using both option 2 and 3 method, I got n = 0.37 shots and n = 0.08 shots respectively.

Lastly, I tried to estimate what the %chance that the new charge design's true-population-mean-perforation-length will meet the required improvement. Taking the mean of the baseline shots to be 8.16 inches and adding 20% to it I found the goal length to be $\approx 9.792$ inches. By taking the sample means of each design we see that none of them averaged to meet the required length. However, I found the 95% confidence interval using a t-distribution for Design 2 to be [3.81 to 14.36] inches. Using excel Goal Seek I believe there is a 31% chance that Design 2's true perforation mean length will meet the 9.792 inch requirement. I'm not sure if my method was correct but maybe you can tell me if I am wrong or right or if interested, I will spell out how I went about that estimation.

## 1 Answer

I didn't check your calculations but finding widely different standard deviations/sample size estimates from very small pilot studies is not surprising. Just like the sample mean, the sample standard deviation is a noisy estimate of its population counterpart. That's why it's not such a good idea to use a small pilot study to compute power or decide on sample size (even if it is, as you noted, standard advice). I discussed this further in these comments and, indirectly, in this answer to an unrelated question.

As to whether or not the pilot data would be included in the final data set, the usual answer would be negative but thinking about it I am not sure if I can see a compelling reason, at least not if you are not using the mean or difference but only the standard deviation in your sample size calculations. On a related note, collecting data and stopping when your CI reaches a certain width is in fact sound (whereas repeatedly testing and stopping when the mean is significantly different from some value is a big no-no). It seems related to the accuracy in parameter estimation approach, which might be relevant to you.

PS: Note that some of your statements about confidence intervals seems a little unclear at times. For example, by construction, 95% of them should contain the true mean, what increasing the sample size does is reduce their width, not change this frequency.

• Laurans - Thank you for the reply, Gael. I will do my best to interpret and investigate your response and try to come back with further updates to this question so that others may learn from this post. – Armadillo Aug 7 '13 at 5:09
• As to the second part of your answer, yes it is also common practice, but reusing pilot data in the "main" calculation introduces bias, even if it is just for stopping at a target CI estimate. One should atleast have an idea of why such bias in the specific case could be deemed small, which is not always the case. – Quartz Aug 7 '13 at 8:23
• @Quartz You could perhaps explain that in more detail (in a comment, by editing my answer or adding another one). I must admit I am not sure I understand what the problem is or how to determine if bias is large or not. – Gala Aug 7 '13 at 9:02
• @Gaël Laurans: Reusing the pilot data introduces dependency among data, so in theory the whole procedure is invalidated. Of course in practice one might still account for that and compensate, but you must know where, how and how much, and afaik already for the latter any dependence estimate will itself be also biased, so the whole procedure is not as trivial as it might seem at first. – Quartz Aug 8 '13 at 12:57