# How to check statistical significance

I am from Civil Engineering background. In an experimental work, I am comparing some property (consider strength) of two materials/ mixtures "A" and "X". I prepared three specimens for each "A" and "X" and tested all of them for strength.

Suppose a, b and c are the values for "A" and x, y and z for "X". I am reporting the strength of theses mixtures as the average of corresponding values (i.e., $[a+b+c]/3$ for "A" and $[x+y+z]/3$ for "X"). Now, I want to check whether the strength of "A" is significantly different from that of "X".

Welcome Goutham. It seems that what you need is a t-test. However, since your samples are (very) small, non-parametric test (like the Wilcoxon-Mann-Whitney U test) might be relevant. You can find more information about those different tests in the following questions:

If you search for the tag, you will find many other relevant threads to help you decide what is the right way to proceed.

• To add to Antoine's response, you should also consider other not strictly statistical aspects, such as the known reliability of your assay technique and your proven record of applying it correctly. The magnitude of the difference is also very relevant to your ability to distinguish between the two materials with such a small sample. – Robert Jones May 30 '15 at 18:02
• To add to Antoine's response, you should also consider other not strictly statistical aspects, such as the known reliability of your assay technique and your proven record of applying it correctly. The magnitude of the difference is also very relevant to your ability to distinguish between the two materials with such a small sample. – Robert Jones May 30 '15 at 18:06

With only 3 measurements in each group you are very unlikely to find any meaningful results without prior knowledge outside of the data set that you can use.

One option is the 2 sample t test, but with only 3 observations per group the accuracy of the t test will be very dependent on the assumption of normality. If you are certain that the processes that produce your data are normal (or very very near normal) then you can use the t test. But this knowledge must be based on scientific knowledge and possibly other data, 6 data points will not give you enough information to guide you in this assumption.

The other main suggestion is usually the Wilcoxon-Mann-Whitney test or other permutation tests. With only 3 observations in each of 2 groups, the smallest possible p-value that you can see from a 1 tailed test is 0.05 (equal to the traditional cut-off) and 0.10 for a 2 tailed test. This means that your only chance of significance is if you a priory believe that one treatment will have higher strength and that all 3 measurements from that treatment are larger than all 3 of the measurements from the other treatment (and you use the traditional cut-off).

You would probably do better to use a larger sample size (do a power analysis ahead of time to determine a good sample size) and/or use prior information with a Bayesian model.

• Thanks for your response. In my case, each measurement is taken through destructive test on each specimen. So each individual measurement requires each specimen. That is why only three measurements are considered as per standards (ASTM, AASHTO etc.) – Goutham Sarang Jun 3 '15 at 6:42