Statistical tests when sample size is 1

Question

I'm a high school math teacher who is a bit stumped. A Biology student came to me with his experiment wanting to know what kind of statistical analysis he can do with his data (yes, he should have decided that BEFORE the experiment, but I wasn't consulted until after).

He is trying to determine what effect insulin has on the concentration of glucose in a cell culture. There are six culture grouped into three pairs (one with insulin and one without) each under slightly different conditions.

The problem is that he only took one sample from each so the there is no standard deviation (or the standard deviation is 0 since the value varies from itself by 0).

Is there any statistical analysis he can perform with this data? What advice should I give him other than to redo the experiment?

Answer 1

Unfortunately, your student has a problem.

The idea of any (inferential) statistical analysis is to understand whether a pattern of observations can be simply due to natural variation or chance, or whether there is something systematic there. If the natural variation is large, then the observed difference may be simply due to chance. If the natural variation is small, then it may be indicative of a true underlying effect.

With only a single pair of observations, we have no idea of the natural variation in the data we observe. So we are missing half of the information we need.

You note that your student has three pairs of observations. Unfortunately, they were collected under different conditions. So the variability we observe between these three pairs may simply be due to the varying conditions, and won't help us for the underlying question about a possible effect of insulin.

One straw to grasp at would be to get an idea of the natural variation through other channels. Maybe similar observations under similar conditions have been made before and reported in the literature. If so, we could compare our observations to these published data. (This would still be problematic, because the protocols will almost certainly have been slightly different, but it might be better than nothing.)

EDIT: note that my explanation here applies to the case where the condition has a potential impact on the effect of insulin, an interaction. If we can disregard this possibility and expect only main effects (i.e., the condition will have an additive effect on glucose that is independent of the additional effect of insulin), then we can at least formally run an ANOVA as per BruceET's answer. This may be the best the student can do. (And they at least get to practice writing up the limitations of their study, which is also an important skill!)

Failing that, I am afraid the only possibility would be to go back to the lab bench and collect more data.

---

In any case, this is a (probably painful, but still) great learning opportunity! I am sure this student will in the future always think about the statistical analysis before planning their study, which is how it should be. Better to learn this in high school rather than only in college.

Let me close with a relevant quote attributed to Ronald Fisher:

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.

Answer 2

Two-way ANOVA with One Observation per Cell

After you finish your important 'lecture' about consulting a statistician before starting to take data, you can tell your student that there is barely enough data here to support a legitimate experimental design.

If the subjects were chosen at random from some relevant population, glucose determinations were made in the same way for each of the six subjects, and if glucose levels are anything like normally distributed, then it seems possible to analyze the results according to a simple two-way ANOVA with one observation per cell.

The data might be displayed is a table like this:

                Insulin
             --------------
Method       Yes         No
---------------------------
     1
     2 
     3

The model is $Y_{ij} = \mu + \alpha_i + \beta_j + e_{ij},$ where $i = 1,2,3$ methods; $j = 1, 2$ conditions (Y or N), and $e_{ij} \stackrel{iid}{\sim} \mathsf{Norm}(0, \sigma).$ You can look at an intermediate level statistics text or introductory level text of experimental design for details.

The two-way ANOVA design would allow for a test whether the two Conditions have different glucose level (almost certainly so if insulin doses are meaningful) and whether the three Methods differ or are all the same.

With only two levels of one factor, only two levels of the other, and only one observation per cell, it would not be possible to take interaction between insulin dose and method into account. [There is no $(\alpha*\beta)_{ij}$ term in the model above; it would have the same subscripts as the error term $e_{ij}.]$

Also, it probably wouldn't be worthwhile to do any kind of nonparametric test (with more than three Methods---perhaps a Friedman test). That is why I made prominent mention normality above.

---

Example using fake data in R:

gluc = c(110, 135, 123,  200, 210, 234)
meth = as.factor(c(  2,   2,   3,    1,   2,   2))
insl = as.factor(c(  1,   1,   1,    2,   2,   2))
aov.out = aov(gluc ~ meth + insl)
summary(aov.out)
             Df Sum Sq Mean Sq F value Pr(>F)  
meth         2   3119    1559   5.193  0.161  
insl         1   9900    9900  32.973  0.029 *
Residuals    2    600     300                 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Insulin effect significant at 3% level.

You could also use just paired glucose measurements for Insulin (Y/N) in a paired t test to get a significant result. (In the ANOVA the Methods provide a bit of interaction, which can't be tested because there is only one observation per cell.)

t.test(gluc~insl, pair=T)

        Paired t-test

data:  gluc by insl
t = -8.812, df = 2, p-value = 0.01263
alternative hypothesis: 
  true difference in means is not equal to 0
95 percent confidence interval:
 -136.92101  -47.07899
sample estimates:
mean of the differences 
                    -92 

Note: See this demo for a $2 \times 3$ ANOVA with several replications per cell, analyzed in detail.

Answer 3

BruceET has described the proper analysis (Two-way ANOVA without interaction), so I'll put a more positive spin on the experiment.

I'm assuming that the design was three pairs, where there is variability between pairs. One of each pair was given insulin and the other without, hopefully randomized. Then each sample (pair X treatment, I call the experimental unit a petrie) was measured once.

1) This is not a bad design. This is probably one of the most commonly used experimental designs in science - it's a complete block design (also called a matched pairs design when the blocks only have two observations). This design generally is superior in power to the even more common completely randomized design (all six experimental units randomized into a set of three that got insulin and three that didn't). The paired design removes variability due to pair-to-pair variability. Seriously, this design is ubiquitous in agriculture, medicine, etc. The only objection I would have is that three pairs might allow too little power. But it is certainly replicated (there are multiple pairs).

2) It appears that the suggestion was that the student should have sampled each petrie multiple times to get replications. This would be a very bad recommendation. Multiply sampling each experimental unit to get replication is an example of pseudo-replication. If the pseudo-replicates are averaged together to yield one measure per petrie dish, you might lower variability somewhat, but you won't gain degrees of freedom in the analysis at all. The subsamples are not independent. So it is good that you didn't recommend that.

NOTE: Yes, with this design you can't get a culture:treatment interaction estimate. But that is also the case if this had been designed as a completely randomized design. The interaction ends up in the noise.

SUMMARY: The design is actually a classical experimental design, highly recommended for this kind of research. It is also easy to analyze. The only objection would be that three pairs might be underpowered.

Answer 4

Delightful question and one with historical precedent. As much as we might fault our budding high school junior scientist for his experimental design, it has a nearly perfect historical precedent.

What some consider the first controlled scientific medical experiment did the same thing. This high school student tested 3 situations with placebo or intervention. Physician James Lind aboard the HMS Salisbury did the same in his famous discovery of the treatment of scurvy. He hypothesized that scurvy might be treated by acids. So he came up with six acids and gave one to each of 6 scurvy-afflicted sailors while each had a matching single control for six more who did not receive the acid. This was basically six simultaneous controlled trials of an intervention on 1 person and no intervention on another. All told, 12 sailors, 6 treated, 6 not treated. Interventions were "cider, diluted sulfuric acid, vinegar, sea water, two oranges and a lemon, or a purgative mixture". How amazingly lucky we are that the one sailor who received the citrus fruits did not incidentally die of something else. The rest, as they say, is history. I've heard this discussed on a few podcasts so I knew the story. Here's a citation which I found with a quick internet search. It may not be the best source, but it'll get you started if you want to read more.

James Lind and Scurvy

-- JS

Answer 5

If the student were willing to make a rather deep dive, you might redirect their interest from sampling variation to uncertainty, and from a hypothesis test to an expanded uncertainty interval. Sampling variation is only one component of uncertainty. While the student is not in position to assess sampling variability, they might learn something from attempting to approximate the uncertainty associated with their measurements. I imagine your student is not up for the investment, but it's a suggestion.

Answer 6

A major problem is the small sample size reducing the degrees of freedom in model selection along with the model's required/sensitivity to normality of error assumption. Preserving degrees of freedom and being robust in methodology appears likely to be the best path. I would even advise generating random errors from possible parent distributions, and with knowledge of the actual parameter values, noting the variation in estimated parameter values and possible changes in test results.

As such, a simple parsimonous model approach would be first to place the data in a regression format in accord with the following reduced model in the variable Methods:

$$ Y_{i,j}-Ymedian = \beta *InsulinDummy_i + \gamma * MethodDummy_j + \varepsilon_{i,j} $$

where the dependent variable is the observed concentration of glucose centered around the population median, and the Insulin Dummy variable (also centered) is 1/2 if Insulin is present in test sample i, else -1/2. The Method Dummy variable is 2/3 for Method 1, else -1/3 for Methods 2&3 (repeat analysis, swapping out Method 1 for say Method 2, and repeat again swapping out Method 2 for Method 3).

Note, the proposed model interpretation of the regression coefficients is that it may aid in accurately determining which side of the median an observation will fall. Given the small sample size, I suggest a probabilistic (even Bayesian) interpretation, whose accuracy can be assessed in simulated model testing.

Next, the introduction of a robust regression analysis, where Least Absolute Deviations (LAD) is an option. Mathematically, LAD is linked to a Laplace distribution of error terms. One can compute coefficients employing iterative weighted Least-Squares, or, especially in the current context with 6 data points, employing the property that the model parameters determine a straight line that passes through two of the observed points in space. This implies examining permutations and testing total sum of absolute deviations. The selected points nearly always avoid outliers (unlike Least-Squares, where ANOVA also rests on a squared error criterion).

To obtain confidence intervals on parameters, bootstrap re-sampling of error terms has been suggested (see this), which can also be assessed on accuracy in simulation runs.

[EDIT] I thought my model is worthy of further exploration, so I built a worksheet based simulation model (convenient for the iterative LAD iteration, which involves examining point shifting, what points absolute errors are converging to zero (indicative of point pairs determining the LAD regression line). Here is a summary of a dozen simulation runs based on a uniform (-0.5 to +.5) error added to the model proposed above.

Actual Underlying Simulated Parameter Values are: 1.250 and 0.100

Simulation Run Values:

Average Observed Values 1.225 0.026

Observed Median 1.224 0.045

Run 1 1.001 0.324

Run 2 1.546 0.297

Run 3 1.350 -0.038

Run 4 1.283 -0.115

Run 5 1.593 -0.113

Run 6 1.498 -0.089

Run 7 0.863 0.151

Run 8 1.090 0.323

Run 9 1.102 -0.435

Run 10 1.166 -0.265

Run 11 1.451 0.128

Run 12 0.761 0.146

My take on the results are that the obtained summary statistics are amazing for my proposed parsimonous model based on 6 points with a uniform error distribution estimating 2 parameters on a data-centered model employing robust regression. Individual runs display, as expected, quite a range on the parameter values, but appear to more likely point to an effect greater than 1 for the first parameter (only 2 out of 12 are less than 1).

Answer 7

While the student does not have type A repeatability measurements, the student may/should be able to estimate the type B error contribution caused by equipment supplied from elsewhere ("For an estimate xi of an input quantity Xi that has not been obtained from repeated observations").

This is detailed in the SI/bipm Guide to Uncertainty in Measurement (there's a NIST equivalent).

This at least allows a route to making some judgement about the results.

The alternative, if the student did have a time series measurement (mentioned in one of the comments) is to estimate the smooth curve shape and hence the measurement error on top of that underlying smooth shape.

And lastly, if all the control groups were actually the same (not clear from the comments) then they could form a single group for the estimation of measurement noise.

Finally, use this as a 'post-mortem' to identify the level of measurement accuracy that would have been required to confirm the hypothesis at risk, and hence the likely number of repeat measurements needed to obtain that accuracy (error in the mean), given particular levels of basic accuracy (error on a single measurement). This at least rescues the student from feeling like it was a complete waste (i.e something learnt!).

Answer 8

What a good example of the old question of bias and random errors in observational errors.

If the biased estimation of the standard deviation is, as you mention:

$ \sigma = \sqrt{\frac{\sum{(x_i-\bar{x})^2}}{n}} = \frac {0}{1}=0$,

the unbiased estimation is

$ \sigma = \sqrt{\frac{\sum{(x_i-\bar{x})^2}}{n-1}} = \frac {0}{0}=undefined$.

So if even you student succeed in drawing some statistical conclusions, these will have an unknown bias.

However, this did not prevent Student to design the t-test, and Fisher to design the ANOVA method for such situations.

What about starting by drawing the three pairs on a scatter-plot, then a linear regression and look at the slope and compare with its standard error?

This is tantamount as BruceET answer, perhaps a bit more geometric and intuitive.