50
$\begingroup$

I'm a high school math teacher who is a bit stumped. A Biology student came to me with his experiment wanting to know what kind of statistical analysis he can do with his data (yes, he should have decided that BEFORE the experiment, but I wasn't consulted until after).

He is trying to determine what effect insulin has on the concentration of glucose in a cell culture. There are six culture grouped into three pairs (one with insulin and one without) each under slightly different conditions.

The problem is that he only took one sample from each so the there is no standard deviation (or the standard deviation is 0 since the value varies from itself by 0).

Is there any statistical analysis he can perform with this data? What advice should I give him other than to redo the experiment?

$\endgroup$
  • 10
    $\begingroup$ I think this is an interesting question, but unfortunately only because I suspect that the answer is "not much" (as in, there's not much to be done other than to repeat the experiment), and I would be very interested if someone were able to give an answer with more content. $\endgroup$ – LSpice Apr 28 at 3:09
  • 1
    $\begingroup$ If you are prepared to take a Bayesian standpoint then there is some hope, but in High School that is a lot of complexity to introduce a student to (and it does degrade the rigour of the scientific process, since you want the experiment to demonstrate the effects regardless of your prior beliefs) $\endgroup$ – JDL Apr 28 at 13:06
  • 6
    $\begingroup$ Also, go find the Biology teacher and whack him with a clue-by-4 . $\endgroup$ – Carl Witthoft Apr 28 at 17:25
  • 3
    $\begingroup$ I got the impression that the Biology teacher wasn't consulted much other than explaining the topic. He's the one who sent the student to me because he didn't know what to do with the data either. As for the experiment design, he has three group pairs: the control group (just nutrient solution without any cell culture), cell cultures whose nutrients are not replenished throughout the experiment (I guess he just added water to keep the volume constant... not really sure. I didn't read the whole procedure), and cell cultures whose nutrients were replenished periodically. $\endgroup$ – Brent Parker Apr 29 at 0:33
  • 1
    $\begingroup$ In each pair, one also received insulin while the other did not. He periodically took one sample from each and measured the glucose concentration to have some idea of the concentration over time. $\endgroup$ – Brent Parker Apr 29 at 0:34
44
$\begingroup$

Unfortunately, your student has a problem.

The idea of any (inferential) statistical analysis is to understand whether a pattern of observations can be simply due to natural variation or chance, or whether there is something systematic there. If the natural variation is large, then the observed difference may be simply due to chance. If the natural variation is small, then it may be indicative of a true underlying effect.

With only a single pair of observations, we have no idea of the natural variation in the data we observe. So we are missing half of the information we need.

You note that your student has three pairs of observations. Unfortunately, they were collected under different conditions. So the variability we observe between these three pairs may simply be due to the varying conditions, and won't help us for the underlying question about a possible effect of insulin.

One straw to grasp at would be to get an idea of the natural variation through other channels. Maybe similar observations under similar conditions have been made before and reported in the literature. If so, we could compare our observations to these published data. (This would still be problematic, because the protocols will almost certainly have been slightly different, but it might be better than nothing.)

EDIT: note that my explanation here applies to the case where the condition has a potential impact on the effect of insulin, an interaction. If we can disregard this possibility and expect only main effects (i.e., the condition will have an additive effect on glucose that is independent of the additional effect of insulin), then we can at least formally run an ANOVA as per BruceET's answer. This may be the best the student can do. (And they at least get to practice writing up the limitations of their study, which is also an important skill!)

Failing that, I am afraid the only possibility would be to go back to the lab bench and collect more data.


In any case, this is a (probably painful, but still) great learning opportunity! I am sure this student will in the future always think about the statistical analysis before planning their study, which is how it should be. Better to learn this in high school rather than only in college.

Let me close with a relevant quote attributed to Ronald Fisher:

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.

| cite | improve this answer | |
$\endgroup$
  • 16
    $\begingroup$ After talking to me, the student has decided to redo his experiment with better data collection techniques and methodology. It's a mistake I don't think he'll make again. Thanks for the thoughtful response. $\endgroup$ – Brent Parker Apr 29 at 0:36
24
$\begingroup$

Two-way ANOVA with One Observation per Cell

After you finish your important 'lecture' about consulting a statistician before starting to take data, you can tell your student that there is barely enough data here to support a legitimate experimental design.

If the subjects were chosen at random from some relevant population, glucose determinations were made in the same way for each of the six subjects, and if glucose levels are anything like normally distributed, then it seems possible to analyze the results according to a simple two-way ANOVA with one observation per cell.

The data might be displayed is a table like this:

                Insulin
             --------------
Method       Yes         No
---------------------------
     1
     2 
     3

The model is $Y_{ij} = \mu + \alpha_i + \beta_j + e_{ij},$ where $i = 1,2,3$ methods; $j = 1, 2$ conditions (Y or N), and $e_{ij} \stackrel{iid}{\sim} \mathsf{Norm}(0, \sigma).$ You can look at an intermediate level statistics text or introductory level text of experimental design for details.

The two-way ANOVA design would allow for a test whether the two Conditions have different glucose level (almost certainly so if insulin doses are meaningful) and whether the three Methods differ or are all the same.

With only two levels of one factor, only two levels of the other, and only one observation per cell, it would not be possible to take interaction between insulin dose and method into account. [There is no $(\alpha*\beta)_{ij}$ term in the model above; it would have the same subscripts as the error term $e_{ij}.]$

Also, it probably wouldn't be worthwhile to do any kind of nonparametric test (with more than three Methods---perhaps a Friedman test). That is why I made prominent mention normality above.


Example using fake data in R:

gluc = c(110, 135, 123,  200, 210, 234)
meth = as.factor(c(  2,   2,   3,    1,   2,   2))
insl = as.factor(c(  1,   1,   1,    2,   2,   2))
aov.out = aov(gluc ~ meth + insl)
summary(aov.out)
             Df Sum Sq Mean Sq F value Pr(>F)  
meth         2   3119    1559   5.193  0.161  
insl         1   9900    9900  32.973  0.029 *
Residuals    2    600     300                 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Insulin effect significant at 3% level.

You could also use just paired glucose measurements for Insulin (Y/N) in a paired t test to get a significant result. (In the ANOVA the Methods provide a bit of interaction, which can't be tested because there is only one observation per cell.)

t.test(gluc~insl, pair=T)

        Paired t-test

data:  gluc by insl
t = -8.812, df = 2, p-value = 0.01263
alternative hypothesis: 
  true difference in means is not equal to 0
95 percent confidence interval:
 -136.92101  -47.07899
sample estimates:
mean of the differences 
                    -92 

Note: See this demo for a $2 \times 3$ ANOVA with several replications per cell, analyzed in detail.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I don't think this will be very helpful. We have six observations and six degrees of freedom. The fit will be perfect, the errors will be zero, and our ANOVAs will divide zero by zero. $\endgroup$ – Stephan Kolassa Apr 28 at 7:07
  • $\begingroup$ @StephanKolassa: It's 1 DF for Condition, 2 DF for Meth, 2 DF for Error. Hardly ideal, but it will run in software, unless there are too many ties.// Just added example and link. // But Sir Ronald's advice seems appropriate. $\endgroup$ – BruceET Apr 28 at 7:27
  • 3
    $\begingroup$ (+1) While not ideal, why isn't the paired $t$-test the obvious thing to do here? "Method" appears to redundant to research question. I appreciate that if it interacts with the effect of insulin we are staffed but that's a different setting. $\endgroup$ – usεr11852 Apr 28 at 19:34
  • 3
    $\begingroup$ Because there are so few data here, no test is ideal. Of mediocre available choices, paired-t is OK, I think 2-way ANOVA is better because, in testing for Insulin (Y/N), it offers protection against possible ruinous variability among Methods.// Nonparametric alternative to paired t is often Wilcoxon SR test on differences, but that's off the table here, because that test can never reject at 5% level with only 3 differences. For similar, slightly more complicated, reasons Friedman test would also be inappropriate here. $\endgroup$ – BruceET Apr 28 at 20:50
  • 2
    $\begingroup$ I mentioned in a comment to AlaskaRon that the student has decided to redo his experiment with better sampling techniques and methodology, but I'm still going to point him to this thread. Your reply was thorough and insightful and will hopefully educate the student. Thanks, BruceET $\endgroup$ – Brent Parker Apr 29 at 0:28
21
$\begingroup$

BruceET has described the proper analysis (Two-way ANOVA without interaction), so I'll put a more positive spin on the experiment.

I'm assuming that the design was three pairs, where there is variability between pairs. One of each pair was given insulin and the other without, hopefully randomized. Then each sample (pair X treatment, I call the experimental unit a petrie) was measured once.

1) This is not a bad design. This is probably one of the most commonly used experimental designs in science - it's a complete block design (also called a matched pairs design when the blocks only have two observations). This design generally is superior in power to the even more common completely randomized design (all six experimental units randomized into a set of three that got insulin and three that didn't). The paired design removes variability due to pair-to-pair variability. Seriously, this design is ubiquitous in agriculture, medicine, etc. The only objection I would have is that three pairs might allow too little power. But it is certainly replicated (there are multiple pairs).

2) It appears that the suggestion was that the student should have sampled each petrie multiple times to get replications. This would be a very bad recommendation. Multiply sampling each experimental unit to get replication is an example of pseudo-replication. If the pseudo-replicates are averaged together to yield one measure per petrie dish, you might lower variability somewhat, but you won't gain degrees of freedom in the analysis at all. The subsamples are not independent. So it is good that you didn't recommend that.

NOTE: Yes, with this design you can't get a culture:treatment interaction estimate. But that is also the case if this had been designed as a completely randomized design. The interaction ends up in the noise.

SUMMARY: The design is actually a classical experimental design, highly recommended for this kind of research. It is also easy to analyze. The only objection would be that three pairs might be underpowered.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks a lot. I'm going to point the student at this whole thread to give him more ideas. After more discussion he has decided to redo his experiment with better methodology but I still think this discussion will be helpful. $\endgroup$ – Brent Parker Apr 29 at 0:24
  • $\begingroup$ You say that this is a common design (and that there is replication) but that is the case when the blocks/conditions are to be considered a random factor. $\endgroup$ – Sextus Empiricus Apr 30 at 13:34
5
$\begingroup$

Delightful question and one with historical precedent. As much as we might fault our budding high school junior scientist for his experimental design, it has a nearly perfect historical precedent.

What some consider the first controlled scientific medical experiment did the same thing. This high school student tested 3 situations with placebo or intervention. Physician James Lind aboard the HMS Salisbury did the same in his famous discovery of the treatment of scurvy. He hypothesized that scurvy might be treated by acids. So he came up with six acids and gave one to each of 6 scurvy-afflicted sailors while each had a matching single control for six more who did not receive the acid. This was basically six simultaneous controlled trials of an intervention on 1 person and no intervention on another. All told, 12 sailors, 6 treated, 6 not treated. Interventions were "cider, diluted sulfuric acid, vinegar, sea water, two oranges and a lemon, or a purgative mixture". How amazingly lucky we are that the one sailor who received the citrus fruits did not incidentally die of something else. The rest, as they say, is history. I've heard this discussed on a few podcasts so I knew the story. Here's a citation which I found with a quick internet search. It may not be the best source, but it'll get you started if you want to read more.

James Lind and Scurvy

-- JS

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

If the student were willing to make a rather deep dive, you might redirect their interest from sampling variation to uncertainty, and from a hypothesis test to an expanded uncertainty interval. Sampling variation is only one component of uncertainty. While the student is not in position to assess sampling variability, they might learn something from attempting to approximate the uncertainty associated with their measurements. I imagine your student is not up for the investment, but it's a suggestion.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Your link (to the Guide to the expression of uncertainty in measurement, aka the GUM) isn't really appropriate. The OP is interested in "what effect insulin has". One could estimate the magnitude of that effect and the uncertainty of that estimate. But it's not clear that such an estimate is a measurement in the sense of the GUM, or that one could using the techniques of the GUM to assess its uncertainty . $\endgroup$ – user20637 Apr 28 at 19:08
0
$\begingroup$

A major problem is the small sample size reducing the degrees of freedom in model selection along with the model's required/sensitivity to normality of error assumption. Preserving degrees of freedom and being robust in methodology appears likely to be the best path. I would even advise generating random errors from possible parent distributions, and with knowledge of the actual parameter values, noting the variation in estimated parameter values and possible changes in test results.

As such, a simple parsimonous model approach would be first to place the data in a regression format in accord with the following reduced model in the variable Methods:

$$ Y_{i,j}-Ymedian = \beta *InsulinDummy_i + \gamma * MethodDummy_j + \varepsilon_{i,j} $$

where the dependent variable is the observed concentration of glucose centered around the population median, and the Insulin Dummy variable (also centered) is 1/2 if Insulin is present in test sample i, else -1/2. The Method Dummy variable is 2/3 for Method 1, else -1/3 for Methods 2&3 (repeat analysis, swapping out Method 1 for say Method 2, and repeat again swapping out Method 2 for Method 3).

Note, the proposed model interpretation of the regression coefficients is that it may aid in accurately determining which side of the median an observation will fall. Given the small sample size, I suggest a probabilistic (even Bayesian) interpretation, whose accuracy can be assessed in simulated model testing.

Next, the introduction of a robust regression analysis, where Least Absolute Deviations (LAD) is an option. Mathematically, LAD is linked to a Laplace distribution of error terms. One can compute coefficients employing iterative weighted Least-Squares, or, especially in the current context with 6 data points, employing the property that the model parameters determine a straight line that passes through two of the observed points in space. This implies examining permutations and testing total sum of absolute deviations. The selected points nearly always avoid outliers (unlike Least-Squares, where ANOVA also rests on a squared error criterion).

To obtain confidence intervals on parameters, bootstrap re-sampling of error terms has been suggested (see this), which can also be assessed on accuracy in simulation runs.

[EDIT] I thought my model is worthy of further exploration, so I built a worksheet based simulation model (convenient for the iterative LAD iteration, which involves examining point shifting, what points absolute errors are converging to zero (indicative of point pairs determining the LAD regression line). Here is a summary of a dozen simulation runs based on a uniform (-0.5 to +.5) error added to the model proposed above.

Actual Underlying Simulated Parameter Values are: 1.250 and 0.100

Simulation Run Values:

Average Observed Values 1.225 0.026

Observed Median 1.224 0.045

Run 1 1.001 0.324

Run 2 1.546 0.297

Run 3 1.350 -0.038

Run 4 1.283 -0.115

Run 5 1.593 -0.113

Run 6 1.498 -0.089

Run 7 0.863 0.151

Run 8 1.090 0.323

Run 9 1.102 -0.435

Run 10 1.166 -0.265

Run 11 1.451 0.128

Run 12 0.761 0.146

My take on the results are that the obtained summary statistics are amazing for my proposed parsimonous model based on 6 points with a uniform error distribution estimating 2 parameters on a data-centered model employing robust regression. Individual runs display, as expected, quite a range on the parameter values, but appear to more likely point to an effect greater than 1 for the first parameter (only 2 out of 12 are less than 1).

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

While the student does not have type A repeatability measurements, the student may/should be able to estimate the type B error contribution caused by equipment supplied from elsewhere ("For an estimate xi of an input quantity Xi that has not been obtained from repeated observations").

This is detailed in the SI/bipm Guide to Uncertainty in Measurement (there's a NIST equivalent).

This at least allows a route to making some judgement about the results.

The alternative, if the student did have a time series measurement (mentioned in one of the comments) is to estimate the smooth curve shape and hence the measurement error on top of that underlying smooth shape.

And lastly, if all the control groups were actually the same (not clear from the comments) then they could form a single group for the estimation of measurement noise.

Finally, use this as a 'post-mortem' to identify the level of measurement accuracy that would have been required to confirm the hypothesis at risk, and hence the likely number of repeat measurements needed to obtain that accuracy (error in the mean), given particular levels of basic accuracy (error on a single measurement). This at least rescues the student from feeling like it was a complete waste (i.e something learnt!).

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

What a good example of the old question of bias and random errors in observational errors.

If the biased estimation of the standard deviation is, as you mention:

    $ \sigma = \sqrt{\frac{\sum{(x_i-\bar{x})^2}}{n}} = \frac {0}{1}=0$,

the unbiased estimation is

    $ \sigma = \sqrt{\frac{\sum{(x_i-\bar{x})^2}}{n-1}} = \frac {0}{0}=undefined$.

So if even you student succeed in drawing some statistical conclusions, these will have an unknown bias.

However, this did not prevent Student to design the t-test, and Fisher to design the ANOVA method for such situations.

What about starting by drawing the three pairs on a scatter-plot, then a linear regression and look at the slope and compare with its standard error?

This is tantamount as BruceET answer, perhaps a bit more geometric and intuitive.

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.