Non-random small sample - What conclusions can I get?

For my thesis I developed a psychotherapeutic method to address social competence in aggressive/withdrawn children.

I have obtained a sample of 10 subjects (not random, the professors chose the children by their behavior in the classroom) which I divided in two groups of 5 each (again not random, I couldn't choose, the children were from different schools and I couldn't, by any means, do a random selection).One of the groups were designed to be the experimental and the other control. I have applied to each children, parent and teacher a questionnaire to assess social problems before the application of the method and in the end.

Now my question is: My objective is not to generalize the results because this is an exploration study but to compare the results of each group in the two moments. Do the sampling problem and the little size of the sample compromise any results or can I do any tests? Is it possible to use parametric tests?

• The brutal answer is that no statistical tests apply because no randomization was performed and there are abundant powerful reasons to believe there are many confounding factors, which means any model-based efforts at statistical reasoning are almost surely just going to be wishful thinking. This does not mean your data are worthless, but it should help direct your analysis in other ways, focusing on exploring and describing the data and formulating hypotheses for future experiments to be conducted with more rigor. – whuber Sep 27 '13 at 15:37
• Having a before-and-after-intervention type of design overcomes some of the issues of confounding effects (since each subject is being compared with themselves, they're their own control), so you may be able to get some useful things out of it, but you need to make arguments that there should be no interaction with possible confounders. You can then set up a model for the response (I mean write a theoretical probabilistic model which includes an unspecified number of terms for counfounding effects) from which confounding effects disappear from the effects you're interested in measuring ... – Glen_b Sep 28 '13 at 0:53
• (ctd)... for example by taking differences of the before and after measurements, and writing a resulting probabilistic model for the differences. (In recent times, using differences tends to be avoided in such before and after studies, but it may actually be a good choice in this case.) -- That is, my advice is to explicitly mention the presence of confounders and show how an appropriate model and (perhaps) reasonable assumptions will deal with them by dint of the pairing, leaving you with the treatment effects plus noise. – Glen_b Sep 28 '13 at 1:01
• (ctd) ... there may still arguably be some possible confounders related to time, but that should be taken care of by the comparison with the control. So I don't think all is lost, but you must argue more carefully. – Glen_b Sep 28 '13 at 1:08
• Hi @Glen_b, I'm not a very experienced statistician, can you explain a more practical way of doing this? How can I write a probabilistic model from the differences? Can you get me an example or some resources? – Unkuiri Sep 28 '13 at 8:09

whuber is right that, technically, statistical inference will not be accurate if randomization was not used. However, in practice, random sampling is often impossible, so it is common (and inevitable and unfortunate) practice that people use inference on non-random samples and generalize the results to the entire population from which the sample was drawn.

The actual conclusions that you can confidently make is to simply describe the sample without generalizing the results to others in the population. For example, let's say group 1 had a higher average score on a certain test than group 2. You can conclude that, among the children you measured, group 1 scored higher on the test than group 2. In this case, you are only comparing the 5 children in group1 with the 5 in group 2, and inference is not used (i.e., p-vaues and confidence intervals would not make sense). Simply calculate descriptive statistics such as mean, median, population standard deviation, etc. to describe your data.

Keep in mind that you can still run various tests or calculate effect sizes without using inference. For example, you can calculate Cohen's d between group 1 and 2 on a test. You can use ANCOVA and find the mean difference of test score between the groups while controlling for the effect of age and gender. I think some people do not realize that things like ANOVA or multiple regression can be used for descriptive statistics.

• I have a big number of variables from the tests I've applied. Can I do ANCOVA tests for every variable to see wich ones have significant differences between groups and then compare the descriptive statistics of the significant ones using a graphical representation for example? – Unkuiri Sep 27 '13 at 18:19
• Sure, you can do it: nothing will prevent your software from running. But the significance levels and p-values will be utterly meaningless. – whuber Sep 27 '13 at 18:27
• This is a whole different topic, a pretty broad one. The answer will depend a lot based on the context. I'd be careful including more than 2 or 3 variables in one model since your n is so low. You have to be very thoughtful of which variable to include in statistical models. One thing certain is that "dump everything in and see what comes out interesting" approach is something you never want to do. Every time you run another test, you increase the chance that you made at least one incorrect conclusion. – Hotaka Sep 27 '13 at 18:32
• Thanks again. I don't want to try everything at random and see what comes up, I just want an easier way to explore the data, because my objective is only descriptive not inferential. So if I understood it, the best way is only to use descriptive statistics? Is there another quicker way to explore this big number of variables? – Unkuiri Sep 27 '13 at 18:46
• There's nothing wrong with using a lot of descriptive stats. But you should always have a purpose or question in mind when looking at data. Without some sort of guidance, there is literally an infinite ways you can look at a set of data. You will probably find many many relationships, most uninteresting, some uninterpretable, and others are useless. Asking your self specific questions and trying to get the data to answer them will make your huge data set a lot less overwhelming. Dont be afraid to not make use of some variables at all. – Hotaka Sep 27 '13 at 20:01

To expand on my comments, here's one approach. Someone more used to the area you're working in (and there are many here) may have a better suggestion for solving the same problems:

1) Assume that confounding variables are of two kinds.

i) The first kind (the main one) are counfounders which are always the same for a given subject. "School effects" and "teacher effects" and socio-economic variables, for example, may be reasonably assumed to be the same before and after for each subject

ii) The second kind (which may not exist for your problem) can change within subjects (these would be time-related things like 'learning effects' from having been tested before rather than from the intervention itself)

2) Assume no confounders interact with any of the effects you're interested in

A model that reflects that could be written as follows:

Let $i$ represent the subject, and let $t$ represent the time (0/1). Let $Y_{it}$ be the response for subject $i$ at times $t$. The variable $\text{Treatment}$ is $1$ for those in the treatment group and $0$ for the control

$\alpha_i$ incorporates all the individual-level counfounders above.

$\gamma$ incorporates any time-counfounders, including the effect of the first round of testing.

$\beta$ incorporates the treatments effects - the difference

$Y_{it} = \alpha_i + \gamma \cdot t + \beta \text{ Treatment}\cdot t +\varepsilon_{it}$

Normally with a model like this I'd be tempted to use mixed effects model with random intercept, but in this case you don't have randomization. Nonetheless, because of the before/after pairing, with the assumptions of no interaction of confounders with treatment you can tease out the treatment effect.

For example, If you take $D_i = Y_{i1}-Y_{i0}$, you get:

$Y_{i1} = \alpha_i + \gamma + \beta \text{ Treatment} +\varepsilon_{i1}$

$Y_{i0} = \alpha_i + 0 + 0 +\varepsilon_{i0}$

$D_i = \gamma + \beta \text{ Treatment} + \eta_i$

where $\eta_i =\varepsilon_{i1}-\varepsilon_{i0}$.

Then - assuming sample sizes are large enough, a straight two-sample test of equality of population means of the $D$'s between control and treatment should arguably pick up a treatment effect.

• It sounds very interesting to me, but is this a correct way to do the things giving this limited sample size? – Unkuiri Sep 29 '13 at 12:17
• How can I do this in practical terms like SPSS or R? (do you know of any tutorial or literature about that?) How can I test this confounding effects and take them into account? Sorry for my ignorance, I'm more or less new to statistics, I have only the basic notions. – Unkuiri Sep 29 '13 at 13:00
• I'm not used to this type of statistical operations. However I seem to understand it more or less. It's some kind of equation. What means "$\varepsilon_{i1}$"? – Unkuiri Sep 29 '13 at 13:14
• So if I understood well, to know the real treatment effect (difference in two moments without noise) we can do: $D_i = \gamma + \beta \text{ Treatment} + \eta_i <=> \beta \text{ Treatment} = D_i - \gamma - (\varepsilon_{i1}-\varepsilon_{i0})$ – Unkuiri Sep 29 '13 at 13:28
• Correct with small sample size? Well, sure if the assumptions hold. Possibly not very informative with small sample size, however, in that effect estimates will have large variability (so nothing is likely to be significant). You can fit models like this via regression, but in this particular case, as I already mentioned, you could look at a two-sample t-test. In R see ?t.test. You can't test the confounding effects. My post shows exactly how to remove their effect, which is 'taking them into account'... (ctd) – Glen_b Sep 29 '13 at 14:39

In this case, I would be much more interested in the individual story of the 10 kids than in any kind of statistics.

History: Statistics with small samples. Cite Mark Twain, "History doesn't often repeat itself, but it rhymes."

Roger Koenker, Dictionary of Received Ideas of Statistics