Non-random small sample - What conclusions can I get? For my thesis I developed a psychotherapeutic method to address social competence in aggressive/withdrawn children.
I have obtained a sample of 10 subjects (not random, the professors chose the children by their behavior in the classroom) which I divided in two groups of 5 each (again not random, I couldn't choose, the children were from different schools and I couldn't, by any means, do a random selection).One of the groups were designed to be the experimental and the other control. I have applied to each children, parent and teacher a questionnaire to assess social problems before the application of the method and in the end.
Now my question is: My objective is not to generalize the results because this is an exploration study but to compare the results of each group in the two moments. Do the sampling problem and the little size of the sample compromise any results or can I do any tests? Is it possible to use parametric tests?
Thanks in advance
 A: whuber is right that, technically, statistical inference will not be accurate if randomization was not used. However, in practice, random sampling is often impossible, so it is common (and inevitable and unfortunate) practice that people use inference on non-random samples and generalize the results to the entire population from which the sample was drawn.
The actual conclusions that you can confidently make is to simply describe the sample without generalizing the results to others in the population. For example, let's say group 1 had a higher average score on a certain test than group 2. You can conclude that, among the children you measured, group 1 scored higher on the test than group 2. In this case, you are only comparing the 5 children in group1 with the 5 in group 2, and inference is not used (i.e., p-vaues and confidence intervals would not make sense). Simply calculate descriptive statistics such as mean, median, population standard deviation, etc. to describe your data.
Keep in mind that you can still run various tests or calculate effect sizes without using inference. For example, you can calculate Cohen's d between group 1 and 2 on a test. You can use ANCOVA and find the mean difference of test score between the groups while controlling for the effect of age and gender. I think some people do not realize that things like ANOVA or multiple regression can be used for descriptive statistics.
A: To expand on my comments, here's one approach. Someone more used to the area you're working in (and there are many here) may have a better suggestion for solving the same problems:
1) Assume that confounding variables are of two kinds. 
i) The first kind (the main one) are counfounders which are always the same for a given subject. "School effects" and "teacher effects" and socio-economic variables, for example, may be reasonably assumed to be the same before and after for each subject
ii) The second kind (which may not exist for your problem) can change within subjects (these would be time-related things like 'learning effects' from having been tested before rather than from the intervention itself)
2) Assume no confounders interact with any of the effects you're interested in 
A model that reflects that could be written as follows:
Let $i$ represent the subject, and let $t$ represent the time (0/1). Let $Y_{it}$ be the response for subject $i$ at times $t$. The variable $\text{Treatment}$ is $1$ for those in the treatment group and $0$ for the control
$\alpha_i$ incorporates all the individual-level counfounders above.
$\gamma$ incorporates any time-counfounders, including the effect of the first round of testing.
$\beta$ incorporates the treatments effects - the difference 
$Y_{it} = \alpha_i + \gamma \cdot t + \beta \text{ Treatment}\cdot t  +\varepsilon_{it}$
Normally with a model like this I'd be tempted to use mixed effects model with random intercept, but in this case you don't have randomization. Nonetheless, because of the before/after pairing, with the assumptions of no interaction of confounders with treatment you can tease out the treatment effect. 
For example, If you take $D_i = Y_{i1}-Y_{i0}$, you get:
$Y_{i1} = \alpha_i + \gamma + \beta \text{ Treatment}  +\varepsilon_{i1}$
$Y_{i0} = \alpha_i + 0 + 0  +\varepsilon_{i0}$
$D_i = \gamma + \beta \text{ Treatment} + \eta_i$
where $\eta_i =\varepsilon_{i1}-\varepsilon_{i0}$.
Then - assuming sample sizes are large enough, a straight two-sample test of equality of population means of the $D$'s between control and treatment should arguably pick up a treatment effect.
A: In this case, I would be much more interested in the individual story of the 10 kids than in any kind of statistics.

History: Statistics with small samples. Cite Mark Twain, "History doesn't often repeat itself, but it rhymes."
Roger Koenker, Dictionary of Received Ideas of Statistics

