Measuring how effective training is

After 15 years post uni formally not using stats I find myself covering for someone at work for the next 6 months and it is rather stats heavy. The memory is very hazy but the mind is willing.

I have to test how effective a training technique has been in my unit to see if it will be rolled out across the organisation. I have before and after data for measurable outcomes (both experimental group (n=2000) and control group (n=650) as well as some demographic info such as age, gender, time in service.

I started using some variations of t-tests to compare means but now find myself at a loss at how to see if the areas like age/gender etc are having an impact. I am looking for help! I don't know what type of analysis to run or even if the t tests were the right thing to do! One of the other guys in the unit mentioned hierarchical regression but I don't know what that is or how to do it.

Any help would be appreciated.

Thanks

• With such a large sample size for that type of problem, I wouldn't advise inferential statistics at all. Any difference that would be practically meaningful would certainly be statistically significant. Therefore, I would rely on descriptive statistics and graphs. Jun 8, 2017 at 2:08

You can use an unpaired t test to check if there is a significant difference in the measurable outcome between the experimental and control group.

A paired t test is used when we measure the outcome on the same subject before and after treatment.

Null Hypothesis: There is no difference in means of the measurable outcome.

Alternate Hypothesis: There is a difference between means of the measurable outcome.

The steps to calculate the t test statistic is given here. http://www.statsdirect.com/help/parametric_methods/utt.htm

Find the 95% confidence interval for the t distribution at the degrees of freedom. If your test statistic lies beyond the confidence interval, you can reject the null hypothesis.

This means that the chance of the test statistic lying in the confidence interval when the actual difference in means is zero is very small, thus leading us to conclude that zero is not the difference in means of the 2 groups and rejecting the null hypothesis.

All of the above steps can be done in an automated way if you have access to tools like SPSS or SAS.

• You lost me at step 1: could you please explain the point of "picking n observations" from each of the groups and how those observations are supposed to be selected? What is the point of not using all the data? And how does your procedure control for the other variables mentioned in the question?
– whuber
Jun 7, 2017 at 21:25

In order to check whether age/gender have an effect:

1. Use a one way anova when your independent variable has two levels (ex: gender - male or female)

2. Use a 2 way anova when your independent variable has more than 2 levels (ex: income ranges - less than 10k, 10k-20k, 20k-30k etc.)