10
$\begingroup$

I don't know much about stats so I'm looking for a starting point here. Any resources or insights would be helpful.

I'm conducting an e-learning experiment, in which students watch videos and then complete a survey which measures cognitive load and user satisfaction, and afterwards they complete a short assessment to test what they have learned.

Because of the issues with pre-testing in learning, I want to stratify the experiment groups by three vairables:

  1. Results of a previous examination
  2. Attitude towards their course (measured by survey)
  3. Attitude towards e-learning (measured by survey)

By doing this I can minimise variance between each group.

If I was stratifying by variable 1, I could ensure that each group had an equal amount of 'A' students, 'B' students etc.

However, I'm not sure how to fairly divide students into groups given that I have three variables to control for. I could just play around with the groups until I get them roughly even, but I was wondering if there are statistical methods for stratifying with multiple variables?

Thanks.

Improve this question
$\endgroup$
8
  • $\begingroup$ Hi there, how many categories do you have for each of your 3 stratification variables? There are options other than complete counterbalancing although they have cons. Having more information on the category sizes would be useful. Also, how many subjects do you think you can get? $\endgroup$
    – Michelle
    Feb 12, 2012 at 4:43
  • $\begingroup$ Is the survey mentioned in variables 2 and 3 a different survey from the one you want to stratify for? You can't stratify on the basis of a variable unless you know its distribution for the whole population, not just your sample. $\endgroup$
    – Peter Ellis
    Feb 12, 2012 at 5:34
  • 1
    $\begingroup$ Even if you could stratify, I think you have too many strata for your number of subjects. You would be better using those variables as predictors in some type of GLM, e.g. regression. $\endgroup$
    – Michelle
    Feb 12, 2012 at 19:36
  • 1
    $\begingroup$ I agree with Michelle. With only 60 subjects one stratum sounds about right (and you have the advantage, presumably, of knowing the grades of everyone in the population!), and I wouldn't try to weight based on the other survey results so much as use model based estimation to improve your total estimates. $\endgroup$
    – Peter Ellis
    Feb 12, 2012 at 19:53
  • 1
    $\begingroup$ After, although the more thinking you've done before the experiment the more likely you are to gather the right data. While the analysis is after the fact, you obviously can't do it if you find you've neglected to collect some crucial bit of info. $\endgroup$
    – Peter Ellis
    Feb 12, 2012 at 23:12

1 Answer 1

Reset to default
4
$\begingroup$

See my comment above re whether variables 2 and 3 really can be used as a basis for stratification (they can't unless the survey you refer to there is a different survey to the one you are discussing the sampling method for now).

If you try to select your sample based on three categorical variables you quickly end up with a lot of strata and complex sampling and weighting problems. You would need to calculate the population in each cell of a three dimensional array, where each cell is a particular combination of the three variables; then specify a proportion of that population you are going to include in your survey (doesn't need to be the same proportion for each cell). You also need to know each potential samplee's values on those three variables as part of your sample selection process.

An alternative to using all three for sampling might be to select your sample on the basis of just one of your variables as strata, and bring the other two in through post-stratification weighting. Further, if you use the raking technique you can get around the problem of so many "cells" in your population array, while still making sure that the weights for each total category of each variable (ie the marginal totals in your three dimensional array) add up to the correct amount, and this can help in keeping the standard errors down to a reasonable size.

If you're doing post-stratification (raking or otherwise) you still need to know the population values for your categorical variables - essential for calculating the right weights.

If I'm right in my suspicion that you don't really know the population values of your variables 2 and 3 (which need to be measured by survey), your best bet will be just to stratify on the basis of previous examination results, and then calculate weights to population based just on that variable.

I've found Thomas Lumley's survey package for R relatively straightforward to use and it has the advantage of being free. I would say this or something equivalent is essential for decent survey analysis. It has a good website and an even better book - you probably need to get hold of the book or an equivalent for all this to make sense

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.