I have a very large data (100+ million records)and I am estimating a complex random coefficients model and I can only comfortably use a million or so records. Iam interested in details in time and space so I have sampled on these attributes so that each space-time cell has 10 records. Consequently some cells are overrepresented compared to the population and some are underrepresented. I also have several other variables -all of them categorical and because I can compare the obtained sample with the know frequencies in the population I can evaluate the under and over representation of these types too.

I want to use sampling weights to rebalance the sample as in the population on a number of attributes simultaneously. Can I use a model whether the dependent variable is in sample or not and predictor variables based on type to derive these weights?

Any help much appreciated.

  • $\begingroup$ How many attributes did you sample on? How many are going to be part of the model.? $\endgroup$ – Steve Samuels Nov 26 '15 at 17:30
  • $\begingroup$ I think I have at the minimum of 4 categories by 2 by 2 if I take interactions into account but this could increase considerably if I take into account time on a month by month bases as a main effect it would be 12 months by 12 year. So if I took into account all of this it would be with interactions 4 * 2 *2 *12 * 12 - but I have a lot of records! Thanks for answering. $\endgroup$ – user55193 Nov 27 '15 at 9:08
  • $\begingroup$ Describe the random coefficients model. From what you've described so far, your sampling method would make difficult any analysis. For example if you have repeated observations on individuals, so that coefficients vary with individual, I'd think that a valid sampling method would have to start with a sample of individuals. $\endgroup$ – Steve Samuels Nov 27 '15 at 21:15
  • $\begingroup$ I am quite comfortable with the problem as listed - the level 1 units are not repeatedly measured - they are however repeated measures of higher level units - but I think I know how to calculate higher level weights and how to use them. What i was looking for was help on the fixed effects ts and how to estimate suitable weights that will overcome the mis-representation of the level 1 units. Thank for taking the time to resdpond $\endgroup$ – user55193 Nov 28 '15 at 13:57
  • $\begingroup$ What was the sampling design for the higher level units? $\endgroup$ – Steve Samuels Nov 29 '15 at 20:37

I'll answer your question based on the information you've already provided. I don't see how these weights can be used in a random effects model, but that's not what you asked for.

1- Constructing sampling Weights

The cells in which you selected 10 observations each are known as "sampling strata". You have 4 x 2 x 2 x 12 x 12 = 2,304 of them. Number for convenience from $i = 1\ldots 2304$. You know $N_i$, the number of records in cell $i$. Then the probability of selection for records in cell $i$ is $f_i=\dfrac{10}{N_i}$, and the sampling weight is

$$ w_i = \frac{1}{f_i} = \frac{N_i}{10} $$

These are the "design" or "sampling" weights.

2- Balancing on other categorical variables known for the entire population.

The standard method for doing this is variously known as "raking", "ratio raking" or "sample balancing". with the first common in the survey literature. The technique is an application of interative proportional fitting (IPF). The commands take the design weights and alter them so them that the resulting weighted totals for categories match the known totals.

References: L Anderson and R Fricker "Raking: an important and often overlooked survey analysis tool" http://faculty.nps.edu/rdfricke/docs/RakingArticleV2.2.pdf

Battaglia, Michael P, David Izrael, David C Hoaglin, and Martin R Frankel. 2004. Tips and tricks for raking survey data (aka sample balancing). Abt Associates 4740-4744.



SAS macros



  • ipfraking (Stas Kolenikov) "findit ipfraking"

  • survgwt rake (Nicholas Winter) "ssc install survwgt"

R package survey (Tom Lumley) https://cran.r-project.org/web/packages/survey/index.html


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.