I am planning to analyse a survey. I have been told that if I clean the data (e.g. data subsetting, value recoding, creating new variables from existing etc.) and afterwards create a survey design object (svydesign function in "survey" package of R with id, strata, weights, fpc), I may get not correct point estimates and CI. I have been advised to make the survey design object first and then clean the data in that object. Can you please explain to me why this is necessary?

  • 1
    $\begingroup$ Can you explain more about what you are doing to 'clean' the data? Could you demonstrate with some code/outputs to show the difference between cleaning before and afterwards? $\endgroup$ – Izy May 31 '19 at 17:36
  • $\begingroup$ @Izy I have not started cleaning data yet. I am planning to work with NHANES data. I looked at some tutorials about survey package (like this: asdfree.com/…) and they where first making survey design object and after data manipulation. Based on that I asked question to one person and have received brief answer (we had not have many time) that as general rule it is better to first create design object and after any data manipulation should be done on that object in order to get correct calculations. My question is why it is true? $\endgroup$ – vasili111 May 31 '19 at 17:44
  • $\begingroup$ @Izy By cleaning data I mean taking subset of observations that I need, some value recoding, creating new variables from existing and etc. $\endgroup$ – vasili111 May 31 '19 at 17:45
  • $\begingroup$ I haven't used the survey package, but I suspect that if a difference exists it may be because the package is doing something to use information about the overall population/sample (e.g. the variance), whereas subsetting beforehand and just analysing the subset would discard the other data. Have you searched for any information in the package documentation to explain what it is doing? - if so please detail what you found (or where you looked and didn't find anything)? Try it both ways round with some dummy data and see what happens? $\endgroup$ – Izy May 31 '19 at 18:53

There are two separate issues here.

Sometimes, including with NHANES data, you do need to subset before defining the survey design object, because not all the records in the data set are part of the sample you are analysing. In NHANES, everyone in the data file will have a health questionnaire, but only a subset will have a clinical examination, and there may be smaller subsets with specific biochemical measurements. You need to remove records from the file that are not part of the sample you are analysing.

For example, I might use something like

nhanesmec <- subset(nhanes, !is.na(WTMEC2YR))

to analyse data from the clinical examination. Records with missing WTMEC2YR are not part of the MEC sample and so should not go into the survey design object.

On the other hand, if you have observations that are part of the sample, you should not remove them even if they have missing or implausible data, and you should not, eg, remove records for men if you want to do analyses only for women.

The reason is complicated, and makes almost no difference for NHANES. However, you asked.

Let's ignore stratified sampling for now and just consider cluster sampling. The survey was designed to sample a specific, preplanned number of clusters. When we're thinking about how different the results could be with a hypothetical replication of the survey [the frequentist definition of sampling uncertainty] we want to think about hypothetical replications that have the same preplanned number of clusters.

If you subset the data and end up removing one of the clusters, you no longer have the preplanned number of clusters. The number of clusters is now random, and you'd need to model the resulting variability due to number of clusters.

The computations to get the correct variances are equivalent to setting the weight to zero when you want to omit an observation, rather than just omitting it; we keep the number of clusters the same. If you look at the output of summary on a subsetted survey design object you can see the object keeps track of how many clusters (PSUs) it has data for (which is random), and also how many it started with (which is fixed). Using Anthony's example



dstrat_after<-svydesign(id=~1,strata=~stype, weights=~pw, data=apistrat, fpc=~fpc)
dstrat_after <- subset( dstrat_after , comp.imp == 'Yes' )

you will see as part of the output

Stratum Sizes: 
             E  H  M
obs         75 17 24
design.PSU 100 50 50
actual.PSU  75 17 24

If you subset before setting up the survey design object, there's no way for the object to know the planned number of observations or clusters, so there's no way for it to get the right standard errors.

And finally: this only makes a difference when your subset has fewer primary sampling units than the full sample. In Anthony's example the PSUs are individual records, so the subset does have fewer. In NHANES the PSUs are cities or counties, so you'd have to remove a lot of observations before you lost a PSU. Also, since the design has only two PSUs per stratum, if you did lose a PSU you'd have other problems with estimating standard errors and would have to look up survey.lonely.psu.

So, for NHANES it's unlikely to actually matter.


subsetting before creating the survey design will give you incorrect standard errors:



x <- apistrat

x <- subset( x , comp.imp == 'Yes' )

dstrat_before<-svydesign(id=~1,strata=~stype, weights=~pw, data=x, fpc=~fpc)

dstrat_after<-svydesign(id=~1,strata=~stype, weights=~pw, data=apistrat, fpc=~fpc)
dstrat_after <- subset( dstrat_after , comp.imp == 'Yes' )

SE( svymean( ~ api00 , dstrat_after ) )
SE( svymean( ~ api00 , dstrat_before ) )

Code from: https://gist.github.com/ajdamico/9b3232a1d986b3460baaa90f5fed3402

  • $\begingroup$ 1. Why this happens? 2. Only subsetting or other types of data cleaning too? 3. If answer on question 2 is "some" which one and why? $\endgroup$ – vasili111 May 31 '19 at 20:47
  • $\begingroup$ @ Anthony Damico, good to see the code. Can you answer the question as to 'why'? @ vasili111, From what I have read of the information for the package, it's because subsetting in your survey object retains information about your overall survey design that you lose if you subset before. I suspect the answer to 2. is 'many', and that to understand which you'll need to read up on how it takes that information into account. $\endgroup$ – Izy Jun 1 '19 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.