# Correct use of the sample weights in a complex survey design for association analysis (Logit OR)

I've doubts about the correct use of sample weights in the NHANES survey, which uses a complex, multistage probability sampling design (1). I'm aware about the importance of the use of the sample weight when the primary outcome of my study is the estimation of a certain prevalence (disease or condition) (2, 3). However, I'm not as confident about the approprietess of using sample weights for other purposes, i.e. association analyses (odds ratio by multiple adjusted logit regression). I read a ton of papers published on top tier journals regarding NHANES data, where the association between two or more conditions has been explored without taking into account the sample weights. For what I understand, when these data are used for cross-sectional association analysis, there is no need to refer to the overall (weighted) population. Indeed, using sample weights would artificially duplicate my observations with potential bias of the association estimate.
Could someone explain whether my conclusion is correct, and when and why use sample weights? Meanwhile, I found this answer from Dr James H. Watt to the same issue on ResearchGate (4):

• Speaking in general, weighted analyses are needed if you want to estimate an absolute prevalence of a mixture of subject types, mixed according to the true population. This is usually not want we need, and you can instead condition on the factors used in the oversampling instead. For example if a survey oversampled females, you can get sex-specific prevalence estimates with no weighting. Thinking of weighted estimates as providing unconditional estimates, which are only useful for global (one size fits all) policy decisions. – Frank Harrell Jun 2 at 11:37
• Thanks @FrankHarrell for your comment. I'm aware about the use of weighted analysis for the prevalence estimation. I'm not as confident about the appropriateness of using sample weights for association analyses. I would examine the association (in terms of Odds Ratio) between exposure to substance (yes/no) and presence of disease (yes/no). I should use weighted sample? – Borexino Jun 2 at 14:34
• I think that for association analysis it is not typical to need to use the weights, but there are exceptions. It depends more than anything on getting the model right. If there is a 3-way interaction that is depending on a variable that is used in weighting, and you want 2-way effects to be representative of the population, you may need to weight. – Frank Harrell Jun 2 at 15:13
• it's also less clear how to use survey weights for things like significance testing and standard errors. Additionally the weights often themselves include implicit linear regression models – probabilityislogic Jun 4 at 13:36
• you need to be clear on what you mean by "bias" and "variance" here - the sampling weights are unbiased with respect to the sample design (ie set of all possible samples). They do not necessarily translate into "unbiased/efficient" with respect to your assumed model – probabilityislogic Jun 5 at 3:34

I think the question you should focus on is the population you want to make inference about. The sample weights are for making estimates of population totals $$\hat{Y}=\sum_i w_i y_i$$. The weights are for getting from the sample to inference about a specific population. What you should think about is whether that is your population of interest.

So, if you wanted to fit say, a linear regression model you need to have all the population sums, sum of squares, and sum of cross products over the population. Using the survey weights gives you an estimate of these quantities.

You could even say that the "population log likelihood" is a sum as we have $$L_p(\theta) = \sum_i \log\left(f(y_i|\theta)\right)$$

For some log likelihood $$\log\left(f(y_i|\theta)\right)$$. Using the sample weights essentially provides an estimate of this quantity - taking $$y_i=\log\left(f(y_i|\theta)\right)$$ in the previous equation for $$\hat{Y}$$.

However, you are likely to run into problems with standard variance estimation in most modelling programs. The standard errors will be far too small. Conceptually it makes sense, if we consider these as estimates of standard errors we would get from fitting models to census data. That is, we expect $$L_P(\theta)$$ to be quite sharply peaked. But the problem is that we are using an estimate for $$\hat{L}_P(\theta)$$ and this estimate has error that needs to be taken account of. Usually there are jackknife/bootstrap weights provided with these kinds of files, and using the variation in these gives you a more reasonable estimate of uncertainty.

This is also a place where your "bayes, model based" / "frequentist, design based" philosophy matters somewhat because your variance estimates depend on what you are conditioning on as fixed. ie is the error from "the sample selected" or from "the predictive model"?

It is also not a bad idea to simply check if using or not using the weights makes a difference to your analysis, noting that you should expect a difference in accuracy measures but not necessarily parameter estimates (such as regression coefficients)