I want to clarify my understanding of using sample weights when comparing means of sub-groups of a sample.

Here is an artificial case with data that can be replicated in r (see code below):

Let’s assume we have a sample of 1,000 respondents, the sample is comprised of two groups “a” and “b” which represent customers of different companies. Each subgroup has 500 observations. Each customer has a gender which takes either the value 0 or 1. Group “a” has more respondents with gender “1”. Group “b” has an almost equal amount of gender 0 and gender 1. Finally, each respondent / customer has a score value ranging between -100 and 100 (the example here represents the net promoter score, but the score value could take any other value range without changing the question at hand).

An unweighted mean comparison of the customer satisfaction value shows

group       mean
a       -2.4
b       -1.5

Let’s further assume that the population is 10,000 people 5,000 with each gender (0/1).

There are basically two ways to make the sample representative of the population.

A variable weight1 can be calculated for the whole sample, making the whole sample representative of the population regarding gender. (Exact calculations for the sample data are provided in the r code below).

In this case the weighted mean1 would look like this:

group       w.mean1
a       -3.1
b       -0.5

Another approach would be to calculate a variable weight2, making each subgroup representative of the population regarding gender.

In this case the weighted mean2 would look like this:

group       w.mean2
a       -4.1
b       -1.4

My understanding is that: If our research question is to estimate how satisfied the people in the population are with each company (a / b) we need to compare the group means weighted by weight2 (which makes each subgroup representative of the population as if we had drawn two separate samples). Using weight1 would yield biased results.

If we were only interested in the question, how satisfied people in the population are with companies like a and b in general (not looking at the group differences / group means) then weight1 would be the appropriate choice.

I usually felt quite sure about my understanding of this issue, but a major market research company uses weight1 when comparing group means, and I really do not understand the reason why they would use biased weights. Maybe my understanding of sample weights is wrong after all.

I appreciate any comments on the issue.

R code:

# set-up
rtnorm <- function(n, mean, sd, a = -Inf, b = Inf){
  qnorm(runif(n, pnorm(a, mean, sd), pnorm(b, mean, sd)), mean, sd)

# generate group a
group  <- sample(letters[1], 500, TRUE) # group indicator
gender <- sample(0:1, size = 500, replace = TRUE, prob = c(.20,.80)) # gender

a = data.frame(group, gender)

# generate group b
group  <- sample(letters[2], 500, TRUE) # group indicator
gender <- sample(0:1, size = 500, replace = TRUE, prob = c(.50,.50)) # gender

b = data.frame(group, gender)

# generate data frame
data <- rbind(a, b)

# add score value
data$score  <- round(runif(1000, min=-100, max=100), 0) # some satisfaction value ranging from -100 to 100

# clear-up

## assumption: population total is 10,000 with 5,000 gender = 1 and 5,000 gender 0
N <- 10000
K <- 5000

# calculation of weights
# here the following formula is applied: w = n/k * K/N with
# w = weight
# n = total number of people in the sample
# k = number of people in the subgroup for which weight is calculated
# K = number of people in the subgroup of the population
# N = total number of people in the population

# weight1 makes the whole sample represetantive of the population
n  <- length(data$gender)
k1 <- length(data$gender)-sum(data$gender)
k2 <- sum(data$gender)

data$weight1[data$gender==0] <- n/k1 * K/N
data$weight1[data$gender==1] <- n/k2 * K/N

# weight2 makes each subgroup (a & b) representative of the population
n   <- length(data[ which(data$group=='a'),]$gender)
ka1 <- length(data[ which(data$group=='a'),]$gender)-sum(data[ which(data$group=='a'),]$gender)
ka2 <- sum(data[ which(data$group=='a'),]$gender)
kb1 <- length(data[ which(data$group=='b'),]$gender)-sum(data[ which(data$group=='b'),]$gender)
kb2 <- sum(data[ which(data$group=='b'),]$gender)

data$weight2[data$gender==0 & data$group=='a'] <- n/ka1 * K/N
data$weight2[data$gender==1 & data$group=='a'] <- n/ka2 * K/N
data$weight2[data$gender==0 & data$group=='b'] <- n/kb1 * K/N
data$weight2[data$gender==1 & data$group=='b'] <- n/kb2 * K/N

## control to check if mean of weights = 1
if (round(mean(data$weight1),5)!=1.0000) {
  warning(" calculation weight1 is wrong")

if ((round(mean(data[which(data$group=='a'),]$weight2),5)!=1.0000) | (round(mean(data[which(data$group=='b'),]$weight2),5)!=1.0000)) {
  warning(" calculation weight2 is wrong")

## mean comparison
dt <- setDT(data)[, j=list(mean=round((mean(score)),1),  
                           w.mean1=round((weighted.mean(score, w=weight1)),1),  
                           w.mean2=round((weighted.mean(score, w=weight2)),1), 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.