Is it valid to compute survey weights using the distribution of a hypothetical/"imaginary" population?

Question

I'm informally reviewing a study by a coworker, but as I'm not a specialist of survey weighting procedures, I need a second opinion about something I disagree with him.

He collected a convenience sample, from a (human) population on which we don't know much – except that this is a population we cannot treat as a finite population. He assigned survey weights to respondents, correcting on the variable "gender". He assumed that the distribution of men and women in the population of interest was (0.5, 0.5), while in the sample there are about 70% women. There are about 500 respondents.

However, his assumption of 50% men and 50% is based on a wild guess, as there's no existing data on this population of interest.

He wants to crosstabulate the variable gender against some others (e.g., variables with 2 or more categories), and then conduct chi-square tests of independence corrected for the weights he computed. He also wants to compute some other statistics based on these weights (e.g., the corrected average age).

His justification for this procedure is that he wants to compare the sample to a hypothetical scenario where there would be an equal number of men and women in the population. I suggested that maybe we wanted to test the hypothesis that the sample was coming from a population where there's a (0.5,0.5) distribution (using a goodness-of-fit test), but that's not what he wants.

He wants to present the results from this procedure to stakeholders who requested the study. He plans to present both the "corrected" and "non corrected" results. To me, as nothing really justifies a scenario of 50% men/50% women (for example, why not 60% vs 40% instead?), I feel that this would bring more confusion than information.

He insists that his procedure is valid, so now I'm wondering if he might have a point. So... does he? Is there any reference on such a procedure?

---

Here is some additional information, as requested in comments:

The population is comprised of tourists in the small towns of a quite touristy area in a European country (about 30 millions of tourists per year in this area). Unfortunately even available data about tourists in general in the country (or in this area) offer no information relative to gender (or other socio-economic characteristics for that matter, barring their country/continent of origin). Even if such data was available, I'd be wary about treating tourists in small towns of this area as being like tourists in general (they might even differ from tourists in larger cities in the same area!).

---

Following another remark in the comment section: I agree it's difficult or impossible to make inference from a convenience sample in the first place - so the question of using this weighting procedure with "imaginary" data may be secondary or irrelevant. However, my worry is that we could try this weighting procedure with many different hypothetical distributions, so why choosing specificallly 50/50%? I have serious doubts about how informative it is (I think there's a risk of involuntarily misleading the stakeholders, by having them thinking that the "corrected" data are more reliable than the "non corrected" data), but I'm also wondering if I'm being overcautious here or if I'm missing something.

Answer 1

As a disclaimer, I don't have any experience with this situation, but it seems like there won't be a clear-cut answer.

There may be justification for a correction. I'm not very familiar with the literature, but I've heard there are studies showing that females have higher survey response rates than males. A little quick searching turned up this article, but you could probably find other resources as well: https://www.sciencedirect.com/science/article/pii/S2451958822000409#bib35

A study like this may provide reasonable justification for weights. For example, if females respond at twice the rate as males, then the weights could be 1 and 2 for females and males, respectively. However, before doing something like this, it would be worth evaluating whether the population in the study is similar to the population you're trying to capture. It's entirely possible that survey response tendencies differ from one population to the next.

Let's imagine you do use a study to calculate weights. If you have very good reason to believe the study population represents your target population (e.g., you find a study comparing response rates of each gender of tourists in your sampled area), then the weighted results will have less bias than the unweighted results. Weighting almost certainly won't remove all of the bias, but your question "why not 60% vs 40% instead?" would have the answer: "maybe 60/40 removes even more bias than 50/50, but the goal is to give some sense about roughly how big the bias is."

However, even with literature-based weights, I still wouldn't call the weighted results "corrected" because that seems speculative and potentially misleading. It would probably be safer to think of a weighted analysis as quantifying the degree of uncertainty in the results, and I might even present the unweighted analysis first as the main results, and the have a section at the end entitled "Uncertainty Analysis" or "Sensitivity Analysis" where you present the weighted results.

For example, if the weighted results agree closely with the unweighted results, then bias from possible gender imbalance may be small. On the other hand, if the two analyses yield very different results, then there is very high uncertainty in the results, and the two sets of results provide some insight into the magnitude of that uncertainty.

Of course, this weighting only removes one type of "nonresponse bias" or "undercoverage bias." It's entirely possible, both before and after adjusting the results with these weights, the results still fail to represent the broader population because of the self-selection bias that can arise in a convenience sample.

As one final note, I find the coworker's stated rationale very unconvincing (i.e., they want to "compare the sample to a hypothetical scenario where there would be an equal number of men and women in the population"). The only reason the coworker is interested in a hypothetical 50/50 population is because they think the true population is closer to 50/50 than to 70/30. I might be wrong, but if you asked your coworker why they think 50/50 is a more interesting hypothetical scenario than e.g. 5/95, presumably they would say "because 50/50 is closer to the true gender distribution." I mention this because it's important to be transparent in your presentation to the stakeholders. IMO, if your office thinks it's worth your time and effort to conduct and present a weighted analysis, then you should also explain to stakeholders how they should interpret that weighted analysis and what their takeaways should be. Calling the 50/50 population "hypothetical" while implying that it's more accurate than 70/30 could be misleading--particularly if the presentation contains no justification for selecting 50/50. That said, it shouldn't be too hard to justify a somewhat reasonable assumption since this is an area of research, as I understand.

In short, the point is not to avoid making any assumption; the point is to be transparent about what assumption was made, what its limitations are, and how the weighted results should be interpreted.

If it were me, I would (1) estimate the true distribution (or alternatively the response rates), (2) justify that estimate based on the best available literature (which might still be pretty rough), (3) explain how that estimate could still be wrong, (4) discuss how the weighted results roughly show the extent of uncertainty associated with possible gender imbalance, and then (5) state the other types of bias/other limitations that likely exist which are not corrected by the weights.