Skip to main content
Cross Validated

Return to Answer

added 25 characters in body
Source Link
jdcrossval
jdcrossval
  • 258
  • 2
  • 7

As one final note, I find the coworker's stated rationale very unconvincing (i.e., they want to "compare the sample to a hypothetical scenario where there would be an equal number of men and women in the population"). The only reason the coworker is interested in a hypothetical 50/50 population is because they think the true population is closer to 50/50 than to 70/30. I might be wrong, but if you asked your coworker why they think 50/50 is a more interesting hypothetical scenario than e.g. 5/95, presumably they would say "because 50/50 is closer to the true gender distribution." I mention this because it's important to be transparent in your presentation to the stakeholders. IMO, if your office thinks it's worth your time and effort to conduct and present a weighted analysis, then you should also explain to stakeholders how they should interpret that weighted analysis and what their takeaways should be. Calling the 50/50 population "hypothetical" while implying that it's more accurate than 70/30 could be misleading--particularly if the presentation contains no justification for selecting 50/50. That said, it shouldn't be too hard to justify 50/50--you could even just use the global population distributiona somewhat reasonable assumption since this is an area of genderresearch, as I understand. The

In short, the point is not to avoid making any assumption; the point is to be transparent about what assumption was made, what its limitations are, and how the weighted results should be interpreted.

As one final note, I find the coworker's stated rationale very unconvincing (i.e., they want to "compare the sample to a hypothetical scenario where there would be an equal number of men and women in the population"). The only reason the coworker is interested in a hypothetical 50/50 population is because they think the true population is closer to 50/50 than to 70/30. I might be wrong, but if you asked your coworker why they think 50/50 is a more interesting hypothetical scenario than e.g. 5/95, presumably they would say "because 50/50 is closer to the true gender distribution." I mention this because it's important to be transparent in your presentation to the stakeholders. IMO, if your office thinks it's worth your time and effort to conduct and present a weighted analysis, then you should also explain to stakeholders how they should interpret that weighted analysis and what their takeaways should be. Calling the 50/50 population "hypothetical" while implying that it's more accurate than 70/30 could be misleading--particularly if the presentation contains no justification for selecting 50/50. That said, it shouldn't be hard to justify 50/50--you could even just use the global population distribution of gender. The point is not to avoid making any assumption; the point is to be transparent about what assumption was made, what its limitations are, and how the weighted results should be interpreted.

As one final note, I find the coworker's stated rationale very unconvincing (i.e., they want to "compare the sample to a hypothetical scenario where there would be an equal number of men and women in the population"). The only reason the coworker is interested in a hypothetical 50/50 population is because they think the true population is closer to 50/50 than to 70/30. I might be wrong, but if you asked your coworker why they think 50/50 is a more interesting hypothetical scenario than e.g. 5/95, presumably they would say "because 50/50 is closer to the true gender distribution." I mention this because it's important to be transparent in your presentation to the stakeholders. IMO, if your office thinks it's worth your time and effort to conduct and present a weighted analysis, then you should also explain to stakeholders how they should interpret that weighted analysis and what their takeaways should be. Calling the 50/50 population "hypothetical" while implying that it's more accurate than 70/30 could be misleading--particularly if the presentation contains no justification for selecting 50/50. That said, it shouldn't be too hard to justify a somewhat reasonable assumption since this is an area of research, as I understand.

In short, the point is not to avoid making any assumption; the point is to be transparent about what assumption was made, what its limitations are, and how the weighted results should be interpreted.

added 465 characters in body
Source Link
jdcrossval
jdcrossval
  • 258
  • 2
  • 7

As one final note, I find the coworker's stated rationale very unconvincing (i.e., they want to "compare the sample to a hypothetical scenario where there would be an equal number of men and women in the population"). The only reason the coworker is interested in a hypothetical 50/50 population is because they think the true population is closer to 50/50 than to 70/30. I might be wrong, but if you asked your coworker why they think 50/50 is a more interesting hypothetical scenario than e.g. 5/95, presumably they would say "because 50/50 is closer to the true gender distribution." I mention this because it's important to be transparent in your presentation to the stakeholders. IfIMO, if your office is doingthinks it's worth your time and effort to conduct and present a weighted analysis, then you should also explain to stakeholders how they should interpret that weighted analysis and what their takeaways should be. Calling the 50/50 population "hypothetical" while implying that it's more accurate than 70/30 could be misleading--particularly if the presentation contains no justification for selecting 50/50. That said, it shouldn't be hard to justify 50/50--you could even just use the global population distribution of gender. The better approach IMOpoint is not to avoid making any assumption; the point is to be transparent about what assumption was made, what its limitations are, and how the weighted results should be interpreted.

If it were me, I would (1) estimate the true distribution (or alternatively the response rates), (2) justify that estimate based on the best available literature (which might still be pretty rough), (3) explain how that estimate mightcould still be wrong, (4) discuss how the weighted results roughly show the extent of uncertainty associated with possible gender imbalance, and then (5) describestate the other types of bias/other limitations that maylikely exist which are not corrected by the weights.

As one final note, I find the coworker's stated rationale very unconvincing (i.e., they want to "compare the sample to a hypothetical scenario where there would be an equal number of men and women in the population"). The only reason the coworker is interested in a hypothetical 50/50 population is because they think the true population is closer to 50/50 than to 70/30. I might be wrong, but if you asked your coworker why they think 50/50 is a more interesting hypothetical scenario than e.g. 5/95, presumably they would say "because 50/50 is closer to the true gender distribution." I mention this because it's important to be transparent in your presentation to the stakeholders. If your office is doing a weighted analysis, you should explain to stakeholders how they should interpret that weighted analysis and their takeaways should be. Calling the 50/50 population "hypothetical" while implying that it's more accurate could be misleading--particularly if the presentation contains no justification. The better approach IMO is to (1) estimate the true distribution (or alternatively the response rates), (2) justify that estimate based on literature, (3) explain how that estimate might still be wrong, (4) discuss how the weighted results roughly show the extent of uncertainty associated with possible gender imbalance, and then (5) describe the other types of bias/other limitations that may exist which are not corrected by the weights.

As one final note, I find the coworker's stated rationale very unconvincing (i.e., they want to "compare the sample to a hypothetical scenario where there would be an equal number of men and women in the population"). The only reason the coworker is interested in a hypothetical 50/50 population is because they think the true population is closer to 50/50 than to 70/30. I might be wrong, but if you asked your coworker why they think 50/50 is a more interesting hypothetical scenario than e.g. 5/95, presumably they would say "because 50/50 is closer to the true gender distribution." I mention this because it's important to be transparent in your presentation to the stakeholders. IMO, if your office thinks it's worth your time and effort to conduct and present a weighted analysis, then you should also explain to stakeholders how they should interpret that weighted analysis and what their takeaways should be. Calling the 50/50 population "hypothetical" while implying that it's more accurate than 70/30 could be misleading--particularly if the presentation contains no justification for selecting 50/50. That said, it shouldn't be hard to justify 50/50--you could even just use the global population distribution of gender. The point is not to avoid making any assumption; the point is to be transparent about what assumption was made, what its limitations are, and how the weighted results should be interpreted.

If it were me, I would (1) estimate the true distribution (or alternatively the response rates), (2) justify that estimate based on the best available literature (which might still be pretty rough), (3) explain how that estimate could still be wrong, (4) discuss how the weighted results roughly show the extent of uncertainty associated with possible gender imbalance, and then (5) state the other types of bias/other limitations that likely exist which are not corrected by the weights.

Source Link
jdcrossval
jdcrossval
  • 258
  • 2
  • 7

As a disclaimer, I don't have any experience with this situation, but it seems like there won't be a clear-cut answer.

There may be justification for a correction. I'm not very familiar with the literature, but I've heard there are studies showing that females have higher survey response rates than males. A little quick searching turned up this article, but you could probably find other resources as well: https://www.sciencedirect.com/science/article/pii/S2451958822000409#bib35

A study like this may provide reasonable justification for weights. For example, if females respond at twice the rate as males, then the weights could be 1 and 2 for females and males, respectively. However, before doing something like this, it would be worth evaluating whether the population in the study is similar to the population you're trying to capture. It's entirely possible that survey response tendencies differ from one population to the next.

Let's imagine you do use a study to calculate weights. If you have very good reason to believe the study population represents your target population (e.g., you find a study comparing response rates of each gender of tourists in your sampled area), then the weighted results will have less bias than the unweighted results. Weighting almost certainly won't remove all of the bias, but your question "why not 60% vs 40% instead?" would have the answer: "maybe 60/40 removes even more bias than 50/50, but the goal is to give some sense about roughly how big the bias is."

However, even with literature-based weights, I still wouldn't call the weighted results "corrected" because that seems speculative and potentially misleading. It would probably be safer to think of a weighted analysis as quantifying the degree of uncertainty in the results, and I might even present the unweighted analysis first as the main results, and the have a section at the end entitled "Uncertainty Analysis" or "Sensitivity Analysis" where you present the weighted results.

For example, if the weighted results agree closely with the unweighted results, then bias from possible gender imbalance may be small. On the other hand, if the two analyses yield very different results, then there is very high uncertainty in the results, and the two sets of results provide some insight into the magnitude of that uncertainty.

Of course, this weighting only removes one type of "nonresponse bias" or "undercoverage bias." It's entirely possible, both before and after adjusting the results with these weights, the results still fail to represent the broader population because of the self-selection bias that can arise in a convenience sample.

As one final note, I find the coworker's stated rationale very unconvincing (i.e., they want to "compare the sample to a hypothetical scenario where there would be an equal number of men and women in the population"). The only reason the coworker is interested in a hypothetical 50/50 population is because they think the true population is closer to 50/50 than to 70/30. I might be wrong, but if you asked your coworker why they think 50/50 is a more interesting hypothetical scenario than e.g. 5/95, presumably they would say "because 50/50 is closer to the true gender distribution." I mention this because it's important to be transparent in your presentation to the stakeholders. If your office is doing a weighted analysis, you should explain to stakeholders how they should interpret that weighted analysis and their takeaways should be. Calling the 50/50 population "hypothetical" while implying that it's more accurate could be misleading--particularly if the presentation contains no justification. The better approach IMO is to (1) estimate the true distribution (or alternatively the response rates), (2) justify that estimate based on literature, (3) explain how that estimate might still be wrong, (4) discuss how the weighted results roughly show the extent of uncertainty associated with possible gender imbalance, and then (5) describe the other types of bias/other limitations that may exist which are not corrected by the weights.