My set up is the following:
- I have a group of people, initially whose gender I do not know.
- I predict the gender of each individual with a gender prediction model and put them into Male and Female group. I know the accuracy/error of this model.
- My test statistic is % of people that like product A. For example % of males that like product A vs. % of females that like product A.
- I'd like to know if the difference is significant, and below is my null hypothesis:
H0: No difference between the two groups.
If I perform permutation test/t-test as it is, I'm only considering the sampling error, and completely ignoring the error from the model.
For example, if my numbers are 47% of males like product A vs. 44% of females like product A, with model error it's really 47% +/- 3.5% of males like product A vs. 44% +/- 4.5% females like product A.
What's the proper way to incorporate model error into significance testing?