My set up is the following:

  • I have a group of people, initially whose gender I do not know.
  • I predict the gender of each individual with a gender prediction model and put them into Male and Female group. I know the accuracy/error of this model.
  • My test statistic is % of people that like product A. For example % of males that like product A vs. % of females that like product A.
  • I'd like to know if the difference is significant, and below is my null hypothesis:

H0: No difference between the two groups.

If I perform permutation test/t-test as it is, I'm only considering the sampling error, and completely ignoring the error from the model.

For example, if my numbers are 47% of males like product A vs. 44% of females like product A, with model error it's really 47% +/- 3.5% of males like product A vs. 44% +/- 4.5% females like product A.

What's the proper way to incorporate model error into significance testing?

  • $\begingroup$ There is always an error component when statistical analysis is needed. $\endgroup$ – Michael Chernick Mar 7 '17 at 0:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.