0
$\begingroup$

I am familiar with using ANOVA to analyze differences among group means. However, I am interested in analyzing differences for metrics other than the mean. Is anyone aware of a statistical test that can be used for this application?

Let's say that I am building a binary classification model to predict if individuals will buy a product. I want to test if a performance metric (e.g. AUC) for my model has significant differences across factor levels in the data. For example, I could test if my model's performance varies across different countries. Given that I have one metric value for each of N countries, how do I test if there is a significant difference among the N values?

One reason that I am struggling to find a solution is that the scope of my problem is very large. I am interested in detecting outliers among the N values, detecting if there is a multi-modal distribution, and also detecting if there is a large variance among the N observations. Also, I am looking to be able to generalize the solution across multiple factors. That is, I could want to partition model performance by country and gender.

My current thinking is that the solution depends on the value of N, as a statistical tests when N = 4 and when N = 100 would probably be very different. When N is large, I can use established tests for outliers and normality. However, when N is small, I am struggling to figure out what to do. Does anyone have any suggestions?

$\endgroup$

1 Answer 1

1
$\begingroup$

To me, this sounds like a problem where quantile regression could come in handy. The reason for this is that you need to compare distributions of model performance metrics (e.g., AUC) across levels of a factor such as country. To achieve this comparison, you could choose to compare certain quantiles of these distributions across the levels of this factor, if that is meaningful in your context.

When you compare a set of distributions, the shapes of those distributions would inform what features of the distributions are amenable for comparison. For example, if all distributions are unimodal and skewed, it would make sense to compare them in terms of center and spread by comparing their medians and interquartile ranges. The quantile regression would help estimate simultaneoulsy the median of each distribution (aka the 0.50th quantile), as well as the 0.25th and 0.75th quantiles, with the latter two being used to derive the interquartile range.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.