Conducting multiple weighted t.tests to count over what fraction of a population a model performs significantly better or worse than another

Question

I have two different model families $A_i$ and $B_i$ each of which predicts a continuous value for an item $i$ at various points in time throughout a given day. For a given day, and item $i$, I can measure the correlation of $A_i$'s $\hat{y}$ and $y$ as well as $B_i$'s. Given a dataset with daily such correlations for both $A$ and $B$ over multiple days for multiple items $i$ I want to test for how many of the $i$ items (if any) are "better predicted" by $B$ given some confidence threshold (say $0.01$).

As this involves running multiple tests, I am adjusting p-values at the end but often finding that with my approach I am hardly every seeing any differences. Am I being overly conservative in the adjustments or am I approaching this incorrectly?

Here is an example that illustrates my point

library(weights)
library(data.table)

set.seed(42)
items <- 1000
dates <- seq(as.Date("2022/01/01"), as.Date("2022/06/01"), by = "day")
x <- data.table(item = rep(1:items, each = length(dates)), date = dates)

## wgt is a weight for a specific date and item
## (e.g. number of measurements observed on that date for that item)
x[, wgt := round(rnorm(.N, 1000, 100))]

## on all but the last 5 items both models A and B perform the same..
x[item <= (items - 5), `:=`(mod.a = rnorm(.N, 0.2, 0.05),
                            mod.b = rnorm(.N, 0.2, 0.05))]

## but B is better on 5 of the items
x[item > (items - 5), `:=`(mod.a = rnorm(.N, 0.19, 0.05),
                           mod.b = rnorm(.N, 0.21, 0.05))]

## we dont know that and we want to test for the effect
## so we reach for weights::wtd.t.test
y <- x[, {
  wtt <- wtd.t.test(mod.a, mod.b, weight = wgt, samedata = TRUE, alternative = "two.tailed")
  c(as.list(wtt$coefficients), as.list(wtt$additional))
}, by = item]

## we did multiple tests so adjust p-values
y[, p.value.adj := p.adjust(p.value, method = 'bonferroni')]

## with unadjusted p.values we get a lot of false positives
y[p.value < 0.01,     .(num.diff = .N, which.diff = paste(item, collapse = ','))]
##    num.diff                                                     which.diff
## 1:       16 9,405,411,597,665,666,691,747,882,901,937,996,997,998,999,1000

## but with adjusted p-values we only detect one of them
y[p.value.adj < 0.01, .(num.diff = .N, which.diff = paste(item, collapse = ','))]
##    num.diff which.diff
## 1:        1       1000

Answer 1

It's hard to see how the proposed method compares the performance of two models A and B in a meaningful way.

Let's start with your performance metric. Correlation is invariant under linear transformations, i.e., $\operatorname{Cor}\{Y,\hat{Y}\} = \operatorname{Cor}\{Y,a+b\hat{Y}\} $. This means that one or both of the models can be mis-calibrated and your chosen metric won't pick up on it.

There are also pitfalls in using null hypothesis significance testing to decide what it means for one model to perform better than another.

One weakness is lack of interpretability. Let's consider a single item first. Do you know what a p-value of p < 0.01 implies for the difference in performance between A and B? Or to turn the question around, do you know what kind of differences in performance between A and B will be detected with a t-test on correlations?

Let's modify your own example to show that the t-test is not sensitive to certain patterns of differences in model performance that might be very meaningful to detect.

compare_model_performance <- function(days) {

  # Model A consistently scores 0.5.
  # Model B scores 0.3 half of the time, 0.7 - the other half.
  # If there are odd number of days, on the final day B also scores 0.5.

  half <- days %/% 2

  A <- rep(0.5, times = days)
  B <- rep(c(0.3, 0.7, 0.5), times = c(half, half, days %% 2))

  t.test(A, B)$p.value
}

# No matter how many days we monitor A and B for,
# the p-value of the t-test is always 1.

compare_model_performance(2)
#> [1] 1
compare_model_performance(30)
#> [1] 1
compare_model_performance(365)
#> [1] 1

And finally, is it useful to report what fraction of a population a model performs better or worse than another. Can you choose between A and B based on this information? It will be more helpful to know how often A performs better, how often B performs better, and how often there is no practical difference. You can only estimate this if you come up with an explicit definition of "better", "worse" and "no practical difference". This is an important step that you shouldn't delegate to the p-value of a t-test. No matter how you decide to adjust the p-values for making multiple comparisons, with the NHST approach the conclusion about how methods A and B perform on item i depends on how many items altogether you assess.