A/B testing for binary classification tasks

Question

I work on project where each user has an IoT sensor and an ML algorithm tries to identify a given situation. The user carries the sensor 24/7 but I split the time into 'sessions'. For each session, I can calculate whether the ML model produced a True Positive, FP, FN or a TN.

I want to run an A/B test where I experiment with a new ML model for some users. So, while the unit of randomization is the user, the metrics are collected over each session (could be many within each day).

Offline, the model is trying to optimize Matthews correlation coefficient and I would like to be able to tell whether the new model performs better on that regard or not.

Since I'm not calculating some numeric metric (e.g. absolute error) at each session but rather a label, I don't know how I can approach A/B testing in this case.

I'm totally new to A/B testing, so any help or directions are appreciated.

EDIT: Assume I have 100 users in total each one having a sensor on him measuring, say, temperature. A users wears this sensor at 8 A.M. and, for whatever reason, puts it out at 11 A.M. This constitutes a session. For each session, the ML model tries to identify whether the user had fever (that's a made-up example). At the end of the session, the user informs us via an app whether he had or not fever, that's how I can label that session as a TP, FP, FN or TN. So, for each user I get multiple TPs, FPs etc.

As I described, I'm trying to optimize MCC which is a calculated based on the above 4 metrics. The thing is, MCC is only optimized when you look across sessions, not within each session.

If I update the ML model for some users, how should I approach that A/B test?

Answer 1

It might be worthwhile to approach this problem as follows:

You have two groups with different treatments: A and B. Both groups have 100 individuals in them. After some session, you can calculate the TP, TN, FP, FN rates in both groups, which are in fact the constituent parts of your MCC. You could then simply choose to directly evaluate whether the A group's rates differ from the B group's rates, for example through a chi-squared test.

Example output could be something like:

   TP TN FP FN
A: 30 30 15 25
B: 35 31 11 23

I can imagine this gets at the hypothesis you are trying to test ('does some technique perform better than the other') without convoluting the setup by calculating the MCC. Of course, there are some complications (for example: what if TP_A > TP_B but TN_A < TN_B). Nonetheless, it seems like a logical avenue to pursue to first examine whether there are actually serious differences in the proportions before evaluating the MCC.

Answer 2

OK, interesting question. There is probably an easier way to do this, but here is a fairly simple way to go about this.

Let's assume you have 100 users, each of which may have multiple sessions. We're really interested in the difference in MCC between models. The difference in MCC is not really important, we maybe want evidence that one is superior to another.

If both models are used to predict on the same data, we can take the difference in MCCs. The difference is between -1 and 1, but we can rescale this to be between 0 and 1 using a min/max scaler.

Now, we need to run an appropriate regression. Since the data are bounded by the unit interval, a beta regression might work. The regression has no covariates, just an intercept, and we are interested in testing the null that the intercept is 0 because this is equivalent to testing if the rescaled difference is 0.5 (e.g. the expected difference in MCC is 0).

Here is an example in R. I've simulated some sessions for 100 users. Each user has a different number of sessions. The predictions between the two models are also simulated by making the outcomes correlated with the truth. Model A has a correlation of 0.5 with the truth, and model B has a correlation of 0.75 with the truth. This means that model B has the superior MCC in truth. Let's take a look at the data.

   users true_outcome predicted_outcome_a predicted_outcome_b
   <int>        <int>               <int>               <int>
 1     1            0                   0                   0
 2     1            1                   0                   0
 3     1            0                   0                   1
 4     2            1                   1                   1
 5     2            0                   0                   0
 6     2            0                   0                   0
 7     3            0                   1                   0
 8     3            0                   0                   0
 9     3            0                   0                   0
10     4            1                   0                   1

Here, each row is a session, and the sessions are considered iid conditional on users. I’ve just shown a sample for each user. In reality, we may have more observations per user, and each user may have a different number of observations. This is just intended to give you an idea of how the data is structured.

We can summarize this data by computing the MCC for each user as follows

  users mcc_a mcc_b     N
   <int> <dbl> <dbl> <int>
 1     1 0.342 0.451    32
 2     2 1     0.667    10
 3     3 0.557 0.823    47
 4     4 0.345 0.6      36
 5     5 0.580 0.861    50
 6     6 0.690 0.777    44
 7     7 0.349 0.535    45
 8     8 0.473 0.804    59
 9     9 0.686 0.750    62
10    10 0.785 0.908    32

Keeping the number of sessions is important since those should contribute more to the likelihood. Let's take the difference and put this in a column called delta. Now, we can run a regression by doing...

betareg(delta ~ 1, data=md, weights = N) %>% 
  summary

Call:
betareg(formula = delta ~ 1, data = md, weights = N)

Standardized weighted residuals 2:
     Min       1Q   Median       3Q      Max 
-10.9961  -3.8136   0.2949   2.9278  29.8707 

Coefficients (mean model with logit link):
             Estimate Std. Error z value Pr(>|z|)    
(Intercept) -0.464651   0.007456  -62.32   <2e-16 ***

Phi coefficients (precision model with identity link):
      Estimate Std. Error z value Pr(>|z|)    
(phi)  19.4514     0.4413   44.08   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

Type of estimator: ML (maximum likelihood)
Log-likelihood:  3023 on 2 Df
Number of iterations: 10 (BFGS) + 2 (Fisher scoring) 

As you can see, we reject the null for the intercept parameter, and the intercept is negative. Since the difference was MCC_a - MCC_b, this implies MCC_b was larger, which is correct.

I'm sure there is a simpler way to assess your model, but this is what I've thought up. Depending on how big the MCCs are, you could probably just get away with NOT doing a beta regression and instead doing a paired t-test. All depends on the context.

Code

library(tidyverse)
library(yardstick)
library(betareg)

users <- 1:100
sessions <- sample(10:64, size=length(users), replace=T)
p <- rbeta(length(users), 30, 80)

d<-tibble(
  users,
  sessions
) %>% 
  uncount(sessions) %>% 
  mutate(
    true_outcome = rbinom(n(), 1, p[users]),
    predicted_outcome_a = rbinom(
                                n(), 
                                1, 
                                true_outcome * (p[users] + 0.5*(1-p[users])) + (1-true_outcome) * (p[users]*(1-0.5))
    ),
    predicted_outcome_b = rbinom(
      n(), 
      1, 
      true_outcome * (p[users] + 0.75*(1-p[users])) + (1-true_outcome) * (p[users]*(1-0.75))
    )
  )


md<-d %>% 
  group_by(users) %>% 
  mutate_at(vars(true_outcome:predicted_outcome_b), factor) %>% 
  group_by(users) %>% 
  summarise(
    mcc_a = mcc_vec(true_outcome, predicted_outcome_a),
    mcc_b = mcc_vec(true_outcome, predicted_outcome_b),
    N = n()
  ) %>% 
  mutate(delta =((mcc_a - mcc_b) + 1)/2)


md %>% 
  arrange(desc(delta))


betareg(delta ~ 1, data=md, weights = N) %>% 
  confint

```