I work on project where each user has an IoT sensor and an ML algorithm tries to identify a given situation. The user carries the sensor 24/7 but I split the time into 'sessions'. For each session, I can calculate whether the ML model produced a True Positive, FP, FN or a TN.

I want to run an A/B test where I experiment with a new ML model for some users. So, while the unit of randomization is the user, the metrics are collected over each session (could be many within each day).

Offline, the model is trying to optimize Matthews correlation coefficient and I would like to be able to tell whether the new model performs better on that regard or not.

Since I'm not calculating some numeric metric (e.g. absolute error) at each session but rather a label, I don't know how I can approach A/B testing in this case.

I'm totally new to A/B testing, so any help or directions are appreciated.

EDIT: Assume I have 100 users in total each one having a sensor on him measuring, say, temperature. A users wears this sensor at 8 A.M. and, for whatever reason, puts it out at 11 A.M. This constitutes a session. For each session, the ML model tries to identify whether the user had fever (that's a made-up example). At the end of the session, the user informs us via an app whether he had or not fever, that's how I can label that session as a TP, FP, FN or TN. So, for each user I get multiple TPs, FPs etc.

As I described, I'm trying to optimize MCC which is a calculated based on the above 4 metrics. The thing is, MCC is only optimized when you look across sessions, not within each session.

If I update the ML model for some users, how should I approach that A/B test?

  • $\begingroup$ You are calculating metrics, it seems that you want to test change in MCC, aren't you? $\endgroup$ – Tim Mar 26 '19 at 15:43
  • $\begingroup$ MCC is calculated based on the TPs, FPs etc. I can only calculate mcc AFTER I get data from many sessions, not just 1 session. $\endgroup$ – Stergios Mar 26 '19 at 15:49
  • $\begingroup$ So for the 2 groups (new vs. old model), I can calculate the MCC but how do I run the hypotheses testing? I won't have an avg. or standard deviation of that metric unless I calculate it for every day (which is an arbitrary time threshold) which will make my metric less stable. $\endgroup$ – Stergios Mar 26 '19 at 15:51
  • $\begingroup$ I am afraid that in I'm not following your description. Could you give us example of your data? $\endgroup$ – Tim Mar 26 '19 at 16:15
  • 1
    $\begingroup$ Yes, but I think you should use pigeonhole bootstrap and sample both users and session (see Owen (2007) for the paper with the same name). $\endgroup$ – usεr11852 Apr 1 '19 at 22:32

It might be worthwhile to approach this problem as follows:

You have two groups with different treatments: A and B. Both groups have 100 individuals in them. After some session, you can calculate the TP, TN, FP, FN rates in both groups, which are in fact the constituent parts of your MCC. You could then simply choose to directly evaluate whether the A group's rates differ from the B group's rates, for example through a chi-squared test.

Example output could be something like:

A: 30 30 15 25
B: 35 31 11 23

I can imagine this gets at the hypothesis you are trying to test ('does some technique perform better than the other') without convoluting the setup by calculating the MCC. Of course, there are some complications (for example: what if TP_A > TP_B but TN_A < TN_B). Nonetheless, it seems like a logical avenue to pursue to first examine whether there are actually serious differences in the proportions before evaluating the MCC.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.