I work on project where each user has an IoT sensor and an ML algorithm tries to identify a given situation. The user carries the sensor 24/7 but I split the time into 'sessions'. For each session, I can calculate whether the ML model produced a True Positive, FP, FN or a TN.
I want to run an A/B test where I experiment with a new ML model for some users. So, while the unit of randomization is the user, the metrics are collected over each session (could be many within each day).
Offline, the model is trying to optimize Matthews correlation coefficient and I would like to be able to tell whether the new model performs better on that regard or not.
Since I'm not calculating some numeric metric (e.g. absolute error) at each session but rather a label, I don't know how I can approach A/B testing in this case.
I'm totally new to A/B testing, so any help or directions are appreciated.
EDIT: Assume I have 100 users in total each one having a sensor on him measuring, say, temperature. A users wears this sensor at 8 A.M. and, for whatever reason, puts it out at 11 A.M. This constitutes a session. For each session, the ML model tries to identify whether the user had fever (that's a made-up example). At the end of the session, the user informs us via an app whether he had or not fever, that's how I can label that session as a TP, FP, FN or TN. So, for each user I get multiple TPs, FPs etc.
As I described, I'm trying to optimize MCC which is a calculated based on the above 4 metrics. The thing is, MCC is only optimized when you look across sessions, not within each session.
If I update the ML model for some users, how should I approach that A/B test?