So I'm having an xgboost model "xgbm" which for a set of features, gives me a prediction between [0, 1]
xgbm(f1, ... fn) = [0, 1]
the model works fine and I measure its success with ROC AUC.
now if i'd wanted to check between different xgbm models, lets say xgbm1 vs xgbm2 the result I would get from the ROC AUC would be sufficient.
My problem is that for the same algorithm, xgbm I'd like to check different data sets. (my objective is to find the best constaint_x that will generate trainx/testx) which I will verify with my model.
which means that with constaint1 I will get a train1/test1 set for the xgbm and with constaint2 I will get a train2/test2 set for the xgbm.
But, alas. I cannot compare between the two because the ROC AUC is not good to compare between 2 different enviorments/datasets, but between different algorithms on the same data set.
Could anyone recommend a measurement that will help me measure which training/testing set (or which constaint) will produce the best result?
a little about the problem:
Its a ranking problem that I address as a 'binary:logistic'. the target is 0 for fail and 1 for success. the ratio on the events (dataset) between 1s and 0s is 0.08.
Please comment anything you need for me to clarify the problem for you.
