Someone built an XGBoost classification model using each pixel in an image (256*256) as a separate feature, plus a few other features. However they only have 500 data points. The target classes were split 2:1.
The P>>N raises alarms to me, but they reported that the performance on a test set was very good (AUC>0.8), which surprises me as I would expect quite a bit of overfitting. When is P>>N is OK for XGBoost, if ever? Perhaps if the data is noise-free? I presume the number of selected features must be <=N?
Ideally they should probably have done some dimensionality reduction beforehand, or in fact, use other ML methods such as CNNs. But taking this as it is, does the P>>N vs the high AUC on the test set seem strange/suggest something strange with the train-test datasets (e.g. duplicated images or overly similar images that would not hold upon deployment)? I'm aware that XGBoost has a regularisation component (and something like Lasso can be used for P>N) but is that sufficient? What are the caveats?