XGBoost when P>>N

Question

Someone built an XGBoost classification model using each pixel in an image (256*256) as a separate feature, plus a few other features. However they only have 500 data points. The target classes were split 2:1.

The P>>N raises alarms to me, but they reported that the performance on a test set was very good (AUC>0.8), which surprises me as I would expect quite a bit of overfitting. When is P>>N is OK for XGBoost, if ever? Perhaps if the data is noise-free? I presume the number of selected features must be <=N?

Ideally they should probably have done some dimensionality reduction beforehand, or in fact, use other ML methods such as CNNs. But taking this as it is, does the P>>N vs the high AUC on the test set seem strange/suggest something strange with the train-test datasets (e.g. duplicated images or overly similar images that would not hold upon deployment)? I'm aware that XGBoost has a regularisation component (and something like Lasso can be used for P>N) but is that sufficient? What are the caveats?

Answer 1

Data is king, so if it works in real life we can't argue with it.

Having said that, I agree with you it's bad practice and will usually not end well.

I can design a dataset where it would work though. Images for example are known for having correlation between close pixels, effectively reducing the number of features with information. Imagine for example a dataset of black balls on a white background and you need to classify images with a ball vs. no ball. In that case, most pixels will not have any information gain so the split will not happen in the tree. All the edge pixels will always be white, containing zero information. Combining that with regularization and image augmentation may leave you with a working model.

Just for my curiosity, do you know if the images are generally similar?

Another phenomenon worth mentioning is double descent. This is more of a NN phenomenon where over parameterized models where $P>>N$ achieve better performance as the number of parameters increases. But it's also been demonstrated in classical models like linear regression.

In summary it's possible, but if I had to do it (with XGBoost) I'd do the following:

1. Dimensionality reduction.
2. Strong regularization.
3. Most importantly - a robust, careful validation procedure.