Dealing with outliers and correlated features for a deep learning based classification problem I am working on a multi-label multi-class classification problem that required me to use deep learning based approach. The data has around 17000 examples where each example has 42 numerical features and 2 labels. I have no additional information about the features like what they are and where they have been collected from. All I have is numbers. The 1st label has 10 classes while the second label has 2 classes. There are a couple of things that I have found upon exploring the data which are mentioned below:

*

*The numerical features have wildly varying ranges. For example, there are some features with a minimum of 0 and a maximum of around 60 and on the other hand, there are some features that have a minimum of 30 and a maximum of 10^7. However, the minimum for all the features is around 25-30.

*There are several features that contain lots of outliers. I am attaching the box plots for such features below. There are also certain other features that do not contain any outliers.







*There are some features that are highly correlated to each other (correlation coefficient ranging from 0.8 to 1).

Now, I have trained the model with different neural network models. Initially, I approached the problem by using two separate models for 2 label. Here I acheived fairly decent results for both labels. Later on, I combined the two labels by simply concatenating the one hot vectors of two labels and treating it as a multi-label classification. Here,  acheive very poor results on the test set even though i used cross validation techniques. (I scaled the data to bring the features into a common range).
I am looking for some advice about approaching this problem, particularly about dealing with outliers and correlated features for better performance. Thanks.
 A: In a real-life practical problem, the answer would of course be to go and find out more about the data generating process, understand each variable and find out how one should interpret these outliers. I assume this is some kind of data challenge, where this is not possible, but if this is a real problem, then finding out more about the data will be a much better idea than to try any of the things I've mentioned below.
If you don't know what the outliers are all about (i.e. you'd just be guessing if you declared them data errors, and they could just as easily be real extreme value), I'd be tempted to explore transformations that make your neural network less heavily "impacted" by them:

*

*Subtracting the training data mean and dividing by the standard deviation is one technique that can work quite well. That is after possibly first log- or square-root transforming variables with extreme tails in the right hand direction - for those with long tails in the other direction e.g. a log(1/x) or sqrt(1/x) transformation could be interesting -, if values are always positive (if they are sometimes 0, then first adding a small number like 0.5 may be necessary before log-transforming).

*One interesting alternative could be a RankGauss transformation as was used in this Kaggle competition winning solution. There it was used to transform numerical features before being put into an denoising autoencoder (which is also potentially worth trying), but this has been since used in other competitions and been a sometimes helpful option. The idea of this transformation is to rank observations and then then to apply the inverse normal cumulative distribution function to the percentile: i.e. the 2.5th percentile of the data would be mapped to about -1.96, the 50th percentile to 0 and the 97.5th percentile to about 1.96. The intuition of this is that neural networks can deal quite well with values that are about N(0, 1) distributed (in the sense that you are unlikely to produce really extreme values in the hidden layers). Whether this really helps for your particular problem is of course another question.

I'm not sure whether there is necessarily much of an issue with features being highly correlated.

*

*That's usually primarily an issue, if you want to interpret regression coefficients for regression models or feature importance for machine learning models. I.e. it might end up being an issue, if you want to provide interpretability help for predictions, but less so if you only care about predicting well.

*However, strong correlations hint that there might be more compact representations of the data, so pre-processing (possibly transformed) data and projecting it to lower dimensions using e.g. PCA could be an idea. With the same rationale self-supervised training to find a representation using a neural network (e.g. with a denoising autoencoder as mentioned above) could be used. On the other hand, a standard neural network for tabular data might just sort that out itself internally.

Another thing to look at is whether you have any categorical features and to use embeddings for those. Especially for high-cardinality categorical features (i.e. those with many levels but without any ordering such as when in sales data there are hundreds of shops) this can be one of the main ways in which neural networks can outperform boosted tree methods (such as xgboost, LightGBM or catboost), i.e. by learning good representations for these categories. However, unless you are required to use neural networks, looking into these alternatives (might have to be separate models for each question) would be an obvious thing to try.
Another thing to look at is to add two separate output heads to your neural network that each have their own loss function. E.g. categorical cross-entropy for the output with 10 classes and binary cross-entroy for the output with 2 classes (of course other loss functions could be of interest depending on what you get assessed on e.g. focal loss, cross-entropy with label-smoothing etc.). Most frameworks let you do this quite easily. The idea is that while you treat the two problems as separate in a sense, within the neural network the hidden layers are "encouraged" to learn a representation that is useful for both problems. This can often work quite well and the NN may end up doing better at both tasks (because what it "learns" from each task for its internal representation helps also on the other task). You may have to look carefully at whether you want to give each loss the same weight or perhaps you want to give different weights (needs experimentation).
More general remark: it's also worth exploring different regularization methods (especially weight decacy aka L2-regularization is usually foreseen as an option and might help), and finding a good learning rate (various frameworks provide their own learning rate finder tools) and learning rate schedules (e.g. experimenting with the one-cycle-schedule or flat-cosine schedules) can help a lot.
Finally, there's some specialized architectures for tabular data such as TabNet or SAINT that might be worth exploring. If you have not tried the basic tabular neural network used in the fastai package, that's also worth looking at.
