0
$\begingroup$

I have a dataset (11000 x 117) with response variable having multiple classes.

Here is a plot of class distribution:

enter image description here

Some of the classes have only 1 sample in the entire dataset and some have 2, 3 and 5.

I have tried:

SMOTE: Smote sampling, it generates sample (with freq = 100, 11000x117) for each of the classes but the model performs very poor on such sampled dataset. May be because SMOTE generate new points in the plane for samples : SMOTE enter image description here

downSample & upSample: Does not give good accuracy, explained below.

Sample/class weights in loss function:

get_class_weights <- function(y){
counter=funModeling::freq(y, plot=F) %>% select(var, frequency)
majority=max(counter$frequency)
counter$weight=pracma::ceil(majority/counter$frequency)
l_weights=setNames(as.list(counter$weight), counter$var)
return(l_weights)
}

This helped increase train and vald accuracy from 0.18 to 0.28 but it is still not enough.

As you can see, the classes are highly imbalanced. Is there any other way this can be done?

I am using Keras networks in R to train a neural network.

SOLUTION I tried:

After removing the observation for which the classes have low frequency( <100 ) I got this class distribution:

enter image description here

and as there is still imbalance, so I tried upSample and it resulted in the following distribution: enter image description here

BUT even after this distribution, the neural net gives only 35% accuracy.

What could be the reason for this?

enter image description here

$\endgroup$
  • 3
    $\begingroup$ Realistically 1 or 2 occurrences in 11 thousands samples do not constitute a category. Just specific type of outliers. As suggested by @black_cat aggregating these together is probably the most sensible options (if not outright dropping them). I would also ask how these categories came to be. Was there a particular continuum that was discretise? $\endgroup$ – usεr11852 Aug 8 at 17:33
  • $\begingroup$ These categories are being generated by a compiler, basically it is a type of compiler flag used for a certain piece of code. and there can be 110 of these flags $\endgroup$ – Obiii Aug 9 at 10:56
  • 1
    $\begingroup$ You will surely need to combine many of these categories to have a solvable problem. $\endgroup$ – Peter Flom Aug 9 at 11:18
  • $\begingroup$ Can you please try not using SMOTE or upsampling/downsampling in general? Also, can you please describe what is the ultimate reason for this classification task? That way we might be able to make more informative suggestions. $\endgroup$ – usεr11852 Aug 11 at 21:26
  • $\begingroup$ @usεr11852 I am trying to predict a combination of tokens (flags) based on some numeric code features (loops, loadInstruction, nestedLoops etc). there are 8 target output all categorical, flag1, flag2... flag8. SO I want model to learn relation between code features and these 8 target variables so that I can predict all 8 of them for unseen code. $\endgroup$ – Obiii Aug 13 at 12:38
1
$\begingroup$

I don't think this is a problem with sampling, rather, it is a problem with your data. You have many responses with too few instances. From your comment, you have 110 different responses - which are some sort of computer flag - and, from your first plot, it looks like about 1/4 to 1/2 of these have so few responses that it's going to be impossible to estimate things.

I don't think any sampling program is going to solve this; you will have to combine some of the flags or else drop some of them.

$\endgroup$
1
$\begingroup$

Weighting of the loss function is another possibility. By giving weights proportional to the class frequencies you penalize the loss more when it misclassifies less frequent classes. This can be done in Keras using the class_weight or sample_weight parameters (at least in Python).

Upsampling and downsampling are a possibility but, depending on the method used (SMOTE, Tomek link, random up/down sampling), they might not preserve the data distribution, leading to poor performances. There are cases where they work well (I'm thinking about computer vision), but in my limited experience every time I had an imbalanced dataset the loss function weighting was always the best method (maybe because I didn't play enough with the SMOTE/Tomek link/etc. parameters).

Another point is that you write

downSample & upSample: generates one sample for each class which is not enough to train.

I'm not familiar with these methods in R, but that shouldn't be the case. When you up/down sample data you should end up with a balanced set, that's the whole point! One way to fix this would be to loop and create (or remove) samples for each minority class (or majority) until balance is achieved.

Your problem is really hard, as you have only one(!) or few samples per class over 11000 records. Depending on your specific problem and what the classes mean, you might bucket some classes together. Again depending on your problem, you could move towards techniques used in rare event classification. For example here an autoencoder is trained only on the majority class; afterwards the data is reconstructed and checked against normal (majority class-related) vs. anomalous (minority class-related) data and the classification is based on the reconstruction error.

Edit after question update

I agree with the suggestions of the other users that you are probably not going anywhere with classes having one or few samples each. So in my opinion you are right on track in having removed the less frequent classes, because you need to start with a simpler problem (few total number of classes) and build from there. When removing be sure that you haven't deleted too many observations from the dataset. If so, bucket classes instead of removing them.

Loss weighting

I'm not completely sure about your code because I don't have experience with R, but it looks correct. The formula for assigning the weights is

weight_class_i = number_of_obs_majority_class/number_of_obs_class_i

and I see that you have something similar in your code, so that should be fine.

However you are looking at the accuracy, and that's the wrong metric when you have imbalanced classification! You should use scores like precision, recall, F1, MCC, ecc (and make sure to compute them correctly in the multi-class case).

Also make sure that your train/validation/test (including Kfold if you are using it) splits are stratified.

Last question

BUT even after this distribution, the neural net gives only 35% accuracy.

In principle there are many candidate answers: from upsample not being able to preserve the original data distribution (see above), to a wrong choice of network architecture/hyperparameters, and so on.

Here's what I'd do in your situation: first of all I'd bucket the classes and get very few classes. I'd build my model on this dataset and check whether it gives me reasonable results. If so I can start building a new model based on the old one with more classes (but still bucketed since there are too many classes with few samples each and that's probably never going to work). In order to do that:

  • I'd ask myself whether I'm using the right network architecture and loss function for this particular problem
  • I'd play (increase/decrease depending on what I see) with the number of layers and nodes of the network
  • I'd increase the number of epochs and define a grid search to find the best hyperparameters. I'd be interested to check the effects of increasing the batch size. I'd also explore a wide range of values for the learning rate
  • I'd try different optimizers in combination with the learning rates

I'd then build my next steps based on the results of this exploration.

$\endgroup$
  • $\begingroup$ I have updated the question. Aded what you suggested. It helped but not a lot. Can you please take a look at the edited question above. Thanks $\endgroup$ – Obiii Aug 9 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.