Weighting of the loss function is another possibility. By giving weights proportional to the class frequencies you penalize the loss more when it misclassifies less frequent classes. This can be done in Keras using the class_weight or sample_weight parameters (at least in Python).
Upsampling and downsampling are a possibility but, depending on the method used (SMOTE, Tomek link, random up/down sampling), they might not preserve the data distribution, leading to poor performances. There are cases where they work well (I'm thinking about computer vision), but in my limited experience every time I had an imbalanced dataset the loss function weighting was always the best method (maybe because I didn't play enough with the SMOTE/Tomek link/etc. parameters).
Another point is that you write
downSample & upSample: generates one sample for each class which is not enough to train.
I'm not familiar with these methods in R, but that shouldn't be the case. When you up/down sample data you should end up with a balanced set, that's the whole point! One way to fix this would be to loop and create (or remove) samples for each minority class (or majority) until balance is achieved.
Your problem is really hard, as you have only one(!) or few samples per class over 11000 records. Depending on your specific problem and what the classes mean, you might bucket some classes together. Again depending on your problem, you could move towards techniques used in rare event classification. For example here an autoencoder is trained only on the majority class; afterwards the data is reconstructed and checked against normal (majority class-related) vs.
anomalous (minority class-related) data and the classification is based on the reconstruction error.
Edit after question update
I agree with the suggestions of the other users that you are probably not going anywhere with classes having one or few samples each. So in my opinion you are right on track in having removed the less frequent classes, because you need to start with a simpler problem (few total number of classes) and build from there. When removing be sure that you haven't deleted too many observations from the dataset. If so, bucket classes instead of removing them.
I'm not completely sure about your code because I don't have experience with R, but it looks correct. The formula for assigning the weights is
weight_class_i = number_of_obs_majority_class/number_of_obs_class_i
and I see that you have something similar in your code, so that should be fine.
However you are looking at the accuracy, and that's the wrong metric when you have imbalanced classification! You should use scores like precision, recall, F1, MCC, ecc (and make sure to compute them correctly in the multi-class case).
Also make sure that your train/validation/test (including Kfold if you are using it) splits are stratified.
BUT even after this distribution, the neural net gives only 35% accuracy.
In principle there are many candidate answers: from upsample not being able to preserve the original data distribution (see above), to a wrong choice of network architecture/hyperparameters, and so on.
Here's what I'd do in your situation: first of all I'd bucket the classes and get very few classes. I'd build my model on this dataset and check whether it gives me reasonable results. If so I can start building a new model based on the old one with more classes (but still bucketed since there are too many classes with few samples each and that's probably never going to work). In order to do that:
- I'd ask myself whether I'm using the right network architecture and loss function for this particular problem
- I'd play (increase/decrease depending on what I see) with the number of layers and nodes of the network
- I'd increase the number of epochs and define a grid search to find the best hyperparameters. I'd be interested to check the effects of increasing the batch size. I'd also explore a wide range of values for the learning rate
- I'd try different optimizers in combination with the learning rates
I'd then build my next steps based on the results of this exploration.