How to train a Neural Network on sparse data?

Question

I am trying to train a sequence model to extract specific substrings. I am working on extremely sparse text data (Sparsity ~ 0.03%, <1000 examples). After training for 500 epochs, the performance remains pretty poor (F1-score ~0.01 on training and test sets, ~98.9% training, validation and test accuracies and losses also being really low).

I am wondering if it's possible to train on F1-score? My intuition tells me it's not possible as it's not a differentiable function as it should use a count function, which itself is not differentiable. Is this right?

Some other methods I am considering to improve performance are: 1. Training on more data. 2. Extract more features to reduce sparsity and improve training due to inter-feature correlation. 3. Training on a deeper model. 4. Using Attention weights.

Do these seem like reasonable approaches? Are there any methods to optimize sequence models on sparse data? Any help appreciated. Thanks in advance.

Answer 1

There was a paper on using F-measure as a training objective. I tried for training a model for very imbalanced classes and didn't work well (oversampling worked better). The idea is simple: for each class, you just compute a soft precision and soft recall and to the harmonic mean.

Your suggestions 1 and 2 sound like a reasonable thing to do.

Regarding the bigger / deeper model: This depends on how you are able to fit your training data. If your model is already overfitting, a deeper model probably would not help, otherwise, it might.

Attention or other changes in architecture indeed can help, but it very much depends on how your current architecture looks like. You certainly cannot say that attention would be a universal solution for data sparsity.