I am trying to train a sequence model to extract specific substrings. I am working on extremely sparse text data (Sparsity ~ 0.03%, <1000 examples). After training for 500 epochs, the performance remains pretty poor (F1-score ~0.01 on training and test sets, ~98.9% training, validation and test accuracies and losses also being really low).
I am wondering if it's possible to train on F1-score? My intuition tells me it's not possible as it's not a differentiable function as it should use a count function, which itself is not differentiable. Is this right?
Some other methods I am considering to improve performance are: 1. Training on more data. 2. Extract more features to reduce sparsity and improve training due to inter-feature correlation. 3. Training on a deeper model. 4. Using Attention weights.
Do these seem like reasonable approaches? Are there any methods to optimize sequence models on sparse data? Any help appreciated. Thanks in advance.