I have dataset with 64 features that I'm using to model 6 labels bounded on the interval [0,1]. These labels are highly related and it therefore seems reasonable to model them all at once rather than individually.
To model these labels I use a weight matrix with a parameter for each feature + label combination and a bias for each label (i.e. 1 neural network layer). To bound the regression I use a sigmoid activation for each label. I optimize using a variant of SGD with a MSE loss function.
I have 200 observations in my training set, 50 in my validation set and 1000 in my test set. I'm unable to change the number of observations in any set. Regularization (dropout) is applied to deal with the low number of observations. I do not care about inference, only prediction performance.
My issue is that for many observations 3-4 of the labels are 0.0 and that in general, the labels are close to zero. The labels each have a mean of ~0.15. This causes my model to favor a solution, where it simply predicts close to 0.0 for all labels for all observations. The final evaluation of performance is the Pearson correlation between the predictions and the true labels, which makes the all-zero solution worthless.
What are some ways in which I can force my model to avoid the all-zero solution? Perhaps another loss function?