1
$\begingroup$

I'm training some net with a supervised learning setting. The inputs are vectors and the outputs (and labels) are non-negative integers that represent a certain amount of times that the input appeared somewhere.

I am using MSE loss and the loss reduces with time (on both train and test set), but I am still not sure whether the net actually learns to predict a label from the input or does it simply learn to output numbers that resemble the labels distribution better.

If I would take labels vector and outputs vector for each epoch and check the correlation/mutual information between them could it give me a sense of whether the nets actually improves? I am not familiar with such work, Is it common to do something of the sort?

Improve this question
$\endgroup$
3
  • $\begingroup$ why wouldn't MSE indicate that the network predicts better? $\endgroup$
    – rep_ho
    Commented Jan 6, 2020 at 15:00
  • $\begingroup$ Well, for example before making some modifications to the net after a few batches it became a constant function, but still improved with training (in each batch it just jumped from one constant function to another). So, it could happen. Now it doesn't become a constant function but I still want to make sure that it actually learns. $\endgroup$
    – Gilad Deutsch
    Commented Jan 6, 2020 at 15:07
  • $\begingroup$ OP, I think you mentioned elsewhere that the target is the number of times an object is observed. I think others will be interested to know that, also any other problem-specific information you can provide. $\endgroup$
    – Robert Dodier
    Commented Jan 6, 2020 at 21:17

1 Answer 1

Reset to default
0
$\begingroup$

Yes, this is common and there are multiple thinks you can do.

First you can find out what MSE intercept only model has therefore you will know if your model is learning something above that baseline.

Second you can use metrics that are invariant to shifts of variables. One possibility is as you mentioned correlation, which will tell you what is the strength of a linear dependence of your predicted and target variables. You can also use Spearman's correlation which is invariant to any monotone shift, similarly to AUC used for classification.

And of course if you really want to know what is going on, you can always plot your data

Improve this answer
$\endgroup$
3
  • $\begingroup$ First, Can you explain what you mean by "First you can find out what MSE intercept only model has"? Second, i'm checking the pearson correlation and see that it is close to zero and does not increase with training, should i conclude that the model is therefore not learning well? (the mutual info doesn't increase with training as well) $\endgroup$
    – Gilad Deutsch
    Commented Jan 6, 2020 at 16:33
  • $\begingroup$ Intercept only is s model that predicts only a constant, so if you predict only a man of your target variable what mse would you get? $\endgroup$
    – rep_ho
    Commented Jan 6, 2020 at 16:36
  • $\begingroup$ It means that it does not improve linear relationship between target and predictions, but it is possible that it is improving bias and calibration $\endgroup$
    – rep_ho
    Commented Jan 6, 2020 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.