0
$\begingroup$

for a video summarization project i use the features of each frame as input to predict if some of these frames are included in the summary or not.

one of the famous implementations i found had treated this task as a regression task but it used a sigmoid output with f-score as evaluation metric and MSE loss.

the problem is that i have read online that MSE_Loss is only used for regression tasks when the target is continuous set of numbers.

  • target values in this case are scores of the frames from 0.0 to 1.0 and the predictions are also set of probability distributions for each video.

fscore is then derived by doing a knapsack to pick the best representative group of scores convert them to 1's and the other groups are converted to 0's then it is compared with a binary summary.

my question is it wrong to use MSE in such case as it clearly looks like a classification problem not a regression one unless i don't really have full understanding of the relation between regression and this task.

Improve this question
$\endgroup$
6
  • $\begingroup$ It is difficult to advise on model evaluation without knowing what is important from an application perspective. Why is F1 relevant, that is normally for detection problems rather than classification tasks? $\endgroup$
    – Dikran Marsupial
    Commented Apr 16 at 12:24
  • $\begingroup$ @DikranMarsupial i can provide more details if needed although i dont quite understand your question. $\endgroup$
    – moha tech
    Commented Apr 16 at 12:35
  • $\begingroup$ basically we need to know how the classifier will be used, are there unequal misclassification costs (e.g false positives worse than false negatives), do you need a hard classification or do you need a probability or a ranking of examples, do you need a reject option, is the relative frequency of classes in operation different to those in the dataset. These are all issues that might influence which performance metrics are important. F1 ignores "true negatves", so you wouldn't want to use that where the classes are symmetric e.g. Cat or dog in images. $\endgroup$
    – Dikran Marsupial
    Commented Apr 16 at 12:43
  • $\begingroup$ @DikranMarsupial task itself is video summarization with the inputs are sequences of features and the labels are binary arrays with 15% or 20% True values that represent the summary of the video the model predicts a probability distribution based on the features of each frame [ .443,.345,.342,....,.877] then these probabilities are processed with the changepoints in video like video segments to give scores to each segment and finally use a (knapsack w/ 15-20 proportion) to keep the most representative segments in the final summary with true values and the unwanted ones with false values. $\endgroup$
    – moha tech
    Commented Apr 16 at 13:12
  • 1
    $\begingroup$ Then the F-score is computed between the final summary and the original labels that are 15%-20% of the original video. I have no problem with the f-score as it seems suitable in this case; however, I was told otherwise in the case of MSE, so I was confused. Also, the extra processing step after prediction got me confused; is that a regression task or a classification task? $\endgroup$
    – moha tech
    Commented Apr 16 at 13:16

1 Answer 1

Reset to default
1
$\begingroup$

The MSE is absolutely appropriate. It is a proper scoring rule (also known as the Brier score), i.e., a loss function for probabilistic predictions that rewards calibrated predictions, i.e., ones that are "correct".

Hard 0-1 classifications are far inferior. Any probabilistic prediction on one side of your threshold will give you the same hard 0-1 classification, but it makes a difference whether an instance has a probability of 0.001 or 0.01 or 0.1 to be of the target class.

A few related threads (the F1 score suffers from the exact same issues as accuracy):

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ "Hard 0-1 classifications are far inferior." this depends on the needs of the application, which we don't know. However even for tasks where a specific threshold is important, it is a a good idea to assess the probabilities as well, via MSE/Brier/Log loss. (+1) $\endgroup$
    – Dikran Marsupial
    Commented Apr 16 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.