What is an acceptable value of square loss in machine learning (using mxnet gluon's square loss function)?

Question

I have checked the gluon's square loss function implementation in mxnet, where it is calculated as follows:

$$L=\frac{1}{2}\sum_i\left|label_i−pred_i\right|^2$$

https://mxnet.apache.org/versions/1.8.0/api/python/docs/api/gluon/loss/index.html#mxnet.gluon.loss.L2Loss

The key here is that the loss is squared and then multiplied by $\frac{1}{2}$.

Now, let's suppose I want to predict a label ∈ {0,1} where P(1)=P(0)=0.5

And instead of doing any calculation I simply always predict that my value is equal to 0.5

If the label is 1, I get a loss $L=\frac{1}{2}\sum_i\left|1−0.5\right|^2$=0.125

And if my label is 0, I also get $L=\frac{1}{2}\sum_i\left|0−0.5\right|^2$=0.125

Does this mean I should aim for my loss to be less than 0.125 for my predictions to be of any use?

Or in this case ( if labels are ∈ {0,1} ), would it be more sensible to switch to softmax model?

Answer 1

Your constant prediction of $0.5$ is a great benchmark. Meteorological forecasters would call it a "climatological forecast", i.e., one that only relies on the overall and unconditional distribution of your target variable.

Any other model should improve on this benchmark. If it doesn't, you are better off with the simple benchmark.

"Of any use" implies, well, putting predictions to use. In some applications, even tiny improvements over trivial benchmarks can be valuable (e.g., in stock price predictions). In other applications, the improvement would need to be larger for us to use the model over the benchmark.

Whether you should switch to a softmax depends on the quality of your final output. If a model with softmax yields better predictions, then by all means, use it.

Answer 2

It’s hard to say what constitutes acceptable performance. For instance, it might sound like awesome performance if your classifier gets $90\%$ of its predictions right, but if the data are the MNIST handwritten digits, such performance is rather pedestrian.

(Note, however, that “classification accuracy” is more problematic than it first appears.)

Being able to beat some baseline is a good start, however, and it mimics $R^2$ in linear regression. In linear regression, the goal is to predict what value you would expect, given some values of the features. The most naïve way of guessing such a value that is sensible is to guess the mean of your $y$ every time. If you can’t do better than that, then why is your boss paying you when she can call np.mean(y) in Python and do better? (This is the “overall and unconditional distribution” in Stephan Kolassa’s answer.)

What you propose uses the same idea. If you know there is a 50/50 chance of each outcome, for your model to be worth using, your model ought to be able to outperform randomly guessing based on that 50/50 distribution of labels.

In the kind of problem you are solving, there are many options for analogues of $R^2$. It is not clear how large any of them should be for your model to satisfy business needs, as this depends on the problem and business requirements (customer demands, regulator demands, investor demands, etc). However, if they show your model is outperformed by a naïve model that always guesses the same value, then you are not making a strong case for your modeling skills.

Links of possible interest:

Why to put variance around the mean line to the definition of $R^2$? By what is this particular choice dictated?

Why getting very high values for MSE/MAE/MAPE when R2 score is very good

Why does putting a 1/2 in front of the squared error make the math easier? (Data Science)

Answer 3

In addition to the other answers, I'll add that usually with machine learning we fit the model by finding the weights that minimise the loss, but we then usually evaluate the model using a different metric that has more "intuitive" meaning (and to be robust this should be on a different dataset that wasn't used to train the model, i.e. a train-test split).

For example with classification we often use Cross Entropy as the loss, however when we evaluate the model we use either accuracy, F1 score or some other metric to actually decide if our model is "good enough". For regression problems such as the one you're working on, we more commonly use squared error loss, and it's fairly common to use something like r-squared as a metric (although it's certainly not a perfect one).