# What is an acceptable value of square loss in machine learning (using mxnet gluon's square loss function)?

I have checked the gluon's square loss function implementation in mxnet, where it is calculated as follows:

$$L=\frac{1}{2}\sum_i\left|label_i−pred_i\right|^2$$

https://mxnet.apache.org/versions/1.8.0/api/python/docs/api/gluon/loss/index.html#mxnet.gluon.loss.L2Loss

The key here is that the loss is squared and then multiplied by $$\frac{1}{2}$$.

Now, let's suppose I want to predict a label ∈ {0,1} where P(1)=P(0)=0.5

And instead of doing any calculation I simply always predict that my value is equal to 0.5

If the label is 1, I get a loss $$L=\frac{1}{2}\sum_i\left|1−0.5\right|^2$$=0.125

And if my label is 0, I also get $$L=\frac{1}{2}\sum_i\left|0−0.5\right|^2$$=0.125

Does this mean I should aim for my loss to be less than 0.125 for my predictions to be of any use?

Or in this case ( if labels are ∈ {0,1} ), would it be more sensible to switch to softmax model?

• What do you mean by a “softmax model”?
– Dave
Commented Dec 20, 2021 at 11:47
• Machine learning and statistical models don't exist in a vacuum. The relevant question isn't "is my loss value good?", but instead "is my model good enough for my particular application?" When the model makes mistakes (which it will, inevitably), you want to know if the costs of those mistakes are outweighed by the value of its correct predictions.
– Sycorax
Commented Dec 20, 2021 at 19:34

Your constant prediction of $$0.5$$ is a great benchmark. Meteorological forecasters would call it a "climatological forecast", i.e., one that only relies on the overall and unconditional distribution of your target variable.

Any other model should improve on this benchmark. If it doesn't, you are better off with the simple benchmark.

"Of any use" implies, well, putting predictions to use. In some applications, even tiny improvements over trivial benchmarks can be valuable (e.g., in stock price predictions). In other applications, the improvement would need to be larger for us to use the model over the benchmark.

Whether you should switch to a softmax depends on the quality of your final output. If a model with softmax yields better predictions, then by all means, use it.

It’s hard to say what constitutes acceptable performance. For instance, it might sound like awesome performance if your classifier gets $$90\%$$ of its predictions right, but if the data are the MNIST handwritten digits, such performance is rather pedestrian.

(Note, however, that “classification accuracy” is more problematic than it first appears.)

Being able to beat some baseline is a good start, however, and it mimics $$R^2$$ in linear regression. In linear regression, the goal is to predict what value you would expect, given some values of the features. The most naïve way of guessing such a value that is sensible is to guess the mean of your $$y$$ every time. If you can’t do better than that, then why is your boss paying you when she can call np.mean(y) in Python and do better? (This is the “overall and unconditional distribution” in Stephan Kolassa’s answer.)

What you propose uses the same idea. If you know there is a 50/50 chance of each outcome, for your model to be worth using, your model ought to be able to outperform randomly guessing based on that 50/50 distribution of labels.

In the kind of problem you are solving, there are many options for analogues of $$R^2$$. It is not clear how large any of them should be for your model to satisfy business needs, as this depends on the problem and business requirements (customer demands, regulator demands, investor demands, etc). However, if they show your model is outperformed by a naïve model that always guesses the same value, then you are not making a strong case for your modeling skills.

Why to put variance around the mean line to the definition of $R^2$? By what is this particular choice dictated?