Interpretation of the consistency property of a loss function

Question

I am looking for an interpretation of the consistency property of a loss function used for classification (e.g., the SVM's hinge loss: $V(t)=\max(0,1-t)$).

I copy from Wikipedia:

Furthermore, it can be shown that for any convex loss function $V(yf_0(\vec{x}))$, where $f_0$ is the function that minimizes this loss, if $f_0(\vec{x}) \ne 0$ and $V$ is decreasing in a neighborhood of $0$, then $f^*(\vec{x}) = \operatorname{sgn}(f_0(\vec{x}))$ where $\operatorname{sgn}$ is the sign function. Note also that $f_0(\vec{x}) \ne 0$ in practice when the loss function is differentiable at the origin. This fact confers a consistency property upon all convex loss functions; specifically, all convex loss functions will lead to consistent results with the $0-1$ loss function given the presence of infinite data. Consequently, we can bound the difference of any of these convex loss function from expected risk.

Could anyone please explain why the fact that a loss function enjoys the consistency property is important?

Answer 1

An estimator is consistent if an estimate for parameter of interest θ, θ* approaches the true parameter θ₀ as the sample size increases. The property is often refereed to as asymptotic consistency. You can look it up on wiki or any intro to stats book. Heuristically, it is obvious that one wants to choose a loss function that will estimate the true θ₀, at least when the sample size is large. There is a lot to be said about proper model structure in addition to loss function and their influence of consistency and efficiency.

If you want to learn about the related information theory , i'd recommend: Burnham & Anderson. "Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach". And any linear model book will tell you about the properties of OLS and associated estimators; including SVM.

That's my best simple interpretation of your question. best of luck.

Answer 2

The theorem states that if the loss function is convex and decreases at around 0, a classifier that minimizes the expected loss will make the same decision as the Bayes optimal classifier. One significance is that we can get P(y|X) by minimizing the expected loss.

You can find more details in the original paper.