I am looking for an interpretation of the consistency property of a loss function used for classification (e.g., the SVM's hinge loss: $V(t)=\max(0,1-t)$).

I copy from Wikipedia:

Furthermore, it can be shown that for any convex loss function $V(yf_0(\vec{x}))$, where $f_0$ is the function that minimizes this loss, if $f_0(\vec{x}) \ne 0$ and $V$ is decreasing in a neighborhood of $0$, then $f^*(\vec{x}) = \operatorname{sgn}(f_0(\vec{x}))$ where $\operatorname{sgn}$ is the sign function. Note also that $f_0(\vec{x}) \ne 0$ in practice when the loss function is differentiable at the origin. This fact confers a consistency property upon all convex loss functions; specifically, all convex loss functions will lead to consistent results with the $0-1$ loss function given the presence of infinite data. Consequently, we can bound the difference of any of these convex loss function from expected risk.

Could anyone please explain why the fact that a loss function enjoys the consistency property is important?


An estimator is consistent if an estimate for parameter of interest θ, θ* approaches the true parameter θ0 as the sample size increases. The property is often refereed to as asymptotic consistency. You can look it up on wiki or any intro to stats book. Heuristically, it is obvious that one wants to choose a loss function that will estimate the true θ0, at least when the sample size is large. There is a lot to be said about proper model structure in addition to loss function and their influence of consistency and efficiency.

If you want to learn about the related information theory , i'd recommend: Burnham & Anderson. "Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach". And any linear model book will tell you about the properties of OLS and associated estimators; including SVM.

That's my best simple interpretation of your question. best of luck.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.