I'm studying up on machine learning basics and the standard high-level approach in supervised ML is to define a hypothesis function that maps inputs to outputs. Then define a "cost function" that gives a measure of how far off the hypothesis function outputs are from the actual output values. And finally to adjust the parameters of the function such that the value of the cost function is minimized.

Given a specific problem we're trying to solve with a specific "hypothesis function" $h_{\theta}(\mathbf{x})$ (where $\theta$ is some generic representation of the parameters of $h$), how do we decide on an appropriate cost function $J$? I've looked at this question: Supervised learning : How did they find the Cost function to minimize?, but I don't feel it addresses the specific criteria that go into deciding an appropriate cost function.

For example, for univariate linear regression, the hypothesis function is $h_{\theta}(x)=\theta_0 + \theta_1 x$. In this case, a commonly-used cost function for a single sample (the $r$-th sample) is $(h_{\theta}(x_r) - y_r)^2$, and the overall cost function is $\frac{1}{2m}\sum_{r=1}^m(h_{\theta}(x_r) - y_r)^2$.

How do we decide this is an appropriate cost function to use - what criteria are used to determine the suitability of a cost function for a particular kind of hypothesis function or for a particular problem we're trying to solve?

Is the choice of the cost function dependent on specific characteristics of $h$, or on the nature of the problem (regression vs. classification)? Or on the features or the output values (range, etc.)? If so, what are those characteristics?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.