1
$\begingroup$

In the linear regression models, the model prediction $y(x, w)$ is given by a linear function of the parameters $w$. In the simplest case, the model is also linear in the input variables and therefore takes the form $y(x) = w^T x + w_0$, so that $y$ is a real number. For classification problems, however, we wish to predict discrete class labels, or more generally posterior probabilities that lie in the range $(0, 1)$. To achieve this, we consider a generalization of this model in which we transform the linear function of w using a nonlinear function $f(\cdot)$ so that

$$y(x) = f(w^T x + w_0).$$

Question: I am taking my first course in machine learning and am unable to wrap my head around the reason for the need to transform the linear function of $w$ into a non-linear function. And also could someone post a simple example of such a transformation.

$\endgroup$

1 Answer 1

3
$\begingroup$

The reason for requiring $f$ is simple: when we're classifying, we want a predicted class, either 0 or 1. But a linear function of the inputs $w^T x + w_0$ won't be equal to 0 or 1; assuming we don't have constraints on $x$, it could be literally any real number.

So, to turn a real number prediction into a class label output, we need to somehow map it into either 0 or 1. The simplest $f$ would be a hard threshold:

$$f(x) = \begin{cases} 1 & \text{if } x > 0 \\ 0 & \text{if } x \le 0\end{cases}$$ (where what to do for $x = 0$ is arbitrary).

If instead we want to predict probabilities, we could use the logistic function

$$f(x) = \frac{1}{1 + e^{-x}}$$

which looks like

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.