Activation Functions

Question

In the linear regression models, the model prediction $y(x, w)$ is given by a linear function of the parameters $w$. In the simplest case, the model is also linear in the input variables and therefore takes the form $y(x) = w^T x + w_0$, so that $y$ is a real number. For classification problems, however, we wish to predict discrete class labels, or more generally posterior probabilities that lie in the range $(0, 1)$. To achieve this, we consider a generalization of this model in which we transform the linear function of w using a nonlinear function $f(\cdot)$ so that

$$y(x) = f(w^T x + w_0).$$

Question: I am taking my first course in machine learning and am unable to wrap my head around the reason for the need to transform the linear function of $w$ into a non-linear function. And also could someone post a simple example of such a transformation.

Answer 1

The reason for requiring $f$ is simple: when we're classifying, we want a predicted class, either 0 or 1. But a linear function of the inputs $w^T x + w_0$ won't be equal to 0 or 1; assuming we don't have constraints on $x$, it could be literally any real number.

So, to turn a real number prediction into a class label output, we need to somehow map it into either 0 or 1. The simplest $f$ would be a hard threshold:

$$f(x) = \begin{cases} 1 & \text{if } x > 0 \\ 0 & \text{if } x \le 0\end{cases}$$ (where what to do for $x = 0$ is arbitrary).

If instead we want to predict probabilities, we could use the logistic function

$$f(x) = \frac{1}{1 + e^{-x}}$$

which looks like