2
$\begingroup$

I’m a bit confused about the concept of using feedforwrd neural networks via backpropagation to model a nonlinear relationship between the input and output variable in a regression setting. Can this be done? I.e., could such an ANN deduce a relationship like $y = \sin(x^2) + x^3$?

If it’s not easy to do this, does it have to do with the fact that in regression problems the output layer activation function is linear?

Thanks.

$\endgroup$
  • 1
    $\begingroup$ If there are non linearities in earlier layers then it doesn’t really matter if the last layer is linear AFAIK $\endgroup$ – kbrose Aug 29 '18 at 23:21
5
$\begingroup$

Yes, feedforward neural nets can be used for nonlinear regression, e.g. to fit functions like the example you mentioned. Learning proceeds the same as in other supervised problems (typically using backprop). One difference is that a loss function that makes sense for regression is needed (e.g. squared error). Another difference is the output layer. If the target outputs are general real numbers, then it makes sense to use a linear output layer. This is because other activation functions are bounded. For example, sigmoidal units (which might be used for output in a classification problem) can only output values between 0 and 1.

If we connect the input layer directly to a linear output layer, network output will be a linear function of the input. So, we'll be performing linear regression. To fit nonlinear functions, we need one or more hidden layers with nonlinear activation functions.

For example, suppose we want to fit your example function. There's a single input $x \in \mathbb{R}$. We feed it through one or more nonlinear hidden layers, each containing multiple units. The output of the last hidden layer (with $p$ units) is $h(x) = [h_1(x), \dots, h_p(x)]$. There's a single, linear output unit with weights $w = [w_1, \dots, w_p]$ and bias $b$. So, the output of the entire network is:

$$f(x) = b + \sum_{i=1}^p w_i h_i(x)$$

We can see from this expression that the output is a weighted sum of nonlinear basis functions, where each basis function is the output of a unit in the last hidden layer. Say we minimize the squared error using backprop. What we're essentially doing is fitting the function by learning an adaptive set of basis functions and their weights.

$\endgroup$
  • $\begingroup$ Hello. Thanks for your excellent response. Is it correct to assume that because of the finite range of sigmoid activation functions, that you would need potentially many nodes in the hidden layers to "learn" this type of regression problem? $\endgroup$ – Thomas Moore Aug 30 '18 at 18:34
  • 1
    $\begingroup$ The range isn't so much of an issue here because multiplying by weights in the output layer can take care of that. The reason to use many hidden units is to properly fit the shape of a nonlinear function. If using a single hidden layer, the basis functions will be rather simple, and we may need to combine many of them to match the target function (shallow, wide network). With multiple hidden layers, the basis functions will be more complicated, and adapt more closely to the target. So, fewer hidden units per layer needed (narrow, deep network). $\endgroup$ – user20160 Aug 30 '18 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.