14
$\begingroup$

I'm going thru the Cousera's DeepAI course (Week3 video 1 "Neural Networks Overview") and Andrew Ng is explaining how each layer in a neural network is just another logistic regression, but he doesn't explain how it makes thing more accurate.

So in a 2 layer network, how does calculating logistic multiple times make it more accurate?

$\endgroup$
2
  • 8
    $\begingroup$ Often to fit a good single logistic regression model, you need to do significant feature engennering. The two layer nn is attempting to do some of that work for you. $\endgroup$ – Matthew Drury Aug 30 '17 at 14:14
  • 1
    $\begingroup$ @msk, have you ever created new features based on your original features to improve your logistic regression? This is what the hidden layer(s) does. $\endgroup$ – Ricardo Cruz Sep 5 '17 at 21:56
11
$\begingroup$

When using logistic activation functions, it's true that the function relating the inputs of each unit to its output is the same as for logistic regression. But, this isn't really the same as each unit performing logistic regression. The difference is that, in logistic regression, the weights and bias are chosen such that the output best matches given target values (using the log/cross-entropy loss). In contrast, hidden units in a neural net send their outputs to downstream units. There is no target output to match for individual hidden units. Rather, the weights and biases are chosen to minimize some objective function that depends on the final output of the network.

Rather than performing logistic regression, it might make more sense to think of each hidden unit as computing a coordinate in some feature space. From this perspective, the purpose of a hidden layer is to transform its input--the input vector is mapped to a vector of hidden layer activations. You can think of this as mapping the input into a feature space with a dimension corresponding to each hidden unit.

The output layer can often be thought of as a standard learning algorithm that operates in this feature space. For example, in a classification task, using a logistic output unit with cross entropy loss is equivalent to performing logistic regression in feature space (or multinomial logistic regression if using softmax outputs). In a regression task, using a linear output with squared error is equivalent to performing least squares linear regression in feature space.

Training the network amounts to learning the feature space mapping and classification/regression function (in feature space) that, together, give the best performance. Assuming nonlinear hidden units, increasing the width of the hidden layer or stacking multiple hidden layers permits more complex feature space mappings, thereby allowing more complex functions to be fit.

$\endgroup$
8
$\begingroup$

One way to see the power of nonlinearity is to note universal approximation theorem.

Though it's not very significant in practice (it's about capabilities of single layer networks), it tells you that if you use (arbitrary long) sums of sigmoids you can in principle approximate any continuous function to any desired level. If you know Fourier theory or remember Weierstrass approximation theorem it shouldn't be surprising.

$\endgroup$
1
  • 3
    $\begingroup$ This was the only person that actually answered the question properly, albeit a little too succinctly. A hidden layer is enough to create powerful new features that combine the original features. The problem is that you might need a hidden layer with too many nodes, and the convergence process works best with a deeper network with hidden layers > 1. $\endgroup$ – Ricardo Cruz Sep 5 '17 at 21:55
5
$\begingroup$

When there are hidden layers exist in the neural network, we are adding non-linear features. Please check my answer here to get some sense.

what makes neural networks a nonlinear classification model?

Specifically, a nested sigmoid function will be more "powerful" than a linear transformation of original features and one sigmoid function (logistic regression.)


Here is an numerical example to address OP's comments.

Suppose we have data frame $X$, it is a $10 \times 3$ matrix (10 data points, 3 features.). If we want to have $7$ hidden unites, then the weight matrix $W$ is a $3 \times 7$ matrix. The output for the hidden layer (output of matrix multiplication $X \times W$) is a $10 \times 7$ matrix, which for each data point, there are $7$ expended features.

$\endgroup$
6
  • $\begingroup$ I'm just wondering, then how does any of the nodes the first layer differ in the output, I.e. Node 1 gets X1, X2, X3, Node 2 also gets X1, X2, X3, if they are all logistic regression, how is their output going to be different? $\endgroup$ – mskw Aug 30 '17 at 14:33
  • $\begingroup$ suppose you have 3 features, and 10 hidden units, then the output of the hidden layer has 10 "engineered features". $\endgroup$ – Haitao Du Aug 30 '17 at 14:35
  • $\begingroup$ Why do you call it "engineered features", also are the features you mentions the X1, X2, X3? $\endgroup$ – mskw Aug 30 '17 at 14:41
  • $\begingroup$ I will edit my answer to address your comments. $\endgroup$ – Haitao Du Aug 30 '17 at 14:47
  • $\begingroup$ Thanks of revising, from your explanation, I'm not sure if you are not answering my question, or I have a knowledge gap where I need to bridge first. Specifically, from your answer does Weight refer to each of the node's output function? If so, how are they different from each of the other nodes on the same layer? $\endgroup$ – mskw Aug 30 '17 at 15:22
3
$\begingroup$

In standard logistic regression we have 1 output in the final layer. However with a single hidden layer neural network, we can have multiple intermediate values each of which can be thought of as an output of a different logistic regression model i.e. we are not just performing the same logistic regression again and again. It is then not a large jump to think that it is possible that the combination of these has greater expressive capabilities than the standard logistic regression model (and also has been shown in practice and theory).

You also mention in the comments about how these nodes have different values in the same layer if they have the same inputs? This is because they should have different weights. Each node in a neural network takes $N$ inputs and produces a value $\displaystyle y_j = f\left(\sum_{i = 1}^N w_{ji} \cdot x_i + b_j\right)$ where $f$ is some chosen function, in our case the sigmoid, $w_{ji}$ are the weights, $x_i$ are the inputs, and $b_j$ is some bias. The weights are chosen by an optimisation algorithm to optimise our objective e.g. minimise classification error. Initialisation is very important for the gradient descent algorithms that are usually used to optimise the weights. See https://intoli.com/blog/neural-network-initialization/ where if all the weights start off at 0, the network is unable to learn.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.