0
$\begingroup$

I'm newbie in machine learning and trying to understand neural network. I know how logistic regression works and on this basis I try to understand how the neural network works. I'm trying to grasp intuitively how NN works. Could you check, please, if I understand correctly:

  • neural network it's roughly saying just extension of logistic regression (in case when we solve classification problem and use sigmoid function);
  • logistic regression trying to aproximate data with sigmoid curve;
  • neural network it's a set of functions which approximate input data (in case with one hidden layer. If NN have more than one hiidden layer i just can't understand how to extend it)

Also I have question: If I understood correctly that NN is a set of functions which approximate input data, why output layer is needed (or more layers)? For example, we have one hidden layer with 4 nodes and output layer with one node. I suppose that we've got 4 functions, which approximates input data in different ways and than output layer approximates output of hidden layer. I don't know how to think about it in context of approximation.

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
1
$\begingroup$

A NN does not necessarily use logistic regression as the activation function (ie, how a neuron activate and what output value it produces), but if you choose a logistic regression, then yes, it's like having as many logistic regressions as neurons.

However, what makes the ANN "magical", is the possibility of combining these functions by weighting the connections, and the ability to learn "automatically" these weights from a training dataset. So yes in your case, you can see your ANN as a combination of logistic regressions.

If you add more layers, you get a combination of combinations of logistic regressions, which can potentially derive more complex features.

In the end, this is this combination of multiple functions that allow for the generation of a non-linear output, which is part of what makes ANN powerful (and sometimes difficult to deal with...).

Improve this answer
$\endgroup$
1
$\begingroup$

I could not comment so add an answer instead. Within Cross Validated, there are already excellent discussions about this topic. For a solid understanding of basics of neural networks, I would recommend the book Neural Networks and Deep Learning by Michael Nielsen.

For your question, logistic regression for binary classification can be viewed as a special structure of neural networks, it's a sigmoid transformation of linear combination of input vector with a decision boundary. A neural network itself as a whole is an approximator for the data, and true, each node's output can be regarded a function of its forward pass, otherwise the backpropagation wouldn't be possible. But I think it is not accurate to say each node as a function approximates input data. My understanding is that each node extracts certain information out of input data such that a proper network structure could approximate the mapping between input and output.

Improve this answer
$\endgroup$
5
  • $\begingroup$ "But I think it is not accurate to say each node as a function approximates input data." --> this is the case, if you initialize all your edges with the same initial weight value, you will end up with the same output for each neuron. That's why the weights must be initialized randomly (or one must add some stochasticity in the backpropagation), in order to differenciate the neurons. $\endgroup$
    – gaborous
    Feb 10, 2018 at 22:29
  • $\begingroup$ @gaborous What you said is absolutely true, but we need a clear definition of "approximate" to relevant it to my comment. According to the dictionary, it means "close to but not completely the same as". If we try to approximate the input data, I took it as "reconstruct" the data, which acts like an autoencoder. This is not realistic for a "subfunction" defined by a single node (from the original question: "4 functions (nodes), which approximates input data in different ways and than output layer approximates output of hidden layer. "). $\endgroup$
    – doubllle
    Feb 11, 2018 at 7:08
  • $\begingroup$ it all depends on the input data. A neural network approximates as long as its structure and activation functions lead to a lower dimensionality than the one of the input data (cf VC dimensions theory). But in practice this might be hard to estimate :-) In any case, the goal of each node is to model the input as good as possible and either restitute it or transform it (depending on activation function), that's the whole basis for backpropagation (and probably of natural neurons, see Sophie Denève's works). $\endgroup$
    – gaborous
    Feb 11, 2018 at 13:34
  • $\begingroup$ Thanks for the comments, I will check that out. I was thinking that the capacity of each single node was far from approximating input data. $\endgroup$
    – doubllle
    Feb 11, 2018 at 16:00
  • $\begingroup$ In general yes but it all depends on the complexity of your input data and of the activation function you use :-) In fact one could even imagine an activation function that would itself be a neural network, in theory nothing prevents that! (although in practice, good luck to learn such a network, but it could even be a ground-breaking discovery if it works, as we are now suspecting a similar topology in brains, named columnar organization). This seemingly basic question is in fact far more exciting than what can be assumed! :-D $\endgroup$
    – gaborous
    Feb 11, 2018 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.