So I've just recently gotten into artificial neural networks, and I have a couple of questions that I can't seem to find addressed anywhere.

Firstly, this one is more specific to image recognition, is it necessary to have a separate output node for each possible outcome and then analyze the results, or is there one output neuron that just gives out the final result? e.g. one neuron that outputs the amount of "dog" in the picture, another that outputs the amount of "cat," and so on vs. a single neuron that determines whether it's a cat or a dog or something else based on the inputs.

Second, based on what I read, in order to do backpropagation you don't alter the weights on the input layer until you achieve a certain level of accuracy with the hidden and output layers. I was just wondering why this is, and what purpose it serves if the current weights already achieved a desired accuracy. Is it more fine-tuning the closer you get to the input?

Third, why is it that "any application for a nn can be done with only 3 layers"? This seems a bit off to me, since I saw some talks on image recognition and they usually have 5+ layers. A quick explanation for the reasoning behind this or a specific source would be great.

Thanks, and if I somehow asked a duplicate question, please let me know and direct me to the original.


closed as too broad by Franck Dernoncourt, Michael Chernick, kjetil b halvorsen, John, Peter Flom Jan 21 '17 at 13:33

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Please write a more informative title. $\endgroup$ – Franck Dernoncourt Jan 21 '17 at 1:42

Adding on to Maxim's answer:

  1. How would you implement a single unit outputting catness, dogness, birdness, chairness, ...? It's not necessarily impossible, but doing separate classification nodes for each class is simple and effective. It's similar to the "one-vs-rest" multiclass classification scheme in more traditional classification methods, where you've trained one classifier for cats, another for dogs, and so on, and you take the most confident one. In the ANN case, you're doing exactly this but with a shared set of feature extractors beforehand (the earlier layers).

    In terms of the rounding approach that Maxim suggested: that scheme is never going to work, as he notes. The earlier layers would need to produce features such that a single linear combination produces "cat" in a certain range, "chair" in another range, "forest" in another; that's a much harder problem than producing features where different linear combinations can recognize "cat", "chair", and "forest."

  2. I've also never heard of this rule, but Maxim's thoughts seem reasonable.

  3. This is probably referring to the fact that shallow, wide neural networks are universal approximators: you can represent any continuous [or piecewise-continuous] function using only a single hidden layer. Though mathematically interesting, this is not really a useful statement beyond the basic sanity check that the representation isn't terrible. Such networks are nearly impossible to learn, and much be extremely wide in order to learn complicated functions, since they're just piecewise approximators. Deep structures can represent certain classes of functions with exponentially fewer parameters, and because they can generalize much more effectively than shallow networks may be learnable with fewer training examples when their architecture is chosen appropriately.


Hope this helps:

  1. Single output node in theoretical concept is possible, as long as you know what is the CUTOFF of the value. For example, let say you want to use 1-layer ANN to build a logic gate, you can use different set of weights to construct an AND gate, and OR gate. And there is only one output node in each ANN (like the dog example you give).

Here it's the OR gate (h(X) is a sigmoid function, which introduce non-linearity to the neural network. The non-linearity is important because that how makes you question 3 possible---using three layers to solve non-linear classification problem): enter image description here

Here it's the AND gate: enter image description here

IMPORTANT!: Usually now what people do in neural network is that they introduce a softamx layer (i.e., logistic regression) in the LAST layer of neural network. Why this is done because let's say you want to have classify 4 animals (cow, cat, dog, chicken), you use softmax last layer and you can OBTAIN 4 PROBABILITIES for being cow, cat, dog, or chicken. Then ANN will calssify your input based on the HIGHEST probability. For example, given your input, the output probabilities are: cow--0.1, cat--0.1,dog--0.1, chicken--0.7. thus, the ANN will classify your input image as chicken.

  1. Let's consider the network in the figure below. It is a ANN with k-layers, as defined as L1,L2,...Lk. The input layer is the first layer, i.e., L1, and the output layer is Lk (we want to classify red dot or blue plus). When you do backpropagation, your delta(error) will be calculated in layer Lk, and it is propagate back to previous layers (started from L(k-1) back to L1). So, the weights, W(1)...,W(k1-1), can be updated. Therefore, in fact, ALL your weights are updated in this case. In your question, you are saying the weight, W(1), will not be updated, right? I have not seen this before. Can you provide a link where you see it and I can try to explain to you. enter image description here

  2. The claim is a big claim :). People have proven that using 1 hidden-layer can make the ANN to do a non-linear classifier, such as making an XNOR gate. That is, using 3 layers (input, hidden, output) can solve both linear and non-linear problem. But that does not mean "using only 3 layers" can solve all the non-linear problems. The current research finding is: to solve difficult classification problem, such as classifying 1000 objects, you need to use DEEP arhitecture. That is, using more than 1 hidden layer (sometime can be more than 10) to build a deep neural network to do classification.

I hope my answer can give you some insight. You questions cover a board range of concepts in ANN. And it is difficult to explain all here. Please do not hesitate to let me know if you have any questions, and I will try to respond when I see them. :) Good luck.

  1. Yes, you can use for example linear output layer which will return some real value by just summing the inputs. You could encode all of your classes like 0 - dog, 1 - cat, 2 - bird etc. and get an out put of, say 1.75, which you will need to round to understand what is your real class label. But I strongly recommend to avoid this - the performance will be horrible. Also you could just have the structure with multiple neurons (each one is for one specific class) and just add one neuron with argmax() function (which will just return the number of the previous neuron with biggest output). It will also work. But I never heard that anyone has problems with the multiple output neurons, why you don't want to use them?

  2. Never heard about such rule, honestly. I could just say that there is a problem with the vanishing gradients in back prop, which means that when you use backprop to train a network with really big amount of layers the weights which are far away from output just don't get any significant updates on each iteration. That means that only several layers close to output layers are actually trained by backprod. There are ways to fight this issue, actually (which are studied within Deep learning domain - LSTM, multi-level hierarachy, usage of pretraining with RBM etc)

  3. I don't know where you've got this quote. It is just not true - deep learning domain could demonstrate cases where you have, say, 20 layers, and they could have different types (some of them convolutional, some are max-pooling layers, some are ordinal layers).

  • $\begingroup$ 1. I wasn't necessarily suggesting a preference, it was just an honest question on which is the better approach since I didn't know that much about how the outputting process worked. Thanks for the answer, makes a lot of sense. 2. That rule was based on my limited reading, and I may have misinterpreted some of the information, but thanks for clearing it up. 3. As Dougal already cleared up, this was another mixup between theoretical math and actual approaches to the implementation, thanks for clearing up some of the deep learning cases. $\endgroup$ – user193860 Jun 22 '15 at 17:49

Not the answer you're looking for? Browse other questions tagged or ask your own question.