I am learning the backpropagtion algorithm, and would like to clarify some concepts.

Suppose my training data set consists of 20-dimensional bit strings that are classified into 5 different classes. Then my neural net has 20 inputs, and 5 outputs:


Is this correct? What if some new test data yields something like $11000$. How does one interpret this? The data can be in class 1 or 2?

Now let's consider a slightly different problem: Suppose I have inputs consisting of 20-dimensional bit strings, and their outputs are 5-dimensional bit strings. For example,

00000000001111111111 --> 01001
11111000001111100000 --> 11000


I want a network to make these sorts of predictions. So my network should still have 20 inputs, but can I still just use 5 outputs? Following the first example, it looks like I should technically have $2^5$ outputs, each corresponding to one of the possible outputs.

What is the relationship between these classification/prediction problems? Are my setups correct?


3 Answers 3


11000 is an invalid class if your classes are mutually exclusive and if you have only five classes consisting of 10000,01000,00100,00010, and 00001.

11000 is no longer an invalid class once you define your class space to be all digits between 0 and 2^5. Ultimately it's up to you to decide what types of classes you would like to classify. Likewise for inputs: you can use however many inputs you would like as long as you are consistent between training and test/predicting.

Your dataset sounds very abstract and that could be what is clouding your thinking. If you're just starting out with machine learning and neural networks, go with a well known dataset first so that you only need to think about the algorithm. A good example is the MNIST handwritten digits dataset. All the classes are digits 0-9 (10 total) and so there is no confusion about classes falling outside of 0,1,2,3,4,5,6,7,8,9.

Google's Tensorflow library has a good MNIST tutorial here: https://www.tensorflow.org/versions/r0.7/tutorials/mnist/pros/index.html.


Each of the 5 classes you are predicting are independent of each other. If you look at the way Neural Networks are constructed, each input has a weight that ultimately ends up activating, or not each output.

When you get a yield of 11000 it means that the data you trained the Neural Network on has cause the network to activate by surpassing the weights trained into the neurons for both classes, which is possible.

In the second example you give, each of the 20 inputs to the neural network have weights to the 5 outputs which could be activated. Those weights are increased or decreased based on the training data, the eta value, and the number of iterations the model used to train.

In short, any number of features could activate, or not, every output class. The activation is simply statistically related to the training data and the configuration of the network. The more training data you have the more potential accuracy.

  • $\begingroup$ This still isn't very clear to me. In the first example, my 5 classes are $10000,01000, 00100,00010,00001$. Is there a difference if I instead set the 5 possible classes as $00000, 00001, 00011, 00100, 00101$? $\endgroup$
    – Harv
    Mar 21, 2015 at 4:22
  • $\begingroup$ If you look at the image on the wiki of the neural network layout (en.wikipedia.org/wiki/Artificial_neural_network) each arrow is a weight. The weights are increased or decreased as data is trained into the system. Each input is connected to each other output. When you go to predict new data there may be an case that will activate 2 outputs because it was never seen in the training data. Neural Networks activate each output node based on the inputs related to each activated node in the training data, not a list of "valid values" $\endgroup$ Mar 21, 2015 at 4:31

Lets take an example to clarify your query. Assume the 20 inputs are pixel intensity values for a 5x4 pixel image, and the neural network is being trained to recognize the digits 1 to 5 from these images. In this case, the interpretation of class 10000 would be digit '1', 01000 would be digit '2' and so on.

But a neural network generally gives a continuous valued output, so the output would be something like 0.9 0.2 0.1 0.7 0.1. From this "prediction", you would have to determine the output class by applying a threshold/cutoff such as 0.5 to get the output class as 10000. Alternatively, you could select the class with highest probability, for example - if the output is 0.1 0.4 0.05 0.1 0.3 then select class as 01000 since the second output has the highest probability.

The answers to your questions are:

  1. Yes, the class representation interpreted by you is correct, but the output would not be a discrete number 0/1, it would be a continuous valued output which you would need to interpret and convert into a class.

  2. Regarding interpreting the output 11000, in case two of the five outputs have exactly the same probability, then, its a symptom of poor training. The remedy is to select better features and/or more training samples to improve the neural network's ability to identify the class appropriately.

  3. In case you need the output to be a five bit string, and if that is your real objective, then the network needs to be trained for such inputs. One such example could be to produce codes from input data, e.g. an auto-encoder which reduces the dimensionality of a given dataset.

  4. The relationship between prediction and classification is that classification is the task of converting a continuous valued prediction into a class membership within a set of discrete classes.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.