0
$\begingroup$

In the following code I am training a multi-layer-perceptron (MLP) with 3 classes using Backpropagation learning algorithm. The data set is arbitrary consisting of 4 features and 8 examples. The training is done using these 8 examples. I have used the Sigmoid activation function. This function maps real valued data to real value. But the class labels are binary valued. In most examples, I have seen sigmoid activation to be used. I used the round function to force the real valued class labels to binary but this gives incorrect class lables.

Activation function is a sigmoid function. When the input pattern is [-0.3 -1 0.2 0.11] the target class output should be 0 0 1 : out=1./(1+exp(-(op_sig*w_ho))); where out variable is the class values. I am getting floating point values as the predicted class:

predicted =

    0.1224     
    0.5390     
    0.5532    

instead of Actual = [0 0 1]';

So, I changed the output class labels using round().

        op_w=in(:,j)'*w_ih;  %multiplying the input pattern with the weights

        op_sig=1./(1+exp(-(op_w+w_bhid'))); %applying the sigmoid %      activation on the bias

        out=round(1./(1+exp(-(op_sig*w_ho)))); %generating class labels

        e=desired_out(:,j)'-out;  %calculating error

The error plot is weird as well given below.

error plot

Problem: What is the correct approach? Do we not convert the floating point values to binary if the class labels are in binary?

$\endgroup$
1
$\begingroup$

In the question, it is stated that only 8 patterns comprise the complete training set. Assuming that the number of hidden nodes is at least 2 (in a one-hidden layer configuration), you have more parameters in your model than training patterns. This fact prevents not only generalization, but also the output nodes do not at all sum to 1, after ended training.

In their pioneering work, Baum and Haussler analyzed the relation between the number of training patterns to the number of weights in a feed-forward neural network [Baum, E., & Haussler, D. (1989). What size net gives valid generalization? Pages 81-90 of' Touretzky, D. S. (ed), NIPS 1].

It is clear that you need several training patterns per weight (per free parameter) in the feed-forward neural network. This in order to guarantee a good generalization, and a low error rate as bound by the Vapnik–Chervonenkis dimension.

When (many) more training patterns are present in the training set than weights in the feed-forward neural network, after succesfull training the outputs approximate the posterior class probabilities, and generally sum to 1 (with only tiny decimal deviations from this). See the reference work by Richard & Lippmann where this result is proven. [Richard M. D., & Lippmann R. P. (1991). Neural network classifiers estimate bayesian a posteriori probabilities. Neural Computation, 3, 461– 483]. Your class indicator output vector was chosen correctly, i.e. Target = [0 0 1]'works fine.

General theory of statistical classification

In 1936, Fischer invented linear discriminant analysis. The goal is to categorize a set of patterns (called observations within statistics), into one of $c$ classes. Within statistics, all is uncertain, also the class membership of each pattern. Probabilities that are purely 0 and 1, that indicates a deterministic mechanism, for which statistics was not developed. As soon as uncertainty comes into play, a deterministic paradigm is left and it becomes statistics and probability theory.

Feed-forward neural networks can represent deterministic mappings (think of the n-parity problem which is often used as a benchmark problem for testing training algorithms. N-parity: does the input pattern contain an evenly number of '1' ?). Nonetheless, by far the most interesting classification application of feed-forward neural networks is that of statistical classification, where the feature distributions overlap, for the different classes. In this case, you train with indicator vectors (Target = [1 0 0]',Target = [0 1 0]' and Target = [0 0 1]'), but the neural network learns the probabilities of the three classes.

In the mathematical annotation of Duda & Hart [Duda R.O. & Hart P.E. (1973) Pattern Classification and Scene Analysis, Wiley], define the feature distributions provided as input vector to the feed-forward neural network as $P({\bf{\it x}}\,\mid\,\omega_i)$, where for example the data vector equals ${\bf{\it x}}=(0.2,10.2,0,2)$, for a classification task with 4 feature-variables. The index $i$ indicates the possible $n$ classes, $i \in \{1,\ldots,c\}$, and $\omega_1,\omega_2,\ldots,\omega_c$.

The feed-forward neural network classifier learns the posterior probabilities, ${\hat P}(\omega_i\,\mid\,{\bf{\it x}})$, when trained by gradient descent. This is the major result proved by Richard & Lippmann in 1991. The hat over the posterior probability indicates the uncertainty as the probabilities are estimated (learned).

$\endgroup$
  • 1
    $\begingroup$ Thank you for your answer. But I think you misunderstood some parts.The code does not return [0 0 1]. It gives real valued numbers for the class labels. Upon eye balling these real numbers and doing round operation, the class labels come out to be 001. Basically, when training and testing I should expect a vector of class labels and not floating point values. So my confusion is whether there is a way to get class labels instead of real numbers and how to apply the onevsall rule -- maximum of the probability to get the classification. $\endgroup$ – SKM Apr 12 '18 at 21:57
  • $\begingroup$ Yeah, I'm aware of the fact that the NN outputs are real-valued numbers. In my answer, I changed the desired output for a pattern belonging to the 3'rd class into 'Target'. The real-valued outputs should converge towards the posterior class probabilities of the classes 1, 2 and 3. (And, in the general case with C-classes, the posterior probabilities of each of these C classes). You can subsequently apply the winner-takes-all rule to choose the most likely class of each pattern. $\endgroup$ – Match Maker EE Apr 12 '18 at 22:03
  • $\begingroup$ Sorry, it is still unclear to me why the error is calculated between the real valued numbers and the binay valued target class and not between similar data types? out=(1./(1+exp(-(op_sig*w_ho))));` %generate class labels which are real valued and if the class labels are in binary say [0 0 1], then would the error be calculated between real and this binary? $\endgroup$ – SKM Apr 13 '18 at 6:34
  • $\begingroup$ See my extended answer above. $\endgroup$ – Match Maker EE Apr 13 '18 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.