1
$\begingroup$

I'm trying to use a neural net for classification and I am getting stumped on how to use it for noisy data. My problem seems simple; I have 1000's of experiments that have features and these features either hit a threshold or don't. I know if the experiments are success' or not, but since the data is noisy I can have two inputs map to different outputs, e.g.,

[1,1,0,0,0,1] -> 1
[1,1,0,0,0,1] -> 0

What I don't understand is how to use neural nets when I have thousands of runs to classify. Most of the examples online deal with XOR functions where there is a 1-1 mapping between input and output like here. How can I use neural nets to classify these inputs where the maps are not 1-1?

If I wanted to use the code in the link how can I do it? The weights would correspond to the same input, no? For example, how would I construct a neural net that could be written as this:

   nn = NeuralNetwork([6,2,1])
    X = np.array([[1,1,0,0,0,1],
                  [1,1,0,0,0,1],
                  [1,1,0,0,0,1],
                  [1,1,0,0,0,1],
                  [1,1,0,0,0,1],
                  ...])
    y = np.array([1, 0, 0, 1, 0,...])
    nn.fit(X, y)
    for e in X:
        print(e,nn.predict(e))

where the ... means more data and of course, I have other inputs and outputs, [1,1,1,1,1,1] -> 1, etc.

$\endgroup$
  • $\begingroup$ Neural networks don't have to be only for perfectly separable data. $\endgroup$ – gung Oct 24 '16 at 18:11
  • 1
    $\begingroup$ @gung is correct. There is no specific requirement that training data be purely-separable. If the data are noisy, this will manifest as more uncertainty in particular decision regions, but this is no different from any other classifier. $\endgroup$ – T3am5hark Oct 24 '16 at 18:16
0
$\begingroup$

You simply feed all input-output pairs into the training algorithm. There is no requirement in neural networks that the training data exhibit deterministic separability as a function of the input values. Most real-world problems will involve training data that are not perfectly separable, which is expected when there is some noise in the responsible generating process.

Most useful classifiers will respond to this condition by reflecting the tendency in a given decision region to be a 1 or a 0 (in a binary classifier). This tends to manifest as greater uncertainty (equivalent to steeper gradient of pre-thresholded classifier output value with respect to the inputs) in regions where greater overlap is seen in the training data. This will be true of most classifiers, including neural networks.

A true probabilistic classifier may express this explicitly in terms of probability. A neural network won't necessarily do this as it depends on structure, although if a logit activation is used for the output layer (just for instance) it will share some characteristics of probabilistic logistic regression. However, one has to be careful with any strong statements about equivalency here because the function of preceding layers will influence outcomes greatly depending on choice of activations, layer depth, number of hidden features, and so forth.

As an aside: your sample data shows no variation in the input feature space (but the outputs vary). That's not likely to result in anything useful in terms of training since these data are purely co-linear (stochastic gradient descent needs to be able to sample many different points in the input space to produce anything other than trivial behavior from the network).

$\endgroup$
  • $\begingroup$ thanks for the reply. My training set is in the thousands and contains all variations of 1's and 0's, I was simply showing the basic code. Let me ask it this way: Let's say my input is [0,1],[0,0],[1,1],[1,0],[1,1] and my corresponding output is [1,0,0,0,1]. The output from the neural net will be the weights. What is the interpretation for the two weights that correspond to the two [1,1] instances? $\endgroup$ – superhero Oct 24 '16 at 18:41
  • $\begingroup$ Depending on what structure you use, there could be a lot more weights than that (corresponding to number of neurons). If you want a very simple, easily-interpretable model, logistic regression will give you simple weights that are easy to interpret (but limited in their ability to describe complex decision regions). Neural networks are great at automatically adapting to very complex decision regions, but can consist of a whole bunch of neurons with lots of weights going every which way. That makes it difficult to characterize what's going on in them apart from the input-output behavior. $\endgroup$ – T3am5hark Oct 24 '16 at 18:44
  • $\begingroup$ In your case, it looks like you have 6 input neurons, 2 hidden neurons, and 1 output neuron (decision classifier). That means that there will be 6x2 weights in the hidden layer, and 2 weights in the output layer (if I'm reading it correctly). The weights at the output layer will just reflect weights on the 2 hidden features. What will the hidden features look like with respect to the input space? They'll look like whatever helps the network get the best performance on the training data. Interpretation of weights in hidden layers is a notoriously complicated question. $\endgroup$ – T3am5hark Oct 24 '16 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.