1
$\begingroup$

I have been using a neural network to classify an object as either 'good' or 'defective' (2 classes). It seems to be doing a decent job of it except in certain unforeseen/unexpected scenarios.

enter image description here

For example, in the above image, 'good' objects are shown in green, and the 'defective' ones are in Red. The second last row has a large object formed by the merger of many objects. Ideally, it should be 'Red', instead the trained network classifies it as 'Good (Green)'. The reason is that such an object was never seen during the training phase. Also, see the left most object in the first row from the top. It should also be 'Red'; but, because it has features never seen during the training, the network classifies it as 'good'. This has been a general trend so far where all such unknown shapes are classified as 'good' with very high classification score.

Now, there could be many such unexpected scenarios with unnatural shapes (see image below).

enter image description here

so, it is not possible to create a training set to train a network. I would instead like to create a network where such unknown shapes would by default be classified to 'defective (Red)' category, or to any other fixed class category of my choice.

I use binary objects for training, so the features are statistical in nature (area, diameter, form factor, convexity etc..). Any suggestions as to how can I accomplish this?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Neural networks learn from data and are able to generalize to examples which are new yet similar in some form. Hence, if you have something completely different and never seen before, the network will not be able to classify it correctly.

One thing that you could do is move the decision threshold for your classes, for example instead of class good being over 0.5 you move it to 0.7. This way only objects which you are more certain are good are classified as such, all other objects are classified as bad.

To decide the best place for your decision threshold you should look at the ROC curves for your classifier, and choose the compromise between the False Positive Rate (FPR) and True Positive Rate (TPR) that best fits your particular problem.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The difficulty I am facing is that the classifier indicates a very high level certainty for such cases. I am consistently getting classification scores > 0.95, which is often more than some of the obvious 'good' candidates. I tried to train the network with more number of 'defective' samples than 'good' samples (40% good - 60% defective) in the dataset to check if the network parameters are somehow biased towards a particular category. But, this has not been useful. $\endgroup$
    – KevalGyan
    Jul 22, 2017 at 16:04
  • $\begingroup$ If that is the case I really don't know how you can prepare for those cases without showing them to your network first... What is stopping you from including these difficult cases in the training set? Since this is probably the best solution anyway... $\endgroup$
    – Miguel
    Jul 22, 2017 at 16:26
  • $\begingroup$ Yes, I know this is the best solution. The only problem is that I am trying to identify the pair of points on the object boundary to separate the individual objects. So, no matter how large a dataset I would prepare, how many such shapes (especially the one in the binary image above) I would consider, there is always going to be a case I would not have/failed to have considered. $\endgroup$
    – KevalGyan
    Jul 23, 2017 at 16:02
  • $\begingroup$ I don't think that would be the case, assuming all none defective objects are almost identical (their features have almost the same values) if you show your network a couple of cases where the objects are defective because their features deviate from what is expected for good objects then your network should be able to learn that objects whose features differ from a given standard are always defective. If you give your network these bad objects during training it should be able to generalize. $\endgroup$
    – Miguel
    Jul 23, 2017 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.