Improving recall in a neural network

Question

I built a neural net in Octave to predict if products are a match or not using a number of features. The net has 8 input features, 1 hidden layer with 8 nodes and outputs either match or no match. I employ feature regualarization and use fmuncg to perform gradient decend.

I trained it with 30,000 examples with +/-10% of those training sets being positve matches.

Here is my problem: The precision on the net is excellent, around 92%. Recall is very low at less than 10%. Playing around lambda in the regularization is not helping much, nor is performing more iterations in fmuncg.

I dont want to balance the training set because that would reduce my data a lot. Is there anything else that I can do?

This is my cost function:

J = (-1/m)*sum(sum(y1.*log(r)+(1-y1).*log(1-r)))+ (lambda/(2*m))*((sum(sum(nTheta1.*nTheta1))+sum(sum(nTheta2.*nTheta2))));

Is there a way that I can change it, to penalise false negatives more?

Answer 1

Your second question has been answered; as such I will attempt to answer your first question "Is there anything else that I can do?":

As time is the commodity of most value, I would recommend spending time on identifying the cause rather than randomly changing things on gut feel (precision and recall do not always answer question on bias and variance nor the appropriateness of the model chosen for classification. As such:

In your neural network implementation determine if you have a high bias or variance (e.g., see here), i.e. is your high precision and low recall due to under fitting High bias or over fitting High variance your positive examples as the methods for solving these issues differ from those for high variance, i.e.:

• Getting more training examples -Fixes high variance
• Trying smaller sets of features -Fixes high variance
• Increasing lambda -Fixes high variance
• Adding features -Fixes high bias
• Adding polynomial features -Fixes high bias
• Decreasing lambda -Fixes high bias

Examining the cases that are incorrectly classified can help determine why the model failed to classify these correctly (graphing and colouring the features on 2d plots can help sometimes (feature scaling / compression may be of use/ can make this possible in some cases).

Other good options include using a different architecture on your neural network, a different algorithm or modified features per below:

Consider treating your problem as anomaly detection, i.e. When normalizing your data (chose functions of features $x$ such that you get a Gaussian distribution i.e.:

• $f_1(x) = log(x +c) $
• $f(x)2 = x.^{1/2}$
• ...
• $f_8(x) = x.^{1/3}$

(check by plotting histograms) also Multivariate Gaussian models may be of value for new features that have some kind of correlation (e.g., see here).

A density detection algorithm could be applied which may better fit your needs than a neural network (a formula for which could be found in Wikipedia per above and in this publicly available article, it is also possible if this method is used to create the 3 subsets of your +- 27 000neg and +- 3000pos examples slightly differently, i.e. allow your training set to have say 60% of the negative examples (16 200) and it would have no positive examples, allow your cross validation set to have (5400 neg and 1500 pos) and your test set to have (5400 neg and 1500pos).

You can use your cross validation set to automatically choose your threshold and allows you to adjust your system and tweak it.

As measures of success consider different evaluation formats and counts (sometimes seeing how items were classified in different measures can help you visualise the problem more clearly):

Calculate your F score (as this may be a good second way to view how well your algorithm is doing since your data set is bias to negative examples the formula for which is simply: $2*((\text{precision}*\text{recall})/(\text{precision} + \text{recall}))$.

And finally use your test set to verify classifiers' ability without any tweaking based on these results.

Answer 2

The part of your loss function related to the prediction accuracy is:

sum(sum(y1.*log(r) + (1-y1).*log(1-r)))

Assuming that y1(n) == 1 indicates that test case n is a positive example. Changing this to:

sum(sum( POS_WEIGHT * (y1 .* log(r)) + (1-y1).*log(1-r))) 

will increase the penalty associated with getting a positive example wrong by a factor of POS_WEIGHT. Play around with this value to find something that works, but I'd start with num_negatives / num_positives and go from there.

This technical report found that adjusting the cost penalty was more effective than either up or downsampling in most situations.