9
$\begingroup$

I'm trying to fit a dense neural network based on tabular data input, where the outputs are two separate classification vectors, with one cross-entropy loss function for each.

Example: given a few input features, for a customer that visits a travel website with the intention of buying a train ticket, the model would predict both the destination of travel and the traveling class (1st class or 2nd class) that the customer is likely to buy.

Problem: it seems as if internally, the network was divided in two at some point in the hidden layers, and each sub-network got specialised in predicting one output vector, ignoring the other. This leads to an overall acceptable accuracy for each output, but the consistency between the two outputs leaves to be desired.

illustration of neural network

For example, for a given entry, the network would predict "London" and "1st Class", because independently, each output makes sense according to the input features, but there isn't a single training point where London and 1st class can be found together, simply because there isn't a 1st class option when travelling to London. The network seems to be completely devoid of any concern for the consistency between the two.

Example, if the passenger is an accountant, 35yo and departs from Brussels, the training set gives a clear winner for destination: London and, separately, also for class: 1st, and so this is what the network will tend to predict, despite this combination being totally absent.

enter image description here

Would there be any way to amend the network and/or the organisation of the loss functions so that the consistency between the two outputs would be taken into account, and the network would avoid combination of outputs that can't be found in the training set, and favor those that are?

More generally, what would be some good approaches to tackle this issue? Note that I would like to avoid resorting to manual rules down the line, if that is possible.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I would put a freeze on the current net work, and cap it with the head, and train the head only to the correspondence. There’s an argument to be made that if you know the correspondence that you only need one output and you can make a simple equation that turns one output into it adjoint. $\endgroup$
    – EngrStudent
    Apr 10 at 13:09
  • $\begingroup$ Thanks. What do you mean by « train the head only to the correspondance » (not clear what the « to the correspondance » bit)? $\endgroup$
    – Jivan
    Apr 10 at 14:54
  • $\begingroup$ @SextusEmpiricus presently, there are two loss functions: one for the destination, one for the class, and they are independent of each other. I’m trying to think of a way to incorporate a loss function for the combination of the two, but not sure how to make that differentiable. $\endgroup$
    – Jivan
    Apr 11 at 13:49
  • $\begingroup$ @SextusEmpiricus for your first question, the answer is that with the present setup, it can’t, and this is the main issue. $\endgroup$
    – Jivan
    Apr 11 at 13:50

3 Answers 3

Reset to default
4
$\begingroup$

Another method would be to build two neural networks. The first NN is trained to predict the destination. For the second NN, include the destination predicted by the first NN as an input feature and train the network to predict the class. The second network should then learn to only predict classes that are options for the predicted destination.

Edited in response to @Jivan's comment.

There are more complex methods of multi-label classification, but I'd keep it simple if possible, and try either @Dikran's or my approach first. They are both standard ways of implementing multi-label classification (see this Medium post). Dikran's method is a Label Powerset and mine is a Classifier Chain. As you've pointed out, there are pros and cons to both these methods. If neither of these produce a good enough result, you could try a variation of the classifier chain, where you build one network to predict one label from the union of destinations and classes. Then train two further networks, one that predicts the destination given a predicted class and the other that predicts the class given a predicted destination. At inference time, you would use the first network to predict either a class or destination, then the appropriate second network predict the other label.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is sound advice, however on reality both outputs can affect the other. Here the causation is clearly destination to class, because people choose their class based on what’s available with the destination. In the real case, it can go both way (people could chose their class and then decide among suitable destinations). $\endgroup$
    – Jivan
    Apr 10 at 14:53
10
$\begingroup$

If consistency is a problem I would make it a single classification task where "London first class", "London second class", ..., "Rome first class" and "Rome second class" were distinct classes, rather than make it two distinct classification tasks. You current network architecture is giving the a-priori hint that they are completely distinct classification tasks, but if e.g. some destinations don't have both classes, then there is a dependence between the two sub-classes. Combining the two classification tasks into one would be the easiest way of putting the dependence back into the model.

At the moment, I think your model is predicting that the customer would opt for a first class ticket if it were available, which is not an unreasonable answer - it is just generalising the idea that people in relatively well paid occupations (e.g. accountant) tend to travel first-class. You could always just ignore the class output where it is not an option.

Does the network really need so many layers? It could be that a single hidden layer may be sufficient for this problem and the layer above that is not actually doing much useful processing, in which case the division of the network may not be that meaningful.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thanks! This is a sound idea, however I observed that it causes another problem, namely that the network is less able to generalise on e.g. destination when a single destination is dominant but scattered among many classes (there are more than two in the real case). There might be 80% London but scattered across 10 classes, and there is another single city with 20% but with a single class. Ideally then, I’d wish the network to say London with 80% probability. Lastly, this is a toy example. In reality, dozens of features, thousands of possibilities for each output. $\endgroup$
    – Jivan
    Apr 10 at 14:49
  • $\begingroup$ @Jivan in that case, why not just sum over the classes that include London as the destination? If there are thousands of classes, is their any heirarchy that you could exploit (e.g. country rather than city of destination) and then have sub-networks for the city? $\endgroup$
    – Dikran Marsupial
    Apr 10 at 16:05
  • $\begingroup$ What do you mean by "sum over the classes that include London"? $\endgroup$
    – Jivan
    Apr 10 at 18:36
  • 2
    $\begingroup$ In your comment, you wrote "There might be 80% London but scattered across 10 classes," I took that to mean there were 10 classes that included London as the destination. Summing the probabilities of those ten classes would give the probability of London being the destination, marginalising over the other aspects of the classification scheme (i.e. ignoring them) $\endgroup$
    – Dikran Marsupial
    Apr 10 at 18:45
3
$\begingroup$

Cost function

In what way would your neural network be able to know that the 1st class with destination London is not feasible? How do you teach that to the network? In what way did you 'punish' the network during training for wrong predictions?

It is important that the training phase allows the network to train the desired features. In your question, you did not tell which cost function you used to train the model.

It is also not clear what type of output is created by your model and what you would desire from it. Do I guess correctly that the output is just a single class prediction? In that case, what class prediction would you favor in the example from the question.

Is 'London 2nd class' a better prediction than 'London 1st class'?

When this cost function only cares about a single error then it is gonna care less about combined errors. That might lead to your problem (I am assuming that this is how your cost function is created, but it is not clear).

Predicting London + 1st class will be wrong in the 89302 cases when the true value is Londen + 2nd class. But the choice to predict the 1st class instead of 2nd class might be rewarded in the 48516 + 41411 + 38186 + 35247 + 28512 cases when the true value is Paris/Rome/Berlin/Madrid/Rotterdam + 1st class (I am not sure, but I guess that your cost function is doing this).

You can punish the system for making predictions about 1st class when it is in London, but at the same time you reward 1st class predictions when the occur in other cities. So you are getting Londen 1st class as result.

Type of output

I mentioned earlier that I am guessing that your model is just giving a single class prediction. I am guessing this based on your situation as well as on the phrase

For example, for a given entry, the network would predict "London" and "1st Class"

If that is the case then you might consider to use a different type of output. Instead of predicting a single class you could have as output a vector of probabilities for all desired combinations of destinations and classes (as well as other aspects that you might have in your model). Then you could value the predictions and perform the training based on a likelihood function of a categorical distribution.

When you apply this model (some online shopping tool or some help for an airline company?) then it will not give a single class as output, but instead it could give a ranking of the top destinations.

Network structure

What kind of dense neural network do you have and how did your train it? It might be imaginable that there should be a node in some of those layers that gets trained to deal with the London + 2nd class case specifically. But, how many layers do you have, how many nodes per layer do you have, how did you do cross-valdiation?

It is imaginable that this error/false-prediction might occur. But it is difficult to say why and how exactly it occurs without details.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks. How would you go about a cost function that takes combination into account? Would you build a lookup table of the frequency for each combination in the training set, and multiply the final loss/cost by a factor of the inverse of this? Or would there be other approaches? $\endgroup$
    – Jivan
    Apr 12 at 9:23
  • $\begingroup$ @Jivan, I was thinking of a cost function that punishes the result if the entire combination is wrong and does not grant half score if you get half of the class correct. How to do this exactly I am not sure, it depends on the problem that you have. Anyway, in this simple example in the question (which may not be your complete problem?) it is not surprising that you get a prediction Londen 1st class if your cost function will reward this prediction in the 89302 cases that you guessed London correct and the 48516 + 41411 + 38186 + 35247 + 28512 cases that you guessed 1st class correct. $\endgroup$
    – Sextus Empiricus
    Apr 12 at 9:59
  • $\begingroup$ @Jivan it is not clear how you value the output of the neural network. Do I guess correctly that the output is just a single class prediction? What class prediction would you favor in the example from the question. Is London 2nd class a better prediction than London 1st class? The London 2nd class will be a perfect prediction in 89302 cases but an extremely bad prediction in 48516 + 41411 + 38186 + 35247 + 28512 other cases. The Londen 1st class will not have these perfect predictions but might do better on average. $\endgroup$
    – Sextus Empiricus
    Apr 12 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.