Picking the right problem class (Regression vs. Classification)

In the problem I am working on, I am dealing with data points which have many features and are mapped to a certain value based on an unknown function. I would like to train a predictor to be able to compare two data points and to predict which of them will have the better (higher) value. I can imagine three approaches for this problem:

1. Classification: Train with pairs of training points labeled by 1 (first training point is better) or 0 (second training point is better).
2. Regression A : Same approach as with the classification, but the label is a continuous value obtained by dividing the value associated with point A by the value associated with point B.
3. Regression B : Train a regressor to predict the values of the hidden function and use this for the comparison.

Apart from implementing all three approaches and comparing their performance, are there any general reasons why one of them might be better?

Let me remove one choice from your list. I think the worst approach is the third one, that is Regression B. This is the reason: You want your model to compare two points, $x_i$ and $x_j$. There might be some useful relationships between features that can be used to facilitate this comparison. For example, consider the case where two samples may be compared according to a subset of their features. Regression B cannot use such possible facilitating knowledge. By a similar argument, among the two first options, I personally prefer the first one (classification approach). The reason is that the second approach tries to learn the division of hidden function values for the input points, while it is not aware of your intention. You want to compare two points, so it is better to make the model aware of your intention by explicitly classifying the training sample pairs. By classification approach, you are guiding the learning algorithm to learn the comparison between input points, that is exactly what you want.

My understanding is that there is a function $F$ that takes examples $x_i, y_i$ and compares them $F(x_i)>F(y_i)$. However, $F$ is unknown, i.e. we do not know the values $u_i=F(x_i)$ and $v_i=F(y_i)$. The only thing we know is the preference $x_i\succ y_i$ or $x_i \prec y_i$. We have data for $i=1,\dots,n$.

Under these assumptions, the only possible approach is 1. because for 2. and 3. we do not have the data $u_i,v_i$. However, you can emulate the Regression B approach in the following way:

• To create a MLP
• The last neuron is a softmax
• There are two neurons that go to the last one.
• Each of them has the same neural network behind it, one taking inputs for $x_i$, the second for $y_i$. The "same neural network" may require changes in the backpropagation. However, this change may not be necessary if you enter each pair twice (first winner, second loser)

Another option could be to use Bayesian network which could lead to easily interpretable results: http://users.dsic.upv.es/~pgupta/pdf/gupta-mtech-thesis.pdf

• Thank you for your answer. For the training set, we also know the function values (F(x) and F(y)) in your example. Feb 8 '17 at 14:15