Supervised learning approaches which can accommodate a supervisory signal composed of multiple dependent continuous variables?

Question

This is a supervised learning problem. Ideally would like to work in R due to having an easy way to pre-process the input data, but could work around that as well.

For each sample, input consists of tens of thousands of features. These are genomics data and will likely need to be reduced to a manageable amount, somehow, before being used to train the classifier.

Supervisory signal consists of 4 dependent continuous values, representing relative composition of the sample.

e.g. continuous between 0 and 1, all 4 summing to 1 for each sample:

Sub012  0.5940594   0.26732673  0.07920792  0.059405941
Sub013  0.5102041   0.34693878  0.08163265  0.061224490
Sub014  0.6521739   0.20652174  0.07608696  0.065217391

Wanted: a regression function capable of predicting the relative composition of a sample in terms of those same 4 dependent continuous values.

The constraints on the supervisory signal are what is causing me pause: the dependence of the variables, being constrained between 0-1 and summing to 1. I was hoping someone might have attempted something similar and could point me in the right direction - packages or approaches which may work or definitely won't work - all thoughts welcomed.

Thank you.

Answer 1

The constraint on the output can be achieved using the softmax inverse-link function used in multi-nomial logistic regression, i.e.

$y_i = \frac{exp\{\nu_i\}}{\sum_{j=1}^n exp\{\nu_j\}}$

where $y_i$ is the $i^{th}$ output of the model and $\nu_j$ is the linear combination of the input features for the $j^{th}$ component.

The model can then be fitted by minimising a suitable likelihood. As the targets are constrained, the likelihood won't be Gaussian, which may be a problem. Some sort of Dirichlet likelihood might be more appropriate? There may not be any R software that does this already, so you will probably have some coding to do.

As you have many input variables, it will be vital to use some form of regularisation to avoid over-fitting, the regularisation parameters can probably by tuned very efficiently my minimising the leave-one-out cross-validation error, see e.g. Generalised Kernel Machines. WHich reminds me, if you have many more features than patterns, then you can perform the computation cheaply without having to do any feature reduction using kernel methods (see the GKM paper).

Update: You may be able to just use some off-the-shelf code for multi-nomial logistic regression, providing the implementation allows a soft (probabilistic) assignment of the targets (most software assumes that the targets specify that each pattern belongs to one particular class, rather than a probability of belonging to each class, as that is what most people need). It will have the softmax inverse-link function, so the required constraints on the output will be in place. The multinomial loss function is probably wrong, but then again so is a Gaussian. I have used logistic regression models for regressing ratios before, it is a bit of a hack, but it did work pretty well.