# Learning multiple output

Can you suggest me an algorithm and probably a real code for multiple output learning, where input of the model is vector of around 10 000 values and output is, for each input vector, an output vector of 1500 dimensions (so, it's kind of a big dataset)?

In my opinion neural network can handle this number of values, but what else? Supervised SOM? Multi-class multikernel learning?

Any examples of the real world data model with the same type of learning (large multiple output?)

Any suggestions and opinions will be very valuable for me.

Update: Data example can be found here - http://pastebin.com/Z4E87d3d

• Wait, is the output 1500 UNIQUE values? Is this a regression problem or a classification problem.
– Zach
Aug 24, 2011 at 19:53
• yes, output is unique label that can have binary (0,1) or other values. Aug 25, 2011 at 15:07
• so output is 0 or 1, with a length of 1500?
– Zach
Aug 25, 2011 at 15:35
• there are two cases: (1) output is binary, (2) output is notbinary (positive integer numbers) Aug 26, 2011 at 5:56
• If it's not proprietary, can I ask what on earth you're doing with 1500 outputs? Just curious.... Apr 5, 2012 at 6:55

Sounds to me like structured SVM could be a good fit. It allows for interdependent outputs, and as long as you have the time to wait to fit the SVM it should work fine. You can also use the kernel trick to handle for nonlinearities of various kinds.

Pretty much every learning algorithm I know could handle a dataset with 10,000 rows. Random Forests, SVMs, boosted trees, penalized linear models, knn, etc. etc. I've fit all of these models on datasets of that size on a pretty standard laptop (4GB ram, core i5 processor).

You can start to hit computational limits on a dataset of that size when you start re-sampling and cross-validating to avoid over-fitting.

I guess the answer to your question is "How much time do you have?" Maybe waiting a few hours to cross-validate a 10x10 grid of parameters is more than you can spare.

Here's some example code in R. Input is a random matrix with 5 columns and 2000 rows (10000 values). Output is a binary vector with 2000 values. I tried a random forest, an SVM, a penalized linear model, a KNN model, and a neural network. All the models fit the entire dataset in under 1.5 seconds. Tuning the models using bootstrapped re-samples took up to ~2 minutes each (although some models were quicker).

Here's the code:

#Setup
rm(list = ls(all = TRUE))
set.seed(1)

#Generate an input matrix with 10,000 values
#2,000 rows, 5 columns
X <- data.frame(replicate(5, rnorm(2000)))

#Construct Y using X
Y <- runif(1)*X[,1]*X[,2]^2+runif(1)*X[,3]/X[,4]

#Convert Y to binary
Y <- as.factor(ifelse(sign(Y)>0,'X1','X0'))

#Create bootstap samples for fitting models
library(caret)
tmp <- createResample(Y,times = 25)
myCtrl <- trainControl(method = "boot", index = tmp, timingSamps = 10)

#Fit models
RFmodel <- train(X,Y,method='rf',trControl=myCtrl,tuneLength=1)         #Random Forest
SVMmodel <- train(X,Y,method='svmRadial',trControl=myCtrl,tuneLength=3) #Support Vector Machine
GLMmodel <- train(X,Y,method='glmnet',trControl=myCtrl,tuneLength=10)   #Penalized linear model
KNNmodel <- train(X,Y,method='knn',trControl=myCtrl,tuneLength=10)      #Nearest-neighbors
NNmodel <- train(X,Y,method='nnet',trControl=myCtrl,tuneLength=3, trace = FALSE) #Neural network

#Assess re-sampled (out of sample) accuracy
resamps <- resamples(
list(   RF = RFmodel,
SVM = SVMmodel,
GLMnet = GLMmodel,
KNN = KNNmodel,
NN = NNmodel
))
summary(resamps)
dotplot(resamps, metric = "Accuracy")

#Assess run-time (in seconds)
resamps\$timings


And here's the results:

Accuracy :
Min. 1st Qu. Median   Mean 3rd Qu.   Max.
RF     0.8972  0.9062 0.9261 0.9205  0.9280 0.9490
SVM    0.8313  0.8437 0.8527 0.8546  0.8634 0.8800
GLMnet 0.6613  0.6808 0.6862 0.6918  0.6995 0.7295
KNN    0.8158  0.8244 0.8344 0.8368  0.8492 0.8661
NN     0.7592  0.7943 0.8003 0.8047  0.8231 0.8352

Run Time (seconds):
Everything FinalModel Prediction
RF          33.80       1.28       0.03
SVM        110.76       1.17       0.01
GLMnet     104.21       0.05       0.01
KNN         17.09       0.00       0.02
NN          76.51       0.51       0.00


As a graph: • yep, a lot of algorithms can handle even more than 10 thousand values as model input, but what's about output? Aug 25, 2011 at 15:09
• @Vladimir Chupakhin So output is a vector of (0,1) with 10,000 values? Almost every algorithm I can think of can handle that.
– Zach
Aug 25, 2011 at 15:15
• @Vladimir Chupakhin: I updated my answer with an example of 10000 values as input and 2000 values as output.
– Zach
Aug 25, 2011 at 16:03
• Thank's a lot! But in my case all of the label in the vector are connected. So the vector is not the vector of independent variable but dependent variable specific for every entry. Aug 26, 2011 at 7:50
• @Vladimir Chupakhin: I don't follow you. Could you provide some example data?
– Zach
Aug 26, 2011 at 13:09

You could try Partial Least Squares (PLS) Regression. There is variant of PLS (PLS-DA) which can also be used for binary outputs. Search online for PLS package

It can output multiple ouputs. And, with its matrix-operations it is very fast too.