# Algorithms, models, recommendations for regression with vector output

I really didn't want to double-post, but it wasn't clear to me which StackExchange forum to post to, and I am looking for all the help I can get with this problem. Here is basically the same question on Data Science StackExchange.

This post's problem seems eerily similar (though we are not using an ANN) (and I don't see any useful answers). @Álvaro, are you still working on this problem?

We have a regression problem that has relatively low dimensional input (say 8 initial relevant features (not including engineered ones)), but high dimensional output vector (not a single value, a vector of reals). During training, a low dimension of features plus 2500 reals as actual output go into the model, then testing samples takes the same types of input features to predict the 2500 reals (a curve).

              inputs                               outputs
_________________________________________ _______________________________________
| a | b | c | ... | (engineered features) | 2500 reals as output (y's of a curve) |


We believe the problem is linear with respect the input parameters, but not 100% sure.

Since we are not machine learning experts, we had some trouble in the beginning even finding an algorithm that supported output vectors as opposed to a single output value.

I'm not looking for generic data science 101 suggestions like: clean your data, subtract irrelevant features, understand the data, etc. We've done that well enough I believe, and we've had some limited success already learning these curves with about a predication rate of ~70% (which sounds rough, but isn't too bad given the nature of this data). However, we are not experts as I said, and I'm still convinced that we are not considering all of our options in terms of algorithms, frameworks, feature engineering techniques, etc. for this specific type of problem; I'd like to squeeze everything we can to get the best possible predictions. Given this data, there isn't a 100% relationship between the input and output - we know there is some expected error because not all relevant measurements could be captured. However, I do think we can get it predicting at a rate of higher than 70%. Maybe. Hopefully. Sometimes the predictions are near perfect, other times they have the "general shape" but are warped in places we can't explain.

We are using Python/SciPy/scikit-learn. We have tried 5-10 models/algorithms that support vector output. We are doing 20-fold cross validation and our current dataset has ~25000 samples.

How would you approach this type of problem? Which algorithms/model, or papers, have you come across / used / are relevant for this type of problem? What should we be keeping in mind for this specific type of problem?

We are very happy with this: