Machine Learning Algorithms for Predicting Entire Regression Functions

From what I've seen in (supervised) machine learning the general idea is to work with some training set $(\mathbb{x_i},y_i)$ and learn the $y_i$ outputs without over-training. In the case of regression this is a continuous problem. However, in the previous definition of my training examples (and in a lot of definitions I see) the $y_i$ is single valued. So I was wondering if it were possible / if there are any good machine learning algorithms currently which can work on continuous vector value output spaces?

To put it more simply, imagine I have a set of inputs, which I regress to a particular output. We can call this model $H_1$. I have new set of inputs, and a new set of outputs, which results in a new model, $H_2$, and so on until I make up to $H_n$ models. I want this set of regression functions to by my training data set, so that if I have new input, I will be able to make an educated guess on what this new model, $H_{n+1}$ should be.

I can only think of possibly extending Boosting to this scenario (if it hasn't already been done before), perhaps using a Gaussian Process, or maybe even some Bayesian parametric method where I integrate over all the models? I don't have much experience using entire regression functions as training inputs, so I'm wondering if someone on here could educate me on the topic / suggest plenty of algorithms and point in the direction of some good sources.

Under the assumption that you're looking at similar data across the different regressions, you're describing Bayesian hierarchical modeling. The classic example of this is the Eight Schools data: students from each school are pre-tested in PSAT scores. Then each of the students' schools provide PSAT coaching to them. Then the students take the PSAT again. Was one coaching experience more effective than the others? How much do the students' scores improve (if at all)? A hierarchical analysis of the PSAT scores can answer this question.

Using the posterior predictive density from the model and some pre-coaching data for a hypothetical ninth school, we can make inferences about how effective coaching might be for this new batch of students.

Hierarchical models are premised on the idea that regression coefficients themselves have some distribution, e.g. $\beta\sim\mathcal{N}(\mu_\beta,\sigma^2_\beta)$ would imply that the coefficients $\beta$ are normal deviates with a mean and variance parameter.

Gelman's Bayesian Data Analysis is a great resource for getting started with hierarchical models.

• Thanks so much! I'll give that reference a look. :) Would you / anyone else have a suggestion of any other methods? Thanks to my teachers I have been given the mindset of there always being a few ways to tackle a single problem. :) – pche8701 Jun 1 '16 at 14:44
• But I am not sure if you could add a new variable say parental income.. and this is what the @pche wants to do .. if i understand his question correctly.. he mentions new set of inputs, Not new values for a given set – ashokragavendran Jun 2 '16 at 2:59
• @ashokrags "parental income" doesn't appear in OP's question, so I'm not sure what you're talking about. Perhaps if OP were concerned, OP would ask for clarification. – Sycorax Jun 2 '16 at 3:02
• @general .. to clarify: i was using your example of PSAT scores.. so if there was an additional variable say "parental income" in the ninth school pre-coaching data not present in the data for all other schools ...i think it would not to possible to generate predictions correct??... i think this is what the OP wants to do... OP can perhaps elaborate here – ashokragavendran Jun 2 '16 at 3:35
• OK, let me try and clarify here. Imagine I have hyperparameters, and parameters. If I set the hyperparameter =1, I get one set of parameters (the regression values). If I set hyperparameter = 2, I get another set of parameters (I new set of regression values = a new curve essentially). I do the same with hyperparameter = 5. Now IF I have new input, with hyperparamter = 4, what should the regression values (parameters) be? I am not changing the actual feature space, but am trying to infer some new regression function, based on other ones. I hope that makes sense. Let me know if it doesn'.t – pche8701 Jun 2 '16 at 3:40