I have a dataset where all observations are measured several times and reported outcomes correspond to those measurements. In other words, my set of data points looks like

$\{x_i, y_{i_1}, y_{i_2}, ... y_{i_n}\}$

I would like to make a predictive model to predict $y$ from $x$. What should I try to predict? The average of $y_i$?

I've heard something about multi task learning, where each observation is considered as $n$ different regression problems and all problems share the same kernel. Can anybody elaborate on this?

• It would be good if you could supply us with more information on your dataset, or just a snapshot of your dataset with the question of interest. – chl Mar 8 '12 at 11:28
• I hope the rephrased question help you to understand the problem. The problem is I have a several Y for a given X and I wonder, what should I target when I am building a predictive model ?! the average of Y? or ... – user4581 Mar 8 '12 at 13:06
• Do you have a deterministic, but noisy process that is yielding your measurements y_i? Or do you have a process with some hidden variable that leads to different measurements y_i? The structure of your problem will determine what you should do to model the measurements. – cape1232 Mar 8 '12 at 14:17
• there are some hidden variable; since the measuring is not exactly unique. but I am not really interested in them. – user4581 Mar 8 '12 at 18:08
• wouldn't this be multiple instance learning rather than multi-task learning (which involves learning different, but related problems with shared attributes, whereas in this case the problems are all the same)? – Dikran Marsupial Jun 6 '12 at 18:38

If observations (for the same subject) are uncorrelated, then one thing you can do is to average observations and use some prediction technique (e.g., regression). However, if these observations are correlated, then the data set is longitudinal.. then you need to use, say generalized linear mixed models...

If the observations are independent, and the ordering of the observations is unimportant, then you could simply estimate the mean as Actuary suggests (+1); however the fact that multiple observations are made suggests that the spread of the observations may be of intrinsic interest, in which case it would be a good idea to model the distribution of plausible values for the observation (for a Gaussian, this would be modelling the conditional variance as well as the conditional mean).

I did some work on predictive uncertainty a few years ago, and the method described in

might be of some interest (MATLAB demo here). It ought to be fairly straightforward to make a basic multiple instance version assuming the observations are conditionally i.i.d.

BTW I think this kind of problem is known as "multiple instance learning" in machine learning, there is a fair amount of research on this indexed by Google Scholar, so hopefully there may be something relevant to your problem there somewhere (too many papers, too little time to read them... ;o).

All of the answers above suggest techniques that assume the data are uncorrelated (with the exception of the later part of user9292's response). I highly doubt that is the case here since you have multiple observations from the same subjects. As a results, you'll likely to take the correlation into account using methods such a mixed models, generalized estimating equations (GEE), or transitional models. This site has plenty of information on all of these methods and how you might be able to apply them.

Basically you can use Pareto Front when choosing between different solutions. One particular approach which extensively uses Pareto front is genetic programming. There are several good frameworks for this stuff where you can setup you problem and see the result without months of coding, check answer to this question with list of frameworks.