Apologies if this question is answered elsewhere---I couldn't find an answer elsewhere but that may be because I don't know what the name for the problem I am trying to solve is.
I have two data vectors, X and Y, which are strongly correlated. An example of a scatterplot of such vectors is below:
What I need to do is to be able to generate more samples of Y given a value of X, that are (ideally) statistically indistinguishable from the actual observed data. Suppose the above are all the observations I have of X and Y, and that I have no prior information about their relation or distributions (for example, I cannot assume the mean of Y is linear in X, or that Y is normally distributed for each X).
For example, at X=0, it appears that Y has some probability distribution, probably centered at or slightly below zero. I need to create a procedure by which I can input X=0, and get a desired number of samples of Y that are distributed in the same way as how the observations of Y near X=0 appear to be distributed (there may not actually be any observations exactly at X=0).
Clearly what I have here is a posterior distribution estimation problem---in fact, one for every possible value of X. This is where it gets tricky. The distribution at X=0 is slightly different from the distribution at X=1. I need to choose which observations will influence my distribution estimation for each X value (i.e. which observations will I input into a Gaussian KDE algorithm, for example). Hence I have a distribution estimation problem combined with a binning problem, fundamentally.
Here is the methodology that I've come up with so far.
- Perform linear regression on X and Y, store the regression line
- Divide the domain according to whatever resolution in X is needed. (Maybe 10 different bins)
- In each bin, calculate the residual of each observation of Y relative to the regression line.
- Upon new input X', sample a residual from the bin which X' is in.
- Add the residual to the value of the regression line at X'. Output the sum.
You can see how the above procedure somewhat solves the problem, but with some issues:
- The fewer bins used, the more observations there are to inform the distribution estimation...at the cost of having poor resolution (X' will be rounded in some sense to the nearest bin)
- The more bins used, the more closely the observations used in distribution estimation correspond to the value X'...at the cost of having fewer observations in the estimation (and hence higher uncertainty in the estimation result).
Is there some general procedure through which the distribution of a variable, and the dependence of that distribution on (possibly more than) one other variable, can be estimated from a set of observations?
I suspect that this may be related to multi-dimensional kernel density estimation...perhaps I apply multi-dimensional KDE to this data and then fix one of the variables to look at the conditional distribution of the other? Is this the correct interpretation of multi-dimensional KDE? If not via KDE, is there some other procedure which can be used to tackle problems of this sort?
