My team has been asked to build a predictive model. We have a very limited dataset, but using a number of rationalizations about bounds on the data and the current behavior (54 data points) I have constructed a Gaussian Process of variable X that seems to make a rational prediction with confidence bands (though they are wide relative to bounds).
On top of this, I am being asked to incorporate some tool that will allow us to "play" with two other variables (supposedly guesses out of management of future values), to see how it influences the prediction on variable X.
The relationship with these other two variables and variable X is very complex. One of the other two variables have been massaged to try to make the relationship simpler for management to digest. So, they have basically just encoded heuristics into their dataset for this one. The other one they have no control over anyways, so letting them hardcode a single belief in just to see the output seems like a lose/lose for me. I would assume adding any wacky behavior would just make the model almost completely random here.
To explain this in a way that does not seem like a matter of opinion, I would like to demonstrate with some mathematical logic why this request makes little sense. I have imagined if I have a full posterior joint probability distribution, I could theoretically build this model. In terms of sample size requirements, I had imagined Hoeffding's Inequality. I know I am mixing inherently different things here, but was wondering if there is some type of proof stating how an empirical distribution would require some minimum sample size in the absence of a reliable prior to narrow in on a reasonable solution with confidence.
I need proof rather than simply an argument.
At this point, I provided a discussion on the curse of dimensionality and how regression is actually curve fitting to a manifold. I think this much will provide a rational explanation that intuitively shows difficulties arising from increased dimensions.
I was hoping something simple such as Hoeffding's inequality requires 30 points, so a 2-dimensional distribution would require 900 points, 3-dimensional -> 27,000, etc. I had just imagined expressing dependency in its entirety would have some statistical generalization for variance and covariance relationships.