0
$\begingroup$

My team has been asked to build a predictive model. We have a very limited dataset, but using a number of rationalizations about bounds on the data and the current behavior (54 data points) I have constructed a Gaussian Process of variable X that seems to make a rational prediction with confidence bands (though they are wide relative to bounds).

On top of this, I am being asked to incorporate some tool that will allow us to "play" with two other variables (supposedly guesses out of management of future values), to see how it influences the prediction on variable X.

The relationship with these other two variables and variable X is very complex. One of the other two variables have been massaged to try to make the relationship simpler for management to digest. So, they have basically just encoded heuristics into their dataset for this one. The other one they have no control over anyways, so letting them hardcode a single belief in just to see the output seems like a lose/lose for me. I would assume adding any wacky behavior would just make the model almost completely random here.

To explain this in a way that does not seem like a matter of opinion, I would like to demonstrate with some mathematical logic why this request makes little sense. I have imagined if I have a full posterior joint probability distribution, I could theoretically build this model. In terms of sample size requirements, I had imagined Hoeffding's Inequality. I know I am mixing inherently different things here, but was wondering if there is some type of proof stating how an empirical distribution would require some minimum sample size in the absence of a reliable prior to narrow in on a reasonable solution with confidence.

I need proof rather than simply an argument.


At this point, I provided a discussion on the curse of dimensionality and how regression is actually curve fitting to a manifold. I think this much will provide a rational explanation that intuitively shows difficulties arising from increased dimensions.

I was hoping something simple such as Hoeffding's inequality requires 30 points, so a 2-dimensional distribution would require 900 points, 3-dimensional -> 27,000, etc. I had just imagined expressing dependency in its entirety would have some statistical generalization for variance and covariance relationships.

$\endgroup$
  • 1
    $\begingroup$ I wonder how convincing a mathematical proof will be to your clients. You already seem to use prediction intervals (which are different from CIs). You could build models using the clients' pet ideas and show how adding useless predictors widens PIs to the point of uselessness. Unfortunately, data science is very prone to snake oil salesmen, and someone can always promise a magic solution. $\endgroup$ – Stephan Kolassa Aug 31 at 16:51
  • $\begingroup$ Thanks Stephan, good point on the PIs vs CIs. The idea of CIs offers me the analogy for explanation for management. My experience thus far is that they are cherry-picking point estimates whenever something is presented. A mathematical proof is intended to protect me, and buy time, because education takes time. $\endgroup$ – Michael Tamillow Aug 31 at 19:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.