I'm going to walk through some logic that I believe I've heard somewhere before...but I can't remember where and I am interested in citing it. Does anyone know where I could cite this type of thinking and its conclusion?
Also, I am interested in whether this line of reasoning is correct, or if there are some holes in it.
I am studying a deterministic process and I want to build a mathematical model for how it works, both for inference and for prediction. The process has the following inputs and outputs.
Input: $p$ binary predictors and I know that nothing outside these $p$ predictors influences the outcome. I'll imagine $p \approx 100$ to create the right flavor of problem.
Output: 1 binary outcome
So, if I observed all $2^p$ possible inputs and their resultant outputs (assuming no measurement error), there would be nothing to do - I would know everything there is to know about this process.
But, I can't really observe all the possible inputs when $p$ is more than about 5 or 6 and I'm interested in situations where $p$ is around 100. So, let's imagine a slightly more realistic situation (but only slightly!).
Imagine I observed "only" $2^p - 1$ inputs and their resultant outputs. How would I predict the output for the final possible input?
It seems there must be at least two possible rules -- one that, if it were true, would give a 0 for the last set of input and one that, if it were true, would give a 1 for the last set of input. And there's no way to choose between them without seeing the last input!
I can only choose between the two (or maybe there are more?) rules based on a priori beliefs about which rules are reasonable/likely/good/useful and which are not.
Though this setup seems fanciful, I would argue it's the easiest possible situation in which a model could be fit.
- The outcome was binary, and any continuous response could be simplified with binarization.
- There are no unknown inputs. I could ignore some of them to return to the situation where most statistical modeling efforts exist -- some inputs are unknown and their effects are sopped up into the residual error term or similar.
- The inputs are binary. Any continuous valued input could be binarized without losing information by turning it into a set of "greater than this cutoff?" "greater than that cutoff?" binary values.