Conjecture on models and real world Context
Assume we train a model with a training set that cointains features $x_1,x_2,...,x_k$. However, we know in advance that there are many hidden variables influencing the output $x_{k+1},x_{k+2},..,x_N$ but during training all these are simply constant $x_j = const$ for $j>k$
However, the hidden variables will change when the model is deployed in the field and thus the prediction performances will drop. I wonder which is the best strategy for training. Whatever I do the model is naturally biased on the $x_1,x_2,...,x_k$ vars I can observe.
Question
Given the need to find a fair trade-off between variance and bias, which are the consequences in this context?
Is better to move towards regularization with less variables $j < k$  or is it better to move towards variance $j=1,...,k$ trying even to add higher order features like $x_j^2, x_i x_j, ...$ and the like?
Conjecture
My conjecture is that adding variance to an acceptable level in the subspace $R^k$ will result in very bad over fitting when deployed in $R^N$
 A: Consider the following toy example:
You want to distinguish men from women by looking at their size.
The distribution looks something like the image below which would lead to an pretty good classifier with a threshold of ~177cm.

Anyway, you know that in your application when the classifier is applied you have an additional feature available. This feature might be the length of the hair of the respective persons, or their IQ, or the temperature outside because the cables of your measurment device have been connected wrongly. You just don't know what this feature is.
How the heck would you base your modeling on that? It could just be anything, from the most useful feature (like the length of the hair), to just random noise. As long as you don't know any more of this features (i.e the correlation to size) the information within this additional features can not be extracted in any way by punshing and transforming of the size feature. 

Mathematically spoken: You have the information of $P(X_k | X_j   = const)$ and would like to know $P(X_j)$. According to bayes 
$$P(X_j) = \frac{P(X_j  = const | X_k) \cdot P(X_k)}{P(X_k | X_j  = const)}$$
you need to know $P(X_j  = const | X_k)$ as well as $P(X_k)$ which both are unavailable.

Hence, as long as you have no additional information about the features in $x_j$ don't use it.
