1
$\begingroup$

Context Assume we train a model with a training set that cointains features $x_1,x_2,...,x_k$. However, we know in advance that there are many hidden variables influencing the output $x_{k+1},x_{k+2},..,x_N$ but during training all these are simply constant $x_j = const$ for $j>k$

However, the hidden variables will change when the model is deployed in the field and thus the prediction performances will drop. I wonder which is the best strategy for training. Whatever I do the model is naturally biased on the $x_1,x_2,...,x_k$ vars I can observe.

Question Given the need to find a fair trade-off between variance and bias, which are the consequences in this context?

Is better to move towards regularization with less variables $j < k$ or is it better to move towards variance $j=1,...,k$ trying even to add higher order features like $x_j^2, x_i x_j, ...$ and the like?

Conjecture My conjecture is that adding variance to an acceptable level in the subspace $R^k$ will result in very bad over fitting when deployed in $R^N$

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
0
$\begingroup$

Consider the following toy example:

You want to distinguish men from women by looking at their size.

The distribution looks something like the image below which would lead to an pretty good classifier with a threshold of ~177cm. enter image description here

Anyway, you know that in your application when the classifier is applied you have an additional feature available. This feature might be the length of the hair of the respective persons, or their IQ, or the temperature outside because the cables of your measurment device have been connected wrongly. You just don't know what this feature is.

How the heck would you base your modeling on that? It could just be anything, from the most useful feature (like the length of the hair), to just random noise. As long as you don't know any more of this features (i.e the correlation to size) the information within this additional features can not be extracted in any way by punshing and transforming of the size feature.

Mathematically spoken: You have the information of $P(X_k | X_j = const)$ and would like to know $P(X_j)$. According to bayes $$P(X_j) = \frac{P(X_j = const | X_k) \cdot P(X_k)}{P(X_k | X_j = const)}$$

you need to know $P(X_j = const | X_k)$ as well as $P(X_k)$ which both are unavailable.

Hence, as long as you have no additional information about the features in $x_j$ don't use it.

Improve this answer
$\endgroup$
3
  • $\begingroup$ An analogy for his case is that the sample was formed such that all people in it had the same breast size (wow! how he did he do it?). He knows that in the "real world" the population will have different breast sizes, and it could become a good predictor. His question is should he think about the breast size in his current sample to increase forecast quality. $\endgroup$
    – Aksakal
    Commented Sep 6, 2016 at 17:38
  • $\begingroup$ There is no difference to my case above. If the variability in the breast size is zero in his gathered data, there is no possibility to learn from it. He just has the information $P(X | breastSize =a)$, wheras he would like to know $P(X)$ which according to bayes rule requires the knowledge of $P(breastSize=a)$ as well as $P(breastSize=a | X)$. $\endgroup$
    – Harald Thomson
    Commented Sep 6, 2016 at 17:46
  • $\begingroup$ There's some information in the fact that his sample is different from "real world", maybe he can collect some more information $\endgroup$
    – Aksakal
    Commented Sep 6, 2016 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.