Is stagewised feature engineering/ selection an invalid approach? What to do when all the features are not ready at one time?

Suppose we want to build a regression or classification model. However, the features (independent variables used) are not all ready at one time.

This is very realistic in business, because the data can be very large and complicated, software engineers and domain experts need to spend a very long time to propose and build features from massive data set.

In such a case, in order to save time, without all the features ready, we want to start to evaluate the goodness of the feature / how well the feature correlated with the label one by one.

Is this approach invalid? When I say invalid I mean following thing can happen:

• At day 1, we found feature $x_1$ is a good one, it has positive correlation with label $y$
• At day 2, when we adding another feature $x_2$, we found feature $x_1$ discovered is completely useless, even we may find $x_1$ has negative relationship with $y$, when we look at $x_1, x_2$ and label $y$ together.

I think this is Simpson's paradox and the "stagewise feature discovery" approach is invalid. Am I right?

Here is a small simulation I built to illustrate what might happen for stagewise feature discovery: - Day 1, we have one feature: hours and income (label), we found that, the more you work the less you earn. - Day 2, we have add one feature: education level, red represents high school and black represents college. So, exact same data, you find the more you work, the more money you make, which is contradiction to day 1's discovery.

If I am right that stagewise feature discovery is an invalid approach, then what is the correct way of doing analysis when all the features are not ready at one time?

• i ran into similar problems and don't think there is magic bullet. I posted a question here on variable selection not too long ago, trying to convey the implicit message that most of the existing variable selection methods tend to do more harm than good. stats.stackexchange.com/questions/215154/… – horaceT Jul 1 '16 at 18:08