In ML, once we remove a feature, can we safely assume that feature will not be important again? This is probably a very simple question.  Let's say we use some metric to remove features, whether that be AIC, regularization like lasso, variable importance, t-tests, etc...
Assuming we use the same technique again as we continue to refine the model, would it be safe to assume that any removed features would not be significant again?  My understanding is that some features may be correlated with other features - but if that's the case, you really only need to include one of them, especially if they are highly correlated.
Other than that, I can't think of a reason why you would add a feature back in to a model, once it has been removed.
The reason I am asking this question is because I am trying to build a large model with many features.  Because I have so much data and limited technology, I am hoping to build a model sequentially.  Train the model on say 10 features, remove the unimportant ones, then rebuild it by adding in 10 more, remove the unimportant ones, and continue until I reach my computer's capacity. 
Are there any issues with that process?
Thanks!
 A: No, you cannot safely assume that. The reason is that conditional independence does not imply independence and vice versa (wiki).
Moreover the forward selection style approach you follow suffers from a fundamental problem: model selection criteria like that usually rely on p-values/t-statistics/... To be based on the  "correct" underlying model.  This however can't be true if you do forward selection and a 'correct' feature is included only later on the process.  That's why usually you should at least do backward selection - if you do any stepwise selection at all.  That way the 'true' model is at least nested in the startimg model for selection.
As has been mentioned in comments above, there are (much) better ways to do feature selection than a stepwise algorithm.  At least try a LASSO approach.
A: You seem to be assuming that the models work in additive fashion, so adding a feature to the model just "adds" some stuff related to this feature alone and does not influence the rest of the model, same with removing the feature. That is not the case. If machine learning models worked like this, then to build a model with $k$ features you would need only to build $k$ models with single feature and find a way of combining them. Here you can find a recent thread, and links to many other questions like this, where including new feature to regression model affects how the model uses the other features. This happens for linear regression, but will also be true for other machine learning algorithms. 
You say that you would assume this to be issue "only" when the variables are correlated, but with real-life data, there always will be some degree of correlation between the variables. Moreover, it is not only about correlations between pairs of variables, but also about relations between all of the variables, where those relations can be non-linear as well. You should be rather talking about independence, and seeing all variables independent is even less likely then seeing them all uncorrelated.
More then this, by adding a new feature your algorithm needs to adapt. Imagine that you have a decision tree with the constraint to have not more then five samples in each final node. You cannot just add new feature to such tree without re-building it, because you already have not more then five samples in each final node, so you cannot split this node any further. In such case, you would need to re-build the whole tree, using different splits, or combinations of splits, so it would use your data in a different way then the initial tree. This would be the case even if the new feature that you are adding is independent of other features. 
What you propose is partially answered also in the Algorithms for automatic model selection thread, that discusses stepwise feature selection algorithms and the most upvoted answer shows how proceeding in such stepwise fashion by adding (or removing) variables leads to ending up with bad models. It simply doesn't work for the reasons discussed above.
