What if Lasso selects transformed terms but not untransformed terms Suppose I have standard normal features $X_i \in \{X_i : i \in \{1,...,1000\}\}$. I extend this set of predictors with transformations as follows: $\{X_i,X_i^2,X_iI(X_i > 0) : i \in \{1,...,1000\}\}$.
What happens if Lasso would pick $X_i^2$ or $X_iI(X_i > 0)$ but not $X_i$ itself. What do I do? Is this even a problem?
 A: You're right: in general, people don't like to put interactions into a model before putting in the primary effects. There is a recent paper that solves this problem for the lasso: "A lasso for hierarchical interactions" by Jacob Bien, Jonathan Taylor, and Robert Tibshirani. Their solution is implemented in the R package hierNet. Hope this helps!
A: It's not a problem strictly speaking, but you should pay attention to whether that interpretation makes sense in the context of your problem. If the squared value is what's important to your dependent variable, then this is what you'd expect to happen because it would have more explanatory power than the raw variable. 
For example, if you're a sports fan, you could use a players age to predict their performance. If you mean-shift that to something like 27, then the age variable can be negative if it is below 27 and positive if it's greater than 27. Well, players performance tends to have an arc that peaks at age 27, so if you put the raw age into the regression, chances are it would show not significant. If you put age^2 in though, it would. And likewise if you gave Lasso the choice between the two, it's a no brainer.
A: As a simple example, consider fitting a model including a squared term and its main effect,
$$Y=\beta _{0}+\beta _{1}X^{2}+\beta _{2}X$$
Suppose that $X$ were to shift in location, say to $X+a$. Then model is unaffected by the change, 
$$Y = \beta_{0}+\beta _{1}( X+a) ^{2}+\beta _{2}( X+a)\\
Y=( \beta _{0}+a^{2}\beta _{1}+a\beta _{2}) +\beta
_{1}X^{2}+( \beta _{2}+2a\beta _{1}) X \\
Y=\beta _{0}^{\ast }+\beta _{1}X^{2}+\beta _{2}^{\ast }X$$
Suppose now fitting the model excluding the main effect,
$$Y=\beta _{0}+\beta _{1}X^{2}$$
After a shift, the model is reparameterized,
$$Y =\beta _{0}+\beta _{1}( X+a) ^{2} \\
Y=( \beta _{0}+a^{2}\beta _{1}) +\beta _{1}X^{2}+2a\beta _{1}X \\
Y=\beta _{0}^{\ast }+\beta _{1}X^{2}+\beta _{2}^{\ast }X$$
So the main effect is reintroduced, but the model has no coefficient for it. Geometrically, removal of the $X$ term means that the quadratic curve is symmetric about $x=0$ and has its turning point at $x=0$.   
Similarly, if $X_{1}^{2}$ and $X_{2}^{2}$ are included in the model then $X_{1}X_{2}$ should also be included. Omitting the $X_{1}X_{2}$ term assumes that the quadric surface is aligned with the coordinate axis and any rotation of the surface will reintroduce the term.
As I understand it, the hierNet package implements the two-way interaction model - that includes all $X_{i}$ and $X_{i}X_{j}, i \neq j$. That is, $X_{i}^{2}$ terms are not included. More general hierarchical structures can be implemented with overlapping groups in the CAP penalty as discussed in Zhao, et al (2009).
