Standardisation when finding the start of the regularisation path I'm having some trouble understanding how standardisation plays into cross-validation with the lasso. Let's say we want to perform standardisation for the cross-validation process.
I can see why we shouldn't standardise before the train/test split, because that would be information leakage. On the other hand, when we run cross-validation for the lasso (or any high-dim method), we run the model along a regularisation path, starting at $\lambda^\text{max},$ which is the exact value at which the first coefficient enters the model.
This value is found using the data (an example is given in section 3.3 in the SGL paper). Hence, the question naturally arises whether we should standardise $X$ when calculating $\lambda^\text{max}$. I've tried looking at the code for the SGL package, but the authors standardise the data set right at the start of the CV function (even before the split), so I don't think this is quite correct anyway.
I've run some experiments and it seems that when I do standardise $X$ for the calculation of $\lambda^\text{max}$, the first model in the path isn't completely sparse. On the other hand, when I don't, the model is completely sparse, so this doesn't quite make sense to me.
 A: I suspect that standardisation before doing anything (even splitting) is something of a standard, even though you are right that there is an information leakage problem. I haven't seen this discussed in the basic lasso literature, but then I don't know all the literature (in much literature variables are assumed to be standardised with no further discussion of the issue). Most people would probably think that the information leakage problem from standardisation is small. Also one could suspect that the leakage may lead to a certain overoptimism in general but wouldn't effect parameter selection of $\lambda$. I'm not sure about this, but let's say intuition doesn't tell me about any expected direction of bias when it comes to selecting $\lambda$ through standardisation of all data together. Of course there may be specific situations in which it makes a difference anyway.
In any case finding $\lambda^{max}$ requires standardisation (values of $\lambda$ are relative to whether data are standardised), unless you don't want to standardise at all. Of course if standardisation is done before anything else, this isn't a problem (apart from information leakage).
I think it is conceivable to do this within cross-validation in such a way that information leakage is avoided, but then obviously there would be different values of $\lambda^{max}$ for different cross-validation instances. I'm not sure how big a problem this is, probably it could be done but would be more complicated than just standardising everything before splitting, which I believe pretty much everyone is doing.
