I'm spending some time learning machine learning (sorry for the recursion :) and I couldn't help be intrigued by the rule of thumb of choosing Gradient Descent over direct equation solving for computing regression coefficients, in the case of multivariate linear regression.

Rule of thumb: if number of features (read coefficients/independent variables) is between $10,000 - 1,000,000$ or above a million, go with Gradient Descent, else matrix inverse computation is fairly manageable on commodity hardware and thus computing the coefficients directly should perform well enough.

Computationally speaking, I get the tradeoff/limitations. But from a statistical standpoint do we really compute models with that many coefficients ever? If I remember my multivariate linear regression classes in grad school, we were cautioned against using too many independent variables since they may have a very negligible impact on the dependent variable or their distributions would not obey the assumptions we make about the data. Even if I did expand my mind to think "many IVs" I still wouldn't have thought of in millions.

Question(s):

• Does this really happen or is it a theoretical issue?
• What's the point of analyzing a million IVs? Does it really give us that much increase in value of information gained as opposed to ignoring them?
• Or is it because, initially we have no idea what is useful, so we just run the damn regression to see what is useful and go from there and possibly prune the set of IVs?

I still believe just because we can analyze "everything" doesn't really mean we should throw it into a solver (or does it) and some of my past questions reflect similar POVs.

I'm yet to finish the course and I may be asking the question to soon, but I just can't get this "WTF/Why" thought out of my head and am trying to understand it to the best of my ability.

I'm spending some time learning machine learning (sorry for the recursion :) and I couldn't help be intrigued by the rule of thumb of choosing Gradient Descent over direct equation solving for computing regression coefficients, in the case of multivariate linear regression.

Rule of thumb: if number of features (read coefficients/independent variables) is between $10,000 - 1,000,000$ or above a million, go with Gradient Descent, else matrix inverse computation is fairly manageable on commodity hardware and thus computing the coefficients directly should perform well enough.

Computationally speaking, I get the tradeoff/limitations. But from a statistical standpoint do we really compute models with that many coefficients ever? If I remember my multivariate linear regression classes in grad school, we were cautioned against using too many independent variables since they may have a very negligible impact on the dependent variable or their distributions would not obey the assumptions we make about the data. Even if I did expand my mind to think "many IVs" I still wouldn't have thought of in millions.

Question(s):

• Does this really happen or is it a theoretical issue?
• What's the point of analyzing a million IVs? Does it really give us that much increase in value of information gained as opposed to ignoring them?
• Or is it because, initially we have no idea what is useful, so we just run the damn regression to see what is useful and go from there and possibly prune the set of IVs?

I still believe just because we can analyze "everything" doesn't really mean we should throw it into a solver (or does it) and some of my past questions reflect similar POVs.

I'm yet to finish the course and I may be asking the question to soon, but I just can't get this "Why" thought out of my head and am trying to understand it to the best of my ability.