Do we really perform multivariate regression analysis with *million* coefficients/independent variables?

Question

I'm spending some time learning machine learning (sorry for the recursion :) and I couldn't help be intrigued by the rule of thumb of choosing Gradient Descent over direct equation solving for computing regression coefficients, in the case of multivariate linear regression.

Rule of thumb: if number of features (read coefficients/independent variables) is between $10,000 - 1,000,000$ or above a million, go with Gradient Descent, else matrix inverse computation is fairly manageable on commodity hardware and thus computing the coefficients directly should perform well enough.

Computationally speaking, I get the tradeoff/limitations. But from a statistical standpoint do we really compute models with that many coefficients ever? If I remember my multivariate linear regression classes in grad school, we were cautioned against using too many independent variables since they may have a very negligible impact on the dependent variable or their distributions would not obey the assumptions we make about the data. Even if I did expand my mind to think "many IVs" I still wouldn't have thought of in millions.

Question(s):

• Does this really happen or is it a theoretical issue?
• What's the point of analyzing a million IVs? Does it really give us that much increase in value of information gained as opposed to ignoring them?
• Or is it because, initially we have no idea what is useful, so we just run the damn regression to see what is useful and go from there and possibly prune the set of IVs?

I still believe just because we can analyze "everything" doesn't really mean we should throw it into a solver (or does it) and some of my past questions reflect similar POVs.

I'm yet to finish the course and I may be asking the question to soon, but I just can't get this "Why" thought out of my head and am trying to understand it to the best of my ability.

Answer 1

Does this really happen or is it a theoretical issue?

It happens, see any popular deeplearning model for computer vision. Say, alexnet has a dense connection between 2048 and 2048 units, that's 4 million coefficients.

What's the point of analyzing a million IVs? Does it really give us that much increase in value of information gained as opposed to ignoring them?

If you're analyzing highly categorical data (say, internet advertisement data), your model has to keep some meaningful 'descriptions' for each category (e.g. city, page id, sitename, advertisement id, user id, etc.), the actual size of 'description' depends on the selected ML model.

Even simple logistic regression will have dozens of thousands of parameters to be fitted (one per category). More advanced models like factorization machines are going to have times more.

Or is it because, initially we have no idea what is useful, so we just run the damn regression to see what is useful and go from there and possibly prune the set of IVs?

Actually, most of fitted parameters in these models can be dropped, but you can not know that beforehand, so you leave the problem of defining which parameters are important to machine learning, and impose some regularizations to put 'soft limit' to the effective number of parameters to stay.

... and I think you'll find such examples later in your ML course.