I thought about something interesting today.
Suppose we have a regression problem where the relationship between the response and the predictor variables is approximately linear.

Let $n$ be the number of observations and $p$ the number of predictor variables.

Which regression methods would be appropriate for each individual case where

$n$ is $10$, $1000$ and $100 000$ times respectively more than $p$.

How would these change if interpretability is an important factor?


Well, here is an opinion, which I do not fully subscribe to, but likely worthy of quoting:

You can build a linear model with all predictor variables and look for p-values. Then, you can keep predictors only with significant p values ( p < 0.05). Yes, you can decrease the dimension of data using this method. But, there is a problem. p values come from the linear modelling which means already you have assigned linear relationship between your predictors and response variables. But that may not be the ‘real’ case, and some of the predictors may have non-linear relationship with response variables, which were mistakenly modelled as linear relations to take the next step. So, it would be an inaccurate method. Concretely, that is why the most famous dimensionality reduction algorithm, principal component analysis, is done in unsupervised way, which is not relating the response and predictors while taking the decision. For your information only, the way you were asking to reduce dimension has a dedicated library in statistical programming language R which is called “relaimpo” (meaning relative importance). This is used when you are sure that your model has to be linear. And, then using it, you can see the relative importances of predictors and remove ones with lower values. But, the core idea to build the library comes from the idea of the significance level of p values or simply put, using Pearson correlations.

With higher dimensions (large # of explanatory variables) with even large number of observations, I believe more sophisticated analysis like principal component regression or a Factor regression model are needed. Some comments per Wikipedia on the latter to quote:

Factor regression model is a combinatorial model of factor model and regression model; or alternatively, it can be viewed as the hybrid factor model,[11] whose factors are partially known.

And also more background comments:

Factor analysis searches for such joint variations in response to unobserved latent variables. The observed variables are modelled as linear combinations of the potential factors, plus "error" terms. Factor analysis aims to find independent latent variables...The theory behind factor analytic methods is that the information gained about the interdependencies between observed variables can be used later to reduce the set of variables in a dataset. Factor analysis is commonly used in biology, psychometrics, personality theories, marketing, product management, operations research, and finance. It may help to deal with data sets where there are large numbers of observed variables that are thought to reflect a smaller number of underlying/latent variables. It is one of the most commonly used inter-dependency techniques and is used when the relevant set of variables shows a systematic inter-dependence and the objective is to find out the latent factors that create a commonality...Factor analysis is related to principal component analysis (PCA), but the two are not identical.[1]


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.