What is the best way/method to model patterns across coefficients and reduce number of coefficients based on these patterns?

We have hundreds of regressors on the same scale and try to reduce the number of coefficients with respect to the regressors' relationship with our outcome variable.

To some extent we try to invert the logic behind finite mixture regression: Instead of estimating different coefficients for different subgroups of our observations, we try to built subgroups of regressors which have a similar effect and are interested in the "overall coefficients" of these subgroups of regressors. Our idea is to estimate both membership to the latent regressors and the final regression parameters for the latent regressors simultaneously and to allow for inference in the end. Our thought was as follows: While a finite mixture regression assumes a mixture distribution of the outcomes and estimates the membership of the observations to the distributions and the regression parameters simultaneously, we rather wanted to transfer this idea to the parameter vector. Instead of assuming a multivariate normal distribution of the regression parameters, our idea was to assume a multivariate mixture distribution at this point, so assign regressors to certain latent regressors and somehow estimate the overall effect for each latent regressor on our outcome.

One easy example would be the following:

Assume we have 500 regressors in total. 250 regressors have a positive coefficient around 100. 250 regressors have a negative coefficient around -100.

Assuming n "latent regressors", our model should (1) identify the regressors which have similar effects on the outcome (in the example of 2 latent regressors, the first 250 regressors should be identified as one latent regressor due to their positive effect, the next 250 regressors should be identified due to their negative effect), (2) should consider the original regressors as n latent regressors in the model and (3) should also give n coefficients and allow for inference for the latent regressors.

Is there any appropriate method? Since we do not come from econometrics in the first place, we unfortunately do not know whether our idea is realistic at all. PLSR that Peter mentioned may indeed be the best way and we have to rethink the method.

  • 1
    $\begingroup$ Point of information: in latent class regression, we assume the latent class is a multinomial dependent variable. We use observed independent variables (not necessarily the same as the latent class indicators!) to predict the latent class. I think what you describe is closer to finite mixture modeling. There, you have one $y$. You assume a regression relationship, e.g. $y = X\beta + \epsilon$. You assume that relationship varies over $k$ latent classes, and you ask your software to estimate the $\beta$s in each latent class. $\endgroup$
    – Weiwen Ng
    Commented Jul 23, 2019 at 20:31
  • $\begingroup$ Thanks for your comment! Of course, you are right! I adjusted my question accordingly. $\endgroup$ Commented Jul 24, 2019 at 8:36

1 Answer 1


First, I recommend using substantive knowledge to do some of this. Think about what all those variables are. Think about how reliable and valid they are. Think about whether they will be easy or hard to collect. Think about which ones will generate missing data. Etc.

Then I recommend partial least squares regression (PLSR).

PLSR is a date reduction method that considers the relationships among the IVs and between the IVs and the DVs. It is similar to principal components, but that (as you note in a comment) does not consider the DV (or outcome variable).

  • $\begingroup$ Thank you very much for your answer! We have also been reading about PLSR, but have rejected this method because it has a strong focus on predicting outcomes. But we are much more interested in describing the relationship between X and Y. I have explained our basic idea in more detail in my question post now. But maybe our idea isn't even feasible and PLSR is actually the best way to do it. $\endgroup$ Commented Jul 24, 2019 at 8:32
  • 1
    $\begingroup$ What is the difference between "predicting outcomes" and "the relationship between X and Y"? I guess you could mean having meaningful X's, but that's going to be a problem with any method of data reduction, I think. $\endgroup$
    – Peter Flom
    Commented Jul 24, 2019 at 11:26
  • $\begingroup$ Everything we read on the topic of PLS focuses on the explained variance or reducing prediction errors. We are not really interested in explaining or predicting y as good as possible but rather in identifying the effect of x on y as causal as possible. $\endgroup$ Commented Jul 24, 2019 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.