0
$\begingroup$

(I'm brand-new here. I have strong mathematical and computing backgrounds, but little knowledge of statistics. If this question belongs elsewhere, I would appreciate a pointer to where.)

I believe I'm really simply searching for the best term to do further research on and perhaps links to helpful resources on that. But any suggestions are welcome.

I'm looking to reduce by an order of magnitude or so the number of factors being used to calculate a result. What terms should I be researching in order to manage this?

For a complex process, we gather 500 - 1000 different factors and feed them into a (mostly) black-box process that chooses sensible defaults for the missing ones and returns a boolean result. We'd like to choose a much smaller set of factors, perhaps 30 - 50 of them, that when fed into the process yields the closest match to the full calculation. (Since we're returning yes or no, I guess "closest match" really means that for a few thousand sample cases the number on which they disagree is the smallest.) We know that some of the factors are closely correlated and some are much more independent, but don't yet have details on those.

Gathering the factors is the expensive part of the process. Running the calculations is fairly quick.

We have to do this for several dozen different large sets of factors, and I will probably write somewhat generic programs to do this for them once I figure out the technique.

I assume this is not a unique requirement. I'd like to know what I should be researching in order to handle this.

A web search on the title yields terms such as "Factor Analysis" and "Principal Components Analysis", both of which sound like good places to start. Is one of those the correct place to focus my research? Is there a different term? Or am I on the wrong track altogether?

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ You're in the right ballpark. "Variable selection" (also called feature selection) may be closer to what you're looking for if you only want to use a subset of the 1000 variables. To start, also take a look at LASSO regression which achieves sparsity in the non-zero estimated parameters (i.e. variable selection) by adding an L1 norm penalty on the parameter estimate to the objective of ordinary least squares regression. $\endgroup$ – Matthew Gunn Dec 2 at 21:54
  • $\begingroup$ @MatthewGunn: Thank you very much. I'll take a look. $\endgroup$ – Scott Sauyet Dec 2 at 22:11
  • 1
    $\begingroup$ Do you have data already? If no, and maybe even if yes, look into experimental design, see experiment-design, maybe start with stats.stackexchange.com/questions/338740/… $\endgroup$ – kjetil b halvorsen Dec 2 at 23:16
  • $\begingroup$ @kjetilbhalvorsen: I don't have the data yet; I know approximately how to gather it. I learned today that some folks in my organization have done this once manually with a lot of trial and error using their subject matter expertise to speed this up. It has taken weeks or even months to do the one case they've tried, and there's no guarantee at all that it's close to optimal. I'm almost certain that I can drop this process down to minutes or hours once I determine proper algorithms. And we can make it a regular routine when there is new data. I will investigate those links, thank you. $\endgroup$ – Scott Sauyet Dec 3 at 2:01
  • $\begingroup$ Good, if you have new questions after reading those links, please edit your post here with those. $\endgroup$ – kjetil b halvorsen Dec 3 at 2:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.