1
$\begingroup$

I'm wondering if there exists a software/library that asks you questions about your dataset and then spits out the appropriate statistical method to use. I have searched online to find mostly that it's hard to choose what method should be used, and the solution is "ask an expert". I wonder, does there exist a program currently that can GUESS what would be best?

$\endgroup$
10
  • 4
    $\begingroup$ Such a software/hardware combination has existed for some time. It's called "Consult a statistician." $\endgroup$
    – JimB
    May 26, 2020 at 0:58
  • 2
    $\begingroup$ I would question the assumption that for any dataset (or, to be more specific, any problem or hypothesis posed with reference to a given dataset) there is a single correct method that should be used. $\endgroup$
    – rolando2
    May 26, 2020 at 1:05
  • $\begingroup$ No, and it cannot exist. It's God like power to know what's underlying process for any data set, because that's what you really need to determine the appropriate methid for a given data set $\endgroup$
    – Aksakal
    May 26, 2020 at 3:21
  • $\begingroup$ @Aksakal are you saying that Google Cloud AutoML and AWS Sagemaker Autopilot are deities :-D ? (I've love to have feedback on my answer if I'm wrong.) $\endgroup$
    – Skander H.
    May 26, 2020 at 3:27
  • $\begingroup$ @SkanderH., yes, and I'm a God too, since I can tell you which model to pick for any given data set. $\endgroup$
    – Aksakal
    May 26, 2020 at 3:56

1 Answer 1

3
$\begingroup$

What you are looking for seems to be halfway between automatic model selection, and automated machine learning (AutoML).

Automatic model selection has been around for a while, but is limited in scope. It is useful when you already know before hand what family of models you should use for your problem (e.g. Polynomial Regression), but not exactly which specific model you should use for your data set. You can use cross-validation, information criteria like the AIC and the BIC, and other approaches, to find the best parameters for you model (e.g the order of the polynomial that should be used for the regression). See here for an example, and Chapter 7 of ESL for the theoretical details.

In theory, you can sometimes use the AIC or the BIC for more than one family of model, but it is tricky, and I've never seen it done in practice. It's also pretty challenging to calculate the AIC or the BIC for something like a neural network or an XGBoost model (again even though theoretically possible), so it is not generally recommended to use Information Criteria for those types of models.

You could use cross-validation for any combination of families of models, but again I've rarely seen it used on production business problems. Usually, domain knowledge and results from the literature are used to narrow down the candidates to one family of models, and then model selection is used to pick the specific parameters or configuration. Just about any statistical computing package should allow you to perform automatic model selection.

Most of the established statistical modeling packages either have model selection routines built in, or have straight forward methods for getting the AIC or the BIC (when applicable), or performing cross-validation, so that it is pretty straight forward to write a search routine that minimizes them over the parameter space.

AutoML is a more recent development that is more promising and more comprehensive than simple model selection, but it has a ways to go in terms of maturity both from the point of view of available tools and libraries, and from the point of view of applicability to more general modeling tasks. It is now pretty much the standard for some computer vision and NLP tasks, but in other domains, it is challenging to pull it off (my team has had a hard team getting to work for time series forecasting, and we just went with domain knowledge + grid-search instead). Almost all AutoML approaches use the following 3 steps to find the best model (or combination models) to use:

  1. Meta-Learning =: Find one or more problems that are similar to the problem you are trying to solve, but for which we already have models that perform pretty well. There are ways to make this search systematic, the challenge is in getting access to a list or historical problems and their solutions. Most commercial AutoMl solutions and open source tools have these lists already baked into their code, this isn't really a problem unless you are trying to build your own domain specific tool. The output of this step is a set of base models that will be used as a starting point for the next step. Some of the more advanced AutoML frameworks, especially in NLP and Image use cases, include or Transfer Learning as part of this phase as well.
  2. An optimization/sesarch procedure, typically Bayesian Optimization or Reinforcement Learning, to narrow down the set of base models from the previous step, to just one or just a small number of models.
  3. Ensembling: If the previous step outputs a small set of models instead of one unique model, then predictions will be done using an ensemble of the output models. It has been noted that using a combination of models to generating predictions performs better on out-of-sample data than trying to find one single model that has the best possible accuracy. Most AutoML models perform some version of this to improve production performance.

enter image description here Image from "Efficient and Robust Automated Machine Learning", Feurer et*. al*, Advances in Neural Information Processing Systems 28 (NIPS 2015)

There are some decent open source tools for doing AutoML, but the best tools are commercial APIs. Organizations might be willing to open source steps 2&3, but the data that is used in step 1 is often proprietary.

There are a couple of things to note about both of these approaches:

  • In both cases, you need to specify the loss metric for your problem. There's no algorithm (yet?) that will look at the data and tell you to use RMSE instead of MAPE, to use precision instead of recall, etc...the loss metric will be determined by your use case, and domain knowledge.

  • Nor is there any algorithm that will tell you whether your problem is better treated as a supervised learning problem, a clustering problem, an reinforcement learning problem, etc...again use case, business objectives, and domain knowledge are the only tools so far.

If the last two points are the level at which you expect the package to spit out a response, then yeah "ask an expert" is really the only way to go.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.