12
$\begingroup$

As I discover machine learning I see different interesting techniques such as:

  • automatically tune algorithms with techniques such as grid search,
  • get more accurate results through the combination of different algorithms of the same "type", that's boosting,
  • get more accurate results through the combination of different algorithms (but not the same type of algorithms), that's stacking,
  • and probably lots more I still have to discover...

My question is the following: there are all those pieces. But is it possible to put them together to make an algorithm that takes as input cleaned data and outputs good results by taking the best out of all techniques? (Of course it will probably be less efficient that a professional data scientist, but he will be better than me!) If yes, do you have sample codes or do you know frameworks that can do that?

EDIT : After some answers, it seems some narrowing has to be done. Let's take an example, we have one column with categorical data, let's call it y and we want to predict it from numerical data X that is either dummies or real numerical data (height, temperature). We assume cleaning has been done previously. Are there existing algorithm that can take such data and output a prediction? (by testing multiple algorithms, tuning them, boosting, etc.) If yes, is it computationally efficient (are the calculations done in a reasonable time if we compare to normal algorithm), and do you have an example of code?

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Probably not until there is strong AI. $\endgroup$
    – gung - Reinstate Monica
    Commented Jul 6, 2015 at 15:28
  • 1
    $\begingroup$ With regards to automatic tuning you might be interested in reading about Hyperparameter Search in Machine Learning. Grid search is a terrible way to optimize hyperparameters. $\endgroup$
    – Marc Claesen
    Commented Jul 6, 2015 at 15:54
  • 2
    $\begingroup$ Do you have to use so much formatting? $\endgroup$
    – Sycorax ♦
    Commented Jul 6, 2015 at 15:54
  • 2
    $\begingroup$ There are products which claim they do what you describe, e.g. Crystal Ball. I personally don't trust them, but as you wrote: they do the job better than someone who has no clue about statistics $\endgroup$
    – Aksakal
    Commented Jul 6, 2015 at 16:01
  • 1
    $\begingroup$ For forecasting auto.arima (from forecast library) can be better than humans - Rob Hyndman mentioned in several times in his presentations. So there are areas where some sorts of "automatic learning" are applied with success. $\endgroup$
    – Tim
    Commented Jul 7, 2015 at 10:48

5 Answers 5

Reset to default
7
$\begingroup$

If you know beforehand what kind of data you will feed in ("these are monthly sales of CPGs, with prices and promotion markers, and I want a point forecast"), so you can tune your setup ahead of time, that will likely be possible and already done, see various "expert systems" for certain specific tasks.

If you are looking for something that can take any kind of data and do "something useful" with it ("ah, here I am supposed to recognize handwriting and output ZIP codes, and there I should do fraud detection, and this input file obviously is a credit scoring task"), no, I don't think that will happen in a long time.

Sorry for an opinion-based answer to what might well be closed as an opinion-based question.

EDIT to address the edited question:

we have one column with categorical data, let's call it $y$ and we want to predict it from numerical data $X$ that is either dummies or real numerical data

This sounds like something that Random Forests are actually pretty good at. Then again, a "general-purpose" algorithm like RFs will likely never beat an algorithm that was tuned to a particular type of $y$ known beforehand, e.g., handwritten digits, or credit default risks.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I've edited my question thanks to you, there is a first part "take whatever input and throw me the result" and your answer is really insightful, and a second part "for a particular type of questions, and data formatting, find me the result". $\endgroup$
    – Pholochtairze
    Commented Jul 6, 2015 at 16:02
  • $\begingroup$ Not sure the last paragraph still holds, given the immense success of deep learning on a large variety of problems. These methods are definitely general purpose, yet they hold records in several application domains such as computer vision and NLP. One might argue that the architectures differ between tasks, but on a theoretical level a fully connected deep network would perform at least as good as, say, convolutional networks, it's just that proper training methods remain elusive. $\endgroup$
    – Marc Claesen
    Commented Oct 8, 2015 at 12:23
  • $\begingroup$ @MarcClaesen: I'd say that "it's just that proper training methods remain elusive" are the point where you will still need domain specific expertise. $\endgroup$
    – Stephan Kolassa
    Commented Oct 8, 2015 at 13:11
7
$\begingroup$

What you describe already exists to some extent, for example in AutoWEKA, and is being researrched actively (e.g. challenges like Chalearn's AutoML).

This is usually considered in the subfield of hyperparameter optimization. Software packages like Optunity, Hyperopt and ParamILS can be used to automatically optimize hyperparameters for a given approach and choose which approach happens to be the best. That said, such optimization problems are not trivial and usually it takes a long time to automatically obtain the best model (or close to it).

You can find an example of using Optunity to automatically determine the best learning algorithm and optimize its hyperparameters at http://optunity.readthedocs.org/en/latest/notebooks/notebooks/sklearn-automated-classification.html

Improve this answer
$\endgroup$
4
$\begingroup$

Advances in hyper parameter tuning and ensemble models are taking a lot of the 'art' out of model building. However, there are two important aspects of model building that hyper parameter tuning and ensembles don't deal with and will keep you from finding the best possible model.

First, certain types of algorithms are better at modeling certain types of data. For example, if there are interactions between your variables, an additive model is not going to find them, but a decision tree will. Knowing how models behave on different dataset, and picking the right one, might require knowledge about the domain to find the best modeling algorithm.

Second, feature engineering and feature extraction is the real 'art' to model building. Your question assumes that the dataset is already prepared. But what you should not assume is that the dataset is the best possible representation of what you are trying to model. This is always and open question. In many cases, where the data set is complex, you can engineer features all day long, but you risk throwing more and more noise into the algorithm. To know which features to add you must know which features make sense from a statistical perspective and which make sense from the perspective of the domain expert.

For those two reasons, I conclude that no, you will not be able to find an algorithm that finds the best possible model automatically. This is also why I'm skeptical of software vendors pitching tools that will replace the need for data scientists.

However, if you are narrowing your ambition to finding the best model from a fixed set of models, with optimal hyper parameters, where 'best' is defined as the highest predictive accuracy on a training set, then yes, this is possible.

Check out the caret package in R as an example of how to automatically tune models. caret uses grid search, which has flaws, and it only builds one model at a time. However, there are functions to compare models and convenient wrappers for a long list of models from many different R packages.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I agree that we're still far away from fully automated machine learning, but not for the reasons you specify. Two things: (i) finding the best model (+ optimize its hyperparameters) from a given set is already possible and (ii) feature engineering and feature extraction is losing importance due to advances in deep learning. The main thing we currently lack is automated ways to incorporate prior knowledge and field specific common sense. $\endgroup$
    – Marc Claesen
    Commented Jul 13, 2015 at 8:13
  • $\begingroup$ I guess I don't see how "prior knowledge and field specific common sense" can improve the ML process except to help (i) find the best model and (ii) find the best features. I tried to distinguish between true models and optimal (max accuracy) models with the second to last paragraph. $\endgroup$
    – brandco
    Commented Jul 13, 2015 at 15:29
  • $\begingroup$ to uncover obvious flaws in the modeling process such as erroneous labels and/or information leaks, how to deal with missing data, identifying the actual learning task (+ a suitable score/loss function) and generally do a full data cleanup (which has always been the main effort on all practical problems I've been involved in). $\endgroup$
    – Marc Claesen
    Commented Jul 13, 2015 at 15:41
2
$\begingroup$

Depends on who you ask.

I recently heard a talk by Scott Golder at Context Relevant. Their product is essentially a feature and model selection robot. The basic underlying model is logistic regression, but the system basically uses machine learning to figure out the right combination of feature selection, dimension reduction, regularization, etc. for generating accurate predictions. It was a very impressive talk, and the details are all very proprietary. Apparently their clients include major financial companies and the system can handle arbitrarily massive data sets.

At least a few people, then, seem to think automated data science is already here, at least for certain applications. And some of those people (Context Relevant's clients) are apparently paying through the nose for access to it.

Improve this answer
$\endgroup$
1
$\begingroup$

No it's not a dream. We've implemented this (the combination of all the techniques you mentioned, including stacked ensembles) in the H2O machine learning library. You can read more about it and find code examples in R and Python here.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.