14
$\begingroup$

I've come across a rumour that some study showed that the performance of predictive models depends more on the expertise of the data analyst with the chosen method than on the choice of the method.
In other words, the claim is that it is more important that the data analyst is familiar with the chosen method than how "appropriate" the method would seem for the problem from a more theoretical standpoint.

This was mentioned in the context of chemometrics, which involves typically problems of many variates (100s - 1000s), multiple collinearity, and of course, too few samples. Prediction may have been classification or regression.

My personal experience suggests that this is plausible, but a study was mentioned (I asked the person who mentioned that by email after a quick but unsuccessful search, but never received any answer). However, also with a more elaborate search, I was not able to track down any papers.

Is anyone aware of such findings? If not, what does the personal experience of Big Guys here say?

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ I'm more of a Little Guy around here, but what I have seen in Neural Networks supports this hypothesis: far from being an "out of the box" tool where "the machine learns" something, successful classification or prediction seems to depend a lot on how smart the person is that tells the network how to learn from the data - most importantly in terms of data preprocessing, but also in terms of network architecture etc. $\endgroup$ – S. Kolassa - Reinstate Monica Feb 7 '13 at 20:35
  • 1
    $\begingroup$ I think that's figure 2.4 from The Elements of Statistical Learning where they compare nearest neighbors with regession-type methods (and of course they provide multiple comparison points throughout the book, too). $\endgroup$ – StasK Feb 8 '13 at 15:08
  • $\begingroup$ @StasK: thanks for the reminder (shame on me for not remembering). They also report that in practice PCR, PLS and ridge regression are very similar, and LDA and logistic regression as well. However, the latter methods are also very similar from a theoretical point of view. $\endgroup$ – cbeleites supports Monica Feb 8 '13 at 20:16
1
$\begingroup$

Actually, I have heard a rumor that decent learning machines are usually better than experts, because the human inclination is to minimize variance at the expense of bias (oversmooth), leading to poor predictive performance in new dataset. The machine is calibrated to minimize MSE, and thus tends to do better in terms of prediction in a new dataset.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ In my experience it is certainly true that humans tend to overfit. However, in my experience you also need a decent expert who chooses the not-overfitting learning machine. Otherwise someone just chooses a learning machine that overfits. $\endgroup$ – cbeleites supports Monica Apr 1 '13 at 10:55
  • 1
    $\begingroup$ MSE in general doesn't protect from overfitting unless you restrict the model very much - and there the expert comes in again. Nevertheless people try to optimize e.g. model hyperparameters. Particularly iterative optimization strategies overfit, (MSE or not), unless you can afford a completely new set of independent test data for each iteration. Maybe I should say that I come from a field where test cases are very rare. And, in any case you may argue that this a not a decent learning machine. $\endgroup$ – cbeleites supports Monica Apr 1 '13 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.