Skills hard to find in machine learners?

Question

It seems that data mining and machine learning became so popular that now almost every CS student knows about classifiers, clustering, statistical NLP ... etc. So it seems that finding data miners is not a hard thing nowadays.

My question is: What are the skills that a data miner could learn that would make him different than the others? To make him a not-so-easy-to-find-someone-like-him kind of person.

Answer 1

I have seen multiple times developers use ML techniques. This is the usual pattern:

1. download library with fancy name;
2. spend 10 mins reading how to use it (skipping any statistics, maths, etc);
3. feed it with data (no preprocessing);
4. measure performance (e.g. accuracy even if classes are totally imbalanced) and tell everybody how awesome it is with its 99% accuracy;
5. deploy in production with epic fail results;
6. find somebody who understand what's going on to help them out because the instruction manual makes no sense at all.

The simple answer is that (most) software engineers are very weak in stats and math. This is the advantage of anyone who wants to compete with them. Of course stats people are out of their comfort zone if they need to write production code. The kind of role that become really rare is that of Data Scientist. It is someone who can write code to access and play with the enormous amount of data and find the value in them.

Answer 2

What it's about

Just knowing about techniques is akin to knowing the animals in a zoo -- you can name them, describe their properties, perhaps identify them in the wild.

Understanding when to use them, formulating, building, testing, and deploying working mathematical models within an application area while avoiding the pitfalls --- these are the skills that distinguish, in my opinion.

The emphasis should be on the science, applying a systematic, scientific approach to business, industrial, and commercial problems. But this requires skills broader than data mining & machine learning, as Robin Bloor argues persuasively in "A Data Science Rant".

So what can one do?

Application areas: learn about various application areas close to your interest, or that of your employer. The area is often less important than understanding how the model was built and how it was used to add value to that area. Models that are successful in one area can often be transplanted and applied to different areas that work in similar ways.

Competitions: try the data mining competition site Kaggle, preferably joining a team of others. (Kaggle: a platform for predictive modeling competitions. Companies, governments and researchers present datasets and problems and the world’s best data scientists compete to produce the best solutions.)

Fundamentals: There are four: (1) solid grounding in statistics, (2) reasonably good programming skills, (3) understanding how to structure complex data queries, (4) building data models. If any are weak, then that's an important place to start.

---

A few quotes in this respect:

``I learned very early the difference between knowing the name of something and knowing something. You can know the name of a bird in all the languages of the world, but when you're finished, you'll know absolutely nothing whatever about the bird... So let's look at the bird and see what it's doing -- that's what counts.'' -- Richard Feynman, "The Making of a Scientist", p14 in What Do You Care What Other People Think, 1988

Keep in mind:

``The combination of skills required to carry out these business science [data science] projects rarely reside in one person. Someone could indeed have attained extensive knowledge in the triple areas of (i) what the business does, (ii) how to use statistics, and (iii) how to manage data and data flows. If so, he or she could indeed claim to be a business scientist (a.k.a., “data scientist”) in a given sector. But such individuals are almost as rare as hen’s teeth.'' -- Robin Bloor, A Data Science Rant, Aug 2013, Inside Analysis

And finally:

``The Map is Not the Territory.'' -- Alfred Korzybski, 1933, Science & Sanity.

Most real, applied problems are not accessible solely from ``the map''. To do practical things with mathematical modelling one must be willing to get grubby with details, subtleties, and exceptions. Nothing can substitute for knowing the territory first-hand.

---

Answer 3

I agree with everything that's been said. What stands out for me are:

1. How few machine learning "experts" are really interested in the subject matter to which they want to apply ML
2. How few truly understand predictive accuracy and proper scoring rules
3. How few understand principles of validation
4. How few know when to use a black box vs. a traditional regression model
5. How none of the "experts" seem to have ever studied Bayes optimum decision or loss/utility/cost functions [this lack of understanding is displayed almost any time someone uses classification instead of predicted risk]

Answer 4

Here are a couple of things to make you stand out from the crowd:

• Understand the application domain or domains. That is, the business environment or other context.
• Understand the big picture. This is very important! People who study machine learning often get lost in the details. Think about the overall picture that your ML models will fit into. Often the ML part is just a small segment of a much larger system. Understand the whole system.
• Study utility and decision theory and Bayesian inference, not just whatever is now considered "the usual" ML models. Bayesian inference is just a way to formalize the notion of bringing all contextual information to bear on a problem. Utility and decision theory is about bringing values into the picture.

The overall message that applies to all three points: Look at the big picture, don't get lost in the details.

Answer 5

I would put out there the notion of "soft skills".

• recognising who the "expert" is for method X, and being able to tap into their knowledge (you shouldn't be able to or expected to know everything about erything). The ability and willingness to collaborate with others.

• the ability to translate or represent "the real world" with the mathematics used in ML.

• the ability to explain your methods in different ways to different audiences - knowing when to focus on details and when to step back and view the wider context.

• systems thinking, being able to see how your role feeds into other areas of the business, and how these areas feed back into your work.

• an appreciation and understanding of uncertainty, and having some structured methods to deal with it. Being able to state clearly what your assumptions are.

Answer 6

Being able to generalize well

This is the essence of a good model. And it is the essence of what makes the best practitioners of the art of machine learning stand out from the crowd.

Understanding that the goal is to maximize performance on unseen data, i.e minimize generalization error, not to minimize training error. Knowing how to avoid both over-fitting and under-fitting. Coming up with models that are not too complex yet not too simple in describing the problem. Extracting the gist of a training-set, rather than the maximum possible.

It is surprising how often, even experienced machine learning practitioners, fail to follow this principle. One reason is that humans fail to appreciate two vast theory-vs-practice magnitude differences:

• How much larger is the space of all possible examples compared to the training-data at hand, even when the training data is very large.
• How much larger is the full "hypothesis space": number of possible models for a problem, compared to the practical "solution space": everything you can think of, and everything your software/tools are capable of representing.

The 2nd magnitude gap is especially incomprehensible. Even for the simplest problem with $N$ inputs and a binary outcome, there are $2^N$ possible input-examples. And this is dwarfed by the exponentially larger "hypothesis space" number which is $2^{2^N}$ possible models.

It is also what most of the above answers said in more specific and concrete ways. to generalize well is just the shortest way I could think of, to put it.

Answer 7

The skill that sets one data miner apart from others is the ability to interpret machine learning models. Most build a machine, report the error and then stop. What are the mathematical relationships between the features? Are the effects additive or non-additive or both? Are any of the features irrelevant? Is the machine expected under the null hypothesis that there are only chance patterns in the data? Does the model generalize to independent data? What do these patterns mean for the problem being studied? What are the inference? What are the insights? Why should a domain expert get excited? Will the machine lead to the domain expert asking new questions and designing new experiments? Can the data miner effectively communicate the model and its implications to the world?

Answer 8

Having done scientific research in Machine learning / Statistical pattern recognition for 17 years - I can come up with a few skills that make a wanted-for data scientist stand out from others.

Machine learning is about:

• Achieving the algorithmic knowledge of learning algorithms out there, and getting the skill of how to apply these learning algorithms successfully to practical ML-problems,
• Gaining the required level of knowledge in probability theory (beginning with Bayes rule), parametric statistics and nonparametric statistics as to assess and compare different types of learnable models, model performance, confidence intervals, sampling theory, and ML-estimation. Don't underestimate the level of statistical knowledge required to become a skilled professional (go though the proof of the central limit theorem, for example, and understand when this theorem is not applicable),
• Understand mathematical approximation theory to a level so you can see why feed-forward neural networks (with 2 hidden layers, or more) are universal approximators - offspring of the theorem of Komolgorov,
• Gain practical experience from learning classifiers on many different training sets, and validate their performance on independent test sets,
• Understand that optimal feature selection and model selection require knowledge of how algorithmics and statistics intertwine - take as an example the branch-and-bound algorithm for feature selection. Recognize that feature selection and model selection always involve a bias-variance trade-off (between performance variance and optimal model fit),
• Go through the derivations by Richard & Lippmann (1991) why neural network classifiers estimate bayesian a posteriori probabilities,
• Get acquainted with major scientific breakthroughs in the last eight decades with respect to the development of statistical and algorithmic prediction models, beginning with linear discriminant analysis (a statistical classifier),
• Embrace the fact that no one machine learning scheme is optimal for almost 'all kinds of problems'. So - neural networks are not better than all other models, neither are support vector machines or random forests for that sake - it all depends on the statistics of the underlying domain. Practical machine learning remains an experimental science, but many relevant theoretical results have been published in literature over the years.

It's hard work to be able to span algorithmics, statistics and mathematical approximation theory. I did my Ph.D. in machine learning and first really became a professional after more than 10 years of work.

A final note is that it is not always necessary to be a programmer to apply machine learning algorithms. ML-suites like Weka or available classifier services like insight classifiers let data scientists apply different ML-algorithms without having to program in for example Python or R.

It is a great discipline - Machine learning.

Answer 9

I see there are two parts while handling machine learning in practice

1. Engineering ( which covers all the algorithms, learning different packages, programming).

2. Curiosity/Reasoning (ability to ask better questions to data).

I think 'curiosity/reasoning' is the skill which distinguishes one from others. For example, if you see the leader boards of the kaggle completions, many people may have used common(similar) algorithms, what makes the difference is, how one logically question the data and formulate it.