# Skills hard to find in machine learners?

It seems that data mining and machine learning became so popular that now almost every CS student knows about classifiers, clustering, statistical NLP ... etc. So it seems that finding data miners is not a hard thing nowadays.

My question is: What are the skills that a data miner could learn that would make him different than the others? To make him a not-so-easy-to-find-someone-like-him kind of person.

• While it's a response to a different question, several of the points in an old answer of mine can be carried over here. Assad makes some similar remarks in his very nice answer below. – cardinal Jun 24 '14 at 10:48

I have seen multiple times developers use ML techniques. This is the usual pattern:

2. spend 10 mins reading how to use it (skipping any statistics, maths, etc);
3. feed it with data (no preprocessing);
4. measure performance (e.g. accuracy even if classes are totally imbalanced) and tell everybody how awesome it is with its 99% accuracy;
5. deploy in production with epic fail results;
6. find somebody who understand what's going on to help them out because the instruction manual makes no sense at all.

The simple answer is that (most) software engineers are very weak in stats and math. This is the advantage of anyone who wants to compete with them. Of course stats people are out of their comfort zone if they need to write production code. The kind of role that become really rare is that of Data Scientist. It is someone who can write code to access and play with the enormous amount of data and find the value in them.

• LOL at "tell everybody how awesome it is with its 99% accuracy" – Jack Twain Jun 24 '14 at 7:51
• +1 I agree with everything. "[...] Data Scientist. It is someone who can write code to access and play with the enormous amount of data and find value the in them.". Which to me suggests it should stay naturally rare, since it seems unfeasible to most mortals to tackle the complex statistical and foundational issues as well as turning something into a production code. It also explains why we have cardiovascular surgeons, anaesthesiologists, neurologists, nurses, hospital administrators etc. Or civil, aeronautic, mining, chemical, mechanical engineers etc. – Thomas Speidel Jun 24 '14 at 22:02
• To me, this is not really a description of what someone needs to be a stand out ML candidate - more of a ML bash. Sounds like you are describing someone who is too focused on getting "the answer" before they are even sure of what the question is. Basically, your "typical ML" person is someone who has poor planning skills, and someone who doesn't discuss what they intend to do with the "client" before ploughing ahead, and coming back with "the answer". This is not something that is helped by having good maths/stats - it is something that requires good communication skills. – probabilityislogic Aug 30 '14 at 13:30

Just knowing about techniques is akin to knowing the animals in a zoo -- you can name them, describe their properties, perhaps identify them in the wild.

Understanding when to use them, formulating, building, testing, and deploying working mathematical models within an application area while avoiding the pitfalls --- these are the skills that distinguish, in my opinion.

The emphasis should be on the science, applying a systematic, scientific approach to business, industrial, and commercial problems. But this requires skills broader than data mining & machine learning, as Robin Bloor argues persuasively in "A Data Science Rant".

So what can one do?

Application areas: learn about various application areas close to your interest, or that of your employer. The area is often less important than understanding how the model was built and how it was used to add value to that area. Models that are successful in one area can often be transplanted and applied to different areas that work in similar ways.

Competitions: try the data mining competition site Kaggle, preferably joining a team of others. (Kaggle: a platform for predictive modeling competitions. Companies, governments and researchers present datasets and problems and the world’s best data scientists compete to produce the best solutions.)

Fundamentals: There are four: (1) solid grounding in statistics, (2) reasonably good programming skills, (3) understanding how to structure complex data queries, (4) building data models. If any are weak, then that's an important place to start.

A few quotes in this respect:

I learned very early the difference between knowing the name of something and knowing something. You can know the name of a bird in all the languages of the world, but when you're finished, you'll know absolutely nothing whatever about the bird... So let's look at the bird and see what it's doing -- that's what counts.'' -- Richard Feynman, "The Making of a Scientist", p14 in What Do You Care What Other People Think, 1988

Keep in mind:

The combination of skills required to carry out these business science [data science] projects rarely reside in one person. Someone could indeed have attained extensive knowledge in the triple areas of (i) what the business does, (ii) how to use statistics, and (iii) how to manage data and data flows. If so, he or she could indeed claim to be a business scientist (a.k.a., “data scientist”) in a given sector. But such individuals are almost as rare as hen’s teeth.'' -- Robin Bloor, A Data Science Rant, Aug 2013, Inside Analysis

And finally:

The Map is Not the Territory.'' -- Alfred Korzybski, 1933, Science & Sanity.

Most real, applied problems are not accessible solely from the map''. To do practical things with mathematical modelling one must be willing to get grubby with details, subtleties, and exceptions. Nothing can substitute for knowing the territory first-hand.

• +1. Out of the box solutions rarely work for a specific business problem of your organisation. You need to tailor and improve, and for that you need to understand what is under the hood. – Zhubarb Jun 24 '14 at 8:08
• @Zhubarb - I think this only only partially true. "Out of the box" solutions can be used to great effect, just not all the time, and also usually not for completing a task from start to finish. The trick is knowing when you can get away with using "out of the box solutions", and when a more tailored approach is needed. – probabilityislogic Aug 30 '14 at 13:19

I agree with everything that's been said. What stands out for me are:

1. How few machine learning "experts" are really interested in the subject matter to which they want to apply ML
2. How few truly understand predictive accuracy and proper scoring rules
3. How few understand principles of validation
4. How few know when to use a black box vs. a traditional regression model
5. How none of the "experts" seem to have ever studied Bayes optimum decision or loss/utility/cost functions [this lack of understanding is displayed almost any time someone uses classification instead of predicted risk]
• Would you mind elaborating on 4? I don't quite understand what you mean – user44764 Jun 24 '14 at 14:51
• I would have thought that that item was the easiest to understand. Here's an example: in a particular field of study suppose that we had previous experience that most variables operate additively. Fitting an additive regression model that does not assume predictors to operate linearly (e.g., using regression splines) will provide an interpretable and useful statistical model. Using svm or random forests on the other hand will be very hard to interpret, not have any separable effects, and will not predict better than the nonlinear additive model. – Frank Harrell Jun 24 '14 at 19:32
• Yea verily to the shallowness of subject matter. Not even in the ML world, I see often this tendency to want to shovel raw data into a magic box and have wonderful insight pop out. What these people are looking for is an artificial brain. – DarenW Jun 27 '14 at 2:11
• +1 for item 1 especially. The evidence of lack of interest in the domain is that applying domain knowledge will require knowing how to 'open' the black box and modify. By black box, I mean for much of entry level ML even basic statistical modeling techniques are in that black box. If the interest/ability is not there then it is much harder to apply domain knowledge. – Meadowlark Bradsher Jun 27 '14 at 17:10
• @DarenW: The tendency is also reflected in the name: "Machine Learning" with the connotation that the machine learns... by itself... just shovel raw data. Contrast the less glamourous name (but more accurate IMO) chosen by Hastie, Tibshirani, et.al.: "Statistical Learning". Different connotation, different explanations, all referring back to statistical principles. – Assad Ebrahim Jun 28 '14 at 8:53

Here are a couple of things to make you stand out from the crowd:

• Understand the application domain or domains. That is, the business environment or other context.
• Understand the big picture. This is very important! People who study machine learning often get lost in the details. Think about the overall picture that your ML models will fit into. Often the ML part is just a small segment of a much larger system. Understand the whole system.
• Study utility and decision theory and Bayesian inference, not just whatever is now considered "the usual" ML models. Bayesian inference is just a way to formalize the notion of bringing all contextual information to bear on a problem. Utility and decision theory is about bringing values into the picture.

The overall message that applies to all three points: Look at the big picture, don't get lost in the details.

The skill that sets one data miner apart from others is the ability to interpret machine learning models. Most build a machine, report the error and then stop. What are the mathematical relationships between the features? Are the effects additive or non-additive or both? Are any of the features irrelevant? Is the machine expected under the null hypothesis that there are only chance patterns in the data? Does the model generalize to independent data? What do these patterns mean for the problem being studied? What are the inference? What are the insights? Why should a domain expert get excited? Will the machine lead to the domain expert asking new questions and designing new experiments? Can the data miner effectively communicate the model and its implications to the world?

• +1 Agreed - although what you describe is called statistics. – Thomas Speidel Jun 25 '14 at 13:37

I would put out there the notion of "soft skills".

• recognising who the "expert" is for method X, and being able to tap into their knowledge (you shouldn't be able to or expected to know everything about erything). The ability and willingness to collaborate with others.

• the ability to translate or represent "the real world" with the mathematics used in ML.

• the ability to explain your methods in different ways to different audiences - knowing when to focus on details and when to step back and view the wider context.

• systems thinking, being able to see how your role feeds into other areas of the business, and how these areas feed back into your work.

• an appreciation and understanding of uncertainty, and having some structured methods to deal with it. Being able to state clearly what your assumptions are.

Being able to generalize well

This is the essence of a good model. And it is the essence of what makes the best practitioners of the art of machine learning stand out from the crowd.

Understanding that the goal is to optimize performance on unseen data, not to minimize training loss. Knowing how to avoid both over-fitting and under-fitting. Coming up with models that are not too complex yet not too simple in describing the problem. Extracting the gist of a training-set, rather than the maximum possible.

It is surprising how often, even experienced machine learning practitioners, fail to follow this principle. One reason is that humans fail to appreciate two vast theory-vs-practice magnitude differences:

• How much larger is the space of all possible examples compared to the training-data at hand, even when the training data is very large.
• How much larger is the full "hypothesis space": number of possible models for a problem, compared to the practical "solution space": everything you can think of, and everything your software/tools are capable of representing.

The 2nd is especially incomprehensible because even for the simplest problem with $N$ inputs and a binary outcome, there are $2^N$ possible input-examples, and an exponentially larger number 2^$2^N$ of possible models.

It is also what most of the above answers said in more specific and concrete ways. to generalize well is just the shortest way I could think of, to put it.

I see there are two parts while handling machine learning in practice

1. Engineering ( which covers all the algorithms, learning different packages, programming).

2. Curiosity/Reasoning (ability to ask better questions to data).

I think 'curiosity/reasoning' is the skill which distinguishes one from others. For example, if you see the leader boards of the kaggle completions, many people may have used common(similar) algorithms, what makes the difference is, how one logically question the data and formulate it.

Having done scientific research in Machine learning / Statistical pattern recognition for 17 years - I can come up with a few skills that make a wanted-for data scientist stand out from others.

• Achieving the algorithmic knowledge of learning algorithms out there, and getting the skill of how to apply these learning algorithms successfully to practical ML-problems,
• Gaining the required level of knowledge in probability theory (beginning with Bayes rule), parametric statistics and nonparametric statistics as to assess and compare different types of learnable models, model performance, confidence intervals, sampling theory, and ML-estimation. Don't underestimate the level of statistical knowledge required to become a skilled professional (go though the proof of the central limit theorem, for example, and understand when this theorem is not applicable),
• Understand mathematical approximation theory to a level so you can see why feed-forward neural networks (with 2 hidden layers, or more) are universal approximators - offspring of the theorem of Komolgorov,
• Gain practical experience from learning classifiers on many different training sets, and validate their performance on independent test sets,
• Understand that optimal feature selection and model selection require knowledge of how algorithmics and statistics intertwine - take as an example the branch-and-bound algorithm for feature selection. Recognize that feature selection and model selection always involve a bias-variance trade-off (between performance variance and optimal model fit),
• Go through the derivations by Richard & Lippmann (1991) why neural network classiﬁers estimate bayesian a posteriori probabilities,
• Get acquainted with major scientific breakthroughs in the last eight decades with respect to the development of statistical and algorithmic prediction models, beginning with linear discriminant analysis (a statistical classifier),
• Embrace the fact that no one machine learning scheme is optimal for almost 'all kinds of problems'. So - neural networks are not better than all other models, neither are support vector machines or random forests for that sake - it all depends on the statistics of the underlying domain. Practical machine learning remains an experimental science, but many relevant theoretical results have been published in literature over the years.

It's hard work to be able to span algorithmics, statistics and mathematical approximation theory. I did my Ph.D. in machine learning and first really became a professional after more than 10 years of work.

A final note is that it is not always necessary to be a programmer to apply machine learning algorithms. ML-suites like Weka or available classifier services like insight classifiers let data scientists apply different ML-algorithms without having to program in for example Python or R.

It is a great discipline - Machine learning.