Having a job in data-mining without a PhD

Question

I've been very interested in data-mining and machine-learning for a while, partly because I majored in that area at school, but also because I am truly much more excited trying to solve problems that require a bit more thought than just programming knowledge and whose solution can have multiple forms. I don't have a researcher/scientist background, I come from a computer science background with an emphasis on data analysis, I have a Master's degree and not a PhD. I currently have a position related to data analysis, even if that is not the primary focus of what I'm doing, but I have at least some good exposure to it.

As I was interviewing some time ago for a job with several companies, and got to talk with a few recruiters, I found a common pattern that people seem to think that you need to have a PhD to do machine learning, even if I may be generalizing a bit too much (some companies were not really looking especially for PhDs).

While I think it's good to have a PhD in that area, I don't think this is absolutely necessary. I have some pretty decent knowledge of most real-world machine learning algorithms, have implemented most of them myself (either at school or on personal projects), and feel pretty confident when approaching problems involving machine-learning / data-mining and statistics in general. And I have some friends with a similar profile who seem very knowledgeable about this also, but also feel that in general companies are pretty shy about hiring in data-mining if you're not a PhD.

I'd like to get some feedback, do you think a PhD is absolutely necessary to have a job very focused in that area?

(I hesitated a bit before posting this question here, but since it seems to be an acceptable topic on meta, I've decided to post this question on which I've been thinking for a while.)

Answer 1

I believe actually the opposite of your conclusion is true. In The Disposable Academic, several pointers are given about the low wage premium in applied math, math, and computer science for PhD holders over master's degree holders. In part, this is because companies are realizing that master's degree holders usually have just as much theoretical depth, better programming skills, and are more pliable and can be trained for their company's specific tasks. It's not easy to get an SVM disciple, for instance, to appreciate your company's infrastructure that relies on decision trees, say. Often, when someone has dedicated tons of time to a particular machine learning paradigm, they have a hard time generalizing their productivity to other domains.

Another problem is that a lot of machine learning jobs these days are all about getting things done, and not so much about writing papers or developing new methods. You can take a high risk approach to developing new mathematical tools, studying VC-dimensional aspects of your method, its underlying complexity theory, etc. But in the end, you might not get something that practitioners will care about.

Meanwhile, look at something like poselets. Basically no new math arises from poselets at all. It's entirely unelegant, clunky, and lacks any mathematical sophistication. But it scales up to large data sets amazingly well and it's looking like it will be a staple in pose recognition (especially in computer vision) for some time to come. Those researchers did a great job and their work is to be applauded, but it's not something most people associate with a machine learning PhD.

With a question like this, you'll get tons of different opinions, so by all means consider them all. I am currently a PhD student in computer vision, but I've decided to leave my program early with a master's degree, and I'll be working for an asset management company doing natural language machine learning, computational statistics, etc. I also considered ad-based data mining jobs at several large TV companies, and a few robotics jobs. In all of these domains, there are plenty of jobs for someone with mathematical maturity and a knack for solving problems in multiple programming languages. Having a master's degree is just fine. And, according to that Economist article, you'll be paid basically just as well as someone with a PhD. And if you work outside of academia, bonuses and getting to promotions faster than someone who spends extra years on a PhD can often mean your overall lifetime earnings are higher.

As Peter Thiel once said, "Graduate school is like hitting the snooze button on the alarm clock of life..."

Answer 2

Disclaimer: I have a Ph.D. and work in machine learning. Having said that, I think other than becoming an academic, you don't need a Ph.D. to work in any field. Getting a Ph.D. helps you develop certain research skills, but

1. You don't need those research skills for most jobs.
2. You can acquire those skills without getting a Ph.D. degree.

Martin Wolf, the chief economic correspondent for the Financial Times, doesn't have a Ph.D. (he has a Master's degree), but his word carries a lot more weight than most Ph.D. graduates. I think to succeed in any field (including machine learning), you have to know how to learn and think thorough stuff on your own. A Ph.D. will help you practice those skills, but it's not an end to itself. Anyone who isn't willing to interview you just because you don't have a Ph.D., is probably not worth working for anyway.

Answer 3

Disclaimer: I do not have a PhD in CS, nor do I work in machine learning; I am generalizing from other knowledge and experience.

I think there are several good answers here, but, in my honest opinion, they do not yet quite make the main issue explicit. I will attempt to do so, but recognize that I don't think I'm saying something radically different. The main issue here pertains to skill development vs. signaling.

With respect to skill development, ultimately you want (as an employee) to be able to get the job done, done well and done quickly, and the employer wants (or presumably ought to) such a person. Thus, the question here is how much extra skill development do you get with the extra couple of years of academic training? Certainly you should be gaining something, but recognize that people who don't continue with graduate school probably don't just sit on their duff until they would have graduated. Thus, you are comparing one set of experiences (academic) vs. another (work). A good bit depends on the quality and nature of the Ph.D. program, your intrinsic interests, how self-directed you are, and what kind of opportunities and support would be available in your first job.

Outside of the effect continued academic training has on skill development, there is the question of the effect and value of the signal (i.e., of having "Ph.D" appended after your name). The signal can help in two ways: First, it can help you land your initial job, and that shouldn't be dismissed--it can be very important. Research has shown that people who are obliged to start out in a first position that isn't as appropriate for them never tend to do as well (career-wise, on average) as people who get to start in a job that is a good match for their abilities and interests. On the other hand, the consensus seems to be that after your first job, your future prospects are much more strongly influenced by your performance in your previous job than your academic credentials.

The second aspect of the signal has to do with the relationship between the analyst and the consumer of the analysis. @EMS does a good job of bringing this point out in a comment. There are a lot of small consulting shops, and they love to have Ph.D.'s to show off to potential clients: in initial meetings trying to land a contract, on letterhead, in presentations of finished work-products, etc. the Ph.D.'s are always there. It's easy to be cynical about this, but I do think there's legitimate value for the consulting firm and the consumer (who may not know much about these matters and can use credentials to help them select a firm that will do a good job for them). Behind the scenes, some of the work may be farmed out to competent people with less credentials, but they want the Ph.D. for the front end, and to sign off on the work-product before it's delivered. I could see something analogous happening with a start-up if they're trying to attract capital and want to reassure investors. On the other hand, if you are going to do work for internal consumption, and your boss is capable of evaluating it, this doesn't matter.

Answer 4

I agree with most that has been said here, but I want to introduce a few practical issues that arise when applying for jobs in finance. Often you will see ads stating that a PhD in statistics or mathematics is required to apply for a particular trading or quantitative developer position. I know there are some particular reasons for this. Mind, I'm not saying this is right, but it is what happens in practice:

• There are many applicants to the job, especially for the most well known companies, and the employer can't possibly dedicate enough time to each candidate. Filtering applications based on the academic background shrinks the population size to a more manageable number. Yes, there will be misses. Yes, it's not the best way to find productive individuals. But on average you are looking at skilled professionals who have dedicated years to learn the craft. They should at least have the discipline to overtake a complex research project.

• The team and the company will be enriched by a number of PhDs to showcase to investors and clients. This will give an image of "oracleous" knowledge to the company and benefit its reputation. The company intangible valuation can rise. The average investor will be more confident in granting their capital to such a knowledgeable team of scientists. You can make a similar point about MBAs.

• Finally sometimes corporate policies dictate that higher academic achievements should have a preferential career path and compensation. I believe this is true for most corporations in different industries, not just finance. It is hard to see John with a BS in computer sciences managing PhDs in mathematics.

Answer 5

Disclaimer: I'm a recruiter and have been since 1982 so I understand your question very well. Let me break it down this way. Your resume is a screening out device. Companies get tons of resumes so they're reading resumes with one question in mind, "Why don't I want to talk to this person?" That reduces their pile to a few candidates who hold the best chances of meeting their needs. So if you're getting interviews and your resume doesn't show a PhD then there's something else going on here. I say that because, just as a resume is a screening OUT device, the interview is a screening IN device. Once they've invited you to an interview then they've already concluded you have enough "on paper" to do the job. So when you're walking in the interview the only quesion they're really asking is "why should I hire you?" The person they hire will be the individual who addresses out they can best serve the company's needs.

My advice as a recruiter is to ask questions throughout the interview to identify their deeper needs. Believe me, the job description rarely resembles the truth so you'll want to probe for their hot buttons then sell directly to those issues. Don't allow the interview to feel like an interrogation, waiting for the end to ask questions. You'll go down in flames and end up being told "you don't have a PhD". Be respectful yet show your willingness to help them solve their problem.

My favorite question is: "What are the traits of the best person you've ever known in this role?" Everyone has a dream team in mind so it's important to figure out what traits they feel are necessary to succeed in this role. Keep in mind, this isn't a question about experience, backgrounds or degrees. See, I can always find a mediocre PhD with tons of experience so this isn't the holy grail. It's just what companies continue to think is best because IMO they don't know how else to write a job description that captures the essence of the person they need.

Answer 6

My 2 cents: No, I don't think so. A PhD per se does not entitle one to be be better for data mining or ML. Take kaggle's own Jeremy Howard. I would even go as far as saying that a PhD says not much about any qualification as there is a huge variability in quality of programs. Perhaps the only thing a PhD proves is for the holder to have a high tolerance of frustration.

Bottom line: If you are interested in that area, knowledgeable, creative and hard-working, why would you need a PhD? It is you that should count, not your titles.

Answer 7

Whether a job requires a PhD or not depends on level of responsibility and the perception of the employer and/or his clients. I do not think there is a discipline that requires a PhD. Certainly data mining can be learned and an employee can do productive work without a PhD. This depends more on the person, his or her ability to learn quickly and adapt as well as being able to understand the literature, than on previous education. This is especially true for data mining which is an evolving field. So even the data miners with PhDs will have more to learn as time goes on.

Answer 8

I have a masters degree in Applied Statistics and worked in Europe as a Data Miner. When I came to the UK nobody had even heard of data mining let alone studied for such a degree. Now it is common place and employers feel that a Phd is necessary for this job. However, it is the statistical knowledge and the modelling aspect which is important for this job. In my experience, most IT people do not understand statistics and are therefore unable to do the job well. I went into teaching and now am registering to do a Phd in Applied Statistics to satisfy these employers. I probably know more than most Phd graduates having studied for my Masters degree in 1980s when the level was very high. I think to be a good data miner, one has to have a background in Statistics.

Answer 9

This totally depends on the job at hand. In my experience (I have a PhD), there are 3 types of jobs. First, as it has been said, most industry jobs these days are oriented towards applied machine learning, i.e. apply-tweak of existing ML algorithms to the domain-specific problem in question. These are by far the most common ML jobs and a Masters degree is more than sufficient for these kind of jobs. A smaller number of jobs, which happen to be in the research wing of companies or universities, institutions are apply-tweak-create ML jobs for the domain specific problem. The experience of creating a new method by looking at existing methods using new mathematics typically takes some time and these experiences are typically gained during the PhD, as the new theoretical result should be sufficiently robust to gain the acceptance of ones peers (a publication). Last and probably the hardest, highest risk and most uncommon type of jobs are the pure theoretical stuff going on at research universities where the focus is to come up with a new algorithm totally, or understand the mathematical properties better of existing algorithms (also has to be good enough to be published). This too is experience typically gained as a PhD. While a PhD student might have had some exposure to all three types of jobs during his/her training (purely due to the length of time of the program and the fact that there are no immediate product deadlines like a real job), the MS student typically is well trained for the first job and would probably have had only minor exposures to the 2nd and 3rd types of jobs. Each one of these jobs are equally important.

Answer 10

I dont think that Phd is required for any machine learning positions. A good masters and an inquistive mind with mathematical curiosity is all what it needs. A Phd biases your approach towards your specialization which is undesirable. I work on core Machine learning algorithms, and codes most of them in the way i want. And i have seen a lot of Phd people with the wrong mindset. Phds are mostly motivated by pure theoretical problems, unlike in industry where the focus is on working solutions in quick time

Answer 11

People who look down PhD training either don't know what a PhD means at all, or just intentionally make untrue comments; most masters training cannot compare with PhD training by any means. the intensity and rigor in PhD training requires unimaginable dedication, self-discipline, learning ability under great pressure, and solid skill sets..., a PhD title already proved all of those, a regular masters degree here in America is not at the same level at all....