73
$\begingroup$

I've been very interested in data-mining and machine-learning for a while, partly because I majored in that area at school, but also because I am truly much more excited trying to solve problems that require a bit more thought than just programming knowledge and whose solution can have multiple forms. I don't have a researcher/scientist background, I come from a computer science background with an emphasis on data analysis, I have a Master's degree and not a PhD. I currently have a position related to data analysis, even if that is not the primary focus of what I'm doing, but I have at least some good exposure to it.

As I was interviewing some time ago for a job with several companies, and got to talk with a few recruiters, I found a common pattern that people seem to think that you need to have a PhD to do machine learning, even if I may be generalizing a bit too much (some companies were not really looking especially for PhDs).

While I think it's good to have a PhD in that area, I don't think this is absolutely necessary. I have some pretty decent knowledge of most real-world machine learning algorithms, have implemented most of them myself (either at school or on personal projects), and feel pretty confident when approaching problems involving machine-learning / data-mining and statistics in general. And I have some friends with a similar profile who seem very knowledgeable about this also, but also feel that in general companies are pretty shy about hiring in data-mining if you're not a PhD.

I'd like to get some feedback, do you think a PhD is absolutely necessary to have a job very focused in that area?

(I hesitated a bit before posting this question here, but since it seems to be an acceptable topic on meta, I've decided to post this question on which I've been thinking for a while.)

$\endgroup$
8
  • 1
    $\begingroup$ There is a couple of quite similar recent questions on this site. You might have a look at them, too. Your question is well stated, though it might be a little better to make it a little less focused on your particular circumstances so as to make it less localized. $\endgroup$
    – cardinal
    May 1, 2012 at 23:45
  • 1
    $\begingroup$ Also, what do you mean you majored in data mining and machine learning in school? I can think of only about one or two universities in North America, for example, where such a degree might exist. I can imagine many M.S. programs where you could have this as a focal point of a degree, but still probably not called as such. $\endgroup$
    – cardinal
    May 1, 2012 at 23:51
  • $\begingroup$ Thanks for the feedback @cardinal (this is my first post here), the only questions I found seemed a bit different, as I'd really like to clarify this no Phd=no machine learning I've seemed to discover in my recent experiences. $\endgroup$ May 1, 2012 at 23:53
  • 1
    $\begingroup$ I have a Master's degree in CS, but my major was in data mining. I'm not originally from the US by the way :) $\endgroup$ May 1, 2012 at 23:54
  • 3
    $\begingroup$ Just thought I'd throw my 2 cents in without putting in an answer. I do data-mining and machine learning and am mostly self taught in the field (besides my bachelors in CS related topics). Academic ML topics are very different from business applications but its good to have a foot in each. $\endgroup$
    – Dan
    May 3, 2012 at 17:25

11 Answers 11

57
$\begingroup$

I believe actually the opposite of your conclusion is true. In The Disposable Academic, several pointers are given about the low wage premium in applied math, math, and computer science for PhD holders over master's degree holders. In part, this is because companies are realizing that master's degree holders usually have just as much theoretical depth, better programming skills, and are more pliable and can be trained for their company's specific tasks. It's not easy to get an SVM disciple, for instance, to appreciate your company's infrastructure that relies on decision trees, say. Often, when someone has dedicated tons of time to a particular machine learning paradigm, they have a hard time generalizing their productivity to other domains.

Another problem is that a lot of machine learning jobs these days are all about getting things done, and not so much about writing papers or developing new methods. You can take a high risk approach to developing new mathematical tools, studying VC-dimensional aspects of your method, its underlying complexity theory, etc. But in the end, you might not get something that practitioners will care about.

Meanwhile, look at something like poselets. Basically no new math arises from poselets at all. It's entirely unelegant, clunky, and lacks any mathematical sophistication. But it scales up to large data sets amazingly well and it's looking like it will be a staple in pose recognition (especially in computer vision) for some time to come. Those researchers did a great job and their work is to be applauded, but it's not something most people associate with a machine learning PhD.

With a question like this, you'll get tons of different opinions, so by all means consider them all. I am currently a PhD student in computer vision, but I've decided to leave my program early with a master's degree, and I'll be working for an asset management company doing natural language machine learning, computational statistics, etc. I also considered ad-based data mining jobs at several large TV companies, and a few robotics jobs. In all of these domains, there are plenty of jobs for someone with mathematical maturity and a knack for solving problems in multiple programming languages. Having a master's degree is just fine. And, according to that Economist article, you'll be paid basically just as well as someone with a PhD. And if you work outside of academia, bonuses and getting to promotions faster than someone who spends extra years on a PhD can often mean your overall lifetime earnings are higher.

As Peter Thiel once said, "Graduate school is like hitting the snooze button on the alarm clock of life..."

$\endgroup$
9
  • 6
    $\begingroup$ When I first read that Economist article when it was published, it sounded from the start like a poorly researched, bitter rant. I wasn't surprised once I got near the end and learned the author was the unhappy recipient of a PhD. My opinion of it hasn't changed much as I've reread it several times since then. Usually the Economist does better, even on academic issues. $\endgroup$
    – cardinal
    May 2, 2012 at 0:18
  • 9
    $\begingroup$ I think it's a great article. It highlights a lot of the incentive problems with graduate school these days. Another interesting article is The Rise of the Post-Doc as Principal Investigator. I'm not sure why you think the article is poorly researched, can you provide some more specific criticisms? I agree that it's important to avoid confirmation bias. But as a reader, I don't much care if the writer is in confirmation mode as long as the sources are good. As far as I'm concerned, there should be more bitter rants about academia. $\endgroup$
    – ely
    May 2, 2012 at 0:47
  • 8
    $\begingroup$ I have several major criticisms of the Economist article, beyond the fact that when reading a supposedly objective analytical piece, the emotion oozing out of it shouldn't be the first thing I notice. Sadly, these don't fit in 600 characters, though they might make an ok blog post. I think the premise of the piece is wrong from the start; a person that seeks to obtain a PhD for economic benefit has already misunderstood the intended purpose. Even so, the wage-premia analysis has a fatal flaw in that it does not account for the fact that a large percentage of PhD holders still go... $\endgroup$
    – cardinal
    May 2, 2012 at 16:35
  • 3
    $\begingroup$ ...into academia. And, this effect alone, will drag down any wage-premia analyses, particularly in a field like math. $\endgroup$
    – cardinal
    May 2, 2012 at 16:36
  • 2
    $\begingroup$ I am (very) familiar with the current academic and industrial job landscapes for math and applied sciences, thanks. I don't intend to be polemic or get into an argument. I can see your passion when responding; this topic appears to touch a nerve. Cheers. $\endgroup$
    – cardinal
    May 2, 2012 at 16:43
47
$\begingroup$

Disclaimer: I have a Ph.D. and work in machine learning. Having said that, I think other than becoming an academic, you don't need a Ph.D. to work in any field. Getting a Ph.D. helps you develop certain research skills, but

  1. You don't need those research skills for most jobs.
  2. You can acquire those skills without getting a Ph.D. degree.

Martin Wolf, the chief economic correspondent for the Financial Times, doesn't have a Ph.D. (he has a Master's degree), but his word carries a lot more weight than most Ph.D. graduates. I think to succeed in any field (including machine learning), you have to know how to learn and think thorough stuff on your own. A Ph.D. will help you practice those skills, but it's not an end to itself. Anyone who isn't willing to interview you just because you don't have a Ph.D., is probably not worth working for anyway.

$\endgroup$
4
  • $\begingroup$ Very interesting to have the advice from someone who has a PhD, I agree that a PhD certainly gives rigor and can be "a plus" for most research-oriented positions, I've noticed that startups are pretty narrow-minded when it comes to this, while big companies seem more open to hire Master's degree for machine learning (again, it depends which companies...). I've found a good compromise currently, and I expect to direct my career more into that path, first by getting some real-world experience at my current position. Thanks for your great advice. $\endgroup$ May 2, 2012 at 5:47
  • 2
    $\begingroup$ @linker cannot agree on the startup thing. I have a master in CS (wrote master thesis in ML though) and my first position was a startup. Startups should not care about titles since they are regularly looking for broad thinkers who are able to get the job done (and play nicely with the other in the company) meanwhile I suspect bigger companies to follow policies carved into stone to avoid too much responsibility for single HR drones (no bitterness, just humor here ;)) $\endgroup$
    – steffen
    May 2, 2012 at 6:29
  • 1
    $\begingroup$ I like it when PhDs do not recommend PhDs, or when Ivy grads recommend not going to Ivy's. It's like Peter Thiel suggesting to not go to a college. The guy got JD from Stanford :) $\endgroup$
    – Aksakal
    Mar 22, 2018 at 1:26
  • 1
    $\begingroup$ It really depends on what you intend to do with the Phd. If your goal is academia, it’s a must, but in industry, it is really not a big plus, unless your goal is to do research. Just look at successful tech companies and their executive teams (including CTO and SVP of Eng if there is one). How many STEM Phds do you find there? They exist, but rarely, and often they’re one of the founders. Having a Phd might open some doors, but for actually accomplishing something, real world experience and deep knowledge of foundations (including the math) are more crucial $\endgroup$
    – user765195
    Aug 19, 2018 at 12:40
19
$\begingroup$

Disclaimer: I do not have a PhD in CS, nor do I work in machine learning; I am generalizing from other knowledge and experience.

I think there are several good answers here, but, in my honest opinion, they do not yet quite make the main issue explicit. I will attempt to do so, but recognize that I don't think I'm saying something radically different. The main issue here pertains to skill development vs. signaling.

With respect to skill development, ultimately you want (as an employee) to be able to get the job done, done well and done quickly, and the employer wants (or presumably ought to) such a person. Thus, the question here is how much extra skill development do you get with the extra couple of years of academic training? Certainly you should be gaining something, but recognize that people who don't continue with graduate school probably don't just sit on their duff until they would have graduated. Thus, you are comparing one set of experiences (academic) vs. another (work). A good bit depends on the quality and nature of the Ph.D. program, your intrinsic interests, how self-directed you are, and what kind of opportunities and support would be available in your first job.

Outside of the effect continued academic training has on skill development, there is the question of the effect and value of the signal (i.e., of having "Ph.D" appended after your name). The signal can help in two ways: First, it can help you land your initial job, and that shouldn't be dismissed--it can be very important. Research has shown that people who are obliged to start out in a first position that isn't as appropriate for them never tend to do as well (career-wise, on average) as people who get to start in a job that is a good match for their abilities and interests. On the other hand, the consensus seems to be that after your first job, your future prospects are much more strongly influenced by your performance in your previous job than your academic credentials.

The second aspect of the signal has to do with the relationship between the analyst and the consumer of the analysis. @EMS does a good job of bringing this point out in a comment. There are a lot of small consulting shops, and they love to have Ph.D.'s to show off to potential clients: in initial meetings trying to land a contract, on letterhead, in presentations of finished work-products, etc. the Ph.D.'s are always there. It's easy to be cynical about this, but I do think there's legitimate value for the consulting firm and the consumer (who may not know much about these matters and can use credentials to help them select a firm that will do a good job for them). Behind the scenes, some of the work may be farmed out to competent people with less credentials, but they want the Ph.D. for the front end, and to sign off on the work-product before it's delivered. I could see something analogous happening with a start-up if they're trying to attract capital and want to reassure investors. On the other hand, if you are going to do work for internal consumption, and your boss is capable of evaluating it, this doesn't matter.

$\endgroup$
2
  • 5
    $\begingroup$ (+1) I think this answer is starting to get closer to the crux of the matter. There are actually two questions from the OP, posed as one (at least, as I see it). The first is (1) Can a person without a PhD do meaningful work in industry in data-mining and/or machine learning? The answer is undoubtedly affirmative. The second is (2) Can one obtain such a position with limited experience in the area and without a PhD (and how hard is it to accomplish this)? Even though the second is probably easier to measure quantitatively, it seems to have more grey area. $\endgroup$
    – cardinal
    May 2, 2012 at 16:17
  • 3
    $\begingroup$ Also, Robin Hanson wrote up some good stuff on this signaling and credential aspect recently, at least w.r.t. consulting. $\endgroup$
    – ely
    May 2, 2012 at 17:06
14
$\begingroup$

I agree with most that has been said here, but I want to introduce a few practical issues that arise when applying for jobs in finance. Often you will see ads stating that a PhD in statistics or mathematics is required to apply for a particular trading or quantitative developer position. I know there are some particular reasons for this. Mind, I'm not saying this is right, but it is what happens in practice:

  • There are many applicants to the job, especially for the most well known companies, and the employer can't possibly dedicate enough time to each candidate. Filtering applications based on the academic background shrinks the population size to a more manageable number. Yes, there will be misses. Yes, it's not the best way to find productive individuals. But on average you are looking at skilled professionals who have dedicated years to learn the craft. They should at least have the discipline to overtake a complex research project.

  • The team and the company will be enriched by a number of PhDs to showcase to investors and clients. This will give an image of "oracleous" knowledge to the company and benefit its reputation. The company intangible valuation can rise. The average investor will be more confident in granting their capital to such a knowledgeable team of scientists. You can make a similar point about MBAs.

  • Finally sometimes corporate policies dictate that higher academic achievements should have a preferential career path and compensation. I believe this is true for most corporations in different industries, not just finance. It is hard to see John with a BS in computer sciences managing PhDs in mathematics.

$\endgroup$
14
$\begingroup$

Disclaimer: I'm a recruiter and have been since 1982 so I understand your question very well. Let me break it down this way. Your resume is a screening out device. Companies get tons of resumes so they're reading resumes with one question in mind, "Why don't I want to talk to this person?" That reduces their pile to a few candidates who hold the best chances of meeting their needs. So if you're getting interviews and your resume doesn't show a PhD then there's something else going on here. I say that because, just as a resume is a screening OUT device, the interview is a screening IN device. Once they've invited you to an interview then they've already concluded you have enough "on paper" to do the job. So when you're walking in the interview the only quesion they're really asking is "why should I hire you?" The person they hire will be the individual who addresses out they can best serve the company's needs.

My advice as a recruiter is to ask questions throughout the interview to identify their deeper needs. Believe me, the job description rarely resembles the truth so you'll want to probe for their hot buttons then sell directly to those issues. Don't allow the interview to feel like an interrogation, waiting for the end to ask questions. You'll go down in flames and end up being told "you don't have a PhD". Be respectful yet show your willingness to help them solve their problem.

My favorite question is: "What are the traits of the best person you've ever known in this role?" Everyone has a dream team in mind so it's important to figure out what traits they feel are necessary to succeed in this role. Keep in mind, this isn't a question about experience, backgrounds or degrees. See, I can always find a mediocre PhD with tons of experience so this isn't the holy grail. It's just what companies continue to think is best because IMO they don't know how else to write a job description that captures the essence of the person they need.

$\endgroup$
1
  • 4
    $\begingroup$ Welcome to the site, @GailPalubiak. Please don't sign your posts with your personal info. Note that your avatar & a link to your user page are automatically added to all your posts. You can post that info there. Since you're new here, you may want to read our FAQ, which discusses topics like these. $\endgroup$ Dec 27, 2012 at 20:11
14
$\begingroup$

My 2 cents: No, I don't think so. A PhD per se does not entitle one to be be better for data mining or ML. Take kaggle's own Jeremy Howard. I would even go as far as saying that a PhD says not much about any qualification as there is a huge variability in quality of programs. Perhaps the only thing a PhD proves is for the holder to have a high tolerance of frustration.

Bottom line: If you are interested in that area, knowledgeable, creative and hard-working, why would you need a PhD? It is you that should count, not your titles.

$\endgroup$
2
  • 2
    $\begingroup$ I totally agree, but I under a strong impression that companies/recruiters seem to think the other way around. Even at my previous job, when I was discussing doing some data mining on some problems, I was told that I am an engineer and not a data scientist and thus it's best if i stay in my focus area. $\endgroup$ May 1, 2012 at 23:51
  • 3
    $\begingroup$ Well said. See my answer for some supporting details. The PhD in this field is not too relevant, and if a company thinks it is, you probably don't want to work for that company. Peter Thiel once said, "Graduate school is like hitting the snooze button on the alarm clock of life..." $\endgroup$
    – ely
    May 1, 2012 at 23:54
6
$\begingroup$

Whether a job requires a PhD or not depends on level of responsibility and the perception of the employer and/or his clients. I do not think there is a discipline that requires a PhD. Certainly data mining can be learned and an employee can do productive work without a PhD. This depends more on the person, his or her ability to learn quickly and adapt as well as being able to understand the literature, than on previous education. This is especially true for data mining which is an evolving field. So even the data miners with PhDs will have more to learn as time goes on.

$\endgroup$
1
  • 4
    $\begingroup$ (+1) One discipline that requires a PhD is university professor. (Of course there are exceptions, but they are few.) $\endgroup$
    – whuber
    May 3, 2012 at 21:07
4
$\begingroup$

I have a masters degree in Applied Statistics and worked in Europe as a Data Miner. When I came to the UK nobody had even heard of data mining let alone studied for such a degree. Now it is common place and employers feel that a Phd is necessary for this job. However, it is the statistical knowledge and the modelling aspect which is important for this job. In my experience, most IT people do not understand statistics and are therefore unable to do the job well. I went into teaching and now am registering to do a Phd in Applied Statistics to satisfy these employers. I probably know more than most Phd graduates having studied for my Masters degree in 1980s when the level was very high. I think to be a good data miner, one has to have a background in Statistics.

$\endgroup$
4
$\begingroup$

This totally depends on the job at hand. In my experience (I have a PhD), there are 3 types of jobs. First, as it has been said, most industry jobs these days are oriented towards applied machine learning, i.e. apply-tweak of existing ML algorithms to the domain-specific problem in question. These are by far the most common ML jobs and a Masters degree is more than sufficient for these kind of jobs. A smaller number of jobs, which happen to be in the research wing of companies or universities, institutions are apply-tweak-create ML jobs for the domain specific problem. The experience of creating a new method by looking at existing methods using new mathematics typically takes some time and these experiences are typically gained during the PhD, as the new theoretical result should be sufficiently robust to gain the acceptance of ones peers (a publication). Last and probably the hardest, highest risk and most uncommon type of jobs are the pure theoretical stuff going on at research universities where the focus is to come up with a new algorithm totally, or understand the mathematical properties better of existing algorithms (also has to be good enough to be published). This too is experience typically gained as a PhD. While a PhD student might have had some exposure to all three types of jobs during his/her training (purely due to the length of time of the program and the fact that there are no immediate product deadlines like a real job), the MS student typically is well trained for the first job and would probably have had only minor exposures to the 2nd and 3rd types of jobs. Each one of these jobs are equally important.

$\endgroup$
2
  • $\begingroup$ When you say "job" do you mean "a regular employment" or "a task or piece of work"? You seem to be switching back and forth between these two meanings and this makes your answer a bit confusing. $\endgroup$
    – amoeba
    Dec 8, 2015 at 23:44
  • $\begingroup$ Is there a difference between the two? Genuinely asking.... $\endgroup$ Dec 10, 2015 at 16:15
2
$\begingroup$

I dont think that Phd is required for any machine learning positions. A good masters and an inquistive mind with mathematical curiosity is all what it needs. A Phd biases your approach towards your specialization which is undesirable. I work on core Machine learning algorithms, and codes most of them in the way i want. And i have seen a lot of Phd people with the wrong mindset. Phds are mostly motivated by pure theoretical problems, unlike in industry where the focus is on working solutions in quick time

$\endgroup$
2
$\begingroup$

People who look down PhD training either don't know what a PhD means at all, or just intentionally make untrue comments; most masters training cannot compare with PhD training by any means. the intensity and rigor in PhD training requires unimaginable dedication, self-discipline, learning ability under great pressure, and solid skill sets..., a PhD title already proved all of those, a regular masters degree here in America is not at the same level at all....

$\endgroup$
1
  • 6
    $\begingroup$ I don't think anyone here "look[s] down on PhD training". Most of us have PhDs. Many of us work in an academic setting where a PhD is mandatory. Some of the responses here (my own, eg) are just recognizing that there can be people w/o the credential who have the skills & that those people can get jobs too. I have a former BS student (no masters) who is working as a data scientist for a consulting firm. $\endgroup$ Mar 15, 2015 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.