Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I've been very interested in data-mining and machine-learning for a while, partly because I majored in that area at school, but also because I am truly much more excited trying to solve problems that require a bit more thought than just programming knowledge and whose solution can have multiple forms. I don't have a researcher/scientist background, I come from a computer science background with an emphasis on data analysis, I have a Master's degree and not a PhD. I currently have a position related to data analysis, even if that is not the primary focus of what I'm doing, but I have at least some good exposure to it.

As I was interviewing some time ago for a job with several companies, and got to talk with a few recruiters, I found a common pattern that people seem to think that you need to have a PhD to do machine learning, even if I may be generalizing a bit too much (some companies were not really looking especially for PhDs).

While I think it's good to have a PhD in that area, I don't think this is absolutely necessary. I have some pretty decent knowledge of most real-world machine learning algorithms, have implemented most of them myself (either at school or on personal projects), and feel pretty confident when approaching problems involving machine-learning / data-mining and statistics in general. And I have some friends with a similar profile who seem very knowledgeable about this also, but also feel that in general companies are pretty shy about hiring in data-mining if you're not a PhD.

I'd like to get some feedback, do you think a PhD is absolutely necessary to have a job very focused in that area?

(I hesitated a bit before posting this question here, but since it seems to be an acceptable topic on meta, I've decided to post this question on which I've been thinking for a while.)

share|improve this question
3  
Community wiki? –  Momo May 1 '12 at 23:41
    
There is a couple of quite similar recent questions on this site. You might have a look at them, too. Your question is well stated, though it might be a little better to make it a little less focused on your particular circumstances so as to make it less localized. –  cardinal May 1 '12 at 23:45
    
Also, what do you mean you majored in data mining and machine learning in school? I can think of only about one or two universities in North America, for example, where such a degree might exist. I can imagine many M.S. programs where you could have this as a focal point of a degree, but still probably not called as such. –  cardinal May 1 '12 at 23:51
1  
I have a Master's degree in CS, but my major was in data mining. I'm not originally from the US by the way :) –  Charles Menguy May 1 '12 at 23:54
2  
Just thought I'd throw my 2 cents in without putting in an answer. I do data-mining and machine learning and am mostly self taught in the field (besides my bachelors in CS related topics). Academic ML topics are very different from business applications but its good to have a foot in each. –  Dan May 3 '12 at 17:25
show 3 more comments

8 Answers

up vote 34 down vote accepted

I believe actually the opposite of your conclusion is true. In The Disposable Academic, several pointers are given about the low wage premium in applied math, math, and computer science for PhD holders over master's degree holders. In part, this is because companies are realizing that master's degree holders usually have just as much theoretical depth, better programming skills, and are more pliable and can be trained for their company's specific tasks. It's not easy to get an SVM disciple, for instance, to appreciate your company's infrastructure that relies on decision trees, say. Often, when someone has dedicated tons of time to a particular machine learning paradigm, they have a hard time generalizing their productivity to other domains.

Another problem is that a lot of machine learning jobs these days are all about getting things done, and not so much about writing papers or developing new methods. You can take a high risk approach to developing new mathematical tools, studying VC-dimensional aspects of your method, its underlying complexity theory, etc. But in the end, you might not get something that practitioners will care about.

Meanwhile, look at something like poselets. Basically no new math arises from poselets at all. It's entirely unelegant, clunky, and lacks any mathematical sophistication. But it scales up to large data sets amazingly well and it's looking like it will be a staple in pose recognition (especially in computer vision) for some time to come. Those researchers did a great job and their work is to be applauded, but it's not something most people associate with a machine learning PhD.

With a question like this, you'll get tons of different opinions, so by all means consider them all. I am currently a PhD student in computer vision, but I've decided to leave my program early with a master's degree, and I'll be working for an asset management company doing natural language machine learning, computational statistics, etc. I also considered ad-based data mining jobs at several large TV companies, and a few robotics jobs. In all of these domains, there are plenty of jobs for someone with mathematical maturity and a knack for solving problems in multiple programming languages. Having a master's degree is just fine. And, according to that Economist article, you'll be paid basically just as well as someone with a PhD. And if you work outside of academia, bonuses and getting to promotions faster than someone who spends extra years on a PhD can often mean your overall lifetime earnings are higher.

As Peter Thiel once said, "Graduate school is like hitting the snooze button on the alarm clock of life..."

share|improve this answer
3  
When I first read that Economist article when it was published, it sounded from the start like a poorly researched, bitter rant. I wasn't surprised once I got near the end and learned the author was the unhappy recipient of a PhD. My opinion of it hasn't changed much as I've reread it several times since then. Usually the Economist does better, even on academic issues. –  cardinal May 2 '12 at 0:18
7  
I think it's a great article. It highlights a lot of the incentive problems with graduate school these days. Another interesting article is The Rise of the Post-Doc as Principal Investigator. I'm not sure why you think the article is poorly researched, can you provide some more specific criticisms? I agree that it's important to avoid confirmation bias. But as a reader, I don't much care if the writer is in confirmation mode as long as the sources are good. As far as I'm concerned, there should be more bitter rants about academia. –  EMS May 2 '12 at 0:47
6  
I have several major criticisms of the Economist article, beyond the fact that when reading a supposedly objective analytical piece, the emotion oozing out of it shouldn't be the first thing I notice. Sadly, these don't fit in 600 characters, though they might make an ok blog post. I think the premise of the piece is wrong from the start; a person that seeks to obtain a PhD for economic benefit has already misunderstood the intended purpose. Even so, the wage-premia analysis has a fatal flaw in that it does not account for the fact that a large percentage of PhD holders still go... –  cardinal May 2 '12 at 16:35
3  
...into academia. And, this effect alone, will drag down any wage-premia analyses, particularly in a field like math. –  cardinal May 2 '12 at 16:36
2  
Also, I absolutely dispute your claim that a large percentage still go into academia, at least in applied science. Virtually none of my peers have stayed in academia past a post-doc. They work for Microsoft, Google, nVidia, Facebook, financial firms, etc. etc. One of the biggest things that our graduate development symposia harp on is the diminishing ability to get tenure and unrealistic prospects for academic jobs. –  EMS May 2 '12 at 16:38
show 7 more comments

Disclaimer: I have a Ph.D. and work in machine learning. Having said that, I think other than becoming an academic, you don't need a Ph.D. to work in any field. Getting a Ph.D. helps you develop certain research skills, but

  1. You don't need those research skills for most jobs.
  2. You can acquire those skills without getting a Ph.D. degree.

Martin Wolf, the chief economic correspondent for the Financial Times, doesn't have a Ph.D. (he has a Master's degree), but his word carries a lot more weight than most Ph.D. graduates. I think to succeed in any field (including machine learning), you have to know how to learn and think thorough stuff on your own. A Ph.D. will help you practice those skills, but it's not an end to itself. Anyone who isn't willing to interview you just because you don't have a Ph.D., is probably not worth working for anyway.

share|improve this answer
    
Very interesting to have the advice from someone who has a PhD, I agree that a PhD certainly gives rigor and can be "a plus" for most research-oriented positions, I've noticed that startups are pretty narrow-minded when it comes to this, while big companies seem more open to hire Master's degree for machine learning (again, it depends which companies...). I've found a good compromise currently, and I expect to direct my career more into that path, first by getting some real-world experience at my current position. Thanks for your great advice. –  Charles Menguy May 2 '12 at 5:47
1  
@linker cannot agree on the startup thing. I have a master in CS (wrote master thesis in ML though) and my first position was a startup. Startups should not care about titles since they are regularly looking for broad thinkers who are able to get the job done (and play nicely with the other in the company) meanwhile I suspect bigger companies to follow policies carved into stone to avoid too much responsibility for single HR drones (no bitterness, just humor here ;)) –  steffen May 2 '12 at 6:29
add comment

Disclaimer: I do not have a PhD in CS, nor do I work in machine learning; I am generalizing from other knowledge and experience.

I think there are several good answers here, but, in my honest opinion, they do not yet quite make the main issue explicit. I will attempt to do so, but recognize that I don't think I'm saying something radically different. The main issue here pertains to skill development vs. signaling.

With respect to skill development, ultimately you want (as an employee) to be able to get the job done, done well and done quickly, and the employer wants (or presumably ought to) such a person. Thus, the question here is how much extra skill development do you get with the extra couple of years of academic training? Certainly you should be gaining something, but recognize that people who don't continue with graduate school probably don't just sit on their duff until they would have graduated. Thus, you are comparing one set of experiences (academic) vs. another (work). A good bit depends on the quality and nature of the Ph.D. program, your intrinsic interests, how self-directed you are, and what kind of opportunities and support would be available in your first job.

Outside of the effect continued academic training has on skill development, there is the question of the effect and value of the signal (i.e., of having "Ph.D" appended after your name). The signal can help in two ways: First, it can help you land your initial job, and that shouldn't be dismissed--it can be very important. Research has shown that people who are obliged to start out in a first position that isn't as appropriate for them never tend to do as well (career-wise, on average) as people who get to start in a job that is a good match for their abilities and interests. On the other hand, the consensus seems to be that after your first job, your future prospects are much more strongly influenced by your performance in your previous job than your academic credentials.

The second aspect of the signal has to do with the relationship between the analyst and the consumer of the analysis. @EMS does a good job of bringing this point out in a comment. There are a lot of small consulting shops, and they love to have Ph.D.'s to show off to potential clients: in initial meetings trying to land a contract, on letterhead, in presentations of finished work-products, etc. the Ph.D.'s are always there. It's easy to be cynical about this, but I do think there's legitimate value for the consulting firm and the consumer (who may not know much about these matters and can use credentials to help them select a firm that will do a good job for them). Behind the scenes, some of the work may be farmed out to competent people with less credentials, but they want the Ph.D. for the front end, and to sign off on the work-product before it's delivered. I could see something analogous happening with a start-up if they're trying to attract capital and want to reassure investors. On the other hand, if you are going to do work for internal consumption, and your boss is capable of evaluating it, this doesn't matter.

share|improve this answer
3  
(+1) I think this answer is starting to get closer to the crux of the matter. There are actually two questions from the OP, posed as one (at least, as I see it). The first is (1) Can a person without a PhD do meaningful work in industry in data-mining and/or machine learning? The answer is undoubtedly affirmative. The second is (2) Can one obtain such a position with limited experience in the area and without a PhD (and how hard is it to accomplish this)? Even though the second is probably easier to measure quantitatively, it seems to have more grey area. –  cardinal May 2 '12 at 16:17
2  
Also, Robin Hanson wrote up some good stuff on this signaling and credential aspect recently, at least w.r.t. consulting. –  EMS May 2 '12 at 17:06
add comment

I agree with most that has been said here, but I want to introduce a few practical issues that arise when applying for jobs in finance. Often you will see ads stating that a PhD in statistics or mathematics is required to apply for a particular trading or quantitative developer position. I know there are some particular reasons for this. Mind, I'm not saying this is right, but it is what happens in practice:

  • There are many applicants to the job, especially for the most well known companies, and the employer can't possibly dedicate enough time to each candidate. Filtering applications based on the academic background shrinks the population size to a more manageable number. Yes, there will be misses. Yes, it's not the best way to find productive individuals. But on average you are looking at skilled professionals who have dedicated years to learn the craft. They should at least have the discipline to overtake a complex research project.

  • The team and the company will be enriched by a number of PhDs to showcase to investors and clients. This will give an image of "oracleous" knowledge to the company and benefit its reputation. The company intangible valuation can rise. The average investor will be more confident in granting their capital to such a knowledgeable team of scientists. You can make a similar point about MBAs.

  • Finally sometimes corporate policies dictate that higher academic achievements should have a preferential career path and compensation. I believe this is true for most corporations in different industries, not just finance. It is hard to see John with a BS in computer sciences managing PhDs in mathematics.

share|improve this answer
add comment

My 2 cents: No, I don't think so. A PhD per se does not entitle one to be be better for data mining or ML. Take kaggle's own Jeremy Howard. I would even go as far as saying that a PhD says not much about any qualification as there is a huge variability in quality of programs. Perhaps the only thing a PhD proves is for the holder to have a high tolerance of frustration.

Bottom line: If you are interested in that area, knowledgeable, creative and hard-working, why would you need a PhD? It is you that should count, not your titles.

share|improve this answer
1  
I totally agree, but I under a strong impression that companies/recruiters seem to think the other way around. Even at my previous job, when I was discussing doing some data mining on some problems, I was told that I am an engineer and not a data scientist and thus it's best if i stay in my focus area. –  Charles Menguy May 1 '12 at 23:51
2  
Well said. See my answer for some supporting details. The PhD in this field is not too relevant, and if a company thinks it is, you probably don't want to work for that company. Peter Thiel once said, "Graduate school is like hitting the snooze button on the alarm clock of life..." –  EMS May 1 '12 at 23:54
add comment

Disclaimer: I'm a recruiter and have been since 1982 so I understand your question very well. Let me break it down this way. Your resume is a screening out device. Companies get tons of resumes so they're reading resumes with one question in mind, "Why don't I want to talk to this person?" That reduces their pile to a few candidates who hold the best chances of meeting their needs. So if you're getting interviews and your resume doesn't show a PhD then there's something else going on here. I say that because, just as a resume is a screening OUT device, the interview is a screening IN device. Once they've invited you to an interview then they've already concluded you have enough "on paper" to do the job. So when you're walking in the interview the only quesion they're really asking is "why should I hire you?" The person they hire will be the individual who addresses out they can best serve the company's needs.

My advice as a recruiter is to ask questions throughout the interview to identify their deeper needs. Believe me, the job description rarely resembles the truth so you'll want to probe for their hot buttons then sell directly to those issues. Don't allow the interview to feel like an interrogation, waiting for the end to ask questions. You'll go down in flames and end up being told "you don't have a PhD". Be respectful yet show your willingness to help them solve their problem.

My favorite question is: "What are the traits of the best person you've ever known in this role?" Everyone has a dream team in mind so it's important to figure out what traits they feel are necessary to succeed in this role. Keep in mind, this isn't a question about experience, backgrounds or degrees. See, I can always find a mediocre PhD with tons of experience so this isn't the holy grail. It's just what companies continue to think is best because IMO they don't know how else to write a job description that captures the essence of the person they need.

share|improve this answer
    
@gung: excellent edit: i was going to do the same. –  user603 Dec 27 '12 at 20:10
2  
Welcome to the site, @GailPalubiak. Please don't sign your posts with your personal info. Note that your avatar & a link to your user page are automatically added to all your posts. You can post that info there. Since you're new here, you may want to read our FAQ, which discusses topics like these. –  gung Dec 27 '12 at 20:11
add comment

Whether a job requires a PhD or not depends on level of responsibility and the perception of the employer and/or his clients. I do not think there is a discipline that requires a PhD. Certainly data mining can be learned and an employee can do productive work without a PhD. This depends more on the person, his or her ability to learn quickly and adapt as well as being able to understand the literature, than on previous education. This is especially true for data mining which is an evolving field. So even the data miners with PhDs will have more to learn as time goes on.

share|improve this answer
3  
(+1) One discipline that requires a PhD is university professor. (Of course there are exceptions, but they are few.) –  whuber May 3 '12 at 21:07
1  
This is also such a statistically unlikely career that most graduate students should discount it, or at least underweight it relative to the rampant beliefs among grad students that they will become tenured professors. The landscape of teaching at a Research 1 university (unless you're content being an adjunct/post-doc) has changed a lot in the past ~15 years in the US. –  EMS May 4 '12 at 11:41
add comment

I have a masters degree in Applied Statistics and worked in Europe as a Data Miner. When I came to the UK nobody had even heard of data mining let alone studied for such a degree. Now it is common place and employers feel that a Phd is necessary for this job. However, it is the statistical knowledge and the modelling aspect which is important for this job. In my experience, most IT people do not understand statistics and are therefore unable to do the job well. I went into teaching and now am registering to do a Phd in Applied Statistics to satisfy these employers. I probably know more than most Phd graduates having studied for my Masters degree in 1980s when the level was very high. I think to be a good data miner, one has to have a background in Statistics.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.