Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I work mainly with non-statisticians in fields such as medicine, social sciences and education.

Whether I am consulting with graduate students, helping researchers with articles or reviewing articles for journals, I often have the problem that someone (client, author, dissertation committee, journal editor) wants to use some relatively well-known technique when it is either entirely inappropriate or when better but lesser-known methods exist. Often, I will explain the alternative technique but then be told "everybody does it the other way".

I'd be interested in how others deal with this sort of difficulty.

ADDITIONS

@MichaelChernick suggested I could share some stories, so I will

Currently I am working with one person who is duplicating a previous paper and adding one independent variable to see if it helps. The previous paper is, frankly, terrible. It treats dependent data as if they were independent; it is tremendously overfit and there are other problems too. Yet he (my client) submitted an earlier version as a dissertation and not only got his degree but was widely praised for the research.

Many times I have tried to convince people not to dichotomize variables. This comes up very often in medicine. I patiently point out that dicohotomizing (say) birthweight into low and normal (usually at 2,500 g) means treating a 2,499 g baby as just like a 1,400 g one; but treating the 2,501 gram baby quite differently. The clinician agrees with me that this is silly. Then says to do it that way.

I had a graduate student client long ago whose committee insisted on a cluster analysis. The student did not understand the method, the method did not answer useful questions, but that's what the committee wanted, so that's what they got.

The entire field of statistical graphics is one where, for many, "this is how grandpa did it" is enough.

Then there are people who seem to just push buttons. I remember one presentation (not by someone I helped!) who had taken an entire questionnaire and factor analyzed it. One of the variables she included was ID number!

Oy.

share|improve this question
6  
Peter, Meta is for questions about this site. I suspect you may have been thinking about "Community Wiki" status, which is for useful, interesting questions unlikely to have an objectively best answer (or which are likely to need collaborative efforts to answer at all). Accordingly, I have interpreted your suggestion as a CW request and implemented that. –  whuber Oct 2 '12 at 16:36
1  
A quick note on dichotomisation: I actually think this is a misplaced intuition from people who know "a little bit" of stats. In the fields you speak of I would imagine a lot of analysis would be geared around decision making (eg, should I start treatment for disease X or Y?). This is dichotomus - and often a useful dichotomy. If you only have a small number of variables, then a proper decision analysis or hypothesis test may well mimic this - we may have "if x>10 take option 1, otherwise take option 2". –  probabilityislogic Oct 3 '12 at 7:44
1  
Also an analysis based on dichotomised variables is very easy to remember if you don't have access to a computer. –  probabilityislogic Oct 3 '12 at 7:46
2  
In psychology people often dichotomise or quantise because they use ANOVA, not regression. This drives me mad. Somewhere in the Vanderbilt stats wiki biostat.mc.vanderbilt.edu/wiki/Main there's a brilliant demonstration of the horrible damage done by dichotomisation. But I can't seem to find it, unfortunately. I think it's there, anyway. Should have Evernoted when I had the chance –  Chris Beeley Oct 4 '12 at 19:29
1  
From my limited experience in the insurance industry, I've found that once a particular predictive model has been developed, it has a life of its own and will continue to be duplicated (maybe with updated coefficients) for years even after the model has become outdated. For example, revamping an old model that sets health insurance policy rates is a big deal since some people's rates may suddenly jump up, the new model will have to be justified to regulatory agencies, etc. –  RobertF Apr 7 at 19:07

5 Answers 5

This is a tricky question!

First, some thoughts on why this happens. I work in an area which does (or at least should) make extensive use of statistics, but where most practitioners are not statistical experts. Consequently one sees a lot of "I put a vector into excel's t-test function and this number fell out. Therefore my paper is supported by statistics."

The main reason I see for this happening is that lack of statistics knowledge starts at the top. If your reviewers and thesis committee don't keep up to date on statistical techniques, then you need to justify use of anything that is "unconventional". For example, in a thesis, I opted to use violin plots instead of box plots to show the shape of a distribution. The use of this technique required extensive documentation in the thesis, as well as a prolonged discussion in my defense where all of the committee members wanted to know what this strange plot meant, despite both the descriptions in the text and the references to the source material. Had I just used a box plot (which shows strictly less information in this case, and can easily deceive the viewer about the shape of a distribution if it is multi-modal) no one would have said anything, and my defense would have been easier.

The point is, in non-stats fields practitioners face a difficult choice: We can read about and then use the correct methods, which entails a bunch of work that none of our higher ups are interested in; or we can just go with the flow, get the rubber stamp on our papers and theses, and keep using incorrect but conventional methods.

Now, to answer your question:

I think a good approach is to emphasize the consequences of failing to use correct techniques. This might entail:

  • Giving a real world example of how someone in their field experienced the consequences of poor inference. This is easier in some fields than others. Examples where careers were damaged are especially good.

  • Explaining that doing incorrect analysis can leave you in a situation where your results are very unlikely to transfer to the real world, which could cause harms (e.g. In my field, if your A.I. system prototype appears statistically better than the competition, but in fact is the same, then spending the next 6 months building a full implementation is a really bad idea.

  • Pick techniques which will save the users lots of time. Enough time so that they can spend what they save explaining the techniques to the higher ups.

share|improve this answer
1  
Good discussion and good answer +1. –  Michael Chernick Oct 2 '12 at 18:39
    
Good points @John –  Peter Flom Oct 2 '12 at 20:33
1  
+1 for pointing out the consequences. It can do wonders for getting people to switch to better methods. –  Leo Oct 3 '12 at 6:35

Speaking from the perspective of a psychologist with only slight statistical sophistication: When you introduce the method, also introduce the tools. If you tell most researchers in my field a long story about a great new method, they're going to spend the whole time worried that the punchline is "and all you have to do is brush up on your differential calculus and then take a two week training course!" (or "and buy a $2000 stats package!" or "and adapt 5000 lines of Python and R code!"). Whereas if there's an implementation of the method available in the stats package they already use, or in a piece of free software with a comprehensible GUI, and they can get up to speed on it in a day or two, they might be willing to give it a try.

I'm aware that this approach can seem venal and unscientific, but it's easy for people to fall into when they're worried about grants and publications, and don't see learning huge amounts of math as likely to help them keep their jobs.

share|improve this answer
2  
@octem Couldn't the investigator trust the statistician to do that part of the collaboration. Why does the investigator need the tool. I liken this with the physician to reversing the tables. How would he feel if I said give me a quick tutorial on how to do this surgery and I will go ahead and do it for you. I think shock and dsimay and it is illegal for me to practice mewdicine without a license. That is probably a good thing. But doesn't the statistician deserve equal respect. Why expect that I can just give him the tool and let him hack away with his lack of training. –  Michael Chernick Oct 2 '12 at 18:37
1  
In social psychology, it's not usual to have a statistician (because it's not usual to have enough funding to pay a statistician). Now I'm in public health / health psychology. Big grants usually include salary for a statistician, but a lot of our work is done on stingy little pilot grants where we can't even afford salary for the PI. That's the perspective I'm coming from... if you're in a field where most projects include a trained statistician, then I agree, this kind of resistance would not be reasonable. –  octern Oct 2 '12 at 18:45
1  
@octem Thanks for the response. Just to follow-up with the analogy. There was a time when abortion was illegal in the US and some women went to other countries or had it done in a back room illegally without sterile conditions and great health risks. It may not seem like a great analogy but not being able to afford a statistician justifies doing an inferior job? I know medicine is more of a life or death thing but bad science has bad consequences too? Misuse of data could lead to practicing bad medicine because an unsafe drug gets used when it shouldn't be. –  Michael Chernick Oct 2 '12 at 19:11
    
@MichaelChernick Look, I'm not happy about the state of statistical analysis in the social sciences either. But the question was how to get researchers to adopt a new statistical approach, and I'm giving an answer that's appropriate to a large subpopulation of researchers -- whether we like it or not. –  octern Oct 2 '12 at 19:33
1  
@octem Fair enough and I think I understood that. I gave you an upvote before I asked the question. The attitude of the investigators to think that the cheap way is the way to go is what I am questioning and not the fact that you recognize the problem exists. It does and I agree with you there. But somehow in the long run I think we just need to get more respect and recognition that our job is not trivial. –  Michael Chernick Oct 2 '12 at 19:39

Thanks for this nice question Peter. I work at a medical research institution and deal with physicians who do research and publish in the medical journals. Often they are more interested in getting their paper published than "doing the statistics completely right". So when I propose an unfamilar technique they will point to a similar paper and say "look they did it this way and got their results published."

There is a problem I think when the published paper is really bad and has mistakes. It is difficult to argue even though I have a great reputation. Some docs have big egos and think they can learn almost anything. So they think they understand the statistics when they don't and can be insistent. It can get frustrating. When it is a t test and Wilcoxon is more appropriate I get them to do a Wilk Shapiro test and if normality is rejected we include both methods and explain why Wilcoxon is better. I sometimes can convince them and often they depend on me for statistics, so I have a little more clout then a general consultant might have.

I also ran into a situation where I did Kaplan-Meier curves for them and we used the log rank test but Wilcoxon gave a different result. It was hard for me to decide and in such situations I think it is best to present both methods and explain why they differ. The same goes for using Peto vs Greenwood confidence intervals for the survival curve. Explaining the Cox proportion hazard assumption can be difficult and they often misinterpret odds ratios and relative risk.

There is no simple answer. I had a boss here who was a top medical researcher in cardiology and he sometimes referees for journals. He was looking at a paper that dealt with diagnosis and used AUC as a measure. He had never seen an AUC curve before and came to me to see if I thought it was valid. He had doubts. It turned out to be appropriate and I explained it to him as best I could.

I have tried to lecture on biostatistics to physicians and have taught biostatistics in public health schools. i try to do it better than others have and produced a book for health science majors introductory course in 2002 with an epidemiologist as coauthor. Wiley wants me to do a second edition now. In 2011 I published a more concise book that I tried to cover just the essentials so that busy MDs might take the time to reasd it and reference it. That is how I deal with it. Maybe you can share your stories with us.

share|improve this answer
    
These are good points @Michael. I will add some stories –  Peter Flom Oct 2 '12 at 20:35
1  
@PeterFlom I think we might have some very similar experiences. I alos think you are getting some other very good responses in other people's answers. –  Michael Chernick Oct 2 '12 at 20:38

There are some nice comments already made here, but I'll throw in my 2 cents. I'll preface this all by saying that I'm assuming we're talking about a situation where using the traditional "canned" techniques will damage the substantive conclusions reached by the analysis. If that's not the case, then I think that sometimes doing an overly simplistic analysis is excusable both for brevity and for ease of understanding when the target audience are laymen. Is it really such a crime to assume independence when the intraclass correlation is .02 or to assume linearity when the truth is $\log(x); \ x \in (1,2)? \ $ I'd say no.


In my career I do a lot of interdisciplinary research and has lead me to work with closely with substance abuse researchers, epidemiologists, biologists, criminologists and physicians at various times. This typically involved analysis of data where the usual "canned" approaches would fail for various reason (e.g. some combination of biased sampling and clustered, longitudinally and/or spatially indexed data). I also spent a couple years consulting part time in graduate school, where I worked with people from a large variety of fields. So, I've had to think about this a lot.

My experience is that the most important thing is to explain why the usual canned approaches are inappropriate and appeal to the person's desire to do "good science". No respectable researcher wants to publish something that is blatantly misleading in its conclusions because of inappropriate statistical analysis. I've never encountered someone who says something along the lines of "I don't care whether the analysis is correct or not, I just want to get this published" although I'm sure such people exist - my response there would be to end the professional relationship if at all possible. As the statistician, it's my reputation that could be damaged if someone who actually knows what they're talking about happens to read the paper.

I admit that it can be challenging to convince someone that a particular analysis is inappropriate, but I think that as statisticians we should (a) have the knowledge necessary to know exactly what can go wrong with the "canned" approach and (b) have the ability to explain it is a reasonably comprehensible way. Unless you're working as a statistics or math professor, a part of your job is going to be to work with non-statisticians (and even sometimes if you are a stat/math prof).

Regarding (a), if the statistician doesn't have this knowledge, why would they be discouraging the canned approach? If the statistician is saying "use a random effects models" but can't explain why assuming independence is a problem, then aren't they guilty of giving in to dogma in the same way the client is? Any reviewer, statistician or not, can make pedantic critiques of a statistical modeling approach because, let's face it - all models are wrong. But, it requires expertise to know exactly what could go wrong.

Regarding (b), I've found that graphical depictions of what could go wrong typically "hit home" the most. Examples:

  • In the example given by Peter about categorizing continuous data, the best way to show why this is a bad idea is to graph the data in its continuous form and compare it with its categorical form. For example, if you're making your response variable binary then plot the continuous variable vs. $x$, and, if it doesn't look an awful lot like a step function, then you know the discretization lost valuable information. If this difference isn't drastic or resulting in any changes in the substantive conclusions, you could also see this from the plot.

  • When the proposed "form" of the model (e.g. linear) is inappropriate. For example, if the regression function "plateaus" like $y = x$ for $x \in (0,1)$ but $y = 1$ for $x > 1$ then a linear fit's slope will be too shallow and, depending on the data, this could push the $p$-value below significance despite there being an obvious relationship between $x$ and $y$.

  • Another common situation (also mentioned by Peter) is explaining why assuming independence is a bad idea. For example, you can show with a plot that positive autocorrelation will typically produce data that is more "clustered" and the variance will be underestimate for that reason, giving some intuition of why the naive standard errors tend to be too small. Or, you could also plot the data with the fitted curve that assumes independence and one can visually see how the clusters influence the fit (effectively lowering the sample size) in a way that is not present in independent data.

There are a million other examples but I'm working with space/time constraints here :) When pictures simply won't do for whatever reason (e.g. showing why one approach is underpowered) then simulation examples are also an option that I've employed from time to time.

share|improve this answer

Some random thoughts because this is a complex issue...

I feel that a big problem is the lack of math education in a variety of professional disciplines and graduated programs.

Without a mathematical understanding of statistics, it becomes a bunch of formulas to be applied according the case.

Also, for getting a real understanding of the matter, professors should talk about the original problems that the original authors were facing at the time they published their approaches. One can learn more from that than from reading thousands books on the subject.

Statistics is a toolbox for solving problems, but it is also an art and faces the same issues than any other art.

One can learn how to make sounds with an instrument. But by being able of "playing" an instrument one does not become a musician.

However, is not uncommon to find people that see themselves as musicians without having studied a single concept of rhythm, melody and harmony.

In the same line, for getting papers published, most people don't need to know nor understand the concepts behind a formula... nowadays scientists just need to know what key they have to press and when it has to be pressed, period.

So this has nothing to do with the "ego" of MDs. This is a subcultural problem, a problem more related with education, customs and values of the scientific community.

What one can expect in an era in which there are thousands and thousands and thousands of useless papers and books being published for fulfilling some academic requisites/policies? In an era in which the amount of papers one publishes is more important than the quality of them?

Mainstream scientists are not worried about the good science anymore. They are slaves of numbers. They are affected (or infected) by the administrative bug of our era...

So, from my perspective, a good course in statistics should include the mathematical, historical and philosophical basis of the approach being studied, always highlighting the several paths one can take for solving a single problem.

Finally, if I were a professor in statistics/probability my first lecture(s) would be dedicated to problems like shuffling cards or tossing a coin. That will put the audience in the right position for listening... probably.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.