2
$\begingroup$

I have been tasked with finding the return on investment of a vaccine. The vaccine is used by farmers for their cows and the number of cows kept varies drastically between farmers (from $1$ to $1,000's$).

I have produced a rather complex model of the death rates based on the number of cows kept (non-linearly) and whether the farmer is vaccinating that predicts the number of deaths a farmer may expect. I can then use this to come up with the value of the vaccine (because a dead cow cannot be sold).

A colleague disagrees and says that from a theoretical point of view, it is incorrect to use the herd to predict deaths and actually I should just be considering each cow separately (you may assume the disease does not pass from cow to cow like a cold would) and calculating the probability of death whether the cow is vaccinated or not. This would, of course, lead to a simple binomial distribution and I am sure that the data does not follow this and have evidence to support my argument.

As you may expect, I am a statistician and he/she is a scientist and so we each support the answer from our respective field.

My questions:

  1. Am I correct that a model that better reflects the data is superior to a model that falls somewhat short but has good theoretical backing for a business decision such as this?
  2. If so, is there a good way to explain this or is there a good example for why this is the case?
  3. If not, why not?

Thank you

EDIT: A good point raised by @whuber , I am not worried about overfitting in this case. The data set is very large.

$\endgroup$
2
  • 1
    $\begingroup$ "Better reflects the data" in (1) can always be accounted for by over-fitting. $\endgroup$
    – whuber
    Feb 12, 2018 at 20:50
  • $\begingroup$ That is a very good point so thank you. My model uses approximately 10 parameters as does theirs. This is for a data set of about 1500 points. There is certainly enough flexibility that I feel that both models still have plenty of predictive power and I'm not worried about this. $\endgroup$ Feb 12, 2018 at 20:54

1 Answer 1

0
+100
$\begingroup$

There is one main question, but is open to many possible answers. The question is: Is a model that better reflects the data is superior to a model that falls somewhat short but has good theoretical backing? There is a second part to it: for a business decision such as this?

The simplified answer is: No. Or rather: Mu -- ( https://en.wikipedia.org/wiki/K%C5%8Dan ) The real answer lies in the definition of "better". If the figure of merit is the return of investment, you have a clear definition by which you can decide. Hypothetically do both and evaluate the difference.

But the determination of the "best fit" of a certain model is a much more involved question. The "goodness-of-fit" can only be determined within certain models, and only in special cases exactly (e.g. chi square fitting). But the comparison of different models, especially of some of which cannot be subjected to this procedure is simply not possible.

It is a common concept in science, that one can find the "right" theory by testing it under the light of data. Or express this in the easy to understand Bayesian expression P(theory|data) = P(data| theory) * P(theory)/P(data), although it's a bit more involved to do an actual calculation. The fundamental problem with the concept is that it's wrong. It may be possible to falsify a certain theory by data (e.g. theory: "no cow ever dies", data: "look at my steak"). But with everything involved there is no way to prove that some theory might be true.

So who is right now? In all probability you are addressing the question from two different directions. My guess is that the statistician is only concerned about the data itself and tries to describe it the best (= shortest, most simple) way; so that technicalities like overfitting etc. don't matter. The scientist starts with the reasoning of what is happening, maybe even from first principles (scientists are known to do this). Everything boils down to the questions how can the probability of cow death (because of disease xYx) be expressed and compared between vaccinated and un-vaccinated animals.

Without more information on the actual problem, it is not possible to say who's right. But clearly you only have the data that you have available. And certainly one would need this data to form/calculate an answer. But if the data is insufficient to extract all needed features, then the statisticians approach might yield a better result. But if the data is sufficient, then the scientist can prove (with a lot of work!) that his theory is statistically the least wrong.

$\endgroup$
1
  • $\begingroup$ Thank you for your answer. I feel it answers it best. The eventual team decision was that, if the data didn't fit the theory, then the data must be wrong. This was my first foray into the science of what my company does and it is messy to say the least, particularly from a stats perspective. I appreciate the effort put in here. Thanks again. $\endgroup$ Feb 16, 2018 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.