1
$\begingroup$

We have a representative digital panel that measures digital media consumption of panelist and then project it for the total population via weighting. Recently I ran a multiple regression on panel data to figure out most explanatory demo characteristics like age, gender, education, profession, etc (that can be controlled in the panel-based measurement) for digital media consumption.

My model output $R^2$ = 0.40 which is good enough in our field study that tries to explain human behavior. Considering that people are just harder to predict than other things/processes.

One of our new stakeholders said the model $R^2$ is so low and he does not trust the result. He is not a statistician and considers himself more a generalist. I tried to explain that in human behavior study, we have an inherently greater amount of unexplainable variation regardless of how many demographic characteristics we add to the model. I also explained that many soft characteristics like individual preference, political view, etc (non-control metrics) will play a role that cannot be simply explained by individual demographic characteristics.

I couldn't convince him. I am wondering how can I explain this concept better for commercial people like him

$\endgroup$
1
  • 1
    $\begingroup$ I am not sure that his misunderstanding is about what $R^2$ means. It seems like it is on you to explain why $R^2 = 0.4$ isn't equivalent to a $40\%$ in school and thus an $\text{F}$ grade. So...why does such an $R^2$ value make the model worth considering? $\endgroup$
    – Dave
    Sep 14 at 18:24
11
$\begingroup$

This is actually a good CV question because it is the pivotal moment when a trained statistician becomes a professional statistician.

Learning moment 1: Never judge a person by their pedigree or training

By you not responding seriously to a critique of your methods because the criticism came from a non-statistician, you are selling yourself short, very short indeed. Instead you should have asked yourself whether the actual matter of the remark made sense. In other words, if John Tukey had said that $R^2$=0.40 was underwhelming, you would have thought quite differently about this scenario.

In my career I've met many non-statisticians whose level of quantitative thinking was a lot better than mine, and they've been able to understand and remark on my work with remarkable detail and clarity, and I'm grateful for their help. It's actually quite easy in the literature to expose one's self to a lot of methods when they're less concerned with implementing those methods. A statistician is more concerned with implementation, and so holistic understanding of methods and their plusses and minuses often ends up being a blindspot for us.

Be gracious that someone cares enough about your work to listen.

Learning moment 2: if you don't understand a person's remark, entice them to say more.

  • Case 1: If they truly haven't mustered statistics, they'll reveal it when trying to explain it in greater detail. Broad criticism is never helpful, but everyone is capable of offering it. That criticism loses credibility when the critic can't pinpoint an idea or opportunity to turn things around in the right direction.
  • Case 2: more likely, you just missed the point and it's a chance to respond intelligently and specifically. Suppose there's another model out there that achieves $R^2$=0.60 you didn't know about, then you graciously thank the commenter and promise that you'll review the methods and respond about the possible impact to your presentation later. Suppose they are contrasting with another study you're knowledgeable about and you can speak to some meaningful differences, etc.

In short, this all seems like a lost opportunity to just ask the person what they meant. Your answer that "human behavior is unpredictable" comes across as glib, and not hanging on to any meaningful differences; it even argues against you ever fitting any model at all. Rather, you can, for instance, feed your model's results into a cost-benefit analysis. Or you can point out that averaged over 1,000,000,000s of users, a 0.40 $R^2$ is actually impressive (did you specify benchmarks? what were the results from prior models if any? If this is the first model of it's kind, you can argue that your model establishes the benchmark). I recall a well funded air pollution model celebrated achieving $R^2$=0.04 for predicting exposure to CO, O3, and other air pollutants, this is because the exposure is SO variable, the model's ability to detect any change was celebrated.

Lastly,

Learning moment 3: There's always room on the collaborator boat

The saying "if you can't beat 'em, join 'em" has never been truer than today. Statisticians are lucky because the stuff of our work (data) practically falls in our lap. In the earlier parts of this answer, I mention some things (e.g. benchmarks, cost-benefit analyses). I think I'm not assuming too much in guessing that these are, in fact, gaps in understanding. And if it's mentioned for the first time at a presentation as a gap in understanding, you can be humble to point it out as such, but it doesn't need to end there. You can show commitment and dedication to both your work and the audience by saying, "This is something that we didn't consider, but there are ways we can reframe the question or provide supporting research on the possible value or room for improvement. And I'd welcome anyone here to reach me afterwards if they're interested to pursue this with me."

$\endgroup$
2
  • 3
    $\begingroup$ +1. One more aspect, which relates to your mentioning benchmarks: it would be good if the OP could point to a measurable, ideally monetary benefit of their model with $R^2=0.40$. If this model is already providing some benefit, the entire discussion gets a much more productive vibe. (If not, that also tells us something.) After that, the question should naturally segue into whether it is feasible to achieve an even better performance. Or whether this is unrealistic. $\endgroup$ Sep 14 at 19:05
  • $\begingroup$ @StephanKolassa agreed, that's what I thought in suggesting a cost-benefit analysis. I didn't get into the nitty gritty. We see this all the time in personalized medicine, though. Some drug confers a 0.70 HR for progression/death, new risk models yield the same 0.70 except that 40% of the treated patients don't take the drug at all (and deal with none of the cost or toxicity either). $\endgroup$
    – AdamO
    Sep 15 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.