I have a dataset, based on forum conversations, with the following variables:

  • count of emotional words
  • count of cognitive words
  • total word count
  • performance evaluation

Now I want to predict performance evaluation, while controlling for total word count. I am doing it in two different ways:

The ratio approach

I divide count of emotional words and count of cognitive words by total word count, so as to get the ratio of specific words to total word count. I then use these new variables as predictors of performance evaluation, like so:

performance evaluation ~ count of emotional words/total word count + count of cognitive words/total word count

This gives an R2 of 0.02

The control variable approach

Here, I simply put in all the variables in a regression, like so:

performance evaluation ~ count of emotional words + count of cognitive words + total word count

This gives an R2 of 0.40

Why do these two approaches give such different results? What different substantive interpretations does each of the ways of "controlling" for word count have? Is one of them a more appropriate way to "control" for word count?

  • $\begingroup$ what is a formula of your first approach? $\endgroup$ – Aksakal Dec 5 '16 at 15:42
  • $\begingroup$ @Aksakal: performance evaluation ~ count of emotional words/total word count + count of cognitive words/total word count $\endgroup$ – histelheim Dec 5 '16 at 15:46
  • 3
    $\begingroup$ They are completely different models so it is not surprising they give differ t outcomes. Pick the one which corresponds to the science rather than expecting to decide statistically. $\endgroup$ – mdewey Dec 5 '16 at 20:49
  • $\begingroup$ @mdewey: it's the difference between the two that I'm trying to understand--hence the question. $\endgroup$ – histelheim Dec 5 '16 at 23:28
  • $\begingroup$ Statement of your objectives/hypotheses and measure f performance evaluation and some contextual details may be useful. $\endgroup$ – Subhash C. Davar Dec 6 '16 at 14:38

You might be looking for a deeper meaning when the straightforward interpretation is the right one: the second model better describes the dependent variable. The second model has three variables, so you better be looking at the adjusted $R^2$, but that wouldn't change things too much since the difference in $R^2$ is so big. In this case it seems that total word count variable explains the performance evaluation best.

I would do two things to confirm my observation.

  1. run a regression

    performance evaluation ~ total word count

and see what's adj. $R^2$, I bet that it's between two of your models.

  1. Look at the definition of the performance evaluation metric. What does it measure? Could it be that from the description itself it becomes clear that the total num of words is the main driver of high performance?

Say, you're trying to understand how do your readers rate your haiku that you publish on your blog. So, you have two theories:

  • the readers simply like tear jerkers, i.e. the length of haiku doesn't matter, only the emotional content
  • the readers like long haiku emotional haiku, i.e. they like it when you have long verses with some emotional content

so you run your two regressions on the ratings of haikus. based on your results, it seems that the first theory doesn't fit the data, so you go with the second. you conclude that they like emotional poems, but they want them really long.

  • $\begingroup$ This makes sense, but what is the conceptual difference of using a ratio vs. a control variable? What does each model "mean"? $\endgroup$ – histelheim Dec 6 '16 at 13:10
  • $\begingroup$ updated an answer $\endgroup$ – Aksakal Dec 6 '16 at 19:24

Both models test fundamentally different hypotheses. In fact, you don't even "control" for total word count in the first model.

The first model tests whether differences in performance evaluation can be modeled by which ratio of words is of a certain type. The second model tests whether performance evaluation can be modelled in terms of word counts.

If that difference isn't obvious to you, consider the following two people:

Jane: emotional : 5 , cognitive : 3, total word count : 15
Emma: emotional : 15, cognitive : 9, total word count : 45

In your first model, both Jane and Emma have a ratio of 33% emotional and 20% cognitive, and your model assumes that Jane and Emma have the exact same performance evaluation. In your second model however, the data for Jane and Emma is vastly different. That model assumes they have a different performance evaluation.

So your question has little to do with statistics. The question is which hypothesis do you want to test. Do Jane and Emma have the same performance evaluation or not?

Moreover, your second model has the obvious problem of collinearity: the amount of cognitive and emotional words is by definition correlated with the total word count. In the case of a (generalized) linear model, this violates the assumptions of the model. Hence, the output of your second model shouldn't be trusted.

I would personally consider a third approach:

Performance evaluation ~ emotionalRatio + cognitiveRatio + totalWordCount

That would be the equivalent of controlling your first model for the total word count. Or, in other words, you model any influence of the total word count on the performance evaluation next to the ratio of emotional and cognitive words.

And judging from the information you gave us, I'm rather confident that the totalWordCount will come out as the most significant contributor in that model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.