10
$\begingroup$

I have a dataset with a dependent and an independent variable. Both are not a time series. I have 120 observations. The correlation coefficient is 0.43

After this calculation, I have added a column for both variables with the average for every 12 observations, resulting in 2 new columns with 108 observations (pairs). The correlation coefficient of these columns is 0.77

It seems I improved the correlation in this way. Is this allowed to do? Did I increase the explanation power of the independent variable by using averages?

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ All you did is run the data through a smoothing filter. This is done all the time in signal processing and it is perfectly acceptable and usually required before the data is even usable. It eliminates the noise which is always prevalent in electronic measurements. However, whether it is acceptable for your particular problem depends on the specifics of what you are trying to achieve and probably to a large degree how much "noise" versus "quality" is in your data. I just noticed "Both are not a time series" so I suspect that what you did is meaningless because changing the order changes results $\endgroup$
    – Dunk
    Commented Sep 15, 2016 at 14:31
  • $\begingroup$ Thank you all. My dependent variable is a serie of monthly results of a betting system (these results are not related). The independent variable is the result of an indicator I constructed. This indicator generates a score regarding how extreme the scores of sportsmatches have been in a particular month (these sportresults are not related). I was suspecting that what I did was meaningless, although it surprised me that the correlation coefficient improved so much. $\endgroup$
    – user2165379
    Commented Sep 15, 2016 at 14:47
  • 2
    $\begingroup$ I'm not certain but I think averaging any data would give similar results. I would think that the averaging reduces the affects of outliers. Thus, the correlation would have to improve. Although, I'll bet that some mathy-geek can come up with well chosen data that would cause the opposite affect, but I wouldn't expect data like that to occur in the real-world. $\endgroup$
    – Dunk
    Commented Sep 15, 2016 at 15:06
  • $\begingroup$ I could not see if you specified what this data was for. However, in general, when presenting your data to your specified audience, providing disclosure on how the data was derived is good practice. $\endgroup$
    – Jon Milliken
    Commented Sep 15, 2016 at 16:29
  • 4
    $\begingroup$ What is the correlation of averaged values intended to represent? It's certainly no longer a reasonable estimate of the correlation between the original variables. $\endgroup$
    – Glen_b
    Commented Sep 15, 2016 at 18:20

3 Answers 3

Reset to default
16
$\begingroup$

Let's have a look at two vectors, the first being 

    2 6 2 6 2 6 2 6 2 6 2 6

and the second vector being 

   6 2 6 2 6 2 6 2 6 2 6 2

Calculating the Pearson correlation you'll get

cor(a,b)
[1] -1

However if you take the average of successive pairs for values both vectors are identical. Identical vectors have correlation 1. 

  4 4 4 4 4 4

This simple example illustrates a downside of your method.

Edit: To explain it more generally: The correlation coefficient is computed in the following way.

$\frac{E[(X-\mu_X)(Y-\mu_Y)]}{\sigma_X\ \sigma_Y}$

Averaging some $X$s and some $Y$s changes the differences between $X$ and $\mu_X$ as well as the difference between $Y$ and $\mu_Y$.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ I've added some mark-up but you can and should define $\mu$ and $\sigma$ terms explicitly. $\endgroup$
    – Nick Cox
    Commented Sep 15, 2016 at 12:31
  • $\begingroup$ Thank you. Does this mean that my results are 'inflated' of flattered by using the averages and that it is always better to use the observations without averaging? $\endgroup$
    – user2165379
    Commented Sep 15, 2016 at 13:29
  • $\begingroup$ For hypothesis testing you should have a look at the data itself and not at averages. In other domains descriptive statistics might be a useful tool. You should also have a look at other meaures of descriptive statistics such as quantiles (especially median) and higher (centralized) moments, such as variance, skewness and kurtosis. However in our case this is not useful. The vectors a and b have the same quantiles, the same moments and the same centralized moments. $\endgroup$
    – Ferdi
    Commented Sep 15, 2016 at 13:39
  • 1
    $\begingroup$ Averaging tends to increase correlations by removing quasi-random scatter but a sufficiently perverse averaging could push correlations towards zero. $\endgroup$
    – Nick Cox
    Commented Sep 15, 2016 at 13:56
  • $\begingroup$ Thank you. So if averaging tends to increase correlationa in general, this implies it is not an improvement? Or is it an improvement because the quasi random scatter is removed? $\endgroup$
    – user2165379
    Commented Sep 15, 2016 at 16:18
10
$\begingroup$

Averaging can be attractive or convenient. It can also be a source of deception, at worst deceit, so tread carefully even when there is a clear rationale for averaging.

Here is a situation it which it is not a good idea. Consider that by careful definition of groups you (usually) could reduce your data to two summary points each distinct on the two variables; and then you would achieve a perfect correlation with magnitude $1$. Congratulations, or not! The improvement here is bogus without a good independent reason for the procedure. You don't need to approach this extreme case to approach the danger.

There are some situations in which averaging can make sense. For example, if seasonal variations are of little or no interest, then averaging into yearly values creates a reduced dataset in which you can focus on those yearly values.

In various fields, researchers could be interested in correlations at quite different scales, e.g. between unemployment and crime for individuals, counties, states, countries (substitute whatever terms make most sense).

The interest, and often also a major source of inference troubles, is in interpreting what is going on at different scales or levels. For example, a high correlation between unemployment rate and crime rate for areas doesn't necessarily mean that the unemployed have a higher tendency to be criminals; you need data on individuals to be clear on that. Data provision can be maximally awkward in data being available only on the least interesting scale, perhaps as a matter of economy or confidentiality.

I note also that that many measurements are in the first place often averages over small time intervals and/or small space intervals, so the data often arrive averaged in any case.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ I echo @Ferdi's answer in underlining that there can be many different ways to average. This creates an extra source of uncertainty. The difficulty is especially acute in aggregating small areas to larger. $\endgroup$
    – Nick Cox
    Commented Sep 15, 2016 at 12:27
0
$\begingroup$

In the case of correlation of averages, it is necessary to be aware of Simpson's paradox.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.