0
$\begingroup$

I have difficulties identifying the correct statistical test for the following. My data show a half-normal distribution, so the histogram would look something like this:

X=0: 200 Y data points 
X=1: 150 Y data points 
X=2: 80 Y data points 
X=3: 40 Y data points 
X=4: 30 Y data points 
X=5: 20 Y data points 
X=6: 10 Y data points 
X=7: 5 Y data points 
X=8: 3 Y data points 
X=9: 2 Y data points 
X=10: 0 Y data point 
X=11: 1 Y data point 
X=12: 0 Y data point 
X=13: 0 Y data point 
X=14: 0 Y data point 
X=15: 1 Y data point

Note 1: X is a positive integer, so discrete/not continuous. 
Note 2: Y data points are normally distributed for each X and continuous.

I have multiple datasets for Y, so you could say I have a dataset Y1, Y2, … Yi. For each Y1, Y2, …Yi, X is distributed exactly as above and only all data points in Y1, Y2, … Yi are different.

Now, I am interested in knowing whether which Y (Y1, Y2, or … Yi) best correlates with the data. I can create a bar graph for each Yi and then do a Spearman or Pearson linear regression analysis to calculate Spearman R and p-value or Pearson R and p-value. Then the Yi that has the largest R and the smallest p-value would best fit the data.

Question: Is this correct at all, since this involves discrete and continuous data, which are half-normally and normally distributed, respectively? This is not a scatter plot type of data, so I have doubts about whether Spearman/Pearson correlation is the best here.

I would appreciate any tips for the best test here.

$\endgroup$
1
$\begingroup$

Pearson correlation is equivalent to $R^2$ in classical OLS linear regression. OLS, in turn, is based solely on two assumptions (see Gauss-Markov theorem):

  • distribution of $Y$ conditional on any $X$ is $\mathcal{N}(\mu(X), \sigma^2)$, where all observations are conditionally independent given $X$, and have the same $\sigma$.
  • $\mu(X)$ is a linear function of $X$.

Please note that these assumptions do not involve distribution of $X$: it doesn't have to be a random variable at all!

The first assumption seems to be satisfied in your case: data points are normaly distributed for each $X$. You can test whether their variance is also equal for each $X$, and if it is not the case, look for heteroschedasticity-compatible methods.

And if it is the case, then OLS makes sense for your problem, and you can use $R^2$ as a goodness-of-fit statistic. If you suspect that relation between $Y$ and $X$ may be nonlinear, you can e.g. include polynomial terms.

$\endgroup$
3
  • $\begingroup$ David, thank you so much for your help! My background is not in statistics, so I had to look up a few of your comments, but it is overall clear what you mean. I would like to ask you a follow-up question, please. You suggested that if the variance for each X is not equal (which is the case for my data), then "look for heteroschedasticity-compatible methods". Would you have a suggestion for which method would be best in this case, please? Thanks again - Peter. $\endgroup$ Feb 24 '18 at 11:45
  • $\begingroup$ Peter, in fact, OLS and R^2 can still work in case of heteroschedasticity (although OLS estimates lose some of their nice properties). But you can also find a way either to rescale your data to make it homoscedastic, or assing different weights to observations (see WLS). $\endgroup$
    – David Dale
    Feb 24 '18 at 11:53
  • $\begingroup$ Thanks again, David. That's very helpful. I will probably go for Spearman correlation tests, because I have thousands of Ys to test, so it will not be practical to, for each Y: (1) check how to best rescale the data and (2) check for the presence of outliers. So, I will prefer Spearman correlations over Pearson correlations. Many thanks! $\endgroup$ Feb 24 '18 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.