New answers tagged

1

That the points all lie within an ellipse is a mathematical restriction: it does not otherwise reveal anything about your data. Generally, when you have two random variables (or data vectors) $Y$ and $Z$ with a correlation $\rho$ between them, the correlations between a third variable $X$ and these two are restricted. Writing these correlations as $\...


0

Consider what you're not seeing - there consistently are no points strongly anticorrelated with mFOXP3 but strongly correlated with hFOXP3. Considering those two are variations on a theme for one underlying function and thus somewhat common structure, it would be surprising if their association with the expression of any gene was strong in opposite ...


1

Basically, there is a correlation among your correlations due to a) spontaneous correlation between mFOXP3 and hFOXP3 and b) using the partial correlations for the multivariate model of $$\text{GENE} = \alpha + \beta_1 +\text {hFOXP3 } + \beta_2 \text{mFOXP3 } + \epsilon.$$ The part that's missing from the axis labels is correlation with what if we put "with ...


2

The problem with your hypothetical data is that it is not standardized. Both $z_x$ and $z_y$ have mean zero, but their standard deviation is not equal to one. Why is this important? In the general case, your formula is calculating the sample covariance: $\hat{cov}(x,y)= \frac{1}{n}\sum_{i=1}^n (x_i-\hat{\mu}_x)(y_i-\hat{\mu}_y)$ where $\hat{\mu}_x$ and $\...


2

The fallacy here is that your hypothetical dataset is impossible. The consequence of standardizing is that each standardized variable has mean 0 and SD 1. You subtract the mean, so the new mean is 0, and you then divide by the SD, so the new SD is 1. Your hypothetical dataset has mean zero on each standardized variable, but its standard deviation on each ...


0

Partial correlation is the inverse of the covariance matrix... So Compute all the partials, write down the matrix and compute the inverse! :)


2

The Gaussian RBF kernel, also known as the squared exponential or exponentiated quadratic kernel, is $$ k(x, y) = \exp\left( - \frac{\lVert x - y \rVert^2}{2 \ell^2} \right) ,$$ where $\ell$ is often called the lengthscale. Remember that for $f \sim \mathcal{GP}(0, k)$, the correlation between $f(x)$ and $f(y)$ is exactly $k(x, y)$. So with a Gaussian RBF ...


4

In terms of $(x_1, x_2, \ldots, x_n, y)$ the correlation matrix is $$\Sigma_n = \pmatrix{1 & 0 & \cdots & 0 & \rho\\ 0 & 1 & \cdots & 0 & \rho\\ \vdots & \vdots & \ddots & 0 & \rho\\ 0 & 0 & \cdots & 1 & \rho\\ ...


1

As already mentioned, the p-value is used to asses the significance of, for example, a correlation. It "evaluates" whether your data provides enough evidence to consider that the correlation is greater than zero. Two things have an effect on the p-value. One is the effect size, how large your correlation is. Larger correlations provide a smaller p-value. ...


1

If you have just 1 independent variable then the beta coefficient of the standardized independent variable is just its correlation with the dependent variable, which you can see matches with your correlations. So nothing has gone wrong with the software or what you clicked if you are worried about that. If you take your unstandardized beta coefficient and ...


1

What do you consider to be a high correlation coefficient? What do you consider to be a low VIF? The VIF is calculated by regressing predictor $i$ on all the other predictors, and then calculating $VIF = \frac{1}{1 - R_i^2}$. If you consider a VIF of 5 to be high, you'd only get a high VIF if $R_i^2$ was greater than or equal to 0.8. Now imagine ...


4

In this general setting, what would be the best approach to calculate corr with null values? Input or discard? I don't know the size of these datasets, but they are generally "big". I'm assuming that you have only 2 variables and you want to compute the correlation between them. Imputing the missing values, if you have no knowledge of why they are missing,...


10

$\newcommand{\Cov}{\operatorname{Cov}}$The multiplication rule for covariances is $\Cov(aX,bY)=ab\Cov(X,Y)$. Note that the constant term ($-\frac{160}{9}$) doesn't affect the standard deviation or covariances of the temperatures, so we can ignore it. Since $T=\frac{5}{9}X$ and $S=\frac{5}{9}Y$, we know that $$\Cov(T,S)=\Cov\left(\frac{5}{9}X,\frac{5}{9}Y\...


0

You are not the only one to have thought correlation useless! John Tukey also had such ideas, see the paper John Tukey and the correlation coefficient by David Brillinger. The paper has many quotes (with refs), but trying to copy quotes here only results in chinese ... so not. Have a look at the paper! Tukey's reason for disliking correlations did not have ...


0

The implication seems to be that the grades are made into noisy predictors by the order of the grading, so what is the correlation between the "denoised" grades and the job prediction? But I don't quite know how one would model the way the grades receive that noise, and even if you do make such a model, I believe you need the actual data on job performance ...


3

Regression can't do everything rank correlation does. If you are talking about simple linear regression on the raw data then Regression makes assumptions that Spearman's does not. Regression results are in terms of the units, Spearman's is not. Regression posits that one variable is dependent and the other is independent. Spearman's does not. Regression ...


11

As a point for further elaboration, here is the explanation in the thread you reference: If it's mathematically possible, [this method] will find an $X_{Y_1,Y_2,\ldots,Y_k;\rho_1,\rho_2,\ldots,\rho_k}$ having specified correlations [between $X$ and] an entire set of $Y_i$. Just use ordinary least squares to take out the effects of all the $Y_i$ from $X$ ...


1

Suppose $x = 1, 2, 3, 4, 5$ and $y = 2x$ and so Pearson's $r$ is $1$. Is that invalid because the joint or marginal distributions aren't normal? The issue raised by non-normality is that P-values may be off. But if you want to assess linearity, stick with Pearson. If you are worried about non-normality, use bootstrapping or a permutation test. (If there is ...


2

A zip code would usually be treated as categorical, since there is (presumably) no meaning to the actual value and difference between the numbers, or ordering. The year of building would usually be numeric, since there is a meaning to the numbers themselves - 1990 is earlier than 1999, and (as I write this in 2020) a house built in 2010 is twice as old as ...


2

Suppose we have a linear regression model $$y=y_{12\ldots p-1}+\varepsilon_{12\ldots p-1}\,,$$ where $y_{12\ldots p-1}=\beta_0+\beta_1 x_1+\beta_2x_2+\cdots+\beta_{p-1}x_{p-1}$ is the part of $y$ explained by $(x_1,x_2,\ldots,x_{p-1})$ and $\varepsilon_{12\ldots p-1}$ is the unexplained part. Parameters $(\beta_0,\beta_1,\ldots,\beta_{p-1})$ are estimated ...


0

The easiest is to use a computer package such as SPSS or STATA to evaluate the significance, I don't think you can easily use tables for this. If you are a programmer you can use the permutation test (which I think is implemented in some computer packages). With observations $(x_1,y_1),\ldots(x_n,y_n)$, randomly generate $m$ permutations $(y_{(1)},\ldots,...


0

Typically when summarizing an distribution with a single number there is loss of information, after all you cannot recover the distribution from just that single number. The question is not if there is loss of information but whether the summary is sensible. The uncertainty coefficient (Theil's U) is a conditional measure: given one variable how well can ...


0

Let $Y_i=\sqrt iX_i$ for all $i$, which directly gives you $$\operatorname{Corr}(Y_i,Y_j)=\frac{\sqrt i\sqrt j\operatorname{Cov}(X_i,X_j)}{\sqrt{i\operatorname{Var}(X_i)}\sqrt{j\operatorname{Var}(X_j)}}=\operatorname{Corr}(X_i,X_j)\quad\forall\,i,j$$ This means the correlation matrix of $(Y_1,\ldots,Y_p)$ is identical to that of $(X_1,\ldots,X_p)$, whence ...


3

Before running an experiment, is it a good practice to plot a scatterchart between # of questions asked in 1 week and average time spent on stackoverflow? It is a very good idea to plot your data because it will give you an idea of what the association looks like. Visualisation is a great general tool in an analyst's chest, and where you have only 2 ...


0

As a guide to the analysis of correlation relationships in your database, I refer one to this comment on the proper use (or misuse) of Pearson’s correlation coefficient, to quote: If a parametric test of the correlation coefficient is being used, assumptions of bivariate normality and homogeneity of variances must be met. We give several examples where ...


2

There is nothing weird about the correlation. It answers one question only: how well could these data be summarized by a straight line? You get a bonus answer, the sign of the slope of that best fit line. The scatter plot makes it clear: no straight line could work well to summarize the entire relationship. The grouping of points is interesting and important ...


5

Because correlation tells you nothing about the magnitudes of variables, you can reverse their relative order by adjusting the magnitudes suitably. Here, for instance, is a scatterplot matrix of some $(Y, B_1, B_2)$ data: Clearly $Y$ is more highly correlated with $B_1$ than with $B_2.$ To help us appreciate the variation in magnitudes, here are the same ...


2

The purpose of the assignment is to see if higher earnings are correlated with a higher percent of university graduates. Correlation does not imply causation, so both scatter plots should be valid. We were instructed to plot the cause on the x-axis and the effect on the y-axis. I'm not particularly pleased with this instruction. As I've noted above, ...


0

What if you look at the partial explained variance. For example run first the intial regression. And then run a regression adding the effect of grading order. I guess you could say something about the change in explained variance


1

To say that the mean of a variable in one group differs from the mean of the variable in another group is to say that what group you're in has a relationship with the variable. For example, if men and women differed in average income, you would say there is a relationship between income and gender. So a test that compares the mean of the two groups is a ...


1

I'll assume that grades are measured on a 0-100 scales, and that there are no edge effects to be worried about (that is, the majority of students do not score near 0 or 100 in their classes). To investigate the effect (here, I should say effect or association since this is not a causal model, but I digress) of MCAT score on med school performance, you can ...


1

Since this is basic statistics, it should be covered in introductory handbooks. Wikipedia quotes one such book by Kenney and Keeping (1962), or just quote Wikipedia itself.


4

Sample correlation is measured for a dataset containing a set of pairs of variables, and it requires at least two pairs of variables to be well-defined (each pair is essentially a single datapoint, and we need at least two of these). Thus, it is possible to compute the sample correlation between counters and time for the entire dataset you have shown, but ...


6

The terms within-individual variance and among-individual variance are not commonly found in the mixed effects model literature. It more commonly arises in the ANOVA literature, and rather than "among", the usual term is "between". Total variance is partitioned into that which is attributable to differences within individuals, for example the natural ...


0

The following pieces of your model are fixed and known: $X_i$ and $Z_i$. The vector $\beta$ is fixed but unknown. You have two random pieces in your model, $b_i$ (I would expect this to be $b$, i.e. shared for all $i$, and I'm going to use $b$ below), and $\epsilon_i$. I suspect the setup of your problem also specifies that $b$ is independent of the $\...


0

Regression slope coefficients are, in essence, partialed correlation coefficients. So there is a strong relationship between the two. However, there is NOT a one-to-one between regression slope coefficients and un-partialed correlation coefficients--the latter can be very misleading, as they are not adjusted for the impact of other predictors. Correlation ...


1

Yes, you can use Kendall's tau (-b) for two binary variables. The result should be similar to determining the correlation with phi. The following is an example in R. if(!require(psych)){install.packages("psych")} set.seed(12345) ### remove set.seed for different samples A = rep(c("A","B"),1,each=100) X = factor(sample(A, 20)) Y = factor(sample(A, 20)...


2

I will throw my 2 cents as I had to battle with exactly the same problem for presenting results for a 77x77 variables correlation matrix. I tried just about anything you can think of in terms of conveniently visualizing a 77x77 matrix by using R, SPSS and Excel. From my experience, there is simply no magical pill/graph that will result in an 82x82 matrix (...


0

If you have enough data for each age-group you are interested in, "facets" may be a better way to convey your message than the first graph proposed by @Frans Rodenburg. Simulate data set.seed(1234) age <- round(rnorm(1000, 40, 10)) sex <- round(runif(1000, 0, 1)) bodyfat <- rnorm(1000, 10 + 5 * sex + age / 10, 3) bloodpressure <- rnorm(1000,...


Top 50 recent answers are included