New answers tagged

3

Putting aside the excellent points made elsewhere regarding how you define accuracy, whether it’s truly out of sample accuracy you’re testing etc etc, there’s also the point that multicollinearity isn’t necessarily a bad thing for prediction accuracy. It can be a nightmare for understanding predictions, eg in simple multivariate regression it can make it ...


2

I think the key here is to utilize the Sine function. One option is to simply calculate the correlation of the Sine of the angles. Another is to calculate the Circular Correlation Coefficient (cor.circular function from the circular library in R). This statistic is designed as a measure of correlation between two angular variables and relies on the Sine ...


0

Setting aside what Cross Validated tends to think of classifier metrics that depend on thresholds,$^{\dagger}$ I think I see your problem. We tends to care very little about machine learning performance on in-sample data. Adding more and more parameters allows us to play connect the dots (so to speak) and memorize the training data. What we care about is ...


0

Have you tried resampling? You can't calculate correlation coefficient for data that is not represented in pairs, so you could, for example, try this: 1. Take a random subsample of 1132 from $V_1$ and $V_3$ variables when correlating it with $V_2$, so that $V_2$ and $V_1$ or $V_2$ and $V_3$ have the same sample size. Note that samples are taken with ...


4

As is often the case, precisely formulating the question helped me work out the answer. My approach makes use of the marginal expectation of the bivariate normal: $$E_X(y) = E(X|Y=y) = \mu_x + \rho\frac{\sigma_x}{\sigma_y}(y-\mu_y)$$ Returning to my notation from the question above, this gives us: $$\begin{split}E(X_1Y_2) & = \int_{-\infty}^\infty \...


0

Old question, old answers. However, I'm using the Unicode character U+2AEB.


1

You can use Bayesian inference to solve your problem. If you answer the question "what is the proportion of Italian recipes?", you have a prior probability of Italian recipe $P(Italian)$. Also, you can get prior probability of having tomato in recipe $P(tomato)$. Then, you can calculate $P(Italian|tomato)$ as the proportion of Italian recipes among the ...


1

I think the right place to start is what correlation means in a nicer setting: Gaussian, of course. In the bivariate Gaussian distribution, the population correlation is given by the $\rho$ parameter. This tells us something about the fit to a line. I have tried fitting marginal uniform distributions to a line and failed, but we can still get at the $\rho$ ...


0

You may fit a statistical model, a random forest, or even a neural network... but best neural networks around are still those wired into your brain. That's why the way to represent relations among variables is plotting. If $y$ is categorical, you can do a two way scatter plot between $x_1$ and $x_2$ using different symbols or colors for data points of ...


0

You could use OLS here. It might also depend on your use case. If you're just explaining the relationship to a colleague, you might use vanilla OLS. If the emphasis is on more accurate predictions, perhaps consider using random forest regression.


4

That is, for the set of predicted values $\{\hat{Y}_1, \hat{Y}_2, ...\}$ and the set of original values $\{Y_1, Y_2, ...\}$, the means of the sets are always equal. The difference between the predicted values and the original values are the residuals $$\hat{Y}_i = Y_i + r_i$$ So you can write $$\begin{array}{} \frac{1}{n} \left(\hat{Y}_1+ \hat{Y}_2+ ...\...


4

In matrix notation, the fitted values can be written as $\hat y=Py$, with the projection matrix $P=X(X'X)^{-1}X'$, wich can be verified by plugging in the definition of the OLS estimator into the formula for the fitted values, $\hat y =X\hat\beta$. Their mean is, with $\iota$ a vector of ones, $$ \iota'Py/n, $$ as the inner product with $\iota$ just sums up ...


1

It's intuitively clear. If you have the correct model as the linear regression, the residuals should be distributed with mean zero. If you take the average on the residuals, you are left only with the predicted values. For example, if your model is $y = c + ax + \epsilon$, where $c$ is constant vector, $a$ is coefficient vector, $x$ is the feature ...


0

That is correct, when your target variable is binary you should use logistic regression (or logit), a linear model for classification instead of linear regression. There are many other models of course but that is a start. You can check an example here: https://stats.idre.ucla.edu/r/dae/logit-regression/.


0

As a practical matter, all of these are de facto 0, even the largest is 0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000059 (5.9e-120 means zero point hundert and twenty zeros five nine). So you cannot use them to distinguish one from the other. More fundamentally, a $p$-value is not ...


0

Yes, nominal predictors can be employed in even standard regression models. Now, as you asked the question: "I would like to know which one of predictors have more influence on the predicted variable", I would simply recommend that you focus on 'Standardized Beta Coefficients'. The standardized coefficients are computed by centering the data (subtracting ...


2

Nomimal predictors are no problem for regression. In most software, the nominal predictors are transformed to dummy variables and their effects are estimated individually. Provided you have good reason to believe that your data conditioned on your variables is normal, then Bayesian regression sounds fine to me. Here is an example of how you might perform ...


0

Thanks for all the answers, after searching through literature I found out, indeed there is a systematic bias in estimating the autocorrelation of time-series with a finite-size. There is a series of old statistics papers discussing this bias and even they derived its analytical form for some cases like: Marriott, F. H. C., and J. A. Pope. "Bias in the ...


2

One idea is to do a jittered scatterplot. I simulated 100 Likert scores x from 1 to 7, and roughly positively associated Likert scores y. They are summarized and tabulated below. summary(x) Min. 1st Qu. Median Mean 3rd Qu. Max. 1.00 3.00 5.00 4.48 6.00 7.00 table(x) x 1 2 3 4 5 6 7 5 8 13 22 24 16 12 summary(y) ...


0

The univariate Pearson correlations between y and X variables is often useless. There are many reasons to it such as confounding, nonlinearity etc. It does have some information though but often very limited


0

Given the visible relationship between the variables represented by the blue curve (y1) and the red curve (y2), you will find a positive value if you calculate the correlation. Linear correlation does not appear to be the best measure for the relationship between the variables y1 and y2. It only measures the linear relationship between them. Something else ...


2

Generally speaking, Pearson correlation measures linear relationship, so naturally nonlinear relationships may not be captured by it. However, having a nonlinear looking time series doesn't mean that you can't calculate Pearson correlation. For example, the two series can be exact multiples of each other, however nonlinear they may look when plotted with ...


0

I have done simulations of linear regression with various levels of multicollinearity and to my great surprise, it seemed to have very little effect. I had a professor tell me that don't even worry about it unless you have > .8 or maybe even > .9 correlation. That turned out to be pretty true in general in my simulations. If I had > .9 I might pick one of ...


4

One of the assumptions of standard OLS regression is that the regressors are not correlated Very very wrong! That's not an assumption of a regression at all. The regressors are almost always correlated unless constructed in a very specific way. You don't want a perfect multicollinarity, which means that they are Pearson correlated 100%. This is undesirable,...


1

Pairwise correlations are not reliable indicators of collinearity in multiple regression but they are useful for knowing the appropriate sign (pos/neg) of that relationship. Wrong-signed variables are a useful diagnostic for the presence of collinearity. Partial correlation matrices are also useful but VIFs and collinearity indexes provided by many software ...


0

We cannot tell what the authors did without reading the paper and I suspect from bitter experience we will be no further forward if we do have it. A confidence interval has to cover a range of values so the upper limit will be greater than the lower limit and in general the point estimate will fall within that range although there are rare circumstances ...


2

Data: I have put your data ($\pm$ typing errors) into R. x = c(1,2,3,3, 4,5,6,3, 10,11,12,19, 20,21,22) y = c(20.5,25.3,29.3,26.0, 32.8,35.2,41.2,26.0, 46.7,68.2,62.8,81.6, 80.4,63.5,100.9) Data summaries: summary(x); sd(x) Min. 1st Qu. Median Mean 3rd Qu. Max. 1.000 3.000 6.000 9.467 15.500 22.000 [1] 7.633261 # SD x ...


1

To quote a source on this topic of correlation relating to dummy variables: A popular approach for dichotomous variables (i.e. variables with only two categories) is built on the chi-squared distribution. We are not interested in testing the statistical significance however, we are more interested in effect size and specifically in the strength of ...


0

Interesting question. Sounds at first like a case calling for mediation, but on closer inspection, it's not. Nor is it a typical moderation (interaction) situation. You could assess to what extent AC and BC are different at different levels of D. You'd have to divide D into categories -- with an overall N of 450, perhaps 3-6 categories would work well. ...


0

Does correlation (any statistical association) correlate with (related to) causation? This question (words in parentheses are mine) is quite general and essentially the answer seem me yes. Causal inference in statistics framework exist exactly because the answer to that question is yes. However in order to give more useful answer we need to better ...


0

Correlation is a special type of association and association is different from causation which can be only inferred from a randomized experiment(the reason would be the confounder). References: 1. Association and Correlation 2. Openintro-Statistics


1

You can test for association between the two columns, so you build a contingency table like: tab = pd.crosstab(testframe['feat1'],testframe['feat2']) tab feat2 0 1 feat1 0 4 0 1 2 4 If there is correlation between you two variables, the counts will populate the diagonal. Like @whuber suggested, you can first try chi-square test: ...


4

Given the intended application, you might be interested in creating realistic modifications of the series of data. Overview This is easier to do than you might think. (See the three-line function decorrelate in the code below.) The idea is to generate the "noise" series realistically, according to any model you like. (You can even use actual data, such ...


2

First, since OP's problem is with relation to stock and index returns correlation, I must note that one thing to be aware of is that if the stock, such as AAPL, is a part of index such as SPX, then the problem is a little more complex, because any change to AAPL will spill into SPX index. So, let's assume that the stock in question is not a constituent of ...


1

So if $\rho_N$ is the desired Pearson correlation, $\rho_0$ is the current correlation with $\rho_N\le \rho_0$, and $\epsilon$ is our uncorrelated noise: $\frac{cov(X+\epsilon,Y)}{\sqrt{var(X+\epsilon)\sigma^2_Y}}=\rho_N$ Solving for variance of $\epsilon$ $\sigma^2_\epsilon=\frac{cov(X,Y)^2}{\rho_n^2\sigma_Y^2}-\sigma^2_X=\sigma_X^2\bigg(\frac{\rho_0^2}{\...


1

One can always come up with examples just as you did. However, such obviously irrelevant variables are not really what Reichenbach has in mind with his dictum. You could preface the Reichenbach principle with this qualifier: among variables that could reasonably be connected, if they are correlated, then there is a causal relationship linking them. This ...


0

This sounds like a mixed model situation. An example is shown in the Case Studies for SPSS Statistics at Using Linear Mixed Models to Fit a Random Coefficients Model. This model fits a common fixed slope and intercept across subjects, as well as random slopes and intercepts for individual subjects to see how they depart from the common values. To show the ...


1

Let $X=(X_1, \dotsc, X_n)^T$ which has covariance matrix $\Sigma=\sigma^2 R$ where the correlation matrix $R$ has diagonal elements $R_{ii}=1$ and off-diagonals $R_{ij}=a, i\not = j$ (here $-\frac1{n-1}\le a \le 1$, see Intuition for near-decorrelation through centering.) Then $\bar{X}=\frac1n 1^T X$ (where $1$ is a column vector of all ones.) The rest is ...


1

This question can also be answered from a linear algebra perspective. Say you have a bunch of data points $(x,y)$. We want to find the line $y=mx+b$ that's closest to all our points (the regression line). As an example, say we have the points $(1,2),(2,4.5),(3,6),(4,7)$. We can look at this as a simultaneous equation problem: \begin{align} & \...


5

This is simply a case where dec is a function of num ---i.e., the value of dec is fully determined by the value of num. That is all it is called --- a function. Functions of random variables are often correlated with the initial random variables, so this is not an unusual situation. The correlation indicates that the two variables are (statistically) ...


2

As Bernhard mentioned, correlation does not have a "from - to" concept. It describes the relationship between to variables. Another useful idea to think about is that if we change (or filter on) one variable, how would another variable change. Think about the relationship between human height and weight, if we focus on tall population, it is very likely ...


7

Correlation does not have a "from" and a "to". It is invariant $Cor(A, B) = Cor(B, A)$. The terms "from" and "to" can make sense in the context of regression, where we speak of "independent" and "dependent" variables or "predictor" and "predicted". Pearson correlation is closely related to linear regression. In Linear Regression again, the first order of a ...


0

You already know how to atribute the summed, yearly eletric use to the real building ; you are only interested in the hourly pattern of electric usage. So I assume you can standardize each hourly electric usage so that it becomes a distribution (i.e. which percentage of the electric usage was used before 8am ; 5pm ; 12pm). Intuitively: You may choose a ...


0

I would like to comment on two issues here. First, we may ask "how many" components should keep instead of "what components". This because PCA is doing a linear transformation of the original feature space. So after transformation, we do not have the original components and we can only select how many to keep, or what to keep in terms of first, second ...


1

If you addressed the multicollinearity issue by mean centering the data, then no issue with short-range prediction. With longer horizons, however, the question is do the historical means, to which you are adding the expected change per the regression model, remain valid? A lesser issue if your interest is in percent change forecast and your variables are ...


3

Like you quoted, multicollinearlity presents a much bigger problem for inference than prediction. Prediction is mostly fine under multicollinearity as long as the variables stay collinear (although if you get close to perfect collinearity, you can get some precision issues with regards to things like matrix inversion). But there is the issue that we don't ...


0

Let's take your second formulation: $$K(h;\theta)=exp(-\theta h^2)$$ The overall intuition is that $h$ is represents the distance between two points. It is always non-negative. As $\theta$ multiplies $h$, the larger $\theta$ is the more negative $-\theta h^2$ becomes. The more negative the exponent becomes, the smaller $K$ becomes. Therefore, ...


0

There is no one-size-fits-all solution for this. The threshold could be judged by the researcher based on the association between the variables. For the high correlation issue, you could basically test the collinearity of the variables to decide whether to keep or drop variables (features). You could check Farrar-Glauber test (F-G test) for ...


2

Consider the case of a simple linear regression using panel data. So, for example, assume we are trying to figure out how age and education affect individuals' income. Also, forget the whole log-linear regression part. To keep this simple, I'm just going to use a simple linear regression. In essence, I want to find the coefficients for the following equation:...


1

Here's something to get you started: First creating some random data that is similar in shape to yours library(geosphere) library(ggplot2) stations <- data.frame(name=LETTERS[1:10], long=rgamma(10, shape=0.1), lat=rgamma(10, shape=0.1)) #name long lat #1 A 1.259977e-07 1.131207e-04 #2 B 6.968209e-03 1.680159e-07 #3 C 2....


Top 50 recent answers are included