Hot answers tagged statistical-significance
7
Statistically significant doesn't imply large $R^2$; with large $n$ even very tiny effects are distinguishable from chance; statistical significance is not practical significance.
As for significance, when you only have a single predictor, either the p-value next to the variable (dChgs.nz) or the p-value for the $F$ for the overall regression - they're the ...
6
$\alpha$ and $\beta$ are related. I'll try to illustrate the point with a diagnostic test. Let's say that you have a diagnostic test that measures the level of a blood marker. It is known that people having a certain disease have lower levels of this marker compared to healthy people. It is immediately clear that you have to decide a cutoff value, below ...
6
The Multiple $R^2$ is the square of the correlation between the response and the fitted values. It tells you how much of the total variance is explained by the model's prediction. The $R^2$ doesn't tell you whether the model is significant or not. Of course: if you want a good prediction model, your aim is to get a high $R^2$. In this case, your model ...
6
Are you interested in being able to say that one of the percentages is greater than the other?
In the cases you want to do it, do they always add to 100%?
In that case, it's easy - you compare one of the percentages to 50%; if it's bigger than 50% the complementary one is smaller than 50%.
If you want to compare two proportions where there are other ...
5
While agreeing with @gung 's comment above, it might be possible to point you to some general ideas, please bear in mind that these aren't complete answers. A good book will help.
The t-values are another effect size measure (like the coefficient) but they are on a standard scale, so that, according to some people you can compare them across variables, ...
3
390 is a very decent sample size. Since you haven't posted any details of the results, all we can do is list some possible reasons:
1) Your model is wrong - the relationship you tested is too weak to detect
2) Your model is wrong - you have left out variables that are important
3) Your model is wrong - you have violated the assumptions in some way that ...
3
You can set any significance level you like before you start looking at the data. The higher you choose to set it, the greater the rate of Type I errors you're prepared to accept (that is, the more frequently you say you'd be prepared to claim a real effect in the presence of nothing but noise)
If you choose a high significance level, even before the fact, ...
3
It is the sample size. For a simple correlation between two variables, the coefficient and sample size are the only pieces of information that you need in order to compute the standard error, and hence assess statistical significance. So if the coefficient does not differ significantly from zero despite being a canonical "medium" sized effect, then the only ...
3
In one case it looks like the variant made the value go up relative to the control (245 vs. 217) and the other time, it went down relative to the control (205 vs. 237). Since the calc for z-score is:
$Z = \frac{x-\mu}{\sigma}$
In the first test you got a positive z-score, and in the second you got a negative z-score.
In both cases the answer is telling ...
2
Intuitively, if the sample proportions exactly matched the proposed proportions, $\chi^2$ would be 0. But there will be random fluctuation.
e.g.
set.seed(239920)
(testmult <- rmultinom(1, 100, c(0.65, 0.27, 0.01, 0.05, 0.02)))
does not yield exactly 65, 27, 1, 5, 2.
On repeated draws, the expected value of $\frac{(O-E)^2}{E}$ will be the degrees ...
2
When performing hypothesis tests, the p-value is the probability (under the null) of a result as, or more extreme than the test statistic you observed.
Here 'more extreme' means 'more consistent with the alternative'.
So which parts of the null-distribution of the test statistic you're interested in depends on your alternative. In particular (for an ...
2
Your interpretation of a low p-value is a very common one, but it is not quite correct. The phrase "is unlikely to be equal" has no meaning in a Frequentist context. Either all of the population means are equal, or they are not. To the extent that something like a statement of probability can be made in this situation, the correct statement would be ...
2
The correlation coefficient is what it is - basically an effect size measure - & any rules of thumb about what's 'small' or 'weak' ignore the context of what real things the variables are measuring. You can test for its statistical significance but its practical/theoretical significance is for a subject-matter expert to determine.
(Spearman's is a ...
2
The "statistical test" your teacher is referring to would be a binomial test. It is an exact test, meaning that it yields the exact probability (p value) of your observed proportion (or a more extreme one) occurring under the null hypothesis. In this case, your null hypothesis appears to be that the proportion of people who pick category A (or alternatively, ...
2
The size of the correlation coefficient doesn't tell you something about the significance of the effect. Look at the Wiki page about the Fisher transformation. As Jake Westfall pointed out, only the correlation coefficient $r$ and the sample size $n$ are used to calculate the standard error $SE=1/\sqrt{n-3}$ and the $z$-value which is then used to calculate ...
2
This approach doesn't have the properties you would have if you fixed the sample size ahead of time.
The situation where you look for a particular result while your experiment continues and have some 'stopping rule' (halt your experiment early if a particular situation is achieved) is a version of sequential analysis; see also SPRT.
You have to take care ...
2
Here's a really basic design for $s=4$ systems. Create a factorial set of trials:
$m$ different inputs
$p=6 = (s-1)(s)/2= 3(4)/2$ comparisons (i.e., System A with B, A with C, A with D, B with C, B with D, C with D)
Create $pm$ trials and present in random order to $n$ participants.
For each participant, record the proportion of the time that system A, ...
1
There is no general relation between alpha and beta.
It all depends on your test, take the simple exemple:
(Wikipedia)
In colloquial usage type I error can be thought of as "convicting an innocent person" and type II error "letting a guilty person go free".
A jury can be severe: no type II error, some type I
A jury can be "kind": no type I but some type ...
1
You can use a likelihood ratio test (LRT) to test whether a random effect is significant. First, fit the random effects model. Then fit the model without the random effect. Extract and store the log-likelihood for each model using logLik and calculate the twice difference between the log-likelhood of the mixed effects model and the normal model. Use a ...
1
@Glen_b examples showed that a strong correlation is not strictly impossible in this situation but I think your intuition is right. Limited variation (for example range restriction) can bias sample correlations toward 0. The problem is not merely one of “significance”, the correlation also appears systematically smaller than it would be if you would consider ...
1
It's possible to get high point biserial correlation even with 27 $1$'s and a $0$. Indeed, you can get as high as 1, so it's not that:
y <- c(0,rep(1,27))
x <- y
cor(x,y)
[1] 1
-- and making x continuously distributed doesn't substantively alter that conclusion:
y <- c(0,rep(1,27))
x=c(rnorm(1,0),rnorm(27,100))
cor(x,y)
[1] 0.9987537
...
1
You could do a permutation/randomization test.
Possibly Wilcoxon rank sum will answer your question as well, although permutation test is probably closer to what you want.
In R there is perm.test in the exactRankTests package that seems made for your problem.
1
I am not sure why you were looking at correlations and similar measures. There doesn't seem to be anything to correlate.
Instead, there are a number of options, none really better than the other, but depending on what you want:
Take the average rank and then rank the averages (but this treats the data as interval)
Take the median rank and then rank the ...
1
Pearson's correlation is usually what is intended when the word "correlation" is used without qualification.
That said, a good starting point is to look at a scatterplot of your two variables. There are plenty of options for how best to assess the negative relationship implied by your question. Depending on distributions you might want to transform your ...
1
The null hypothesis for the F-test in ANOVAR, say, is that all the group means are equal. The alternative is that any are unequal. So by including more groups you could be reducing the power of the F-test* - it becomes harder to detect non-zero effects if you include more groups with zero effects - but not reducing its size - the significance level is ...
1
Seems to me the answer is more in the game theory of research than in the statistics. Having 1% and 5% burned into the general consciousness means that researchers aren't effectively free to choose significance levels that suit their predispositions. Say we saw a paper with a p-value of .055 and where the significance level had been set at 6% - questions ...
1
I think you are interested in equivalence testing. See this other question on testing a hypothesis of no group differences.
There are various approaches that can be adopted to assess whether the null hypothesis is true. In general, the absence of statistically significant effect is very week evidence for the truth of the null hypothesis.
Three common ...
Only top voted, non community-wiki answers of a minimum length are eligible
