New answers tagged statistical-significance
1
I will try to give an answer anyway (I looked at the image and you need the sample standard error of the mean):
The standard error is the standard deviation of the sampling
distribution of a statistic. Wikipedia
So
$$
SE_\bar{x}\ = \frac{s}{\sqrt{n}}
$$
and
$$
S = \sqrt{S^2} := \sqrt{\frac{1}{n-1} \sum_{i=1}^n{(X_i-\bar{X})^2}}
$$
So regarding the ...
0
Setting your stopping condition based on the significance of your interim analyses is, in general, not a great idea. The worst possible thing you could do would be to re-run your analysis after every page view and stop as soon as you got a significant result. You've decided, by setting your $\alpha=0.05$, that you're willing to tolerate a 5 percent chance of ...
0
I'm not sure I would use the term bias here.
Following the procedure you suggest, you are accepting a risk of inappropriately selecting the green button on inadequate evidence.
To quantify that risk, you would need to calculate the probability of an unfavorable outcome (presumably Blue > Green) of an n=7663 test GIVEN (Bayes Rule) the observed outcome at ...
1
First you need to decide whether to treat the responses as if they are at the interval or ordinal level of measurement. "Interval" would assume that the difference between 1 and 2 is the same as the difference between 2 and 3 (and so on all the way to the difference between 9 and 10). "Ordinal" would relax that assumption.
With data treated as interval, ...
5
In addition to @gung's answer, I'll try to provide an example of what the anova function actually tests. I hope this enables you to decide what tests are appropriate for the hypotheses you are interested in testing.
Let's assume that you have an outcome $y$ and 3 predictor variables: $x_{1}$, $x_{2}$, and $x_{3}$. Now, if your logistic regression model ...
3
What exactly are you trying to find out? One way to approach this would be to look at the problem differently: Simply add a predictor (0 for regular observations, 1 for observations you would remove) and estimate/test the interaction between this new predictor and the other term of interest.
1
And do you have reason to believe that this data would show positive or negative autocorrelation? i.e. do you need to do a one or two sided test? Often a two sided DW test is simply carried out as two one sided tests and the $\alpha$ (type I error) for the two sided is simply doubled. So for example for the test:
$H_o: \rho = 0$
$H_a: \rho \neq 0$
...
0
I would use a Wilcoxon signed-rank test, separately for each question.
The paired t-test is appropriate if either 1) the data are distributed normally or 2) the sample size is large enough that the central limit theorem plausibly applies. I don't think either of these things is true here. 30 might be big enough for the CLT to apply or it might not.
I ...
2
Not exactly an answer but perhaps can help start a discussion.
To me the design of the questions seems problematic. The pre-question measures "determination" and the post-question measures "experience." Someone may be too anxious to decide to be focused, but then experienced a very focused session. I am not sure what you can get out of that. It feels like ...
0
Based on the suggestions proposed by Glen_B and COOLSerdash I did an ordinal logistic regression using 375 instances each comprised of 10 binary features and the results are as followes.
values |std error| T-Value | P-Value | 2.5% | 97.5% |
-3.40728 | 0.0750 | -45.4461| 0 | -3.55 | -3.26 |
-2.61808 | 0.0590 | -44.3806| 0 | -2.73 | -2.50 |
...
0
The only correct number is .04284731
...which is a flippant response intended to mean that the choice of .05 is essentially arbitrary. I usually just report the p value, rather than what the p value is greater or less than.
"Significance" is a continuous variable, and, in my opinion, discretizing it often does more harm than good. I mean, if p=.13, ...
1
Seems to me the answer is more in the game theory of research than in the statistics. Having 1% and 5% burned into the general consciousness means that researchers aren't effectively free to choose significance levels that suit their predispositions. Say we saw a paper with a p-value of .055 and where the significance level had been set at 6% - questions ...
0
This result does not mean that the different blocks are the same on the dependent variable, only that there is not sufficient evidence to reject the notion; more precisely, it says that, in a sample of this size, an effect of this size would have arisen more than 5 percent of the time if, in the population from which the sample was drawn, there really was no ...
3
390 is a very decent sample size. Since you haven't posted any details of the results, all we can do is list some possible reasons:
1) Your model is wrong - the relationship you tested is too weak to detect
2) Your model is wrong - you have left out variables that are important
3) Your model is wrong - you have violated the assumptions in some way that ...
2
Here's a really basic design for $s=4$ systems. Create a factorial set of trials:
$m$ different inputs
$p=6 = (s-1)(s)/2= 3(4)/2$ comparisons (i.e., System A with B, A with C, A with D, B with C, B with D, C with D)
Create $pm$ trials and present in random order to $n$ participants.
For each participant, record the proportion of the time that system A, ...
3
You can set any significance level you like before you start looking at the data. The higher you choose to set it, the greater the rate of Type I errors you're prepared to accept (that is, the more frequently you say you'd be prepared to claim a real effect in the presence of nothing but noise)
If you choose a high significance level, even before the fact, ...
7
$\alpha$ and $\beta$ are related. I'll try to illustrate the point with a diagnostic test. Let's say that you have a diagnostic test that measures the level of a blood marker. It is known that people having a certain disease have lower levels of this marker compared to healthy people. It is immediately clear that you have to decide a cutoff value, below ...
1
There is no general relation between alpha and beta.
It all depends on your test, take the simple exemple:
(Wikipedia)
In colloquial usage type I error can be thought of as "convicting an innocent person" and type II error "letting a guilty person go free".
A jury can be severe: no type II error, some type I
A jury can be "kind": no type I but some type ...
2
Likert scaled items are probably about as good as you are going to get for somewhat amorphous questions like these - it's not like you can measure these on an interval scale.
To correlate ordinally scaled variables you can use Spearman's rank correlation.
1
You can use a likelihood ratio test (LRT) to test whether a random effect is significant. First, fit the random effects model. Then fit the model without the random effect. Extract and store the log-likelihood for each model using logLik and calculate the twice difference between the log-likelhood of the mixed effects model and the normal model. Use a ...
1
@Glen_b examples showed that a strong correlation is not strictly impossible in this situation but I think your intuition is right. Limited variation (for example range restriction) can bias sample correlations toward 0. The problem is not merely one of “significance”, the correlation also appears systematically smaller than it would be if you would consider ...
1
It's possible to get high point biserial correlation even with 27 $1$'s and a $0$. Indeed, you can get as high as 1, so it's not that:
y <- c(0,rep(1,27))
x <- y
cor(x,y)
[1] 1
-- and making x continuously distributed doesn't substantively alter that conclusion:
y <- c(0,rep(1,27))
x=c(rnorm(1,0),rnorm(27,100))
cor(x,y)
[1] 0.9987537
...
6
Are you interested in being able to say that one of the percentages is greater than the other?
In the cases you want to do it, do they always add to 100%?
In that case, it's easy - you compare one of the percentages to 50%; if it's bigger than 50% the complementary one is smaller than 50%.
If you want to compare two proportions where there are other ...
2
The "statistical test" your teacher is referring to would be a binomial test. It is an exact test, meaning that it yields the exact probability (p value) of your observed proportion (or a more extreme one) occurring under the null hypothesis. In this case, your null hypothesis appears to be that the proportion of people who pick category A (or alternatively, ...
1
I think you are interested in equivalence testing. See this other question on testing a hypothesis of no group differences.
There are various approaches that can be adopted to assess whether the null hypothesis is true. In general, the absence of statistically significant effect is very week evidence for the truth of the null hypothesis.
Three common ...
3
If you are trying to relate number of clicks or number of impressions by group, I suggest a count regression model (Poisson or negative binomial, for instance) with "test #" as a categorical independent variable.
2
The size of the correlation coefficient doesn't tell you something about the significance of the effect. Look at the Wiki page about the Fisher transformation. As Jake Westfall pointed out, only the correlation coefficient $r$ and the sample size $n$ are used to calculate the standard error $SE=1/\sqrt{n-3}$ and the $z$-value which is then used to calculate ...
3
It is the sample size. For a simple correlation between two variables, the coefficient and sample size are the only pieces of information that you need in order to compute the standard error, and hence assess statistical significance. So if the coefficient does not differ significantly from zero despite being a canonical "medium" sized effect, then the only ...
0
(Initially written as a comment but it was becoming too long, I can elaborate if there is interest)
It seems that there are two errors in the original Excel file that add to the confusion. What it calls the p-value is in fact the value of the cumulative distribution function. It then uses it to perform two one-tailed tests at the specified error rate, ...
3
In one case it looks like the variant made the value go up relative to the control (245 vs. 217) and the other time, it went down relative to the control (205 vs. 237). Since the calc for z-score is:
$Z = \frac{x-\mu}{\sigma}$
In the first test you got a positive z-score, and in the second you got a negative z-score.
In both cases the answer is telling ...
0
In the first case the variant gives you a higher proportion of sign-ups than the control; in the second case it give you a lower proportion: so significance is being assessed in different directions.
7
Statistically significant doesn't imply large $R^2$; with large $n$ even very tiny effects are distinguishable from chance; statistical significance is not practical significance.
As for significance, when you only have a single predictor, either the p-value next to the variable (dChgs.nz) or the p-value for the $F$ for the overall regression - they're the ...
6
The Multiple $R^2$ is the square of the correlation between the response and the fitted values. It tells you how much of the total variance is explained by the model's prediction. The $R^2$ doesn't tell you whether the model is significant or not. Of course: if you want a good prediction model, your aim is to get a high $R^2$. In this case, your model ...
1
You could do a permutation/randomization test.
Possibly Wilcoxon rank sum will answer your question as well, although permutation test is probably closer to what you want.
In R there is perm.test in the exactRankTests package that seems made for your problem.
5
While agreeing with @gung 's comment above, it might be possible to point you to some general ideas, please bear in mind that these aren't complete answers. A good book will help.
The t-values are another effect size measure (like the coefficient) but they are on a standard scale, so that, according to some people you can compare them across variables, ...
0
If I understand your statement of the problem, the resulting Earth Movers Distance (EMD) has a formal probabilistic interpretation referred to as the Mallows Distance (among other names). See the reference:
E. Levina and P. Bickel, “The earth mover's distance is the Mallows distance: some insights from statistics,” Computer Vision, 2001. ICCV 2001. ...
2
Your interpretation of a low p-value is a very common one, but it is not quite correct. The phrase "is unlikely to be equal" has no meaning in a Frequentist context. Either all of the population means are equal, or they are not. To the extent that something like a statement of probability can be made in this situation, the correct statement would be ...
2
The correlation coefficient is what it is - basically an effect size measure - & any rules of thumb about what's 'small' or 'weak' ignore the context of what real things the variables are measuring. You can test for its statistical significance but its practical/theoretical significance is for a subject-matter expert to determine.
(Spearman's is a ...
0
Do you mean something like this?:
in R:
R <- matrix(c(1.0, 0.1, 0.1,
0.1, 1.0, 0.1,
0.1, 0.1, 1.0), nrow=3)
N <- 100
chi <- -2*log(det(R)^(N/2))
df <- nrow(R)*(nrow(R)-1)/2
p <- 1 - pchisq(chi, df)
chi
p
Or in Excel:
Where the matrix is in cells C26:E28, and N is 100:
=-2*(LN(MDETERM(C26:E28)^100/2))
And the ...
Top 50 recent answers are included
