Statistical significance refers to the probability that, if, in the population from which this sample were drawn the true effect were 0 (or some hypothesized value) a test statistic as extreme or more extreme than the one gotten in the sample could have occurred.
3
votes
1answer
2k views
How can I calculate margin of error in a NPS (Net Promoter Score) result?
I'll let Wikipedia explain how NPS is calculated:
The Net Promoter Score is obtained by asking customers a single
question on a 0 to 10 rating scale, where 10 is "extremely likely" and
0 is ...
2
votes
1answer
1k views
How to test the statistical significance for categorical variable in linear regression?
If in a linear regression I have categorical variable... how do I know the stastical signifance of the categorical variable?
Let's say the factor $X_1$ has 10 levels... there will be 10 different ...
12
votes
1answer
41k views
How to interpret and report eta squared / partial eta squared in statistically significant and non-significant analyses?
I have data that has eta squared values and partial eta squared values calculated as a measure of effect size for group mean differences.
What is the difference between eta squared and partial eta ...
6
votes
1answer
2k views
Cross-correlation significance in R
How do you tell if the correlations at different lags obtained from the cross-correlation (ccf function) of two time series are significant.
33
votes
2answers
1k views
What are good references containing arguments against null hypothesis significance testing?
In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...
15
votes
2answers
2k views
Do we need a global test before post hoc tests?
I often hear that post hoc tests after an ANOVA can only be used if the ANOVA itself was significant. However, post hoc tests adjust p values to keep the global type I error rate at 5%, don't they? So ...
4
votes
4answers
318 views
Checking whether accuracy improvement is significant
Suppose I have an algorithm that classifies things into two categories. I can measure the accuracy of the algorithm on say 1000 test things -- suppose 80% of the things are classified correctly.
Lets ...
3
votes
2answers
5k views
Threshold for correlation coefficient to indicate statistical significance of a correlation in a correlation matrix
I have computed a correlation matrix of a data set which contains 455 data points, each data point containing 14 characteristics. So the dimension of the correlation matrix is 14 x 14.
I was ...
2
votes
6answers
398 views
How to assess effect of intervention in one state versus another using annual case fatality rate?
I am a beginner in statistics with just basic knowledge. I have these data: cases, deaths and CFR (Case Fatality Rate-deaths per 100 cases) of a disease for 17 years (1994-2010) from 2 neighbouring ...
0
votes
2answers
1k views
How can you have a non-significant multiple regression model w/ significant predictors? [duplicate]
Possible Duplicate:
Not-significant F but a significant coefficient in multiple linear regression
How can a regression be significant yet all predictors be non-significant?
Significance of ...
2
votes
3answers
676 views
Determining sample size
If I have a population and want to draw samples from there, what are the steps to determine the number of samples to draw from the population? In other words, if I have population N and want to draw k ...
6
votes
2answers
2k views
Significance test on the difference of Spearman's correlation coefficient
(Thanks a lot for the quick responses! I did a poor job of asking the question, so let me retry.)
I do not know how to find out whether or not the difference between two Spearman's correlations is ...
8
votes
2answers
1k views
Do you reject the null hypothesis when $p < \alpha$ or $p \leq \alpha$?
This is clearly just a matter of definition or convention, and of almost no practical importance. If $\alpha$ is set to its traditional value of 0.05, is a $p$ value of 0.0500000000000... considered ...
4
votes
2answers
530 views
How to test for significant differences in means across four within subject conditions where dependent variable is a proportion?
I have a dataset with four within-subject conditions. Participants gave binary answers to 8 questions in each condition, and these were averaged to form a proportion for each condition. I would like ...
3
votes
2answers
108 views
What test should I use to determine if a policy change had a statistically significant impact on website registrations?
A client's website was operating under a certain policy for membership sign ups for over a year. At the start of October 2012 the client implemented a new policy for sign ups that was supposed to ...
2
votes
2answers
1k views
Comparing two classifier accuracy results for statistical significance with t-test
I want to compare the accuracy of two classifiers for statistical significance. Both classifiers are run on the same data set. This leads me to believe I should be using a one sample t-test from what ...
0
votes
2answers
254 views
Significance tests
I have two sample means from two different samples. I purport that the true population means are identical.
$H_{0}$: $\mu_{1}=\mu_{2}$ and
$H_{1}$: $\mu_{1}\ne\mu_{2}$
What significance test would I ...
14
votes
4answers
3k views
Correcting p values for multiple tests where tests are correlated (genetics)
I have p values from a lot of tests and would like to know whether there is actually something significant after correcting for multiple testing. The complication: my tests are not independent. The ...
6
votes
3answers
4k views
How to test group differences on a five point variable?
I have a series of observations that fall into bins (or "scores"); that is, the data can be 0, 1, 2, 3 or 4. There are two groups of such data, control and treated. I know the number of individuals ...
5
votes
4answers
871 views
How to interpret the margin of error in a poll?
Recently the media reported on a political poll that stated that "46% of Republican voters in Mississippi think that interracial marriage should be illegal". One example story (of many around the ...
4
votes
3answers
850 views
Calculating the $p$-value of an $F$- statistic
I am trying to implement an algorithm for calculating $p$-values of $F$-tests and I need this method to be highly precise. It is easy to implement this with $z$- or $t$- tables, however I don't know ...
3
votes
4answers
316 views
Comparing smoke and hormones level in two groups of people. Which test?
I'm totally not a pro in statistics so i hope the question is not too simple!
I have two groups of patients. One group are smokers, the second are non-smokers.
I want to compare the level of some ...
3
votes
6answers
839 views
How to assess whether a coin tossed 900 times and comes up heads 490 times is biased?
A coin is tossed 900 times and heads appeared 490 times. Does the result support the hypothesis that the coin is unbiased?
2
votes
1answer
275 views
Significance test for differences in baseline and remeasurement rates between two groups
I need help in knowing which test to perform with this analysis.
I have two groups (lets say group A and B). A study was performed with initial rates of heart failures in group A and B (i.e, ...
1
vote
1answer
408 views
Statistical significance of a survey/poll?
Let's pretend I'm conducting a poll/survey. It's a simple yes/no poll (i.e. everyone only gives 1 of 2 answers). I have asked N people so far, and X of them have said "yes".
I would like to stop ...
0
votes
2answers
550 views
How to test for equality of means for proportions? And two-sided or one-sided?
I have the following problem: two groups A and B.
The proportions of married and single are ($p_1$) and ($p_2$) respectively, and the standard deviations ($s_1$) and ($s_2$). The sample consists of ...
-1
votes
1answer
405 views
Question regarding sampling, estimation and accuracy [closed]
Let's say there are N chunks of metal, each are 4 inches thick. They are to be hammered and reduced to following best possible sizes: 1, 2, 3 or 4 inches. If we were to use sampling to get an estimate ...
9
votes
2answers
3k views
How can one do an MCMC hypothesis test on a mixed effect regression model with random slopes?
The library languageR provides a method (pvals.fnc) to do MCMC significance testing of the fixed effects in a mixed effect regression model fit using lmer. However, pvals.fnc gives an error when the ...
24
votes
5answers
955 views
Regarding p-values, why 1% and 5%? Why not 6% or 10%?
Regarding p-values, I am wondering why 1% and 5% seem to be the gold standard for "statistical significance". Why not other values, like 6% or 10%?
Is there a fundamental mathematical reason for ...
8
votes
3answers
2k views
Comparing and contrasting, p-values, significance levels and type I error
I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error.
I understand that p-values are defined as "the probability of ...
11
votes
3answers
893 views
How do I compare bootstrapped regression slopes?
Let us assume I have two data sets with n observations of data pairs of independent variable x and dependent variable y each. Let us further assume I want to generate a distribution of regression ...
10
votes
1answer
675 views
Sequential hypothesis testing in basic science
I'm a pharmacologist and, in my experience, almost all papers in basic biomedical research use Student's t-test (either to support inference or to conform to expectations...). A couple of years ago it ...
4
votes
3answers
868 views
What is a good introduction to statistical hypothesis testing for computer scientists?
I was recently exposed to some statistical hypothesis testing methods (e.g. Friedman test) at work, and I would like to increase my knowledge on the topic.
Can you suggest a good introduction to ...
9
votes
4answers
3k views
Examples of studies using p < 0.001, p < 0.0001 or even lower p-values?
I come from the social sciences, where p < 0.05 is pretty much the norm, with p < 0.1 and p < 0.01 also showing up, but I was wondering: what fields of study, if any, use lower p-values as a ...
6
votes
1answer
465 views
What is a good index of the degree of violation of normality and what descriptive labels could be attached to that index?
Context:
In a previous question, @Robbie asked in a study with around 600 cases why tests of normality suggested significant non-normality yet the plots suggested normal distributions. Several people ...
4
votes
1answer
584 views
Test the significant differences between the coefficients obtained by logistic regression on two data sets.
So I run logistic regressions of same dependent variables and independent variables on two sets of data $S_1$ and $S_2$. I obtained two sets of coefficients $\beta_1$ and $\beta_2$. Now I suspect that ...
4
votes
4answers
622 views
What dependence is implied by a chi square test for independence?
I have a question on subject chi-square test for independence.
I have, for example, two events A and B. If chi square test is not passed: is A dependent on B (A|B) or B on A (B|A)? Or does be valid ...
3
votes
1answer
291 views
Characterizing datasets and estimating statistically significant differences?
I have four datasets: D1, D2, D3 and D4. Each dataset contains elements (different sample size in each dataset) that can be described by 100 categories (1 represents that the element belongs to that ...
2
votes
1answer
579 views
How to determine statistical validity of results
I have two exclusive groups of people and a counter of how many events happened for each group.
Lets say group 1 has 7000 people and group 2 has 3000 people.
group 1 had 50 events and group 2 had 40 ...
10
votes
5answers
2k views
Assessing the significance of differences in distributions
I have two groups of data. Each with a different distribution of multiple variables. I'm trying to determine if these two groups' distributions are different in a statistically significant way. I ...
5
votes
3answers
926 views
Is it ever good to increase significance level?
Suppose you are using $\alpha = 0.01$ as the significance level. IF you are not getting significant results, is it better to increase the significance level to $\alpha = 0.05$?
4
votes
3answers
479 views
Effect size and statistical significance
Using Cohen's d, I am getting small amd medium effect sizes for results that are statistically non-significant (p>.05). Does this make sense?
4
votes
3answers
414 views
What is the best way to determine if pageviews are trending upward or downward?
Given the following dataset for a single article on my site:
...
4
votes
2answers
333 views
Is there a way to determine the significance of a change in a d' score?
I have a series of scores in a signal detection task. For each block of scores (i.e. a set of scores from one participant on one day) I have calculated a d' score which I am using as an indicator of ...
3
votes
1answer
154 views
Test whether count of events is significantly greater in a feature list than the general feature population
I begin with a matrix of features (~30000 rows) vs cases (columns). I have binary - TRUE or FALSE - data in each cell indicating whether there is an event spanning the feature for that case. Next, I ...
3
votes
2answers
663 views
How to test whether correlation measures differ when controlling or not for a third variable?
If the correlation between demographic dissimilarity and satisfaction is $r=.-14$ and the partial correlation, with career development partialled out, between demographic dissimilarity and ...
3
votes
1answer
3k views
How do I calculate if the degree of overlap between two lists is significant?
If I have two lists A and B, both of which are subsets of a much larger list C, how can I determine if the degree of overlap of A and B is greater than I would expect by chance?
Should I just ...
2
votes
1answer
70 views
How to compare two datasets using metrics drawn from unknown distributions and with small sample sizes?
I have two datasets consisting of metrics from several experiments. Dataset 1 is the collection of results of experiments E performed by user A on product A, repeated N times. Dataset 2 is the ...
2
votes
1answer
242 views
What happens if you reject normality of residuals when estimating with least square ?
What happens if you reject normality of residuals when estimating with least square ?
Is it too important to have normality on the residuals?
9
votes
1answer
153 views
Does significance test make sense to compare randomised groups at baseline?
Many randomised controlled trial (RCT) papers report significance tests on baseline parameters just after/before randomisation to show that the groups are indeed similar. This is often part of a ...
