New answers tagged

4

Yes, a permutation test makes sense here. Strictly speaking, the null hypothesis is that the distributions are the same, not just that they have the same means. (If they had same means but difference variances, the test would have the wrong Type I error rate.) Here's example code > df<-data.frame(group=factor(rep(c(1,2,3),c(11,6,5))), + ...


-2

Finally, what I did. I changed the R function corr.test: r <- r/1.001 t <- (r * sqrt(n - 1))/sqrt(1 - r^2) p <- -2 * expm1(pt(abs(t), (n - 1), log.p = TRUE)) a) I changed the Formula: where n-2 became n-1 in order to remove "0" from numerator. b) I divided the whole r correlation coefficient matrix with r/1.001 in order to ...


1

The one-measure-at-a-time approach is throwing away all the useful information from the correlations among the various measures of "quality," while leading to a massive multiple-comparison problem from the 10 separate regressions. And I'd be skeptical if you considered a particular predictor to be "significant" if it only was associated ...


3

If you have data on $n_1 = 700,000$ in Group 1 and $n_2= 10,000,$ then I wonder about two issues: (a) Unbiasedness. Were the observations randomly taken in order to represent the groups fairly? Or are they self-selected subjects who may not be representative. On the positive side, are these samples so large that they essentially exhaust their respective ...


1

One of the libraries in R has a runs.test procedure, which you can explore. My purpose here here is to give an idea how looking at runs can help you decide whether your observations are randomly sampled from the same population. To begin we look specifically at sequences of Bernoulli trials, as mentioned in your Question. (Randomness tests for other ...


0

I think the answer as to "Why are p-values uniformly distributed under the null hypothesis?" has been sufficiently discussed from a mathematical perspective. What I thought is missing is a visual explanation of this and the idea of thinking of p-values as areas to the left of a set of quantiles under a given distribution. By quantiles I mean cut-...


2

Suppose you used a 1-sample t test at level 5% to test $H_0: \mu = 100$ vs. $H_a: \mu \ne 100.$ You used a sample of size $n = 20$ without checking the power against various alternatives. Now, somehow you know that the population from which you sampled is $\mathsf{Norm}(\mu=104,\sigma=15).$ So now you know you failed to reject $H_0,$ when it is false, a Type ...


2

The false discovery rate (FDR) is the expected proportion of type I errors, which is where you incorrectly reject the null hypothesis. α is the p-value you choose to use. You have defined it as 0.05, which is a common choice. At this p-value, there's a 5% chance of a Type 1 error.


1

You can use the Neyman-Pearson lemma to determine the most powerful test to apply. First, we want to move into testing two simple hypotheses: $$ H_0: \quad \alpha=0\\ H_1: \quad \alpha=\hat{\alpha} $$ where $\hat{\alpha} \in \{ \alpha : \alpha > 0 \}$. Please bear in mind that $\hat{\alpha}$ is a fixed value, so we have the full distribution defined. We ...


6

This is too long to be a comment, but just to add to the excellent answer by Sextus, one issue with "significance" is the arbitrary nature of the significance level(s). Often these are dictated by whatever is common practice in a particular field. Also, when a researcher performs a test and finds a p-value of, say 0.0499999 they may claim to have ...


6

Warning pessimistic/cynical post We do not want to move away from significance. That is a false premises that lead to your question. Recently I have found that many statisticians are speaking of moving away from significance. .. Since we want to move away from significance... We do not want to move away from significance. Significance is important. It is ...


1

There are many help sites on the Internet for doing two-factor ANOVAs. I will leave that part to you. The rest of this is in case you find that residuals of your ANOVA are not normal (or differ widely in variance among the four treatment combinations). Below is an example of two (simulated) bimodal samples, and a permutation tests to see if their means ...


2

dat$mean_diff_pval <- pt(dat$mean_diff_tstat, dat$mean_diff_dof) This is wrong. The p-value is $P(|t|>|t^*|)$ and you can calculate it by: pt(abs(dat$mean_diff_tstat), dat$mean_diff_dof, lower.tail=FALSE) # P(t > |tstat|) + pt(-abs(dat$mean_diff_tstat), dat$mean_diff_dof) # P(t < -|tstat|) or dat$mean_diff_pval <- 2*pt(-abs(...


0

This is an old thread but as lots of people are reading it... From the description it sounds like you have a skewed distribution, not a heavy tailed distribution as those terms are commonly used. This matters as the remedies are different. For a skewed distribution, a common remedy is to transform the data to something like a normal distribution. Taking ...


2

If your group differences are sufficiently blatant, very small $p$ values can easily result. For instance: Here, we get $p\approx 2.3\times 10^{-15}$. ("$\approx$" because you shouldn't take this number too seriously.) R code: set.seed(1) # for reproducibility group_1 <- rnorm(120) group_2 <- rnorm(60,3,2) t.test(group_1,group_2,var.equal=...


1

Your first approach will be fine; see for example here for more background. The KS test works for unbalanced data sets. (And yes, you ought never to average p-values, at least not by taking an arithmetic mean. If you do decide to run multiple draws, consider Fisher's method. But you don't need the sampling approach.)


1

The Bonferroni correction (or its more powerful but equally sensitive Holm modification) controls the family-wise error rate, the chance that any of your nominally significant results is a false positive. If that's what you want to control, then you need to consider all 40,000 comparisons. You could instead try to limit the false-discovery rate, the fraction ...


0

You can test the fit of a model with that factor and it’s levels omitted against the full model. That’s basically what’s going on to calculate the p-values of each individual parameter. The model with the factor and it’s levels is going to fit better, but then the test, in some sense, says if the increase in fit is worth the additional model complexity. You’...


0

The underlying data are counts associated with each of a set of 40 substrates, so putting aside the coding problem (which isn't really on-topic here) the approach has two problems: t-tests aren't correct for count data, and serially comparing each substrate against the mean of all the other substrates isn't a correct way to do these tests. Count data are ...


0

Significance Levels and Confidence Intervals Confidence intervals (only) provide a plausible range for population parameters. For instance, the population proportion would be within the range $(p_1, p_2)$ with 95% confidence level. That's if we take $N$ samples of the same sample size, 95% of the time the sample proportion would be within that range. Note ...


1

You need to correct for all of the comparisons you are doing. So if that's 300,000,000 comparisons you need to correct for that many multiple comparisons. But consider what some standard corrections for false-discovery rates (FDR) and family-wise error rates (FWER) protect you from. Say you have data in which there are no true associations but you do a lot ...


0

CIs give us a range estimate for value of parameters. Meanwhile, p-values allow us to reach conclusion such as whether a parameter is non zero, or positive. Moreover, p-values have more general usage, eg in all use cases where hypothesis testing involved.


1

There may be an issue on the cycle length of the random number generator, which could be an issue for large n! Apparently, cycles repeat at some point. Try adding a random choice to which of two (or more) possible new random #s you will select. If a repeating cycle issue, this will add randomization. See if you can detect any improvement.


0

For others who might visit this page, it's important to provide some detail on the underlying studies and the meaning of the plot. These analyses start with individual IC50 values for a drug (a measure of sensitivity to a drug, smaller is better) on each of a large number of cell lines, label the cell lines with respect to several hundred known genomic and ...


5

Please try not to be too concerned with p-values. They don't tell you anything about practical significance. I am wondering if the interpretation is as simply as, for example, there is a significant difference in the maximum depth (max_depths) reached between feeding and non-feeding dives, with maximum depth taking higher (deeper) values in feeding dives. ...


0

I did not read the paper. But one way to get something like this with R is using regression. Suppose you have 3 portfolios and need to test P3 - P1 # portfolios w1=c(A=.4,B=.5,C=.1) w2=c(D=.3,D=.1, E=.4, F=.2) w3=c(G=.3,H=.5,P=.2) #some returns library(mvtnorm) NS=500 set.seed(123) ret = rmvnorm(n = NS,mean=runif(10,min = 0.02,max = 0.025),sigma=diag(runif(...


1

In my opinion you have to perform a one-sample Kolmogorov-Smirnov Test. You have a sample of the random variable Y and you want to check if the random variable is two-parameter APE distributed. So I've updated your cdf such that it has a further argument for the values of the sample, called argument y. However, you have some problems performing the KS-Test. ...


Top 50 recent answers are included