I tried several standard tests: A test of two binomial proportions,
an equivalent chi-squared test in R with a simulated
P-value (on account of the small sample taking the drug), and Fisher's exact test. No P-values below 5%. The drug gives a
higher survival rate, but not enough larger to be statistically
significant at the 5% level.
Looking toward the ...
With this approach, you are effectively treating everyone who did not complete the survey as though they had answered No Allergies. I can't see any reason you'd want to do that - it would be equally (in)valid to take the opposite approach, and treat them all as if they had answered Yes Allergies.
In this situation, there's no theoretical or empirical reason ...
A test of two binomial proportions in R, seems appropriate to test $H_0: p_1=p_2$ against $H_a: p_1 \ne p_2.$ The two estimated
proportions are $\hat p_1 = 40/300 = 0.13$ and $\hat p_2 = 200/1000 = 0.20,$ so the observed proportions are different.
Then prop.test in R gives a P-value $0.009 < 0.01 = 1\%,$
so the difference is statistically significant at ...
Here is a output from Minitab for a test of binomial proportions.
(I used the version in which the two samples are taken separately
to estimate the standard error.)
Test and CI for Two Proportions
Sample X N Sample p
1 320 10010 0.031968
2 275 9912 0.027744
Difference = p (1) - p (2)
Estimate for difference: 0.00422388
95% CI ...
One possibility is to consider binomial proportions of False responses in the two groups. Is the difference in the observed proportions false (roughly $0.098$ and $0.108,$ respectively), significantly different between the two groups?
In R, this test (which uses a normal approximation) is done using the prop.test procedure. (I have opted not to use ...
You can use logistic regression with an ordinal predictor variable. By choosing the encoding system for the predictor, you can get the information presented in a useful form. Here is a useful UCLA page overview for different categorical encoding systems, using R (there are similar pages for other languages.) For your purposes, maybe a successive differences ...
$G^2$ and $X^2$ are very similar. They have the same null hypothesis and they have power against the same set of alternatives. More than that, when the departure from the null is not very large and the sample size is not very small, they are numerically close together. (technical phrase: locally asymptotically equivalent)
$G^2$ is the likelihood ratio test: ...
You could use: Bayesian Models Comparison based on Bayes Factor.
In this case, you have the relative probability between the various models; below I provide an R implementation.
Also, I found an article that uses Bayes Factor for a problem similar to your:
[The Bayes Factor Against Equiprobability of a Multinomial Population Assuming a Symmetric Dirichlet ...
When you have a 2 factor category independent variable and a continuous dependent variable then look at performing a t-test. In this case you have a categorical independent variable and count data for the dependent variable, i.e. number of wins and the number of losses; thus the $\chi^2$ test is the better test.
If you have a continuous independent ...
Your choice of a chi-square test is standard for this type of problem. With respect to the number of degrees of freedom (df), the Wikipedia page provides guidance for a goodness-of-fit test:
For a test of goodness-of-fit, df = Cats − Parms, where Cats is the number of observation categories recognized by the model, and Parms is the number of parameters in ...
I think you are right to doubt the output. I tried to replicate the result by implementing the Chi-square test as it is supposed to work for categorical data, e.g. see this link. I don't get the result that comes from sklearn.feature_selection.chi2, here's my code:and outputs:
import numpy as np
from scipy.stats import chi2
from sklearn import ...
The G-test has become a favorite of mine for proportion testing. ;)
The way it works is by fitting a (multinomial) logistic regression model with a factor variable as a predictor, plus an intercept. Then it fits an intercept-only model. Finally, it performs a likelihood ratio test of the factor variable by comparing the two models. (I have verified this with ...
Software engineering: Avoid 'magic numbers' in the code. Define the constants in the code and re-use them. That makes it easier to change the code to run with different values.
Self-education: Have you tried changing any of your 'magic numbers'?
Statistics: $\chi^2$-test checks for differences in ratios (relative frequencies). It would be ...
The study on the various techniques mentioned in the original question above led me to conclude that the right approach is a binomial logistic regression with subject as a random effects. The R code I used is the following
model <- glmer(correct ~ emotional_intent * musical_expertise + (1|subject),family=binomial(link='logit'),data=scrd)
The "exactness" of Fisher's test depends on the assumption that each of the row totals and column totals (the "margins") is given as part of the experimental design. That's what's meant by "fixed marginal distributions." In that case the distribution of counts among cells exactly follows a hypergeometric distribution.