David Wright
  • Member for 10 years, 11 months
  • Last seen more than a month ago
How to assess the similarity of two histograms?
26 votes

The standard answer to this question is the chi-squared test. The KS test is for unbinned data, not binned data. (If you have the unbinned data, then by all means use a KS-style test, but if you only ...

View answer
What is a manifold?
10 votes

In this context, the term manifold is accurate, but is unnecessarily highfalutin. Technically, a manifold is any space (set of points with a topology) that is sufficiently smooth and continuous (in a ...

View answer
Is it logical to use correlation between percentages
10 votes

One reason that someone might assert that "you cannot run correlation on percentages" is that percentages are bounded by [0, 1], and the underlying assumption of the Pearson r test is that values are ...

View answer
Should the logistic regression equation always result in a value between 0 and 1?
Accepted answer
7 votes

No. Your linear function could return any real value. You are supposed to put that value into the logistic function to get a probability value between 0 and 1. That's really the whole trick of ...

View answer
Calculating error of mean of time series
Accepted answer
6 votes

Clearly you have good statistical intuition, because you are exactly right! Because of correlations between the individual terms, the standard error of the mean of the observations is not an accurate ...

View answer
Interpretation of dummy variables
Accepted answer
6 votes

You can reconstruct what you want from your fit using the population fractions of male and female $f_M$ and $f_F = 1 - f_M$. The expectation of $Y$ for males and females is given by: $$ E[Y|M] = a + ...

View answer
What concepts in optimization do I need for machine learning?
Accepted answer
5 votes

Conceptually, the only thing you need to know to understand machine learning algorithms is "there is an optimum, and we can find it". Practically, it's always useful to have some idea how optimization ...

View answer
In regression, what if a factor is very sparse?
5 votes

If by "sparse", you mean that the factor value is unknown for 95% of your rows, then there is not much you can do besides not including that factor in your regression. If by "sparse", you mean the ...

View answer
Central Limit Theorem and t-test
4 votes

Short answer: your colleague is right. In the end, the t statistic depends only on the mean and variance of the two samples. The CLT says that (under most circumstances) those rapidly become normal ...

View answer
Permutation version of Ftest
Accepted answer
4 votes

You can't shuffle values, because H0 does not assume the two distributions to have the same mean. You can shuffle deviations from the mean, because those are assumed to be equal under H0 (well, under ...

View answer
How to calculate or approximate the integral $\int_{-\infty}^{\infty}\sigma(x)(1-\sigma(x))\mathcal{N}(x|0,1)dx$
4 votes

Mathematica doesn't give an analytic result, but it does give the numerical result 0.206621. If you want some sort of semi-analytic treatment, you can expand $$\sigma(x) ( 1 - \sigma(x) ) = \frac{1}{...

View answer
Is this formula for Standard Error just for a sampling distribution that's normal?
Accepted answer
3 votes

It is the standard deviation of the sample mean for any distribution, not just the normal. This follows from the theory of cumulants. Also, it is exact for any $n$, not just asymptotic. It is not the ...

View answer
How to compute a confidence interval for the mean of count data?
3 votes

It's certainly asymptotically correct, which is as much as you can say for the usual formula for CIs, which assumes that the sampling mean of a continuous distribution is normal. You are essentially ...

View answer
Generates variates of $F^s$ given a known algorithm for variates of $F$
3 votes

Given a CDF $P(x)$ and a RNG that generates uniform deviates $U \in (0, 1)$, the standard way of producing non-uniform deviates that follow the CDF is to take $P^{-1}(U)$. So you would generate $X = F^...

View answer
Am I interpreting logistic regression coefficient of categorical variable correctly as a probability?
3 votes

No. Start with the two-category case, which is presumably easier to understand. There is one indicator variable and one coefficient. Say the categories are sex, F and M, and we code F as 0 and M as 1....

View answer
Reverse Engineering game. What sample size do I need?
Accepted answer
3 votes

The formula you want is $$\delta p = \sqrt\frac{p(1-p)}{n}$$ which comes from the variance of a Bernoulli distribution. So if your observed fraction is $p = 0.08$ with $n=100$ observations, the error ...

View answer
How to use $\chi^2$ test to test if my data follow a $\chi^2$ distribution?
3 votes

You don't really want to use a $\chi^2$ test for this, since that would require you to needlessly and arbitrarily bin your data. You want to do a Kolmogorov-Smirnov test. The KS test takes as input a ...

View answer
How did the stattrek calculator compute the t-score without the population mean?
Accepted answer
3 votes

The formula you quote relates t to quantities derived from a data set. The calculator uses the theory behind the t-distribution to determine P(t) given t, or t given P(t). This will only accurately ...

View answer
What are some valuable Statistical Analysis open source projects?
3 votes

Meta.Numerics is a .NET library with good support for statistical analysis. Unlike R (an S clone) and Octave (a Matlab clone), it does not have a "front end". It is more like GSL, in that it is a ...

View answer
Uniform distribution testing
Accepted answer
2 votes

Use a Kuiper test. It's location shift invariance makes it particularly suited to testing circular distributions.

View answer
Median larger than mean in a Weibull distribution in scipy.stats
Accepted answer
2 votes

I checked your results against those given by the Meta.Numerics library and got exactly the same values. A Weibull with shape parameter 6.0 has skewness -0.37. If you look at the graphs of Weibull ...

View answer
Lognormal parameters knowing GDP per-capita, Gini coefficient and quintile shares
Accepted answer
2 votes

So you know the mean $m$ and Gini $g$ of a distribution, and, assuming it is a lognormal distribution, you want to get the lognormal parameters $\mu$ and $\sigma$? That is pretty straightforward. The ...

View answer
Observation Independence Assumption in KS-Test
Accepted answer
2 votes

No, it is not. The derivation of the null distribution for the KS statistic relies on IID. (It may be robust against some violations of this assumption, but I don't know of any studies of that.) If ...

View answer
Why do qq-plots appear to show normal residuals from a GAM when the underlying distribution is not normal?
2 votes

There is nothing about a Q-Q plot that is intrinsically tied to the normal distribution. It is a plot of the quantile points of one distribution against the quantile points of another. If those two ...

View answer
Is cumulative sum large deviation an alarm for bad normal pseudo-random number generator?
Accepted answer
2 votes

Your $s_t$ should be normally distributed. If a variable is normally distributed, you expect it to fall more than $2\sigma$ from the mean 4.5% of the time, and more than $3\sigma$ from the mean 0.27% ...

View answer
What to conclude about the results if the p-value is same in 2 Kolmogorov-Smirnov tests?
Accepted answer
2 votes

These P-values aren't the same, they are just both less than $2.2 \times 10^{-16}$. From this I would conclude: As whuber said, neither acc15 nor acc16 is a good fit to acco. Your software isn't very ...

View answer
Example of danger of ecological correlation of rates?
2 votes

One example, which was true for many decades but might not be true right now, is the correlation between income and political party in the US. If you look at state level data, you would think that ...

View answer
How to generate Gaussian random vectors on a hyperplane?
Accepted answer
2 votes

Diagonalize $\Sigma$, i.e. find its eigenvalues $\sigma_i^2$ (which will be non-negative) and eigenvectors $\vec{v}_i$. Generate $(n-1)$ normal deviates $z_i \sim N(0, \sigma_i)$. Form $\vec{u} = \...

View answer
Statistics for changes in odds ratios
2 votes

This is paired 2X2 experiment, yes? And you want to know whether the observed difference of the odds ratio from 1 is statistically significant? The answer to this question is a McNemar test. Be ...

View answer
Predicting variable Time Series - Unknown Frequency
Accepted answer
2 votes

The thing that is called a "time series" in normal statistical parlance always has regular values. There are techniques for filling in missing values (e.g. you want to analyze a daily temperature ...

View answer