70

I am going to change the order of questions about. I've found textbooks and lecture notes frequently disagree, and would like a system to work through the choice that can safely be recommended as best practice, and especially a textbook or paper this can be cited to. Unfortunately, some discussions of this issue in books and so on rely on received wisdom....


46

It gave the warning because many of the expected values will be very small and therefore the approximations of p may not be right. In R you can use chisq.test(a, simulate.p.value = TRUE) to use simulate p values. However, with such small cell sizes, all estimates will be poor. It might be good to just test pass vs. fail (deleting "no show") either with ...


42

If the population is known to be normal, a 95% confidence interval based on a single observation $x$ is given by $$x \pm 9.68 \left| x \right| $$ This is discussed in the article "An Effective Confidence Interval for the Mean With Samples of Size One and Two," by Wall, Boen, and Tweedie, The American Statistician, May 2001, Vol. 55, No.2. (pdf)


36

I remember reading that using the percentile confidence interval for bootstrapping is equivalent to using a Z interval instead of a T interval and using $n$ instead of $n-1$ for the denominator. Unfortunately I don't remember where I read this and could not find a reference in my quick searches. These differences don't matter much when n is large (and the ...


33

It is very common for extremely simple forecasting methods like "forecast the historical average" to outperform more complex methods. This is even more likely for short time series. Yes, in principle you can fit an ARIMA or even more complex model to 20 or fewer observations, but you will be rather likely to overfit and get very bad forecasts. So: start ...


28

Sure there is. Use a Bayesian paradigm. Chances are you have at least some idea of what $\mu$ could possibly be - for instance, that it physically cannot be negative, or that it obviously cannot be larger than 100 (maybe you are measuring the height of your local high school football team members in feet). Put a prior on that, update it with your lone ...


25

The issue is that the chi-square approximation to the distribution of the test statistic relies on the counts being roughly normally distributed. If many of the expected counts are very small, the approximation may be poor. Note that the actual distribution of the chi-square statistic for independence in contingency tables is discrete, not continuous. The ...


23

In my view the principled approach recognizes that (1) tests and graphical assessments of normality have insufficient sensitivity and graph interpretation is frequently not objective, (2) multi-step procedures have uncertain operating characteristics, (3) many nonparametric tests have excellent operating characteristics under situations in which parametric ...


23

There is the rule of three saying if a certain event did not occur in a sample with $n$ subjects, the interval from $0$ to $3/n$ is a 95% confidence interval for the rate of occurrences in the population. You have $n=6$, so says $[0, 3/6=0.5]$ is a 95% confidence interval for the binomial $p$ of transmission. In non-technical language: 6 non-events is ...


21

I am again using a question as an opportunity to learn more about time series - one of the (many) topics of my interest. After a brief research, it seems to me that there exist several approaches to the problem of modeling short time series. The first approach is to use standard/linear time series models (AR, MA, ARMA, etc.), but to pay attention to certain ...


21

You should rarely trust any single significant result. You didn't say why you were using a one-tailed instead of a two-tailed test, so hopefully you have a good reason for doing so other than struggling to be able to claim a statistically significant outcome! Setting that aside, consider the following from p. 261 of Sauro, J., & Lewis, J. R. (2016). ...


20

The names "$t$-test" and "$z$-test" are typically used to refer to the special case when $X$ is normal $\mbox{N}(\mu,\sigma^2)$, $\hat{b}=\bar{x}$ and $C=\mu_{0}$. You can however of course construct tests of "$t$-test type" in other settings as well (bootstrap comes to mind), using the same type of reasoning. Either way, the difference is in the $\mbox{s.e....


18

Here is my take on it, based on chapter 16 of Efron's and Tibshirani's An Introduction to the bootstrap (page 220-224). The short of it is that your second bootstrap algorithm was implemented wrongly, but the general idea is correct. When conducting bootstrap tests, one has to make sure that the re-sampling method generates data that corresponds to the null ...


17

Neural networks, in vast majority of cases, need lots of data. If you have 20 observations, neural network is clearly a bad choice. With that small sample size, network would easily memorize the data and overfit. Even cross-validation with that small sample size is disputable, because you'd be validating the results on just few samples at a time. With that ...


16

I think the cardinal principle here is that you can and should show all the individual values. Even if the detail is not obviously interesting or useful, there is no reason not to show it, or to oblige the reader to decode (say) a histogram in which the bars might represent just one or two values. I offer here a small composite. Top left is a dot or strip ...


16

In theory if all the assumptions of the t-test are true then there's no problem with a small sample size. In practice there are some not-quite-true assumptions which we can get away with for large sample sizes but they can cause problems for small sample sizes. Do you know if the underlying distribution is normally distributed? Are all the samples ...


16

Imagine yourself to be in a situation where you're doing many similar tests, in a set of circumstances where some fraction of the nulls are true. Indeed, let's model it using a super-simple urn-type model; in the urn, there are numbered balls each corresponding to an experiment you might choose to do, some of which have the null true and some which have the ...


15

For such small counts, you could use Fisher's exact test: > fisher.test(a) Fisher's Exact Test for Count Data data: a p-value = 0.02618 alternative hypothesis: two.sided


14

If you are provided with small sample size (as a sidelight, what is "small" seems to depend on some underlying customary rule in each research field), no bootstrap will do the magic. Assuming a database contains three observations for each of the two variables under investigation, no inference will make sense. In my experience, non-parametric bootstrap (1,...


14

A small simulation exercise to illustrate whether the answer by @soakley works: # Set the number of trials, M M=10^6 # Set the true mean for each trial mu=rep(0,M) # Set the true standard deviation for each trial sd=rep(1,M) # Set counter to zero count=0 for(i in 1:M){ # Control the random number generation so that the experiment is replicable set.seed(i)...


13

I have a friend who used to work for the US defense department (long time ago, cold war era) and was once asked to answer a question using a single data point. When he insisted that he needed more data he was told that the person who had provided the single data point had been caught and executed for espionage shortly after providing the single data point, ...


13

Other answers criticise the performance of bootstrap confidence intervals, not bootstrap itself. This is a different problem. If your context satisfy the regularity conditions for the convergence of the bootstrap distribution (convergence in terms of the number of bootstrap samples), then the method will work if you use a large enough bootstrap sample. In ...


13

Rand Wilcox in his publications and books make some very important points, many of which were listed by Frank Harrell and Glen_b in earlier posts. The mean is not necessarily the quantity we want to make inferences about. There maybe other quantities that better exemplifies a typical observation. For t-tests, power can be low even for small departures from ...


13

If the original statement doesn't limit the conditions under which it applies pretty substantially, Field is just wrong on this. Responding to the quoted section: In effect, this means it does much the same as the Mann–Whitney test! No, it really doesn't. They really test for different kinds of things. As one example, if two close-to-symmetric ...


12

No, I do not believe there's any formal test that you can apply. (By the way, the usual term for such samples is pooled.) You are right that repeating the blot will not tell you more about the biological variance in population. You could try to look up the variance obtained by others in similar, but non-pooled experiments (if there is any), and use it to ...


11

No, There is no best univariate extrapolation method for a short time series with $T \leq 20$ series. Extrapolation methods need lots and lots of data. Following qualitative methods work well in practice for very short or no data: Composite forecasts Surveys Delphi method Scenario building Forecast by analogy Executive opinion One of the best methods that ...


11

Sounds like you are severely overfitting. Basically, you need to use a simpler model than the one you are currently using or collect (a lot) more data. Generally, the more data you have, the more complex a model you can fit without overfitting. I do not think you are going to get meaningful results using a CNN on such a small dataset. Start with a simple ...


10

The notion of "topics" in so-called "topic models" is misleading. The model does not know or is not designed to know semantically coherent "topics" at all. The "topics" are just distributions over tokens (words). In other words, the model just capture the high-order co-occurrence of terms. Whether these structures mean something or not is not the purpose of ...


10

@Glen_b is right about the nature of the normality assumption in regression1. I think your bigger problem is going to be that you don't have enough data to support 4 to 5 explanatory variables. The standard rule of thumb2 is that you should have at least 10 data per explanatory variable, i.e. 40 or 50 data in your case (and this is for ideal situations ...


9

Yes, you surely can do that. I don’t know applications in ecology, but you may be interested to know that this is widely used in genetics (epidemiology and population genetics), with $n \ll p$, typically $n = 1000$ or $5000$ individuals and $p = 500\,000$ genotypes. To adjust analyses for population mixture, the first 10 or 50 PC are used. The first two PC ...


Only top voted, non community-wiki answers of a minimum length are eligible