I have a dataset consisting of 1000 samples (rows). Each sample consists of 300 categorical data points (columns). The process of generating these data points has a known probability of generating an erroneous result, $pErr$.

Let $pErr = 0.1$ for this example. As each of the 300 columns consists of 1000 categorical data points, the number of data points, which can be considered as errors in each column, is $1000 * pErr = 100$. Consider this value $Expected$.

I need to know the lowest number of errors per column, which is considered statistically significant. I run $\chi^2$ tests, starting with $Observed = Expected + 1$, and increasing $Observed$ by $1$, until the test statistic is greater than the $\chi^2$ value, for one degree of freedom. In this example, an $Observed$ value of 119 is statistically significant: test statistic = 4.01 > $\chi^2$ = 3.84.

I then examine each of the 300 columns for categorical data points occurring less than 119 times, and consider them to be errors.

Question: Do I need to apply a multiple comparisons correction to the analysis? If so, where?

Edit: Each data point (each column for each row) is independent. I am not interested in any overall significance or testing. I am only interested in each column as an entity. The number of errors for each column are considered separately from any other column.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.