Skip to main content
Cross Validated

Return to Question

Notice removed Draw attention by T.J. Gaffney
Bounty Ended with jvbraun's answer chosen by T.J. Gaffney
Changed "qualitative" to "quantitative"
Source Link
T.J. Gaffney
T.J. Gaffney
  • 187
  • 1
  • 1
  • 9

Let's say I have a categorical variable. I try to test the null hypothesis that each category has the same count (of something) using a Pearson's Chi-Squared test. I may not be able to reject null hypothesis using just the categorical variables, but if I group the categories together in the right way, I can reject the null hypothesis. (For example $\{a,b,c\}$ have a higher count than $\{d,e,f\}$.) It seems though, that if I choose my groupings based on my sample distribution, then I'm overfitting. In simulations, I've been able to group categories of counts from a uniform distribution in the correct way to reject the null hypothesis too often for my significance level. However, I want to be qualitativequantitative about this error/abuse I'm committing. For example, I may be willing to group $\{a,d,e\},\{b,c,f\}$ but no other partition would make sense in my context. In this case I would be more confident in making the choice to group or not group then if I considered all possible partitions.

Is there some way to quantify this type of overfitting? I thought it might be hiding in the degrees of freedom, or maybe it's a type of parameter and something like AIC or BIC might be useful.

Let's say I have a categorical variable. I try to test the null hypothesis that each category has the same count (of something) using a Pearson's Chi-Squared test. I may not be able to reject null hypothesis using just the categorical variables, but if I group the categories together in the right way, I can reject the null hypothesis. (For example $\{a,b,c\}$ have a higher count than $\{d,e,f\}$.) It seems though, that if I choose my groupings based on my sample distribution, then I'm overfitting. In simulations, I've been able to group categories of counts from a uniform distribution in the correct way to reject the null hypothesis too often for my significance level. However, I want to be qualitative about this error/abuse I'm committing. For example, I may be willing to group $\{a,d,e\},\{b,c,f\}$ but no other partition would make sense in my context. In this case I would be more confident in making the choice to group or not group then if I considered all possible partitions.

Is there some way to quantify this type of overfitting? I thought it might be hiding in the degrees of freedom, or maybe it's a type of parameter and something like AIC or BIC might be useful.

Let's say I have a categorical variable. I try to test the null hypothesis that each category has the same count (of something) using a Pearson's Chi-Squared test. I may not be able to reject null hypothesis using just the categorical variables, but if I group the categories together in the right way, I can reject the null hypothesis. (For example $\{a,b,c\}$ have a higher count than $\{d,e,f\}$.) It seems though, that if I choose my groupings based on my sample distribution, then I'm overfitting. In simulations, I've been able to group categories of counts from a uniform distribution in the correct way to reject the null hypothesis too often for my significance level. However, I want to be quantitative about this error/abuse I'm committing. For example, I may be willing to group $\{a,d,e\},\{b,c,f\}$ but no other partition would make sense in my context. In this case I would be more confident in making the choice to group or not group then if I considered all possible partitions.

Is there some way to quantify this type of overfitting? I thought it might be hiding in the degrees of freedom, or maybe it's a type of parameter and something like AIC or BIC might be useful.

Tweeted twitter.com/#!/StackStats/status/538530171159343104
Notice added Draw attention by T.J. Gaffney
Bounty Started worth 100 reputation by T.J. Gaffney
Source Link
T.J. Gaffney
T.J. Gaffney
  • 187
  • 1
  • 1
  • 9

Choice of grouping in Chi-Squared test

Let's say I have a categorical variable. I try to test the null hypothesis that each category has the same count (of something) using a Pearson's Chi-Squared test. I may not be able to reject null hypothesis using just the categorical variables, but if I group the categories together in the right way, I can reject the null hypothesis. (For example $\{a,b,c\}$ have a higher count than $\{d,e,f\}$.) It seems though, that if I choose my groupings based on my sample distribution, then I'm overfitting. In simulations, I've been able to group categories of counts from a uniform distribution in the correct way to reject the null hypothesis too often for my significance level. However, I want to be qualitative about this error/abuse I'm committing. For example, I may be willing to group $\{a,d,e\},\{b,c,f\}$ but no other partition would make sense in my context. In this case I would be more confident in making the choice to group or not group then if I considered all possible partitions.

Is there some way to quantify this type of overfitting? I thought it might be hiding in the degrees of freedom, or maybe it's a type of parameter and something like AIC or BIC might be useful.