Skip to main content
Added explanation for ROCAUC 50% for binary classifier
Source Link
Dudelstein
  • 373
  • 2
  • 9

I would start with a simpler model, for example logistic regression, in which you can look at the chi-square test to see how the addition of X variables improves the model, as well as at the individual p-values. If you do not see statistically significant results especially for the entire model, this would suggest the variables are not very informative of y. If you have reasons to believe there are interactions or higher-order terms (e.g. squares), you can add these to the model.

You could then progressively try more complex models (e.g. random forest or neural networks) and show that even with more flexibility (interactions etc.), the results do not improve. As per @Dave's helpful comment: to make the general statement that X is not informative of y, you could use a measure such as ROC-AUCROCAUC, which does not depend on a specific threshold. 

If itROCAUC is very close to 0.5, it means that for all possible decision thresholds, it performs on averageyour classifier is not better than randomly flipping a coin flip. More precisely, if we let the classifier rank data points (highest ranking for most sure about positive class), the chance that a randomly chosen positive class will be ranked above a randomly chosen negative class is just 50%, which is no better than random order.

Finally, to make the case that X is not strongly related to y, it would help greatly to discuss what is in the data and why X might not be informative at all. For example, if you are predicting lung cancer for patients and X data contains job occupations, it could be the relationship is weak -- job occupation does not connect with lung cancer unless perhaps in very specific jobs like mining. However, if X contains demographic data or information about smoking, it seems strange if it could not predict lung cancer to some extent.

I would start with a simpler model, for example logistic regression, in which you can look at the chi-square test to see how the addition of X variables improves the model, as well as at the individual p-values. If you do not see statistically significant results especially for the entire model, this would suggest the variables are not very informative of y. If you have reasons to believe there are interactions or higher-order terms (e.g. squares), you can add these to the model.

You could then progressively try more complex models (e.g. random forest or neural networks) and show that even with more flexibility (interactions etc.), the results do not improve. As per @Dave's helpful comment: to make the general statement that X is not informative of y, you could use a measure such as ROC-AUC, which does not depend on a specific threshold. If it is very close to 0.5, it means that for all possible decision thresholds, it performs on average not better than randomly flipping a coin.

Finally, to make the case that X is not strongly related to y, it would help greatly to discuss what is in the data and why X might not be informative at all. For example, if you are predicting lung cancer for patients and X data contains job occupations, it could be the relationship is weak -- job occupation does not connect with lung cancer unless perhaps in very specific jobs like mining. However, if X contains demographic data or information about smoking, it seems strange if it could not predict lung cancer to some extent.

I would start with a simpler model, for example logistic regression, in which you can look at the chi-square test to see how the addition of X variables improves the model, as well as at the individual p-values. If you do not see statistically significant results especially for the entire model, this would suggest the variables are not very informative of y. If you have reasons to believe there are interactions or higher-order terms (e.g. squares), you can add these to the model.

You could then progressively try more complex models (e.g. random forest or neural networks) and show that even with more flexibility (interactions etc.), the results do not improve. As per @Dave's helpful comment: to make the general statement that X is not informative of y, you could use a measure such as ROCAUC, which does not depend on a specific threshold. 

If ROCAUC is close to 0.5, it means your classifier is not better than a coin flip. More precisely, if we let the classifier rank data points (highest ranking for most sure about positive class), the chance that a randomly chosen positive class will be ranked above a randomly chosen negative class is just 50%, which is no better than random order.

Finally, to make the case that X is not strongly related to y, it would help greatly to discuss what is in the data and why X might not be informative at all. For example, if you are predicting lung cancer for patients and X data contains job occupations, it could be the relationship is weak -- job occupation does not connect with lung cancer unless perhaps in very specific jobs like mining. However, if X contains demographic data or information about smoking, it seems strange if it could not predict lung cancer to some extent.

Switch F1-score to ROC-AUC curve not dependent on specific threshold
Source Link
Dudelstein
  • 373
  • 2
  • 9

I would start with a simpler model, for example logistic regression, in which you can look at the chi-square test to see how the addition of X variables improves the model, as well as at the individual p-values. If you do not see statistically significant results especially for the entire model, this would suggest the variables are not very informative of y. If you have reasons to believe there are interactions or higher-order terms (e.g. squares), you can add these to the model.

You could then progressively try more complex models (e.g. random forest or neural networks) and show that even with more flexibility (interactions etc.), the results do not improve. Since your dataAs per @Dave's helpful comment: to make the general statement that X is unbalanced, I would consider metrics such as Recall and Precision or their harmonic meannot informative of - F1-score. By the wayy, if you could use a metricmeasure such as F1ROC-scoreAUC, which does not depend on a specific threshold. If it is quite robustvery close to an imbalanced dataset0.5, the results might improveit means that for all possible decision thresholds, it performs on average not better than randomly flipping a coin.

Finally, to make the case that X is not strongly related to y, it would help greatly to discuss what is in the data and why X might not be informative at all. For example, if you are predicting lung cancer for patients and X data contains job occupations, it could be the relationship is weak -- job occupation does not connect with lung cancer unless perhaps in very specific jobs like mining. However, if X contains demographic data or information about smoking, it seems strange if it could not predict lung cancer to some extent.

I would start with a simpler model, for example logistic regression, in which you can look at the chi-square test to see how the addition of X variables improves the model, as well as at the individual p-values. If you do not see statistically significant results especially for the entire model, this would suggest the variables are not very informative of y. If you have reasons to believe there are interactions or higher-order terms (e.g. squares), you can add these to the model.

You could then progressively try more complex models (e.g. random forest) and show that even with more flexibility (interactions etc.), the results do not improve. Since your data is unbalanced, I would consider metrics such as Recall and Precision or their harmonic mean - F1-score. By the way, if you use a metric such as F1-score, which is quite robust to an imbalanced dataset, the results might improve.

Finally, to make the case that X is not strongly related to y, it would help greatly to discuss what is in the data and why X might not be informative at all. For example, if you are predicting lung cancer for patients and X data contains job occupations, it could be the relationship is weak -- job occupation does not connect with lung cancer unless perhaps in very specific jobs like mining. However, if X contains demographic data or information about smoking, it seems strange if it could not predict lung cancer to some extent.

I would start with a simpler model, for example logistic regression, in which you can look at the chi-square test to see how the addition of X variables improves the model, as well as at the individual p-values. If you do not see statistically significant results especially for the entire model, this would suggest the variables are not very informative of y. If you have reasons to believe there are interactions or higher-order terms (e.g. squares), you can add these to the model.

You could then progressively try more complex models (e.g. random forest or neural networks) and show that even with more flexibility (interactions etc.), the results do not improve. As per @Dave's helpful comment: to make the general statement that X is not informative of y, you could use a measure such as ROC-AUC, which does not depend on a specific threshold. If it is very close to 0.5, it means that for all possible decision thresholds, it performs on average not better than randomly flipping a coin.

Finally, to make the case that X is not strongly related to y, it would help greatly to discuss what is in the data and why X might not be informative at all. For example, if you are predicting lung cancer for patients and X data contains job occupations, it could be the relationship is weak -- job occupation does not connect with lung cancer unless perhaps in very specific jobs like mining. However, if X contains demographic data or information about smoking, it seems strange if it could not predict lung cancer to some extent.

Source Link
Dudelstein
  • 373
  • 2
  • 9

I would start with a simpler model, for example logistic regression, in which you can look at the chi-square test to see how the addition of X variables improves the model, as well as at the individual p-values. If you do not see statistically significant results especially for the entire model, this would suggest the variables are not very informative of y. If you have reasons to believe there are interactions or higher-order terms (e.g. squares), you can add these to the model.

You could then progressively try more complex models (e.g. random forest) and show that even with more flexibility (interactions etc.), the results do not improve. Since your data is unbalanced, I would consider metrics such as Recall and Precision or their harmonic mean - F1-score. By the way, if you use a metric such as F1-score, which is quite robust to an imbalanced dataset, the results might improve.

Finally, to make the case that X is not strongly related to y, it would help greatly to discuss what is in the data and why X might not be informative at all. For example, if you are predicting lung cancer for patients and X data contains job occupations, it could be the relationship is weak -- job occupation does not connect with lung cancer unless perhaps in very specific jobs like mining. However, if X contains demographic data or information about smoking, it seems strange if it could not predict lung cancer to some extent.