Testing if single features have predictive power alone I am working on a classification task where the target variable is binary. The feature space is huge (about 280 features) and I am almost certain that some of them are completely irrelevant for  classification.
Now, I am thinking about training one neural network for each feature. If the cross validation accuracy for a certain feature is above 50% I suggest the feature provides some important information. If it is at 50% or below it is basically not providing any important information as the result is not better than assigning classes randomly.
However, it came to my mind that it might be possible that one feature does a bad job at classification on its own but is very important for the accuracy in combination with other features. Is this possible?
So for instance: Feature 1 on its own provides 49.98% accuracy, Feature 2 on its own provides 64% accuracy but Feature 1 & Feature 2 together provide 74% accuracy. Is this possible?
 A: Here is an example, simulated in R.
library(ggplot2)
set.seed(2021)
N <- 1000
a <- -2
b <- 2
x <- runif(N, a, b)
g <- rbinom(N, 1, 0.5)
y <- x - 2*x*g + rnorm(N, 0, 0.1)
d <- data.frame(x = x, g = as.factor(g), y = y)
ggplot(d, aes(x = x, y = y, col = g)) + 
    geom_point() +
    theme_bw()
Lx <- lm(y ~ x)
Lg <- lm(y ~ g)
L <- lm(y ~ x*g)
summary(Lx)$r.squared # = 0.000466123448518558, so basically  
                      # no predictive ability
summary(Lg)$r.squared # = 0.000428737021642939, so basically 
                      # no predictive ability
summary(L)$r.squared # = 0.992562429058087, so outstanding 
                     # predictive ability

Here, the x and g alone are awful predictors. However, using both together creates an excellent predictor. While, yes, I did have to engineer the interaction term, the question concerns a neural network, which would be able to figure out (or at least approximate) such a feature.
Interestingly, this example does not even feature much correlation between x and g.
