Having predictor variables such as:
- learning rate(continous, range 0-1), number of iterations(continous), number of hidden nodes(continous), LossFunction(categorical)
and predicted variable
- Accuracy (continous)
find, which of the predictor variables have strongest impact on Accuracy and which don't.
The relationship is not linear - after some threshold, increasing number of iterations leads to overfitting.
I was thinking about polynomial, and I see the categorical variable would be ok for this: What is the role of a categorical predictor in polynomial regression? . Is that correct thinking?
How to handle interactions in this model? I know each of the variables interact in the way, that having 0 hidden nodes / learning rate can greatly decrease the Accuracy, but that is just edge case I'm not interested in. Other than that, I don't suppose there are any strong interactions and I see it can be detrimental: https://stats.stackexchange.com/a/211662/96761 .
Will the part of variance of one variable be substituted from the impact of the other predictor variable? I'm pretty sure, that's how multivariate regression works, just making sure.
(optional) Currently I'm using Python statsmodels.api as it gives good statistics, especially statistical significance and t score. Unfortunately it does not yet support polynomial regression. on the other hand, scikit-learn has vary limited output, especially I can't find t and P