How can one determine which inputs are not useful in a machine learning algorithm?

Question

There is an infinite amount of inputs that one could supply to a machine learning algorithm, be it neural network or just logistic regression, and the algorithm will spend equal time processing each input.

How can one determine which inputs are not useful for a certain regression/classification and remove them from the input field?

Intuition suggests that by looking at the parameters/weights associated with that input we would be able to assume that an input is not useful if the magnitudes of the parameters/weights are very small.

Is there a formal approach to this problem?

Answer 1

First off, you can do dimensionality reduction of features independent of any particular prediction problem, i.e. representation learning.

In the context of prediction problems, sparsity-promoting regularization can be used to automatically perform feature selection. This is commonly accomplished using $L_1$ penalties such as LASSO for linear regression (and also in deep learning).

($L_1$ regularization is also used in representation learning, such as sparse coding and sparse autoencoders).

Answer 2

I can only guess by your affirmations that your question is on a multivariate setting, it will always depend on your data and the problem you are looking to tackle. Domain experience it's the best way to accomplish this task but on very very large datasets it can be prohibitive.. if that's the case and you have plausible reason to acknowledge co-linearity then dimensionality reduction techniques can help (principal component analysis, factor analysis, manifold learning).

If it can be applied please do try to apply a penalty to your objective function, namely a L1/Lasso penalty, a laplacian prior.

Regarding your comment about the amplitude of the weights it does make sense but it's a naive way to judge the importance of each variable, two variables can have really high value when combined (interactions) and close to none if used alone.. if you are considering a non linear model or interactions take that into account.