I am planning to use a neural network for prediction. For example, to predict whether a student will pass a course based on his previous academic records or characteristics. I was wondering how to choose meaningful data as input for my neural network, as opposed to irrelevant data.

I remember reading that it's better if the input data is not strongly correlated. Is that true?


The most important thing is that the inputs should carry information about the variable you want to predict. Use as much prior knowledge as you can to choose meaningful inputs, including consulting with domain experts who are familiar with your type of data. It may be the case that some inputs jointly carry relevant information, but alone they don't. It's not always possible to know ahead of time which features are meaningful, and many algorithms can tolerate some amount of 'noisy' inputs that don't carry relevant information. But, performance can break down as you add more of these. 'Feature selection' is a broad class of methods for identifying relevant features, but may involve either heavy assumptions or heavy computation. Some algorithms (including some neural nets) can identify relevant features as part of training.

For neural nets (and other methods), the way you preprocess your data is important. Common preprocessing steps include centering the data and normalizing it (e.g. dividing by the standard deviation or range). One reason is to avoid saturating units in the network, which can slow or halt training. This can happen if inputs are too large. Another reason is to avoid creating long, narrow valleys in the loss function, which can happen if input dimensions have different scales, or if they're strongly correlated. Neural nets are often trained using gradient-based procedures like stochastic/minibatch gradient descent. On each step, they adjust the weights in a direction that reduces the loss function. If the loss function has a nice, symmetric bowl shape, the direction of steepest descent points toward the minimum, and stepping in this direction will give fast convergence. If the loss function has elongated, sloping canyons, the direction of steepest descent points toward the canyon floor, but not toward the actual minimum, so the network will zig-zag through weight space and training will be slower. Data can be preprocessed using PCA, to give uncorrelated inputs.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.